Rena Palloff and Keith Pratt are well-known in the e-learning world for their workshops and publications designed for distance education practitioners. Unfortunately, their recent publication, Assessing the Online Learner: Resources and Strategies for Faculty, is surprisingly weak. This 2008 release is one of the newest in the Jossey-Bass Guides to Online Teaching and Learning series and although learner assessment is a timely topic, this text provides only a bare bones overview of the subject, even for novice e-teachers.
The text is presented in two sections. Part One: Assessment Basics is as advertised, the basics. Chapter 1, “How Do We Know They Know?” steps all the way back to course design and development, clearly and concisely demonstrating how assessments should evolve naturally from established program competencies, course outcomes, and unit objectives. The authors do a nice job of tying specific, performance-based objectives to higher order thinking and student engagement, but they choose an unfortunate model to support their ideas on active learning. Most of us have seen one or more lists that allegedly illustrate how much one will learn or remember based on the type of instruction encountered (e.g., 5% from lecture, 10% from reading, etc.), and it would truly be wonderful if learning were, in fact, that simple. Palloff and Pratt commit a serious error when they present one of these questionable pyramids (p. 19) suggesting that retention can be categorized in this manner. This kind of error would be understandable in a presentation or article by someone outside the field, but is disappointing when it occurs in support of “engaged critical teaching.” (For an enlightening read about these pseudoscience models and their amazing longevity, check out Will Thalheimer's blog at http://www.willatworklearning.com/2006/05/people_remember.html)
Later in chapter 1 the authors discuss grading, a frustrating process for many instructors as they attempt to condense the myriad aspects of learning into a single letter or number. This section, however, is much too short and contains little about the actual process of determining who gets an “A.” Surprisingly, there is no mention of normreferenced versus criterion-referenced grading, although if that was considered too esoteric the topic could at least have been presented as the pros and cons of “grading on a curve,” a topic many instructors are familiar with. Nor do the authors explore the notion of assessment activities that do not contribute (point-wise, say) to a final grade and why one might integrate these into their instruction.
The second chapter, “Assessment Online,” takes the instructional design concepts presented earlier and applies them to online environments, emphasizing the alignment of assessment strategies with desired outcomes. Several principles for effective assessment are discussed, such as designing activities that incorporate collaborative peer review and using rubrics for clarity and consistency. This chapter also includes a section on plagiarism and cheating, and expands on earlier points specifically regarding cheating. However, for a book about assessment, dedicating less than two pages to both plagiarism and cheating is surprising, considering that these are such hot topics. It would have been helpful to read, for example, about technological deterrents to cheating that go beyond randomizing test questions, such as browser lockdown software. And, although plagiarism detection software is mentioned, there is really no discussion about how it works, or about the accompanying ethical and pedagogical issues to consider when choosing to use it or not.
“Course and Program Evaluation,” chapter 3, covers course evaluation in some detail, and (refreshingly) separates the evaluation of the course itself from the instructor's teaching behaviors. However, the perception that online courses should be evaluated differently than traditional courses is puzzling, considering that almost all of the criteria cited could be used for a web-enhanced face-to-face course, a blended/hybrid course, or a fully-online course. Granted, many traditional course evaluations are flawed, but this calls for an improvement to those evaluation instruments and/or methods, not the establishment of an additional, semi-redundant process. Sadly, online instruction is still viewed by many academics as less effective than traditional instruction; aren't we reinforcing that idea by suggesting that online classes need rigorous evaluation if we're not concurrently advocating this for traditional courses? An additional concern with this chapter is that course and program evaluation are substantial topics in their own right and warrant the development of a separate text. The audience looking for direction on these topics—administrators, typically—would be unlikely to look for it in this book.
Part 2: “The Assessment and Evaluation Toolkit,” provides examples of different types of assessments, drawn mainly from practitioners in higher education. This part of the text could be an excellent resource for those wondering how others have integrated specific assessment activities into their online coursework. Unfortunately, this half of the book has the earmarks of a hurry-up job that received little or no editing, and flaws in its visual and organizational presentation ultimately diminish its utility. For example, there are 14 assessment/evaluation techniques included on the initial list (p. 67), but only 13 are actually covered, and these appear to be in random order. The first section covers rubric design and development, the second the use of student feedback, and the third jumps back to the first topic with an example rubric designed to assess course interactivity. Why not include this example in the first section to illustrate the use of rubrics for evaluation? Or, if the plan was to separate assessment from evaluation, why not include the rubric with the sections on course evaluation and faculty evaluation that are presented as the two final topics?
The visual design elements also work against the user trying to locate a specific section quickly. It's not obvious where one section ends and another begins, illustrative graphics aren't designated as numbered exhibits (except for Exhibit 4.1, and it was left out of the List of Exhibits on page vii), and assessment examples are given mostly as narrative text, leaving the reader to wonder what they looked like on-screen for students, how much text was presented per page, or how feedback was displayed. Irrelevant information included with the examples makes it even more difficult to find a specific type of assessment. Do we really need “CHAPTER 23” included within an example to get the pertinent information about the design of a specific assignment? And would one or two arithmetic examples suffice in the section on tests, rather than including two full pages of them?
Additionally, instructors using some of the example techniques may encounter issues for which they are unprepared, because potential problems were not included in the discussion. For instance, the introduction to blogs and wikis explains that using an external/public site allows outside experts to contribute more easily than using a password-protected course management system. However, neither the possible student privacy (FERPA) issues that could arise when using a public space for course assignments are mentioned, nor the need for a back-up plan if an externallycontrolled server is taken offline unexpectedly. Short shrift is also given to possible issues related to peer or collaborative assessment, for example if a student files a grade appeal and claims that peer assessments unfairly influenced his or her final grade. How does an instructor deal with this? Are there ways to forestall such complaints? These are not trivial concerns; they are real and should have been addressed.
Overall, I found Assessing the Online Learner disappointing. I realize that Jossey-Bass wants to keep the texts in this series concise (i.e., affordable), but in this case that brevity resulted in too little substance. If there were concerns with length, chapter 3 (“Course and Program Evaluation”) could have been eliminated completely and the first two chapters beefed up. This may have also been the case with the “Additional Resources” section appended following the “Assessment and Evaluation Toolkit.” Surely there is more than one web site or other resource about self assessment, rubric development, or authentic assessment worth exploring, but each of these topics merits only one recommended resource. Those e-learning professionals looking for information about how to assess learning gains would be wise to look elsewhere.
