Evaluating College Students' Efforts in Asynchronous Discussion: A Systematic Process

Whether distributed as a listserv or centrally-located on a bulletin board, asynchronous computer-mediated discussion can be a valuable component of the higher education classroom. Superficially, in Web-based instruction, asynchronous discussion serves as a reminder to students that they aren't alone in the educational process (Weiss, 2000). More substantively, through asynchronous discussion, students have an opportunity to develop a more thorough understanding of course content. As students share their ideas, perspectives, and experiences, they are broadening their classmates' frame of reference for interpreting course content (Jonassen, Davidson, Collins, Campbell, & Haag, 1995; Morrison & Guenther, 2000). As Hacker and Neiderhauser (2000) note, this type of collaboration makes learning more “deep and durable” (p. 53) and can increase student motivation.

However, evaluating students' efforts in online discussion is sometimes awkward. After all, a valuable online discussion is based on students' initiative—their willingness to share their understandings of course content and react to their classmates' understandings—so it should be no surprise that students view professor-centered evaluations as antithetical to the very types of initiative that make online discussions educationally sound. Also, evaluating students' efforts is awkward because online learning environments don't always “support” authentic assessment (Marra, 2002). Because of the limitations of many online learning environments, evaluation does not happen naturally in asynchronous discussion. Rather, designers of online environments and professors who use asynchronous discussion must integrate evaluation strategies into asynchronous discussion.

In this paper, I offer a process-based framework for evaluating students' efforts in online discussion. Importantly, this framework is systematic in that it is based on inputs and corresponding outputs (see Table 1 for a summary). These inputs and outputs create a cyclical process. Professors' and students' inputs lead to analysis of evaluation criteria and application of those criteria. Analysis and application lead to outputs, which can further inform students' participation in the asynchronous discussion itself. Thus, evaluation is not tacked on to the end of the discussion; it is integrated into the discussion process and designed to improve the quality of the discussion itself (see Figure 1 for a summary of this cycle). Both the systematic and cyclical elements of this process are derived from a variety of already-existing literature—including articles dealing with assessment theory, writing-across-the-curriculum literature, cognitive learning theory, and constructivism.

I begin by delineating criteria for evaluating students' contributions to an online discussion. I then describe processes for implementing student-centered evaluations and professor-centered evaluations. Throughout this paper, implications for online pedagogy will be relatively obvious; so, in the last section of this paper, I offer implications for future theoretical and empirical endeavors.

Professors should establish criteria for evaluating students' efforts in online discussions and communicate those criteria to students. Professors sometimes assume that criteria and standards for evaluation are self-evident (Speck, 1998); this assumption is problematic. When professors fail to communicate standards for participation, students have no choice but to revert to their own past experiences to help them define acceptable levels of performance (Knowlton & Knowlton, 2001). Therefore, professors must establish clear criteria that can serve as the basis of student-centered and professor-centered evaluations, which will be discussed later in this paper. But what criteria should professors rely on for evaluating the products of online discussion? I begin this section by discussing two mistakes that professors routinely make when determining criteria as a basis for evaluation. Then, I offer criteria that can become a basis for evaluating students.

The first mistake that professors often make when selecting criteria as a basis for student evaluation is that they confuse research rigor with pedagogical rigor. This tradeoff of privileging criteria based in research rigor instead of criteria that are pedagogically useful can be destructive. Evaluating students, for example, from a linguistic perspective (cf. Baym, 1996; Murray, 2000) or assessing students based on cultural (cf. Herring, 1999) or social (Rourke, Anderson, Garrison, & Archer, 1999) factors is instructive for researchers and theorists. But these same criteria are relatively useless for students who are trying to determine if their contributions to an asynchronous discussion evidence an understanding of key concepts that are specific to a discipline, like literature, engineering, or biology.

In pointing to the need for professors to develop criteria that are pedagogically useful, I am not suggesting that research is unnecessary. I am not even suggesting that criteria related to linguistics, culture, or social factors are useless. After all, such criteria can provide students with opportunities to think metacognitively about their participation in asynchronous discussion—a topic addressed later in this paper. I am suggesting that the primary criteria that will become the basis of student evaluation should be useful to students as they evaluate the degree to which they understand the content of a course, not the linguistic, cultural, or social cyber-milieu of the discussion itself.

A second mistake that professors often make when selecting evaluation criteria relates to the consistency between the criteria and the nature of online discourse. For example, Bauer and Anderson (2000) suggested that professors focus on the degree to which a student “uses complex, grammatically correct sentences on a regular basis.” They also urge professors to consider whether or not a student “expresses ideas clearly, concisely, cogently, in a logical fashion” (p. 68). These criteria suggested by Bauer and Anderson imply that contributions to an asynchronous discussion will be revised and polished into formalized writings, but online discourse is more similar to conversation than to formal writing. Students rarely engage in substantive revision when participating in online discussion. Typically, students write a contribution for the discussion, perhaps read over it once, and submit it to a discussion forum. In some respects, then, Bauer and Anderson's criteria are not consistent with online discussion contributions.

Similarly, Morrison and Ross (1998) argue that points should be deducted for off-topic comments. But online discourse is a type of writing in which students often learn about their own perspectives as they write. When they begin a contribution to an online discussion (and sometimes even as they end one), students may not know their own exact point. They are engaged in a type of “writing to learn” (see Lindemann, 1995, and Adams & Hamm, 1990, for a discussion of “writing to learn”). In fact, the more text students generate in a contribution to discussion, the more likely those students will generate ideas from which they can learn. Therefore, criteria for evaluating students' contributions to online discussion should encourage students to explore their ideas in writing even if the exploring creates digressions and seemingly off-topic comments. In other cases, the relevance and scope of asynchronous discussion emerge as different students contribute. In asynchronous discussion, knowledge is collectively created and socially distributed (Moller, 1998). To penalize individual students for seemingly off-topic comments, then, is to undermine the collective and social nature of knowledge.

As I've argued, criteria for evaluating students' contributions to asynchronous discussion should provide students with an understanding of the degree to which they understand key course concepts. Criteria for evaluation also should be consistent with the “rough draft” nature of most discussion contributions and enhance the opportunity for relevance to emerge throughout the course of a discussion. Even with this consideration of consistency, though, criteria may be idiosyncratic to specific professors and disciplines. So, the criteria in this section can serve as a starting point for consideration, not as a definitive list of criteria that will work as a basis for all asynchronous discussions. After outlining criteria for asynchronous discussion, I describe cognitive approaches for helping students understand and internalize the criteria.

When writing initial contributions to a discussion, professors might urge students to consider the following questions:

• Is the contribution mechanically clear enough for readers to understand your points?

• Does the contribution cite assigned readings or other resources as a means of addressing the specific topics and/or questions?

• Does the contribution seem to be based on factually-correct ideas that are generally accepted as discipline “truths”?

• Does the contribution contain “critical thinking” that is indicative of the paradigms in the field?

Because replies in asynchronous discussion are unique in their educational purpose, a separate set of criteria should govern their quality. For evaluating responses to classmates, professors might suggest that students consider the following criteria:

• Does the content of the reply adopt a respectful tone towards the ideas in the contribution to which you are replying?

• Does the reply inspire further discussion among the class?

• Does the reply develop the scope of the discussion in a meaningful way?

Clearly articulating the criteria may not be enough; professors may need to provide resources or facilitate educational activities that will help students understand the nature of the criteria. For example, critical thinking is abstract, and students may not know how to operationalize and apply it in their discussion contributions. Professors, therefore, may need to provide a resource that would help students with their applications. A discipline-based heuristic that defines critical thinking can help scaffold students' thinking about course content. In economics, for instance, Gwartney and Stroupe's (1995) eight guideposts for economic thinking can help students gauge the degree to which they are thinking “like an economist.”

Sometimes, though, a resource may not be enough; students may need to participate in activities to gain a more full understanding of evaluation criteria. For example, in teaching an online English composition course, I required students to participate in an online discussion that developed strategies for including critical thinking in their contributions. Some strategies that students developed were relatively obvious, but they introduced some novel strategies, as well. One education major, for example, introduced the class to Bloom's cognitive taxonomy (Bloom, Englehart, Furst, Hill, & Krathwohl, 1956). She suggested—and her classmates and I agreed—that contributions on higher levels of the taxonomy (application, analysis, synthesis, and evaluation) might be more indicative of critical thinking than contributions that only demonstrated knowledge and comprehension of course content. To some extent, then, these students were able to operationalize the criteria to meet their own needs. Through trying to understand a criterion like critical thinking, students renegotiated the scope of that criterion. Anderson (1998) has suggested the value of this type of negotiation. Involving participants in discussions about criteria and their scope can help those students understand the ways that educational goals can be met through online discussion.

Professors should actively engage students in evaluation processes. Students should not only be recipients of evaluations, but also they should serve as evaluators. In this section, I describe processes for students' self evaluations and peer evaluations. Within each, I discuss professor inputs that likely will promote productive student-centered evaluations.

Self evaluation provides students with an opportunity to critically re-examine their own contributions to an asynchronous discussion. Self evaluations are only successful when students understand the criteria and practice applying the criteria to their work (Knowlton & Knowlton, 2001). Two strategies are particularly useful for helping students evaluate their own contributions to a discussion. The first is the use of checklists. The second is the use of generative self evaluations.

First, both Likert-scale and dichotomous checklists can help students evaluate their own contributions to a discussion. For example, by marking a scale ranging from “strongly agree” to “strongly disagree,” a student can react to a criterion statement: “My contributions to the discussion contain critical thinking.” Or, by simply marking a “yes” or “no,” students can check that they referenced assigned readings in their contributions. Checklists can be advantageous because they help students focus specifically on the salient criteria. That is, the close-ended nature of the checklists can help students evaluate their contributions in narrow and discrete terms. A disadvantage of checklists, though, relates to this same close-ended nature. Students easily can become entranced with simply checking positive responses, without closely analyzing the fit between their contributions to the discussion and the criteria (Knowlton, 2001).

Second, students can be asked to complete a generative self-evaluation. Because generative self-evaluations take more time than completing a checklist, they should be used sparingly. But, at key points during the span of a discussion, professors might require students to write a paragraph demonstrating how they met each criterion in past discussions or how they will change their participation to better meet that criterion in future discussions. Professors even might require students to cut and paste excerpts from their contributions as evidence of meeting the evaluation criteria. Similarly, some discussion environments might allow students to “tag” their discussion contributions with additional comments—similar to the “comment” function in word processing. This type of tagging would create a type of meta-discussion that could be seen easily by both students and the professor. A generative self-evaluation moves students beyond passively marking a checklist. Generative self-evaluations obligate students to actively synthesize their own contributions to a discussion with evaluation criteria.

Lin (2001) argued the need for strategy training as a means of promoting metacognition among students. I am arguing that a generative self evaluation is one such strategy that would be useful in online discussion. Generative self evaluations can extend students' thinking beyond the immediate content and engage them in issues of ontology and epistemology. They consider what they really do know and how they have come to know it. As students engage in generative self evaluations, they are thinking about their own thinking. This approach is consistent with Prestera and Moller's (2001) discussion of evaluation because it forces students to examine their own process, not just their products. Such an approach is also consistent with Marra's (2002) vision that online discussion has “the potential to call into question learners' existing epistemic beliefs” (p. 23).

Online discussion creates a self-contained rhetorical context (Anderson, Benjamin, Busiel, & Parades-Holt, 1998) and gives students an authentic audience (Fabos & Young, 1999) that can provide feedback in ways not typical of other writing contexts (Kemp, 1998). To some extent, then, the natural development of an online discussion is a form of peer evaluation. By reading responses to their contributions, students learn about the quality of their own ideas and gain insights into how others receive those ideas.

Though online discussion does have a type of built-in peer feedback mechanism, students also can benefit from more formalized peer evaluations that require students to focus specifically on the pre-established evaluation criteria. Obviously, checklists similar to those described for self-evaluation can be used for peer evaluations. Also, students could be required to complete a generative analysis by answering open-ended questions about their classmates' participation. But, peer evaluation in itself should involve discussion and significant exchange of ideas. For example, professors can create “peer feedback groups” to allow participants to achieve more depth in evaluating each other's contributions to asynchronous discussion. The smaller the peer feedback groups, the stronger the opportunity for obtaining depth in peer evaluations; the more often professors require students to participate in peer feedback group, the more specifically students will be able to refine their own understanding of how to contribute to asynchronous discussions.

As professors themselves give feedback in online discussion—a topic discussed in the next section of this paper—they certainly should act as model evaluators. Students can learn how to give feedback by being the recipient of carefully-constructed feedback. Beyond modeling, though, professors might find it helpful to scaffold students' abilities to give constructive feedback to their peers. For some students, direct practice of peer evaluation can be useful. To this end, professors can ask students to evaluate fictional students' contributions to a discussion as a means of providing non-threatening practice for peer evaluation.

More directly relevant to the task of peer evaluation, professors might consider giving students a job aid that offers guidelines for productive peer evaluations. Students can gain insights from simple advice for avoiding personal attacks and ideas for using Socratic questioning as a part of peer evaluations. Effective job aids also might guide students through the peer evaluation by including specific questions for students to consider. These questions might be fairly straightforward, such as asking students to identify three strengths and three weaknesses of contributions written by classmates in their peer feedback group. But, these questions, though, might be much more specific and ask students to pick a contribution of a classmate and evaluate it based on each discussion criterion.

Though self-evaluation and peer-evaluation are important processes that can add considerable depth to the quality of online discussion, the professor holds the ultimate responsibility of grading. Often, students view a grade—and the anticipation of receiving a grade—as punitive. Therefore, professors should consider offering meticulous formative feedback throughout the span of a discussion. Formative evaluations will better establish a context for summative evaluation.

As shown in Figure 1, professors should offer formative feedback throughout the online discussion, even as students are engaged in student-centered evaluations. This placement of formative feedback within the evaluation system is consistent with the notion that professors have the responsibility to “frame” students' online activities (cf. Knowlton, 2000, p. 11). Formative feedback should not only be a preliminary judgment of students' contributions, but also formative feedback should offer students advice for improving their contributions. A simple letter grade, number, or other marking (like a check mark) does not serve the function of giving students insights into the strengths and weaknesses of their own contributions. To help students see strengths and weakness in their contributions, professors should explain to students how they reacted—as an authentic reader—to the student's contribution. For example, professors might describe how a student's discussion contribution inspired them to think more (or differently) about the topic. Or, the professor might offer feedback that suggests different connections to course readings. The professor can describe how a consideration of a specific reading could add to the discussion in substantive ways.

My experiences suggest that students initially will be overwhelmed by authentic feedback from a professor. Students often aren't accustomed to professors treating their work with legitimacy. They are more accustomed to professors handing down clear-cut judgments that culminate in a grade. Therefore, students initially may find the authentic feedback to be esoteric and needlessly obtuse. Over time—perhaps the course of a semester, for example—students will become more adept with using authentic reactions from a professor to improve their contributions to the discussions.

Professors should meticulously connect the grade to established discussion criteria, evaluations from peers, and formative feedback from the professor. One way to help students see the relationship between pre-established criteria and a grade is to use grading rubrics. For example, in one class that I taught, I used online discussion as an impetus for connecting textbook readings with students' real world experiences. The criteria used were similar to the criteria described in section one of this paper. Table 2 shows the rubric that I used for summative evaluation of students' initial contributions to a particular discussion about the text.

Notably, the issue of a contribution being “on time” is evident in the rubric. The particular course in which I used this rubric was divided into weekly modules. Each discussion lasted the span of one or two weeks. So, if students waited until late in the week to offer their contribution to the discussion, other students didn't have time to respond. That is, the deadlines ensured opportunities for interaction among students.

In this same course, I used a separate rubric to summatively evaluate students' replies within a discussion. This rubric can be found in Table 3. Importantly, in this particular course, students were required to offer a minimum of three replies per week. So, a minimum number of replies is an additional criteria listed in this particular rubric but not discussed earlier in this paper. As I've noted, criteria may be idiosyncratic to particular professors and courses.

Establishing clear criteria, creating opportunities for self evaluation, facilitating peer evaluation, and offering formative feedback can form the cornerstones of strong evaluation procedures for online discussion. Notably, constructivist, metacognitive, and generative principles of learning theoretically frame the evaluation system that I've described in this paper. Perhaps more substantively, this paper connects asynchronous discussion to the writing-across-the-curriculum literature by introducing the idea of pre-established evaluation criteria, self evaluation, and peer evaluation. This paper seems to be one of the first to implement connections between the constructivist nature of online discussion and principles from the writing-across-the-curriculum movement into a systematic approach to evaluation.

Pedagogically, the approach described in this paper allows evaluation to become an integral component of the learning process in online discussion. No longer is evaluation tacked on to the end of a discussion; evaluation contributes to students' learning and becomes a part of the educational experience. This idea of integration of evaluation into discussion should be useful to designers, professors, and faculty development specialists. Beyond the immediate pedagogical implications, the process of evaluation described in this paper raises questions that should be addressed in future theory and through empiricism.

In articulating the role of self-evaluation, peer evaluation, and professor-centered evaluations, I am making the assumption that the evaluation process will engage all students; but just as some students have the propensity to lurk in online discussion, students may tend to lurk during peer evaluation. As Lave and Wenger (1991) have noted, legitimate reasons may exist for participants to stay on the periphery of educational activities. What are those reasons? Can learning theory account for them? Can instructional prescriptions guide students toward more active participation throughout the evaluation process? Theoreticians need to address many questions about the behavior of passive participants. Theoretical connections between current learning theory and asynchronous discussion can begin this question-answering process.

Another area for theoretical investigation relates to how these evaluation processes transcend disciplinary lines. My pedagogical experiences are mostly in social sciences and humanities. Does this process that I describe apply equally well to online discourse in the hard sciences? Are self evaluations and peer evaluations appropriate in business courses? I argue that they are, but I would suspect that the processes manifest themselves slightly differently in various disciplines. Theorists across disciplines should offer prescriptions for implementing self evaluation and peer evaluation into online discussion.

The systematic evaluation process that I describe in this paper is based on the assumption that more evaluations in different varieties will result in improvements to the quality of the discussion. But, does more evaluation necessarily equal a stronger quality of discussion? Admittedly, quantity is probably not the issue; the specific self evaluation and peer evaluation strategies that are used are probably a stronger predictor of improvements in discussion quality. Which self- and peer-evaluation strategies promote the greatest improvement in students' contributions to asynchronous discussion? Quantitatively, different peer evaluation and self-evaluation strategies need to be compared. Are dichotomous checklists, for example, any less effective than generative evaluations?

Qualitatively, researchers should examine students' understandings of formative evaluation. Do they understand the formative feedback? Do they know how to use it toward the goal of improving their own work? What types of feedback do they view as helpful? Part of this type of empirical investigation should focus on students' metacognitive competence. But, part of this investigation might depend on gaining an understanding of students' perceptions of the role of a professor in online discourse. Rhetorical analysis, students' personal narratives, and case studies would be useful in understanding students'perspectives.