Introduction
The original goal of this investigation was to identify a potential tool that would consistently document student achievement of a major program goal: reflective and flexible thinking about professional tasks. A rubric based on Bloom's Taxonomy of Educational Objectives was used to code student posts in asynchronous discussion board (ADB) forums. Data consistently demonstrated gradual increases in the relative proportion of higher order content in discussion board posts from class members from the first to final week of a semester. Results were replicated in two successive years following the original pilot study. The author plans to use this tool in two ways: as part of ongoing assessment of student learning during a semester, and investigating the influence of such assessment on the quality of instructor discussion board contributions.
A primary academic goal of any instructor is to facilitate student achievement of course objectives. How does the instructor assess course components that are designed facilitate student achievement? To answer this question, the instructor must both assess student learning and evaluate individual course components where learning should occur. Thus, pedagogical assessments have dual objectives: to assess learner performance and to assess interventions aimed at eliciting certain outcomes.
In the Deaf Education personnel preparation program at a major Midwestern university, a primary objective is that students develop reflective and flexible thinking about professional tasks. Teachers of the deaf must constantly acquire new knowledge in a rapidly changing field and immediately apply it across a broadly heterogeneous population of students, evaluate the efficacy of their intervention strategies and actively identify the need for additional knowledge. Reflection and flexible thinking must occur continuously.
One advantage of participating in ADBs is the opportunity to engage in reflective thinking, defined by Lipman (2003) as “thinking that is aware of its own assumptions and impli cations as well as being conscious of the reasons and evidence that supports this or that conclusion” (p. 26), which results in conversational turns that are more carefully considered than might take place in a face-to-face classroom discussion (Wegerif, 1998). According to Rovai. Ponton, and Baker (2008), reflective thinking “promotes the ability of learners to construct deep personal knowledge” (p. 94).
The primary course component used to build reflection and flexible thinking is the ADB. Scholars of distance pedagogy have devoted considerable attention to best practice for student and instructor interaction on asynchronous discussion board (Jahng, Nielsen, & Chan, 2010; LaPointe & Gunawardena, 2004; Mazzolini & Maddison, 2002; Swan, 2002), because it is prevalent in online courses (Drouin, 2008; Woods, 2002).
In a review of 36 studies of online discussion participation, Hrastinski (2008) found that approximately one-third examined message quality, each using a different measure. Where higher order thinking was included as part of larger studies of participation, some studies did not clearly define “higher order thinking” and others used a variety of cognitive skills frameworks (Hrastinski, 2008). Given that each study based its coding scheme on a unique course, with unique course objectives, it is not surprising that multiple strategies were utilized.
All of these studies evaluated student discussion in discrete time periods as a measure of ADB efficacy. In contrast, the goal of this study was to evaluate a tool that would reflect the structure of the course, using ADB posts to observe student acquisition of new knowledge at the beginning of the semester and demonstrate increasing use of higher order thinking to manipulate that knowledge over time.
Method
Participants in this study were 14 students in two sections of an online teacher preparation methods course taught in two separate semesters. Each participant was a graduate student holding a valid initial teaching credential.
To observe student progress, an established taxonomy of higher order thinking was selected: Bloom's Taxonomy of Educational Objectives (Bloom, Engelhart, Furst, Hill, & Krathwohl, 1956), the standard for curriculum design in the cognitive domain (Anderson et al., 2001). The investigator created a rubric specific to ADB posts by completing a content analysis using all student ADB posts from a previous year's offering of the target course. By using student data, the investigator insured that actual thought processes, which typically appeared in the course discussion, supported each level of the research rubric. For each post, the elements of that post were described according to function, regardless of content. A single post might include one or more elements: an example from the student's experience, a request for clarification, comparison and contrast of two concepts, and/or a supporting reference to the course preparation materials. These functional descriptions were then sorted into groups consistent with the descriptions and questions commonly associated with levels of Bloom's Taxonomy and provided as examples of post characteristics at each level. The columns in the rubric contain all of the original elements judged as related to a given level.
The investigator assigned nominal values to elements of each post, corresponding to Bloom's Taxonomy. The investigator calculated the percentage of response components from each rubric level for each week. If students increased their use of higher order thinking, then the proportion of level 3, 4, 5, and 6 components in a forum should increase over time, and the proportion of 0, 1, and 2 level components should decrease, relative to the total number of components for that week.
Over two years, outside coders (two coders in Year 1 and one coder in Year 2), graduate and undergraduate research assistants from related fields completing directed research under the supervision of the investigator, were trained in the coding process using anonymous posts from the previous year. Each post was coded and then discussed with the investigator to identify potential areas of confusion.
During Year 1, both undergraduate coders completed each forum, and then met to check inter-rater agreement. The team used the intraclass correlation to assess inter-rater agreement across the twelve forums. Both raters rated all 12 forums in common. Inter-rater agreement was excellent (ICC = .94, p < .001).
During Year 2 a graduate student research assistant was trained on the first year posts and the assigned codes. When the graduate assistant achieved 90% or greater agreement with the previous year's coders, she continued throughout the semester to code independently. The investigator re-coded random discussion threads to maintain coding quality.
Results
In both years, Level 3 represented the largest percentage of total elements for each week; however, in Year 2, percentages of elements coded as Levels 4 and 5 increased over time while the percentage of elements at Levels 0, 1, and 2 declined.
Discussion
After 2 years, discussion board posts coded using a rubric of response characteristics categorized using Bloom's Taxonomy demonstrated an increasing proportion of higher order thinking elements over the course of the semester. Asynchronous discussion board posts contained increasing percentages of components that reflected application, analysis, and synthesis of course content. Results based on the rubric (designed as a tool to assess discussion board effectiveness) consistently suggested that students met the goal of using higher order thinking, and therefore increased their level of reflection on discussion board. Despite numerical differences in performance between classes, the trend from lower to higher levels on Bloom's Taxonomy as defined in the rubric was consistent during both years. Therefore, there is initial evidence that the rubric may provide a valid way to observe one type of student learning during the course.
This investigation represents only a single look at the potential efficacy of a rubric based on levels of higher order thinking as a tool to assess a primary course and program goal of increased flexible and reflective problem solving on the part of students. The number of participants was limited by the low class size accessible to the researcher in this discipline. The results are preliminary, and apply primarily to online courses in a particular program, where significant amount of course pedagogy takes place through asynchronous discussion board. Until a wider population and several courses can be included in the study, the rubric cannot be validated. Instructor participation in ASD also needs to be correlated with the student posts.
Finally, student activities and test grades and field experience outcomes need to be studied to determine the extent to which the increase in higher order thinking seen in this investigation influences the skills demonstrated by students during practicum and after graduation.
Complex teaching and learning technologies may call for increasingly rigorous tools for instructors to use in assessing the impact of their course designs and activities on student achievement of program goals. This study represents one effort to develop and test such a tool in a grounded manner in relation to a single course component within the context of a graduate level program and course conducted online.
