Determining Utility of Formative Assessment Through Virtual Community: Perspectives of Online Graduate Students

The rise in online graduate-level courses has created a unique challenge for educators: provide a superior education in this environment. Physical distance between graduate students can cause feelings of isolation, and is associated with student dissatisfaction, higher attrition rates, and decreases in measured learning outcomes in comparison to traditional courses (Ludwig-Hardman & Dunlap, 2003; Rovai, 2002a,2002b). Researchers have found successful online courses offset these negative outcomes by reducing the psychological distance between learners (So & Brush, 2007; Tyler-Smith, 2006). Defined as a sense of virtual community, feelings of trust, belonging, commitment, and shared goals between online learners are vital to students' experience in online courses and support student retention and success (Lui, Magjuka, Bonk, & Seung-hee, 2007; Rovai, 2002a,2002b; Shea, Sau Li, & Pickett, 2006).

A sense of virtual community has been called class culture or classroom community in face-to-face settings, but is more difficult and arguably more important to establish in an online environment (Brown, 2001; Chinnappan, 2006). Thus, researchers have begun to investigate possible improvements in cultivating virtual community in a number of online settings. One call for research has focused on implementing online formative assessment in a way that allows students to sense virtual community (Andresen, 2009; Challis, 2005; Dawson, 2006), with emphasis placed at the graduate level of instruction (Exter, Korkmaz, Harlin, & Bichelmeyer, 2009).

We aimed to address this research need by qualifying students' perceptions of formative assessment within an online graduate program. The research question asked what factors contributed towards students' experiences with online formative assessment while pursuing a master's degree in mathematics with a teaching emphasis. We found students determined the utility of formative assessment by the amount of virtual community the task allowed. The information from this study could help instructors make informed decisions to improve online formative assessment tasks to allow virtual community. Armed with such knowledge, policymakers and program developers could achieve maximal benefits when implementing formative assessment (Challis, 2005; Miller, 2009). The next section summarizes similar findings on virtual community and online formative assessment to provide a context for our study.

Research offered numerous reasons why graduate students' sense of virtual community affects the outcome of online courses. Given the physical isolation between participants, some researchers claimed virtual community was necessary to create an experience more familiar to the student to compensate for the lack of social and visual cues that cause a sense of disconnection in the participants (Lui et al., 2007). According to multiple views of learning, feelings of isolation or the inability to communicate with others can hinder students' education, since social interaction is often deemed essential in how students construct knowledge (Brook & Oliver, 2003; Gergen, 1995).

In online graduate education and leadership courses, Rovai (2002b) found a positive relationship between students' sense of community and perceived cognitive learning, claiming, “online learners who have stronger sense of community and perceive greater cognitive learning should feel less isolated and have greater satisfaction with their academic programs, thereby resulting in fewer dropouts” (p. 329). Lui et al. (2007) reported similar findings with graduate students in an online master's in business administration program, when strong feelings of virtual community were correlated with increased student learning outcomes, satisfaction, and engagement. Many researchers, including the ones in these studies, stress the need for online instructors to acknowledge the importance of virtual community in online learning environments (Rovai, 2001,2002a,2002b; Shea et al., 2006).

In addition to the instructor being an important component in the establishment of virtual community, researchers identified the technology implemented and the learners themselves influence students' (in)ability to sense virtual community (Barcelona, 2009; Rovai, 2002a; Vonderwell, 2003). Less examined is how formative assessment tasks contribute to graduate students' sense of virtual community, with many researchers warranting studies in this area of online education (Dede, Ketelhut, Whitehouse, Breit, & McCloskey, 2009; Lui et al., 2007; Signer, 2008).

Formative assessment has been named an essential component of online classes at the university level, and technology offers unprecedented opportunities for educators to provide quality formative assessment tasks to assess students learning (Challis, 2005; Miller, 2009). For example, computer-based assessment (CBA) allows a number of new formative tasks to be available for use including discussion boards, model answers, electronic feedback systems, reflections, and online smallgroup discussions (Thelwall, 2000).

Researchers have explored the learning benefits associated with formative CBA. Formative CBA can be completed at a time and place convenient for the student, allowing quicker, sometimes immediate feedback (Miller, 2009; Ricketts & Wilks, 2002). For instructors, formative CBA allows new and powerful modes of assessment evaluating a wide array of student abilities, and offer a more in-depth and current view of students' development. Some formative tasks also allow increased student-instructor options, such as chat boxes and discussion boards, that provide opportunities to record student interactions (Barajas, 2002; Conole & Warburton, 2005). In graduate engineering classes, researchers found formative CBA supported student learning, provided feedback, contributed to deeper learning, and increased student satisfaction of the course (Burrow, Evdorides, Hallam, & Freer-Hewish, 2005).

The benefits of formative assessment are not automatic; researchers caution that considerations are needed to ensure successful online formative CBA (Dobbs, Waid, & del Carmen, 2009). The need to carefully create discussion board tasks for students was one identified challenge, as ineffective tasks have been found to disadvantage students in undergraduate online courses (Downing, Lam, Kwong, Downing, & Chan, 2007). Other factors negatively influencing the effectiveness of CBA are the screen layout, mode of presentation, and amount of scrolling (Challis, 2005; Rickets & Wilks, 2002). Occasionally personal issues and quality of interaction were sometimes associated with using formative CBA in both undergraduate and graduate courses (Walker & Kelly, 2007). In particular, the type of feedback students find useful has been identified as an important area for future studies as the need for additional formative CBA research is admitted (Conole & Warburton, 2005; Downing et al., 2007; Miller, 2009).

Determining graduate student perceptions of their online courses constitutes a great research need in online education, particularly in the area of formative CBA (Burrow et al., 2005; Dobbs et al., 2009; Handley & Cox, 2007). Researchers comment this information would fill the gap in literature and assist instructors of online courses, who are one of the largest influencers on the course experience (Barcelona, 2009; Dawson, 2006). Armed with the knowledge of how students view these tasks, instructors could make more informed decisions to cultivate the positive learning outcomes of formative assessment and avoid the pitfalls discussed earlier (Signer, 2008). Graduate student perceptions of online courses have also been used as a quality insurance process and provided valuable assessment information to the institution and instructor for future course design (Lao & Gonzales, 2005; Young & Norgard, 2006). While undergraduate online teacher education programs are starting to be examined (Alexander, Lignugaris-Kraft, & Forbush, 2007; Carey, Kleiman, Russell, Venable, & Louie, 2008), the next section describes our efforts to meet this research need within the context of a graduate online teacher education program.

Our phenomenological study aimed to understand graduate student experiences with formative assessment within online mathematics education courses. The phenomenological approach was best in answering our research questions, since it aims to describe the experience and precipitating factors influencing the experience. Using a phenomenological approach also allowed us to examine student perceptions of formative assessment from different perspectives, thus helping identify themes and patterns in the data (Merriam, 1998). We chose to use the social constructivist epistemology because the theoretical framework seeks to understand how people view the world in an atmosphere where social aspects contribute to the education experience (Creswell, 2009). To address our research questions, we conducted interviews, observations, and open-ended surveys to capture participant perceptions of three online formative tasks.

All three researchers of the study were in a PhD program for mathematics education. Glassmeyer worked as a teaching assistant (TA) for the professor who taught the online courses in this study. His dual roles of both researcher and TA were identified to possibly influence participant responses, although effort was taken to make participants aware of the distinction between these positions. Dibbs had no direct contact with the participants but assisted with data analysis and provided researcher triangulation. Jensen had little prior knowledge of online courses and formative assessment but assisted in the design of the study and data collection. The research team compensated for each other's roles and biases throughout the project in addition to being reflexive during this study.

We examined two online courses in a two-year program designed for in-service teachers attaining a master's degree in mathematics. Professors from two universities in western states ran the program, which was in its second year of operation. Our participants, the first cohort of the program, started in the previous summer with a mix of face-to-face and online classes before switching to entirely online courses for fall 2009 and spring 2010 semesters. The participants were enrolled in two courses during the spring 2010 semester: a pedagogical content knowledge course on geometry and an action research project course. Most students in the program were enrolled in both courses, totaling 23 distinct individuals overall. Each course met once a week as a group via Elluminate, and used Blackboard for file posting and discussion board activities throughout the week. The courses reflected the program standard that all courses include “activities that engage teachers in collectively analyzing, exploring and understanding culturally responsive mathematics pedagogy” (course syllabus). Dr. Tanner, the instructor of both courses, attempted to reflect the standard by incorporating a variety of activities, including polling, break-out sessions, and discussion boards.

The typical Elluminate session incorporated two types of formative assessment: polling and break-out sessions. Near the beginning of most classes, Tanner posed a multiple choice question over the assigned reading, such as:

Each student in the class had a chance to choose one of the options. The results were usually displayed to the entire class in a bar chart that Elluminate produced at the instructor's command; occasionally the results were held privately for only the instructor to view.

Breakout sessions were an Elluminate tool the instructor used to place students into smaller groups of two to four to allow smallerscale conversations to take place. Tanner prompted the class with an open-ended question, such as “What do you think students may gain out of this type of an activity?”, before placing students in break-out groups to discuss for a few minutes with their peers. Normally each class had two or three break-out sessions, and afterwards a short whole-class discussion would ensue to summarize the main concepts. A 5-year veteran instructor of online courses, Tanner had taught both courses multiple times, yet the current semester was his first time using Elluminate.

The Blackboard website had a section devoted to discussion boards, where Tanner posted a prompt over the readings with instructions regarding student expectations. Because of their infrequency, there was no general establishment of how discussion board posts were used in the course.

The participants of the study were students enrolled in both online courses during the semester. Participants encompassed in-service teachers at both the middle and high school levels from two neighboring states. We e-mailed all the students three short surveys during the course with open-ended questions about their experience with a type of formative assessment recently encountered to provide baseline data (see Appendix A for the break-out session survey). We asked seven of these students to participate in interviews based on this data and six agreed to participate. We employed maximal variation sampling (Creswell, 2007) in the interview selection process, using the criteria of survey responses, state of residency, and teaching experience (Table 1). The six participants all taught at different schools, had attended different undergraduate institutions, and offered a wide variety of responses on their surveys. We also interviewed the instructor of the courses to gather data on the program and his specific views of formative assessment in the online classes he was teaching.

We conducted interviews online via Elluminate that lasted between 15 and 20 minutes; both Glassmeyer and Jensen were present during the majority of the interviews. We tailored the interview questions to probe and follow up items from the participant's surveys. The interviews were audio-recorded and later transcribed using an alias for each participant. Glassmeyer also observed both hour-long class sessions each week to triangulate the data, focusing on the six participants selected for interviews. After administering each interview and survey, the researchers wrote analytic memos and maintained an audit trail (Corbin & Strauss, 2008; Creswell, 2009).

We separately developed in-vivo code words from the data collected (Creswell, 2007), then collaborated to agree on the coding process. From the code words we constructed categories before turning to literature to assist in the development of themes. The survey responses and interview transcripts of the six participants were recoded using these themes to help answer our research questions. We invited all six participants to member check the themes and our results model; three participants chose to do so, and we used this information to structure our findings in the following section.

We found students' perceived utility of formative assessment tasks related to the sense of virtual community the task allowed. Students indicated virtual community and formative assessment were important parts of the courses, and furthermore connected the two course aspects. While they explained their views of formative assessment, the students brought up virtual community for a number of reasons. This section summarizes students' overall comments about virtual community, how those comments contributed to the utility of the formative assessment tasks, and the student-indicated relationship between formative assessment and virtual community.

Even though we focused our questions on formative assessment, we found that students frequently brought up virtual community in their responses. These statements presented a basis for the criteria participants used to judge formative assessment tasks. Students mentioned feelings of connecting and communicating were vital to their online education experience, especially sharing perspectives with one another.

All six participants expressed a desire to communicate with others in the course, which is typical for students in online courses (Palloff & Pratt, 1999,2005; Rovai, 2002b). Three participants stated the physical distance contributed to their increased desire to communicate, corroborating evidence that this need stems from the physical isolation between the online learners, possibly compensating for the lack of social and visual cues that cause a sense of disconnection in the participants (Dennen, 2005; Lui et al., 2007). Almost every student also stated communicating and connecting were important because, “it's human nature to want to do that” (Clayton). Angela offered a similar statement restricted to her cohort: “I kinda think that's why we're in the program because we all collaborate together and that's who we are that's the type of people we are.?

Within comments on communication students stated that sharing perspectives contributed to their learning experience. Four students shared comments similar to Camille's: “I really like the opportunity to hear what other people have to say and to really listen to their viewpoints and so I think that's why I really benefit from these [online classes].” Daisy went so far as to say, “I need to communicate with other students and have my thoughts validated or corrected” (Survey 1). Daisy and Camille's statements reiterate the importance students place on sharing opinions and perspectives in online classes (Rovai & Lucking, 2003).

Three participants indicated communicating in online courses was sometimes challenging. For instance, Daisy said, “there is this constant need to be able to talk and we're finding ways around the way the class is set up.” Other participants explained methods to overcome the communication challenges, such as presenting clear prompts for discussion boards and other options identified by researchers (Dennen, 2005; Lock, 2002; Nussbaum-Beach, 2007). While a few students said they had met face-to-face with other classmates to work outside of class together, almost all participants stated their main method of collaboration was through technology.

The instructor was cognizant of students' need to communicate; Tanner commented in his interview that the students valued communication in his courses. We observed Tanner occasionally trying to facilitate feelings of virtual community during the course, though he never commented any further about aspects of virtual community in his interview.

Realizing the importance students placed on aspects of virtual community and how they would use technology to meet these needs, we began to suspect students viewed formative assessment tasks in light of how well the task allowed communication and other aspects of virtual community to take place. Indeed, the perceived utility of virtual community reappeared later as an important factor in participant views of three formative assessment tasks.

From our observations, we found students had opinions on three formative assessment tasks: polling, breakout groups, and discussion boards. Students generally believed polling and breakout sessions were types of formative assessment and were associated with positive views; students claimed these positive views were because the tasks effectively facilitated aspects of virtual community. We found the same results concerning discussion boards even though initial student opinions differed greatly.

The main reason all six participants gave for the positive view of polling was the ability to share classmates' perspectives through discussion. Participants stated multiple comments indicating the value of communicating through polling, such as, “it [was] very easy to start good group conversations after polling” (Clayton, Survey 3). Participant responses indicated students valued interacting and communicating with each other and these feelings dominated how students felt about the polling task. Literature explains this may be because peer interaction helps support virtual community (Lock, 2002; Nagel, Blignaut, & Cronjé, 2009).

Two themes emerged as participants explained their experience with breakout sessions, both stemming from how the task allowed virtual community to take place. Every participant indicated breakout sessions allowed communication to take place, benefiting the students from the sharing of perspectives and thoughts on class material. Participants often said their favorite part of the task was hearing, talking, and sharing with others to gain new insight about the material. Angela added that collaboration with others in the breakout session helped answer questions on the material. Daisy and Clayton stated similar experiences that breakout sessions helped “form questions which I may not have realized that I had” (Daisy, Survey 1). Through their comments, participants clearly believed learning took place through these discussions, and contributed to the strong positive feelings towards breakout sessions.

The second theme was every participant felt more comfortable when communicating in the small group breakout sessions. Ted was particularly adamant about this, saying, “I think that those smaller groups definitely facilitate discussion a lot easier … in the smaller groups everybody gets a chance to speak.” Other participants said speaking to the small group was less intimidating and offered the opportunity to hear ideas from classmates who “don't speak up to the whole group as much” (Clayton). The participants explained the breakout sessions were a positive experience because of the increased confidence the task allowed in sharing with a few individuals rather than taking the time from the entire group. This “comfort that develops over time among members of a group” (Conrad, 2005, p. 1) is a vital aspect of virtual community, which Daisy illustrated in her interview:

Daisy's comment emphasized the value participants placed on communication and how the breakout sessions allowed students an opportunity to converse and develop a sense of virtual community (Barcelona, 2009; Li, 2008-2009).

Not all comments regarding polling and breakout sessions were positive; we found students identified problems that hindered a sense of virtual community and offered solutions to such obstacles. For instance, students stated a strong dislike of the poll answers being public, where everyone in the class could see what particular people voted. Instead participants preferred anonymous polls with secret answers, where the participants could see the overall class results, but not individual responses. In explaining her view, Diane said she felt this way because “it is a little embarassing to post your solution especially if you do not agree with all of your classmates, or if you get the answer wrong” (Survey 3). Clayton expressed a similar view that the answers should remain hidden because “for certain topics [it] might be more effective… many people just follow the responses given by the majority of the people” (Survey 3). Participant preferences over the specifics of a task related to the social aspects of polling, such as saving face and obtaining genuine responses. Participant preferences also seemed geared towards providing authentic and comfortable online environment, which is a theme other researchers have found in examining virtual community (Akyol, Garrison, & Opzden, 2009).

When speaking about breakout sessions, four participants expressed issues with group interaction that could be altered to achieve a more effective small group. Participants had negative views about ineffective groups characterized by nonparticipating members with frequent technological problems. Other studies corroborate these views of how the construction and implementation of small groups influences communication and collaboration in online settings (Cox, Carr & Hall, 2004; Nagel et al., 2009; Wilson, Ludwig-Hardman, Thornam, & Dunlap, 2004).

Overall participants expressed positive comments towards the tasks of polling and breakout sessions, focusing on how the tasks allowed students to experience virtual community. The negative comments aimed to improve the utility of the task by allowing students to experience a stronger sense of virtual community. Student comments concerning discussion boards corroborated these themes, though the overall impression of the task differed greatly.

Through the observations, we realized the instructor chose to use discussion boards infrequently. The instructor acknowledged this fact himself, which he explained was because, “now with the weekly webinars, we sort of get a rich discussion and so I haven't really thought about using the discussion boards in that way.” Participants clearly held a similar view of the task, stating discussion boards were not implemented during the current semester in a way they thought was formative assessment. Participants drew upon previous experiences in the online program where discussion boards, in their opinion, had been used effectively, and contrasted the experience with their current situation. The major problems participants cited for the low satisfaction with discussion boards during the current semester was because of improper implementation of the task and technological frustrations with the operating system used in the course. Researchers have found technological concerns about screen layouts and mode of presentation hinder the perceived effectiveness of formative tasks in online courses and may explain our participants' strong comments concerning discussion boards (Challis, 2005; Rickets & Wilks, 2002).

Despite difficulties, participants showed a desire for discussion boards to allow communication with peers and offered guidelines to ensure success in the future. Angela emphasized this desire when she said discussion boards “can be a pain when the topic is not very difficult or not stimulating. If the topic is a challenge it is nice to communicate with a variety of people” (Survey 2). Other researchers have found the same preferences, possibly stemming from a desire to maintain a sense virtual community in online courses through discussion boards (Downing et al., 2007; Rovai, 2002a,2002b). Overall, we believe comments about discussion boards corroborate the finding that students determine utility of formative tasks by the amount of virtual community the task allows.

All participants exhibited a strong understanding of formative assessment, perhaps due to a lesson observed early in the course where the instructor defined formative assessment. Most participants included some aspect of discussion into their explanation in their overall definition of formative assessment. Since the instructor's lesson lacked any inclusion of discussion in the definition of formative assessment, student comments indicate this aspect of virtual community was important enough to qualify as a distinguishing characteristic of formative assessment. The definition Ted gave encompassed all other participants' responses:

Every participant offered definitions in terms of what the instructor gained from formative assessment, with peripheral aspects of what the assessment provided the students. Including discussion in the definition of formative assessment corroborated participants' statements that polling, breakout sessions, and discussion boards should be designed for communication to take place.

Overall, most participants were pleased with their online classes, with participants favorably acknowledging formative assessment was being implemented in the courses. This positive perception of computer-based assessment and an overall positive perception of the online courses is a common characteristic in these types of courses (Denton, Madden, Roberts, & Rowe, 2008; Downing et al., 2007; Holmes, 2006; Walker & Kelly, 2007); however, including aspects of virtual community when defining formative assessment is a new finding not seen in many other studies. Likewise, specific comments on the three formative tasks revealed students defended their views of the task by the feelings of virtual community the task allowed. We believe a hole in the literature may have been filled by revealing a relationship between students' perceived utility of formative assessment and sense of virtual community.

Our study connected virtual community and students' perceived utility of formative assessment. Before accepting the relationship between these aspects of online learning, a few key limitations must be considered. Data concerning discussion boards largely contained information from the previous semester. These experiences are absent from our study because the experiences could not be validated via our observations, and other factors may have influenced the data. (Such factors included a different instructor teaching the previous semesters and using a different software.) Also, since Glassmeyer helped design and implement some of the formative assessment tasks, participant responses may have been influenced because of his position. Finally, participants in our study were in-service mathematics teachers in a graduate level program. Thus, the findings should be considered carefully before being applied to other populations, such as undergraduates or students in different disciplines, since the scope of our study remained on a group of professionals in a two-year graduate program.

By inquiring about student perceptions of formative assessment, our findings corroborated a number of other studies concerning online courses. Online formative assessment was viewed positively, with participants expressing a desire to communicate with others and thought the sharing of perspectives was vital to their learning (Barcelona, 2009; Denton et al., 2008; Downing et al., 2007; Holmes, 2006; Li, 2008-2009; Rovai, 2002b; Walker & Kelly, 2007). Participants indicated aspects of virtual community were a challenge to develop, and suggested improvements to classroom procedures to overcome these obstacles (Akyol et al., 2009; Dennen, 2005; Lock, 2002; Nussbaum-Beach, 2007).

Literature could not explain the connection we found between formative assessment and virtual community, which offers a number of implications to administrators, policymakers, and instructors. Participants in our study revealed a connection between the utility of formative assessment tasks and the amount of virtual community the task permits. Considering the multitude of benefits virtual community affords online education, such as increased learning and satisfaction (Buchanan, 2000; Challis, 2005; Garrison, Anderson, & Archer, 2001), online instructors should structure formative assessment tasks to allow interaction, collaboration, and trust between students. Policymakers and administrators who choose the software for online classes need to be aware of possible student technological frustrations that come with the program, and opt for technology that permits student interaction. We believe this to be important because we saw poor design and layout of discussion boards had a strong negative effect on the task, while activities such as breakout sessions were viewed by students as both academically valuable and enjoyable.

Although not measured in our study, we suspect the instructor was a main determiner of students' sense of virtual community. In his interview, the instructor identified that students valued virtual community, and in our observations he made efforts to contribute to that sense when possible. We believe because of these efforts, the students were able to sense virtual community, and contributed to satisfaction and learning in these online courses (Lui et al., 2007; Palloff & Pratt, 1999,2005; Rovai, 2002b). The positive findings are encouraging because other research warns that low levels of perceived virtual community can result in negative perceptions and learning outcomes of the online course (Rovai, 2002a). Considering the essential role the instructor plays in developing a sense of community (Barcelona, 2009), we believe professional development would help instructors be cognizant of virtual community's importance, and would support instructor efforts to create and maintain virtual community though formative assessment in their online classes (Signer, 2008).

We also recommend two directions for future research. In this article we have proposed virtual community and formative assessment are related, but we know little about this relationship. A future qualitative study examining this relationship would be valuable in determining how online instructors can incorporate formative assessment in ways that promote students' sense of virtual community. Secondly, a quantitative study measuring formative assessment in relation to student satisfaction and achievement could determine the strength of the relationship, and offer instructors another indication of the importance of these aspects of online education. We hope this information helps educators take the next steps towards improving the quality of online education, especially within the context of graduate teacher education courses.

This research was partially supported by the National Science Foundation under Grant No. DUE0832026. Any opinions, findings and conclusions or recommendations expressed in this material are those of the authors and do not necessarily reflect the views of the National Science Foundation.

Recall that each week in class we usually split everyone into groups of 3-4 students for discussion. The following questions will refer to this activity as a break-out session.

1. My overall feelings toward the formative assessment task of being in a break-out sessions has been:

2. What do you think is the purpose or the goal of break-out sessions is within this course on teaching geometry or the action research course?

3. Did the task achieve that goal? How?

4. Did you find the break-out sessions helpful in your learning? Why or why not?

5. I enjoyed being in a break-out session boards because …

6. The use of break-out sessions in online instruction can be improved significantly by …

7. What is the best thing about this task? Feel free to list two to three things with an explanation for your response.

8. What is the worst thing about this task? Feel free to list two to three things with an explanation for your response.

If you have any other comments about this, please just put them below.

Thank you so much for your time!