Data Dashboards to Support Facilitating Online Problem-Based Learning Free

Problem-based learning (PBL) is an instructional approach that begins with a complex problem. In PBL, students learn collaboratively as they engage in cycles of problem formulation and analysis, self-directed learning, integration and evaluation of their ideas; instructors act as a facilitator to help support and guide the collaborative learning (Hmelo-Silver, 2004). Although PBL relies on instructors engaging in the labor-intensive practice of facilitation, there has been mounting interest over the last decade in bringing PBL and other types of inquiry learning into online environments (Tsai & Chiang, 2013). This study is designed to develop knowledge to support the design and use of technology with the goal of extending skilled facilitation resources to support multiple collaborative groups (Steinkuehler, Derry, Hmelo-Silver, & DelMarcelle, 2002). To accomplish this, we have developed and deployed an online learning environment, HOWARD (Helping Others With Argumentation and Reasoning Dashboard) to support both students and instructors in enacting PBL (Kazemitabar et al., 2016).

In online learning environments, students generate large amounts of data and metadata that can be mined, analyzed, and visualized; these visualizations are designed to provide deeper understandings of student activity and inform instructional decision-making.

Learning analytics (LA), a recently emerging field, is built on such practices (Baker & Inventado, 2014). LA tools, which synthesize and condense student-generated data, typically aim to identify struggling students to aid instructors in providing them with support (Arnold & Pistilli, 2012). LA tools typically condense student data into a visualization, making it easier for the instructor to understand, and dashboards are often the visualization of choice (Gašević, Dawson, & Siemens, 2015). Owing to challenges encountered with approaches centered on predicting student success, LA researchers have called for investigations that take a more theoretically informed approach aimed at improving instruction (Gašević et al., 2015).

In collaborative learning environments, dashboard visualizations are functional tools with visualized images produced by real-time analysis of participation to support collaborative learning (Janssen, Erkens, Kanselaar, & Jaspers, 2007). However, to generate actionable information, these data need to be in a form that can provide a basis for real-time instructional decisions.

Our study investigates the use of an LA dashboard to assist instructors in an asynchronous, online PBL workshop for medical students learning about medical communication, specifically learning how to deliver bad news to patients. While LA dashboards would seem to offer substantial support to instructors, little is known about how instructors make sense of student activity in asynchronous online PBLs. Furthermore, little is known about how instructors’ decisions are influenced by dashboard visualizations of student activity. We investigate these issues through iterative design-based research (Bannan-Ritland & Baek, 2008; Cobb & Gravemeijer, 2008; Sandoval, 2004).

This study was a small part of a larger design-based research effort concerning use of technology in medical education, with one line of research focused on the use of technology to aid medical students’ education in the delivery of bad news (Hmelo-Silver et al., 2016; Lajoie et al., 2014). The dashboard tools are described in more detail elsewhere (Kazemitabar et al., 2016); this article is focused on instructors’ use of a LA dashboard designed to aid instructors’ support of student learning.

PBL is a pedagogical approach in which students begin their learning with an ill-structured problem. In small groups, students iteratively cycle through a tutor-facilitated process that includes key phases, such as problem definition, identification of relevant facts, hypothesis generation, identification of gaps in knowledge, self-directed learning, and abstraction of newly generated understandings (Hmelo-Silver, 2004; Savery, 2006). Instructors take the role of tutor, and aid students through their problem solving process by facilitating group discussions and guiding students toward intended learning objectives (Wood, 2003).

Instructors in PBL are called facilitators and/or tutors. They provide support and scaffolding to students by engaging in a repertoire of strategies, which include open-ended questioning, revoicing, encouraging summarization, hypothesis generation and evaluation, creation of learning issues, and consensus building (Hmelo-Silver & Barrows, 2006). Because PBL problems are specifically designed to be ill-structured and complex, they require a high degree of support, scaffolding and tutor interaction to help manage the complexity of the problem (Hmelo-Silver, Duncan, & Chinn, 2007; Kirschner, Sweller, & Clark, 2006).

As an instructional model, PBL was designed to be implemented with a low student-to-teacher ratio, making instructor attention a critical factor in determining resource costs and scalability (Albanese & Mitchell, 1993). Several strategies have been adopted to make PBL available to traditional-size courses. These include training peers to function as facilitators (Duch, Groh, & Allen, 2001; Schmidt, Van Der Arend, Kokx & Boon, 1994); one instructor facilitating multiple groups (Hmelo-Silver, 2004), and combining the use of peer tutors with a single facilitating instructor (De Simone, 2008). Other unique pedagogical approaches in midsize classrooms include problem-based inquiry, which involves creating multiple small groups where each group has a different role in an open-ended problem; the knowledge generated by each group is then shared with the entire group (Brush & Saye, 2008; Saye & Brush, 2002). The current study uses visualizations to increase an instructor’s ability to facilitate multiple groups.

PBL has been in use in medical education for nearly 40 years, and has also been adopted for use in other contexts, including K–12 education in the sciences and humanities, higher education, and even police and military training (Walker & Leary, 2009). The concept of “asynchronous online PBL” is, in other contexts, labeled as distributed PBL (Barrows, 2002), distance PBL (Cheaney & Ingebritsen, 2005), asynchronous PBL (Hmelo-Silver, 2002), PBLonline (Jennings, 2006), and simply online PBL (Orrill, 2002). For simplicity, we will use the term asynchronous PBL (aPBL) in this article.

Researchers have investigated both synchronous and asynchronous approaches to PBL. Synchronous efforts have included using student-selected avatars in virtual worlds to encourage multiple perspective-taking (Annetta, Murray, Laird, Bohr, & Park, 2008), as well as video conferencing software deployed with rich video cases (Hmelo-Silver et al., 2016; Lajoie et al., 2014). Although synchronous online PBL designs can provide insight into offering PBL across geographic distance, it does not address the asynchronous design needs for distance education.

Asynchronous PBL is often conducted in a threaded discussion format, which encourages replies to an idea; however, such threads are often difficult for a facilitator to track the development of the discussion (Orrill, 2002). Also, students in aPBL discussions tend make fewer but longer statements, which has also been described as more reflective (Hmelo-Silver, 2003; Lan, Tsai, Yang, & Hung, 2012). These threaded discussions present challenges for aPBL facilitators, because they have less opportunity to provide immediate feedback in the appropriate discursive context (Hmelo-Silver & Derry, 2007). Other challenges include limited student participation (Hewitt, 2005), low levels of discourse (Guzdial, 1997), and a tendency for discussions to get off-track (Ellis, 2001; Dennen, 2005).

Research on asynchronous learning discussions has offered some insight into how these challenges can be addressed. Hew and Cheung (2008) summarize research on the actions that instructors can take to address such issues: similar to PBL facilitation, asynchronous online discussions can be motivated through instructor encouragement, revoicing, providing response guidelines, and summarizing discussion (Beaudin, 1999). Other effective approaches include responding to a majority of student posts, directing comments to specific individuals, and significant instructor participation (Tagg & Dickinson, 1995). However, all of these strategies are time intensive, and depend upon an instructor’s ability to understand what is happening in student groups.

LA is an approach to analyzing studentgenerated data, and using that data to inform instructional decisions, ultimately in hopes of increasing student support and understanding. As a type of educational data mining, LA approaches collect, analyze, and illustrate learner-generated data to “inform and empower instructors and learners,” with the goal of informing educational decision making (Siemens & Baker, 2012, p. 253). LA as a field is emerging, and recently developed frameworks have worked toward identifying critical elements of LA (Greller & Drachsler, 2012). LA is united by use of analysis of complex data sets to predict student outcomes (Ferguson,2012). Several forces have led to the rise of this field, including the availability of big data, the applicability of online learning in different contexts, the various needs of participants, and the possible enhancement of decision-making processes (Ferguson, 2012; Greller & Drachsler, 2012).

Despite the focus on prediction and informing administrative level decisions, LA researchers also take an interest in course-level approaches that provide instructors with information that informs instructional decisions (Arnold & Pistilli, 2012; Ferguson, 2012; Siemens & Long, 2011). These efforts showed initial promise but tended to focus on superficial data and provided low-level support for students, which led to calls for approaches more substantially informed by instructional theory (Gašević et al., 2015). LA researchers have also conjectured that cuing support as an “intervention”may result in instructor passivity, and that LA tools should be studentcentered, cuing instructors into processes of student learning, inquiry, and reflection (Kruse & Pongsajapan, 2012).

The use of LA tools in PBL instruction has included combinations of log and self-report data that identify students who are not as engaged in their PBL groups (Templaar, Heck, Cuypers, van der Kooij, & van de Vrie, 2013). Other efforts have aimed at understanding student collaborations in small groups by using social network analysis (SNA) (DeLaat, Lally, Lipponen, & Simons, 2007), and used of a combination of tools to identify most frequent and representative words in the group discourse (word clouds) to demonstrate the evolution of group discourse in a dynamic manner (Cui et al., 2010). In designing LA for PBL, it is important to consider ways in which LA can provide deeper indicators of high quality engagement that can extend skilled facilitation.

We use PBL as a theoretically motivated instructional approach to connect with LA, in an attempt to help instructors scaffold students’ collaborative learning processes in an online learning environment, using the Helping Others With Argumentation and Reasoning Dashboard (HOWARD).

The purpose of this study is to investigate instructor’s understanding of LA dashboards in PBL. LA dashboards hold potential for supporting instructors facilitating multiple groups when using PBL (or other inquiry pedagogies) online. We conjectured that investigation into instructor use and interpretation of student activity could provide a foundation for building theory to inform how LA dashboards should be designed and used in student-centered learning environments.

Although previous research provides a multitude of tools and visualizations for understanding student-generated data, little is known about how instructors understand, analyze, and facilitate student activities with the assistance of visualization tools in online, inquiry-oriented pedagogy. This study investigated the following research questions:

1. How do PBL facilitators, aided by LA dashboards, understand and facilitate student activity in asynchronous online PBL?

2. How do instructors understand dashboard visualizations of student activity?

3. What challenges and opportunities exist for LA tools in PBL environments?

The context for this study was learning to deliver bad news in a medical context using a heuristic for the delivery of bad news to patients (Lajoie et al., 2014). This took place in an asynchronous PBL environment, a workshop that was 1.5 weeks in duration with medical students from North America and China.

The learning environment we developed has two main facets: (1) a student environment in which students engage in the instructional activities, and (2) an instructor dashboard, which condenses and visualizes student activity.

As shown in Figure 1, the student environment/interface is composed of three components: (a) a video feed, for receiving, selecting and viewing video cases and other additional supportive videos that familiarize students with the interface, and help students identify the central problems. (b) A semi-threaded discussion space to support collaboration in a chat space for students to brainstorm, exchange ideas and negotiate ideas, (c) a section with a “whiteboard,” a document that all participants (both instructors and students) can edit that serves as a shared space for keeping track of where they have been and where they need to go in their problem solving. To enhance participation, we designed a video-annotation tool for students to make comments in selected sections of the video from the two contrasting video case-based scenarios. Students could choose cases from the video feed and click to annotate the video, which leads them to a separate page to make the annotation. The whiteboard serves as a metacognitive scaffold, which is designed to help students think through the problem with which they are confronted (Reiser, 2004). Additionally, students have a checklist provided as a guide to help them structure and regulate their activities.

The instructor interface consists of a LA dashboard, as well as the ability to see the student interface. The LA dashboard incorporates a number of features, some of which are visible, and others that function behind the scenes as a part of the database.

The database system tracks and records nearly all activity in HOWARD (see Table 1). The database records this data on an individual level, and is then manipulated to show information about group-level interactions. Individual level data includes information about what action a student has taken, the content of that action, and the time it was performed. For example, if a student starts to watch a video, the database records the action as: name of the student, time of the action, and name of the action (e.g., started video x). Similarly, if a student types a reply to a comment in the chat space, the database will record the name of the student, which student was replied to, and the content of the comment. Types of actions recorded, and their frequency, can be seen in Table 1.

These data were condensed and processed into several visualizations. These visualizations aimed to provide information about students’ most recent actions, collaborative interactions, content discussed, whether the students were on-task, and the content students discussed. The visualizations were designed such that the facilitators can look at dashboards quickly to inform decisions about needed facilitation moves (e.g., asking open-ended questions, encouraging the students, etc.). These visualizations are divided into three sections (from left to right of Figure 2): visualizations of participation, a newsfeed, and visualizations of interaction patterns and content.

Visualizations of participation included (1) a pie chart visualization of student output. This chart shows the text output of all students in a group, in relationship to the text output of other students in the same group, and (2) A two progression bar with meters, including one showing the total text output of the group in comparison with other groups, and a task completion meter that shows the percentage of checklist tasks students of that group have completed. This can also be clicked to show the completed tasks of the individual students in the group.

Visualizations of student action included (3) a newsfeed that provides the instructor with a running list of the actions students take within the group. This newsfeed includes the time, the name of the student, and a title of the action taken. Hovering over the action taken provides more detailed information, such as the content of a post. The newsfeed is scrollable, allowing an instructor to scroll through the entire history of the actions taken in a group. These actions can also be filtered to only display actions the instructor has taken. The dashboard also provides the ability to track students’ completion of assigned tasks; this feature was not used during the implementation.

Visualizations of interaction. The third section included (4) a SNA of interactions between students. The graph showed the connection between students, and between students and instructors, by displaying text output as size, and the thickness of the connecting line corresponding to the number of direct replies between all participants in a group discussion. Lastly, (5) a word cloud displayed the most common words entered by students, to provide a general picture of what students discussed. See Figure 2 below, or Kazemitabar et al. (2016) for greater detail.

In short, different categories of data were represented in different ways. Some visualizations represent only one type of activity. For example, the pie chart establishes the ratio of student output within the group. Participation and task completion meters represent measures of student output. The word cloud and newsfeed hint at content of student output, and the SNA analysis shows interaction relationships. In short, these visualizations separate different aspects of students’ learning activity and draws specific data out of the learning activities.

Students were introduced to the problem of bad news delivery via a rich video case featuring a fictional patient who is anticipating bad news from her physician. The case was designed to induce an emotional investment on the part of the students, and to orient students to the problem space. The driving question, delivered at the end of the video, is “how do you tell her?”

Students were prompted to explore the problem space by identifying facts, ideas, learning issues, and forming action plans in a shared PBL whiteboard (the rightmost panel in Figure 1). Additional rich video cases were used to provide an authentic context for the problem, showing examples of bad news delivery in Hong Kong and Canada. Students were later provided with a video-based overview of the SPIKES bad news delivery heuristic (Baile et al., 2000). Throughout, instructors supported student discussion through facilitation. A daily checklist directing students’ attention to the group conversation and resources provided further support.

Design-based research evolved out of the work of Brown (1992) and Collins (1992) and their emphasis on education as a design science; their description of design experiments have evolved into an established research framework (Bannan-Ritland & Baek, 2008; Cobb & Gravemeijer, 2008; Sandoval, 2004). It is similar to design and development as defined by the Institute of Educational Sciences and the National Science Foundation, research that draws on existing theory and evidence to design and iteratively develop interventions or strategies, including testing individual components to provide feedback in the development process (Earle, 2013, p. 9). This is distinct from design and development research as defined by Klein (2014), which instead systematically investigates processes of design, development, and evaluation in order to inform models of such processes (Klein, 2014; Richey & Klein, 2007).

The goal of this study is to gain an emic perspective—that is, perspective of the participants in the study, in this case, the instructors’ interpretation of their experience. In this sense, it is similar to interpretive research (Erickson, 1986), which is focused on the “particularizability” of research and theory within a context. This approach allows us to describe, analyze, and interpret features of a specific situation, preserving its complexity and communicating perspectives. To investigate our research questions, both instructors were interviewed after the completion of the workshop.

The participants were 17 medical students, and two instructors (one from Hong Kong, China, the other from Canada. Both are veteran physicians and PBL facilitators). Both instructors were experienced medical educators and PBL facilitators. Six students were from Hong Kong and 11 were from Canada, though some of the students attending the Canadian school were American. Student participants were volunteers recruited by the instructors via e-mail. Low participation occurred at the very beginning of the workshop owing to technical issues and time conflicts with students’ courses. Because of this, the students were initially assigned to one of four groups. To encourage more participation, we regrouped 10 students into a focused group, including three from Hong Kong and seven from Canada. Among them, six were active in group-discussions in either the whiteboard or semithreaded discussion, or both. Initially, each instructor facilitated separate groups but after regrouping, the instructors took turns, based on other commitments and travel. All participants discussed in this article are given a pseudonym.

Although difficult to obtain, perceptions of moment-to-moment interaction data can be obtained by using stimulated recall (Calderhead, 1981). Stimulated recall was used with the instructors to learn the rationale and motivations for the specific kind of facilitation they provided to students. Instructors were individually shown a history of the PBL tutorial sessions, and asked about instances where they provided scaffolding or feedback. Questions included, “what was your understanding of this student’s comment?” and “why did you respond as you did?” Follow up questions were also included to probe answers more deeply. The facilitators were asked questions pertaining to the design of the online environment and LA tools therein, and the instructional design of the course. Interviews totaled 3.5 hours of audio, and were transcribed, then independently open coded at the sentence level by two separate researchers.

The open coding created one to four word summaries of each sentence, generating an initial 537 codes. These codes were collapsed down to 101 codes by researchers synchronously reviewing the codes, discussed in light of research goals, and debated until both researchers agreed on the appropriate code. After codes were agreed upon, interviews were reviewed again, and codes were categorized by theme, resulting in 11 emerging themes. To triangulate our understanding of the interviews, and more deeply understand the meaning of these themes, the researchers also analyzed log data of student and instructor activity during the course as additional evidence. As was done previously, two researchers independently analyzed the data, looking for meaningful patterns. These were then compared and coded until both researchers agreed upon the possible meanings of the data. These findings were then compared to the interview codes, and synthesized into our research findings.

As an intervention, the Breaking Bad News workshop provided grounds for discussing the often-neglected topic of medical communication. Asynchronous discussions provided several benefits. First, as other authors have found, students tended to create few posts, but those that were posted were substantial, lengthy responses (Hmelo-Silver, 2003; Lan, Tsai, Yang, & Hung, 2012). Dr. Gordon (pseudonym) remarked, “I got a sense, when I was reading a long post, that this person is having a soliloquy, that this person is writing their own reflections,” and went on to describe student contributions as nuanced and sophisticated, that reflected not only on events within the video case examples, but on the nature of communicating with patients:

Dr. Gordon also described students’ critiques of the video cases as novel and insightful:

Students were also receptive to the context provided in the rich video case we used to present the problem. The case showed a patient nervously anticipating bad news, and also showing family life scenes. Student responses indicate this case was successful in sensitizing students to the emotional concerns of a patient awaiting bad news. As Dr. Gordon remarked, referring to the SPIKES heuristic for delivering bad news:

Students’ ability to relate to the prospective case patient may indicate the rich case was effective in achieving its purpose, or that students’ entered the workshop with previous experience delivering bad news. The length and reflectiveness of their responses, while not a standard in face-to-face PBL discussions, may have provided students with the ability to respond more substantially.

The asynchronous nature of the workshop, combined with low participation, created some challenges. Long response time in between student posts was a concern, particularly as the PBL model heavily emphasizes conversations that build collective understandings. With students posting longer and more reflective posts, this made it difficult for students to progress in the intended manner. For the design and use of student-generated data, this is an important consideration, as students may be learning and generating new ideas without proceeding in the expected manner; this may influence not only the practicality of visualizations, but also how instructors evaluate student progress. Our major findings, listed below, expand on these issues.

Instructors primarily acted upon data related to student output, specifically their typed responses, rather than activity-level data. Log data indicated that students generated a substantial amount of data related both to student activity (utilizing provided resources, using checklists, reading the comments of other students), and data related to student output (submitting reflections to instructor, commenting, replying to comments, writing in whiteboard). The data are summarized in Table 1.

Despite the prevalence of activity level data available on the dashboard, instructors seldom considered activity level data as a factor to identify students’ engagement or as a way to track student progress. As an example, two of the students “lurked,” and participated by viewing the video resources and discussion, but did not contribute to the discussions. The instructors considered these lurkers to be disengaged. Instead, students’ engagement was evaluated on the basis of their output (rather than all of their learning activities as a whole). As Dr. Gordon noted:

This was echoed by the other instructor, Dr. Ray (pseudonym), who discussed use of the newsfeed feature:

Dr. Ray went on to articulate how a student with activity but no output appears disengaged: “I notice that some students actually log onto the website, look around, click here and there, then they didn’t submit anything or join the discussion.” Log data, however, indicates that some students were using materials such as the checklist, as an instructional prompt, checking off a task, completing the task, and then revisiting the checklist.

Analysis of instructor prompts also indicates focus on student output, with nearly all instructor comments referenced only the content of student comments and/or clarifications of how to organize the whiteboard. Typically, instructor prompts were focused on details of provided content, as in the example by Dr. Ray:

Other instructor prompts focused on encouraging interaction among students by encouraging discussion on a single topic. As Dr. Gordon posted, “There is a controversy going on among many of you: What should Sara (the patient in the rich video case) be told over the phone, and what should be told in a face-to-face meeting?” Instructor prompts and comments addressing activity level data were almost entirely absent.

Although our data do not suggest that instructors did not consider activity level data, it does suggest that instructors did not consider activity level data to be evidence of engagement.Students who generated only activity level data were considered disengaged.

Visualizations of data categorized by type of activity did not produce understandings of student learning progress. We expected that aggregated visualizations would enhance instructors’ understanding of learning activity. However, instructors found some aspects of the visualizations to be confusing. For example, SNA visualizations are designed to present interaction among participants and imply their communicative patterns for instructors. During our workshop, neither instructor recognized SNA as a functional representation that might help them detect connections between students’ interaction and group participation. Dr. Ray remarked, “I look at it (SNA) just to see who has a bigger circle, I don’t really know how to make better functional use of it.”

Dr. Ray went on to describe participation meters, which displayed amount of text output for the group, and task completion (based on progress on checklists):

Thus the simple quantified indicator did not have much meaning to the instructor without knowing what the benchmarks were for good performance.

The use of word clouds aimed to illustrate the high frequency words generated from students’ conversation on a daily basis. HOWARD’s word cloud visualizations were designed to quickly give a sense of what students were talking about, and to provide an easy means of determining if students were off topic in discussions. However, Dr. Ray remarked that this was not helpful because students stayed on topic, leaving the instructor unsure of the meaning of the visualization: “I think, what we are seeing there is, this word cloud tells you if they are talking about what you want them to talk about, and they were, but beyond that, that’s all it does.” These findings suggest that context is important for instructors attempting to understand visualizations; visualizations that strip student activity from context may frustrate instructors’ sense of student learning progress.

Instructors’ focus on student output was related to interest in conversation building as evidence of student progress. Despite students generating more activity level data than output level data, and visualizations primarily directed at activity level data, instructors focused on student output. Specifically, instructors focused on the students participating in and developing the conversation. This concern was related to the pace of student replies. While the average rate of student output (submitting a reflection, posting or replying to a comment, writing in the whiteboard) was 6.09 per day, this did not occur at a continuous rate; some days had several active students, other days had no student activity. This caused some concern over conversation building, and changes in PBL facilitation strategies. As Dr. Ray remarked,

He went on to say, “sometimes I find it difficult to build up a dialogue with such a lengthy wait. That I need to wait for a day before I can get a response from that person.” He also reported that such a wait time gave the impression that students were not enthusiastic about their learning. This change in question strategy was not simply a change in the pace of questioning, it was a change in soft scaffolding strategy, a change in how he would “jump in” to the discussion. Dr. Ray described this as different from a typical synchronous session, where “I will ask the second question at certain time after I ask the first one, depending on what kind of difficulties they run into.”

Dr. Gordon also reported feeling unsure of where students were in their progress, making it difficult to find the right place to “jump in” to facilitate, and thought this might also be difficult for students:

PBL relies on meaningful conversations. The instructors, as experienced PBL instructors, were strongly focused on building conversations as a measure of student engagement and progression of understanding. HOWARD, however, provided summaries of student activity and output. In sum, the dashboard was less useful for conversation building owing to the difference between PBL instructor focus, and the focus of the dashboard.

LA dashboards present many possibilities for expanding PBL and other forms of inquiry to online settings in which one instructor needs to facilitate multiple groups. Our study indicates that there are still challenges to be overcome in terms of finding the right combination of data analysis and visualizations that will go farthest in assisting instructors using PBL or other pedagogies rooted in student dialogue.

Instructors focused on the student output data and paid little attention to students’ activity level data as evidence of engagement. While activity level data were visible on the dashboard, the visualizations did not specifically identify lurking students, and this oversight may explain why instructors considered activity level data insufficient for showing engagement. In other words, instructors were not aware of which students were failing to fully participate in the learning activities. Deeper investigations into student learning trajectories, and instructor assessment of such trajectories in asynchronous PBL, may provide data for more relevant visualizations.

Our second finding was unexpected. It was a reasonable assumption that more information about student activity would help inform instructors’ decisions, but our findings indicate that visualizations that pulled data out of its context may frustrate rather than aid instructor understandings of students’ learning. Investigations into instructor understandings of incontext student activity may provide insight into the design of LA visualizations for collaborative learning environments. Research investigating instructor understandings of interactive visualizations may provide further recommendations for their design. Alternatively, visualization that focuses on relationships between learning activities and learning trajectories may provide better guidance for instructors both at the individual and group level rather than visualizations that focus on isolated summaries of student activity. In other work, we have created timelines that allow researchers to view log data, student activity, and relevant aspects of discussions in parallel (Hmelo-Silver, Chernobilsky, & Jordan, 2008; Hmelo-Silver, Jordan, & Sinha, 2013). The technology available at the time made constructing these timelines somewhat laborintensive and they could only be accomplished after all the research data had been collected. As technology has advanced, creating more sophisticated representations of data are increasingly possible. Having such relational visualizations available in real time might be one approach that would provide the context that the instructors felt they needed.

That instructors were focused on building conversations is expected, as PBL relies heavily on collaborative learning. Interestingly, instructors changed facilitation strategies not only based on the content of student responses (as would be expected), but also made facilitation changes based on factors related to the asynchronous aspect of the workshop (such as the length of time between student posts). This change in scaffolding strategy was a major concern for instructors, but was not considered in the design of the dashboard. In order to gain better insight into relationships between dashboard visualizations, student activity, and instructor concerns, more investigation is needed.

There are several limitations to this study. Although facilitators in this study were experienced PBL facilitators, and actively engaged in research on PBL, the ratio of instructors to students was more than we planned in the original research design. Unfortunately, the technical delays resulted in a larger group of students that each facilitator managed in multiple groups. The timing of the study was at the end of an academic semester and this proved to be a barrier to learning. In this regard it was difficult to ascertain whether student challenges were from the tools, activity structures or competing academic demands of busy medical students. Finally, interview questions for the facilitator were geared toward understanding instructor’s perspectives and improving HOWARD, rather than on instructor impressions of the positive aspects of the environment.

Moving toward research and development of LA informed by instructional theory is a step toward the construction of tools and information that are truly helpful for instructors. However, actually doing so requires a much greater focus on what instructors actually do with the information in practice. What information do instructors actually need? What types of information do instructors understand, value, and act upon? How are those values enacted in instructional practice? Without extensive consideration of these questions, LA as instructional aids may struggle to deliver their intended benefit. While LA research has produced new understandings of students, transforming LA into instructional aids must be done carefully and with great consideration of instructors, variations in data, and how those variations are visually presented to instructors. Without such care, LA applications are in danger of being “a technology looking for a problem to solve.”

A core issue for LA is that its development is rooted in the availability of data for use, rather than driven by explicit needs of instructors and students. As Ferguson (2012) points out, “The first driver, then, is a technical challenge: how can we extract value from these big sets of learning-related data”? (p. 4). Pardo’s (2014) description of LA designs also show that, while many LA designs do consider the experience of those using the tools, the focus remains on extracting meaning from data (and visualizing it); ultimately a technology-centric approach. This focus assumes that additional information (and visualizations) will be helpful. In practice, that extra information may be confusing. LA built around theoretically informed pedagogical models and principles may produce unintended results, necessitating an approach that involves combinations of iterative development and empirical research. Lastly, use of visualizations of student activity requires some professional development to support data literacy. A picture may be worth a thousand words, but only if the user understands what it is trying to represent and how to use it to make instructional decisions.