What Learning Analytics-Based Prediction Models Tell us About Feedback Preferences of Students Free

In educational settings, an enormous volume of potentially valuable information is generated by both students and educators. Such information may include academic performance, tracking data from online learning environments, e-mails, and social network data. In recent years, the term “learning analytics” has emerged as educational institutions and corporate learning started to harness this wealth of information to provide real-time feedback to students while offering valuable insights for educators to improve teaching quality (Siemens, Dawson, & Lynch, 2013). In the corporate world, learning analytics (LA) can help learning and development of professionals by identifying successful learning activities and patterns, with clear indications of the learning progress of its employees. In a higher education context, students and teachers may benefit from personalized and adaptive learning experiences (Knewton, 2016). To better catalyze the processes of learning for individuals and collectives, Buckingham Shum and Crick (2012) have proposed a dispositional learning analytics infrastructure that combines learning activity generated data with learning dispositions, values and attitudes measured through self-report surveys which are fed back to students and teachers through visual analytics. Tempelaar, Rienties, and Giesbers (2015) have investigated the predictive power of learning dispositions, outcomes of continuous formative assessments, and other system-generated data on modeling student performance and their potential to generate informative feedback. The study found that computer-assisted formative assessments can best detect underperforming student and academic performance.

In learning theory, monitoring and evaluation play a crucial role, as they provide feedback on how activities coordinate across several stages of studies (task definition, goal setting and planning, and enacting study tactics and strategies) (Winne & Hadwin, 1998). Feedback assesses the level of understanding of learners and can provide cues for reinforcement.In a meta-study by Hattie (2013), feedback is considered one of the most powerful tools in enhancing the learning experience. In the past, traditional formal feedback is limited to taking the form of a grade, which is available only after finishing all learning activities. However, the involvement of educational technology allows us to gather feedback on learning-in-progress activities, which provides a real-time assessment to both students and teachers. For instance, a study by Duffy and Azevedo (2015) revealed that students in the “prompt and feedback” condition deployed more self-regulated learning strategies and spent more time viewing relevant science material compared to students in the control condition, in which learners did not receive any support. Additionally, McLaren, van Gog, Ganoe, Karabinos, and Yaron (2016) categorized different feedback modes into worked examples, erroneous examples, tutored problems, and problem solving. Their study showed clear efficiency benefits of the use of worked examples in a web-based learning environment: equal levels of test performance were achieved, with significantly less investmentof time and effort during learning. Given the importance of feedback and the advancement in assessment technology, the investigation of the effects of feedback use by students on their academic performance suggests being a promising research trajectory in learning analytics.

This study examines how learning dispositions and feedback preferences affect academic performance. The article is organized as follows. The next section (Section 2) introduces the context of the study and its instruments. This is followed by Section 3, which presents the results, and is followed by the discussion in Section 4. Finally, Section 5 concludes the study and discusses the implications of big data in education and LA for online learners/instructors, and how this study bridges the gap between existing LA literature and pedagogy.

The educational system in which students learn mathematics and statistics is best described as a “blended” or “hybrid” system. The main component is a face-to-face instruction that employs problem-based learning, in small groups (14 students), as an instructional strategy. As part of the problem-based learning approach, learners are coached by a content expert tutor (Schmidt, Van der Molen, Te Winkel, & Wijnen, 2009). Participation in these tutorial groups is required, as is the case for all courses based on the Maastricht problem-based learning system. Within the online component of the blended learning, students can optionally make use of the two e-tutorials Sowiso (mathematics) and MyStatLab (statistics) (Tempelaar et al., 2015; Tempelaar, Heck, Cuypers, van der Kooij, & van de Vrie, 2013). This choice is based on the philosophy of student-centered education, placing the responsibility for making educational choices primarily on the student. However, the use of e-tutorials and achieving good scores in the practicing modes of the MyLab environments is stimulated by making bonus points available for good performance in the quizzes. Quizzes are taken every 2 weeks and consist of items that are drawn from the same item pools applied in the practicing mode. We chose this particular constellation as it stimulates students with limited prior knowledge to make intensive use of the MyLab platforms. The bonus is maximized to 20% of what one can score on the exam.

The student-centered characteristic of the problem-based learning instructional model requires, first and foremost, adequate, informative feedback to students so that they are able to monitor their study progress and their topic mastery in absolute and relative sense. The provision of relevant feedback starts on the first day of the course when students take a diagnostic entry test for mathematics. Feedback from this entry test provides a first signal of the importance of using the digital learning platforms made available to the students. Next, the Sowiso and MyStatLab environments take over the monitoring function: at any time, students can see their progress in preparing the next quiz, get feedback on the performance in completed quizzes, and on their performance in the practice sessions.

Participants in this study were 1,069 students in a blended introductory quantitative course at a public university in the Netherlands during 2015–2016. A large diversity in the student population was present: only 24% were educated in the Dutch high school system. The largest proportion, 46% of the students, was educated according to the German Abitur system. High school systems in Europe differ strongly, most particularly in the teaching of mathematics and statistics. Therefore, it is crucial that the first module offered to these students is flexible and allows for individual learning paths (Tempelaar et al., 2013; Tempelaar et al., 2015). In the investigated course, students work an average 10 hours in Sowiso, and 25 hours in MyStatLab, which represents 12.5% to 31% of the available time of 80 hours for learning on both topics.

In this empirical study, we investigate the relationships between course performance measures, learning management system (LMS) trace variables, student information system (SIS) based variables, and learning disposition variables measured in six self-report surveys. Most learning dispositions incorporated in this study are assumed to be relative context independent. Examples of such are attitudes and learning styles. These are relative stable constructs, not impacted by the specific learning activity the student is in: traitlike type of variables. For that reason, these self-report surveys were all administered at the start of the course, to make their data available as early as possible. On the other hand, learning emotions are context dependent: they relate to emotions of students in specific learning activities. These statelike variables cannot be measured at the start of the course, since students need to have sufficient experience with the learning context in order to be able to assess their contextual learning emotions. To differentiate between test emotions and learning emotions, the measurement should also not take place too late in the course, and therefore, we opted to do so exactly half way the course. Thus, it gives students sufficient experience with the topics and the learning activities, without being in danger that the approaching exam would strongly impact learning emotions. In the subsections that follow, several instruments are described to provide the groundwork for our analysis.

The ultimate aim of the predictive modeling endeavor is to understand how student dispositions and learning activity relate to four relevant course performance measures: performance on the exam, both for mathematics (MathExam) and statistics (StatsExam), and the aggregated bonus for both topics, which was based on performance in the three quizzes: MathBonus and StatsBonus, combined with the mastery level achieved in the etutorials for each topic: MathMastery and StatsMastery.

Three different digital systems have been used to organize the learning of students, and to facilitate the creation of individual learning paths: Blackboard (learning management system), and the two e-tutorials Sowiso and MyStatLab. Students worked in the two e-tutorials for all 7 weeks, practicing homework exercises selected by the module coordinator. The e-tutorial systems track the mastery score achieved in each task, which is measured as the number of successful attempts (MathMastery and StatsMastery), time on task (MathHours and StatsHours), the total number of attempts required to get to the mastery level achieved (MathAttempts and StatsAttempts), the number of fully worked-out solutions called for (MathSolutions and StatsSolutions), and the number of hints asked for (MathHints and StatsHints). In this study, feedback preferences imply the use of fully worked-out solutions and the use of hints. Overall, students who see more fully worked-out solutions, and who ask for more hints, perform better. These data were aggregated over the on average 25 weekly tasks for mathematics, and about 20 tasks for statistics, to produce 10 predictor variables, 5 for each topic, for each of the 7 weeks, and next, aggregated over all 7 education weeks. Less aggregated data sets have been investigated, but due to high collinearity in data of individual tasks, these data sets produced less stable prediction models.

The preliminary results from this study suggest that the five types of track data for both topics appear to be collinear: in general, active students spend more time in the e-tutorials, making more attempts, achieving higher mastery, and in doing so, they use more hints and examples. Due to this collinearity, the added value of time on task and number of attempts in predicting course performance appeared to be minimal, with mostly nonsignificant betas. Therefore, in the final version of prediction models, only mastery level, the use of hints and the number of examples were included. In this article, we are particularly interested in which factors influence the way students use feedback (fully worked-out solutions versus hints), and how different feedback modes can help to explain students’ academic performance.

The Maastricht University SIS provided four further variables, which are used as controls. Standard demographic variables are Gender (an indicator variable for female students), Studytrack (economics and business economics, fiscal economics, and international business) and MathMajor (indicator for the advanced mathematics track in high school). Distinguishing between national and international students is key, given the strong focus on statistics in the Dutch high school system (with a large variation in other countries, but never as extreme as in the Dutch case). The MathMajor indicator is constructed on the basis of distinguishing prior education preparing for sciences, or for social sciences. Students in the sample are from 45 different national high school systems, all being very different, but in all cases differentiating between advanced and intermediate level mathematics track (students of basic mathematics track are not admitted into the program).

Upon entering the course, students were required to do a mathematics diagnostic entry test (MathEntryTestScore), of which the scores were added to the SIS data.

Attitudes toward the learning of mathematics were assessed with the SATS instrument (Tempelaar, Gijselaers, van der Loeff, & Nijhuis, 2007), based on the expectancy-value theory (Wigfield & Eccles, 2000). The instrument contains six mathematics related attitudes, and two general attitudes. However, in this study, we only focus on the two general learning related self-perceptions referred to as RiskTaking, how strong risk seeking and how less risk avoidant students are, and Procrastination, the tendency to avoid doing learning activities.

Vallerand et al. (1992) propose three main categories of motivations in learning: intrinsic, extrinsic, and amotivation. First, intrinsic motivations (IM) refer to the pleasure and satisfaction derived from doing the task itself. IM consists of (1) intrinsic motivation to know (IMknow), that refers to the satisfaction while learning or trying to understand something new, (2) intrinsic motivation toward accomplishments (IMacc), in which individuals get pleasure from accomplishing or creating something, and (3) intrinsic motivation to experience stimulation (IMstim), referring to the fulfillment from engaging in the activity. Second, extrinsic motivations (EM) pertains to a wide variety of behaviors which are engaged in as a means to an end and not for their own sake. EM can be differentiated between (1) EM external regulation (EMext), which refers to rewards or constraints, (2) EM introjection (EMint), in which individuals begin to internalize the reason for his or her actions, and (3) EM identification (EMiden), in which the behavior is perceived as valuable and important for oneself. Third, individuals are Amotivated when they are neither intrinsically or extrinsically motivated. They perceive their behaviors are caused by forces that are out of their control.

In this study, we combine IMknow, IMacc, IMstim, and EMiden into a new construct called Autonomous, which is the total average of the mentioned motivations. In addition, Control is also created by taking the mean of EMintro, and EMext.

Help seeking can be conceptualized as a general problem-solving strategy that allows learners to cope with academic difficulties in gaining the assistance of others. Nelson-Le Gall (1985) draws a distinction between “executive” or dependency-oriented help seeking and “instrumental” or mastery-oriented help seeking. The former refers to those instances in which the student's intention is to have someone else solve a problem or attain a goal on his or her behalf, whereas the latter is limited to the amount and type of assistance needed for the student to solve the problem independently. Avoidance of help-seeking is a situation in which help is needed, but the student refuses to seek help. Perceived benefits of help seeking are students’ beliefs about the outcomes of help-seeking activities, such as interest or learning. In addition, the source of help can also be distinguished between Formal source and Informal source. The former refers to institutional resources such as instructors, or tutors, while the latter refers to noninstitutional resources such as classmates, friends, and family members (Knapp & Karabenick, 1988).

The learning processing and regulation strategies that shape self-regulated learning are based on the Inventory of Learning Styles instrument (Vermunt, 1996). Our study focuses on two of the four domains or components of learning of Vermunt’s model:cognitive processing strategies, and metacognitive regulation strategies. Each of these components are composed of three scales. Different processing strategies include Deep processing strategy, in which students relate, structure and critically process the new knowledge they learn, Stepwise processing strategy (also called surface processing) based on memorizing, rehearsing and analyzing, and Concrete processing strategy, focusing on making new knowledge concrete, and applying it (Vermunt, 1996). Likewise, three metacognitive regulation strategies are Self-Regulation of learning processes and learning content, External Regulation of learning processes and learning results, and lastly, Lack of Regulation: the absence of regulation be it by the student or out of the environment.

Epistemic emotions distinguish from activity emotions in that they are related to cognitive aspects of the task itself (Pekrun, 2011). Prototypical epistemic emotions are curiosity and confusion. In this study, epistemic emotions were measured with the Epistemic Emotion Scales (Pekrun & Meier, 2011), including Surprise, Curiosity, Confusion, Anxiety, Frustration, Enjoyment, and Boredom.

In order to examine how learning dispositions and feedback preferences affect academic performance, the following research questions were posed:

• Q1: How do feedback preferences influence academic performance?

• Q2: How do learning dispositions influence feedback preferences?

• Q3: To what extent do feedback preferences mediate the relationship between learning dispositions and academic performance?

The data analysis steps of this study are all based on linear, multivariate models, making use of Sobel-Goodman mediation analysis (Figure 1). In the first step, we investigate the direct effects of the four performance measures, the feedback preferences data derived from LMS, and several types of disposition data, with SIS data as controls. For space limitations, we restrict ourselves to static models that are estimated on all available, aggregated track data, rather than dynamic models estimated on weekly data. In the second step, we focus on the indirect effects of dispositional data on academic performance through feedback preferences track data: the mediation effect is calculated as the product of the coefficients of dispositional data and feedback preferences, and feedback preferences and academic performance. In the final step, the total effect is computed as the sum of direct effect and indirect effect.

Figure 2 summarizes the relationship between feedback preferences, as revealed by student behavior which represented through their actions within the tools, and academic performance. MasteryLevel (See Section 2.2.2 LMS Trace Data) in both tools, that is the average number of exercises successfully finished, is strongly positively related to all performance measures. Most strongly for performance in quizzes, with MathBonus and StatsBonus with betas of .78 and .92, respectively, and somewhat less strong for performances in the exams, with MathExam and StatsExam with betas of .40 and .53, respectively (p < .01). This difference in explained variation is easily interpreted using the strong tie between quizzes and practicing in the tools. Second, the average number of fully worked-out solutions asked for per exercise, MathSolutions and StatsSolutions, are associated with a significant decrease in Mathexam (B = –.16, p < .01) and Statsexam (B = –.31, p < .01), respectively. This may seem counterintuitive, but is to be interpreted in a multivariate context: given the same MasteryLevel, students who requested less fully worked-out solutions are the better performers, and therefore, scored higher on the final exam. The effect of Stats-Solutions is salient on StatsBonus whereas MathSolutions has an insignificant impact on MathBonus. Third, while the average number of hints asked for per exercise (Mathhints) has no significant effect on Mathexam and MathBonus, the Statshints variable is negatively correlated with Statsexam (B = –.14, p < .01) and Statsbonus (B = –.08, p < .01). Its interpretation follows the multivariate context: given the same level of mastery in Statistics, students who asked for fewer hints are the ones who perform better on the quizzes and the final exam.

In terms of academic performance, there are no significant differences amongst study tracks and revealed feedback preferences, except for Economics students, for whom the performance in Mathexam is significantly higher than International Business students (B = .05, P < .1).

The indicator for prior mathematics schooling, MathMajor, impacts both academic performance and feedback preferences (Table 1). The beta weights of advanced level prior education are .12 for MathExam, .07 for StatsExam, and .09 for MathBonus. Evidently, the benefits of having more prior knowledge in mathematics are greater on Mathematics related performance than Statistics related performance. Regarding feedback preferences, students with MathMajor level asked for less fully worked-out solutions than non-MathMajor students in both mathematics and statistics, with the stronger effect on the former. Similarly, MathEntryTestScore also demonstrated similar patterns with stronger effects on MathExam and MathBonus than on StatsExam and StatsBonus.

While the difference in academic performance across gender is not significant, there are some interesting patterns in feedback preferences between females and males. On average, female students use more fully worked-out solutions and have higher mastery score than male students in Mathematics. However, in the multivariate model, the beta of the indicator Female is negative. This is to be understood by the gender difference in MathMajor, the main predictor of the Solutions variable. Female students are underrepresented in the MathMajor category, but within both categories, female students use fewer Solutions than male students.

After carrying out the analysis of the direct effects of revealed feedback preferences on academic performance, and how SIS system data impact feedback preferences, we are interested in investigating how learning dispositions influence feedback preferences, and to what extent feedback preferences mediate the relationship between learning dispositions and academic performance. In order to do so, we once more apply Sobel-Goodman mediation tests to measure the indirect effect of any learning disposition on academic performance, multiplying the coefficient of the learning disposition on feedback preference, as well as the coefficient of feedback preference on academic performance.

Direct effects of dispositional attitudes on performance measures are limited, with only one significant relation: students with higher levels of Procrastination perform on average worse on Mathexam (B = –.06, p < .1). In contrast, indirect effects through feedback preferences are resilient (Table 2). Procrastination hinders all student activity, and above all, has deteriorating effects on mastery in the tools. Due to the strong tie between mastery in the tools and bonus (the score on the quizzes), major negative indirect effects are those from Procrastination through Mastery to Bonus score. There is a weak positive indirect effect, composed of two negative paths, from Procrastination through both types of Solutions to Exam and Bonus scores.

Long-term orientation has no direct effects on performance and a very weak indirect effect through Hints for both statistics performance types.

In reporting the role of autonomous, controlled, and lack of motivation, indirect effects are of a very limited size, and absent for both Mastery and Hints variables, as shown in Table 3. The only significant indirect effect is through the Solutions variable. Autonomously motivated students more often follow “their own learning plan” by calling fully worked-out examples, rather than solving the problems themselves, both in mathematics and statistics. Amotived students do too, for mathematics. This negatively impacts performance scores, mostly for the exam scores. These negative indirect effects add to direct effects, also negative, of all motivation types: Autonomous and Amotivation, as well as Controlled motivation, producing all negative total effects. It is especially the exam component of performance, Mathexam and Statsexam that is most strongly affected.

In help-seeking dispositions, we find more instances of opposite directions of direct and indirect effects. The direct effect of the preference to solve problems independently (Instrumental) is positive for Mathbonus, Statsexam, and Statsbonus (Table 4). The indirect effect for performance in statistics, is, however, negative and about the same size, making the total effect indeterminate. The same mechanism is at work for the Executive help-seeking disposition, the preference to have someone else solve the problem on one’s behalf. For performance in statistics, its direct effect is positive and its indirect effect negative. It is only in students who are in need of help but refuse to seek it (Avoidance), that negative direct effects add to negative indirect effects.

Indirect effects are mainly through lower levels of mastery in both tools. Concerning feedback preferences, students with an Executive help-seeking disposition ask for more worked-out solutions in Statistics (B = .10, p < .05), which lead to lower performance (Stats-Bonus & StatsExam). By comparison, students whose help-seeking is Perceived to support Learning, search for help that is beneficial for their learning, and ask for less worked-out solutions (B = –.07, p < .05).

The effects of self-regulated learning strategies on academic performance are summarized in Table 5. First, students with performing a Deep processing style, who tend to relate elements of the subject matter to each other and to prior knowledge, structure these elements into a whole, and form a critical view on the materials, performing focused better in Mathexam, Statsexam, and Statsbonus. In this case, direct and indirect effects are reinforcing: both are positive. The Step-wise processing style, focused more on memorizing and analyzing the subject matter, and the Concrete processing style, where students have the tendency to apply the subject matter in practice, are unrelated to performance measures, lacking both direct and indirect effects.

This pattern of reinforcing direct and indirect effects repeats itself with metacognitive learning regulation styles. Students with a Self-regulated learning style, who prefer to regulate their learning process themselves, do less well, both as a direct effect, and an indirect effect through mastery. In contrast, students with an External-regulated learning style, who prefer to orient on tutors and peers in the regulation of learning, do slightly better due to positive indirect effects through mastery. Neither Solutions nor Hints play any role in the indirect effects of learning styles.

The direct effects of epistemic emotions for mathematics, reported in Table 6, are mostly in line with expectations: positively valenced epistemic emotions have positive effects, such as Curiosity, and negatively valenced epistemic emotions have negative effects, such as Anxiety and Frustration. The emotion without a straightforward valence, Confusion, carries a small positive effect. Surprisingly, Enjoyment comes with a negative effect on performance in statistics (but not in mathematics). Its reason may be in the specific constellation of high school mathematics education in Europe: students who enjoy mathematics will typically opt for advanced mathematics tracks, but such tracks do not include statistics, whereas students not enjoying mathematics may opt for social science oriented tracks that do contain statistics.

In the indirect effects, the Solutions variables act as mediator variables. In the statistics domain students high in Frustration call for more worked-out Solutions, lowering on average their performance scores in statistics. In mathematics, students with high levels of Confusion call for fewer worked-out Solutions, increasing on average their Exam score.

Trace data from the two e-tutorials, Sowiso and MyStatLab, are incorporated in all models with a consistent pattern: mastery levels in the tools are by far the strongest predictor of all performance types, whilst number of fully worked-out solutions called for (and in some cases the number of hints called for), negatively impact performance. These findings are in line with previous studies (Tempelaar et al., 2015). These negative betas may surprise, since all bivariate relationships between the number of hints, and the number of solutions, demonstrate positive correlations with each of the four performance types. Overall, students who see more fully worked-out solutions, and who ask for more hints, do perform better. However, in the context of multivariate prediction equations, the favorable effect of intensive practicing is already contained in the mastery variables, reducing the impact of the “hints” and “fully worked-out solutions” variables on conditional relationships. The negative betas tell us that for students with a given mastery level, requiring more hints to reach that mastery level, or requiring more worked-out solutions, lowers the expected performance for each of the performance categories.

The findings also indicate the stronger effect of fully worked-out solutions compared with the use of hints as a feedback channel. Furthermore, the use of hints has little to no impact on mathematics performance, while the effect is more salient in statistics performance. Our results confirm the advantage of fully worked-out solutions in multimedia learning environments as indicated in previous literature (Hoogerheide, Loyens, & Van Gog, 2014; McLaren et al., 2016; Renkl, 2005). It especially addresses a common limitation of the methodology of the aforementioned studies, which is the generalizability from lab/controlled settings to authentic settings. In real life, the effects of feedback preferences are interlinked rather than being isolated and individually examined. Thus, LA helps resolves this issue by using trace data that reflect actual user behaviors.

Out of 25 dispositions, only nine have a statistically significant impact on feedback preferences (Figure 3). Overall, learning dispositions have stronger and more significant impact on fully worked-out solutions, when compared to hints. Students who are inclined toward Autonomous and Amotivation types of academic motivation, the Executive help-seeking disposition or the Frustration emotion use more fully worked-out solutions. In contrast, the Concrete learning strategy, Procrastination attitude, Formal help-seeking disposition, or the Confusion emotion are associated with the lower use of fully worked-out examples. Procrastination and Longterm are the only two measurements which have a salient impact on the use of hints, in which the use of hints is lower in the former and higher in the latter.

Our findings contribute to the development and implications of educational policies concerning learner/instructor data by bridging the existing gap between LA and pedagogy (Gašević, Dawson, & Siemens, 2015). Most studies at the early stage of LA have built upon data extracted from both institutional SIS and the log data retrieved from digital platforms that organize and facilitate learning, such as LMSs and e-tutorials (Arnold & Pistilli, 2012; Macfadyen & Dawson, 2010). While these studies provide important markers on the potential of LA in education, most are still unable to go beyond the descriptive function of LA, largely based on demographic data, grades, and trace data. Hence, effective instructional and intervention practices are hindered by the lack of pedagogically based findings. Using dispositional characteristics of students, this study has addressed some of the limitations of conventional LA research by providing educators with “actionable feedback,” which not only describes how students prefer certain feedback types but also explains why students follow certain behavioral patterns based on their learning dispositions.

In general, fully worked-out solutions appear to be a better mediator than hints. The mediating effect is stronger for performance in statistics (the topic about which most students had little prior knowledge) than for mathematics. Next, we find several cases of feedback preferences where direct and indirect effects of dispositions have opposite directions. In general, the use of more Hints has little to no impact on performance measures beyond the effect already included in the mastery level, whereas the use of more worked-out Solutions tends to have a negative effect on performance levels. In all situations where the learning disposition is positively related to the mediator variable Solutions, the indirect effect (being the product of a positive and a negative beta), becomes negative. An example is the Executive feedback disposition: the tendency to use others to solve your own (academic) problems. Students who score high on Executive feedback tend to call for more worked-out Solutions, which, for example, in turn lowers their expected performance scores. However, this small indirect effect is completely offset by the positive direct effect of Executive feedback on performance in the statistics quizzes. Apparently, in the end, it pays to have the disposition to let others work for you.

A crucial conclusion relates to the role of systematic comparison of direct and indirect model effects, and the diverging outcomes, to which such a comparison may lead. LA models are typically of input-output kind, directly relating performance components, (the outputs) to measured input variables. Restricting to direct effects only, surpassing the process effects visible in an input-process-output type of model, would leave all indirect effects unobserved. As the example above indicates, this could lead to incorrect conclusions, and incorrect interventions, derived from an inputoutput prediction model.

A second important finding from this research is similar in nature: it relates to the importance of a systematic comparison of bivariate and multivariate relationships. Simple correlations, often applied in LA applications, do not provide proper insight. In our context correlational analyses would have led us to the conclusion that feedback modes, the use of hints, and the use of fully worked-out solutions, all contribute positively to all performance types. This would suggest that positive instructional strategies would include stimulating students to use more hints, and having students use more worked-out solutions in their learning. However, these bivariate relationships are confounded by overall student activity in the e-tutorials. When correcting for this confound, by looking at multivariate relationships, we find opposite conclusions: the use of hints is completely neutral, and the use of worked-out solutions is, in fact, detrimental to learning outcomes. Another striking example of the divergence between bivariate and multivariate modeling outcomes relates to gender differences in revealed feedback behavior of students. Within the Dutch context, empirical research into mathematics education suggests that female learners may profit more from example-based education (Tempelaar, Rienties, & Nguyen, 2016). Based on this finding, one would expect female students to more often make use of worked-out solutions than would male students. And indeed, in a bivariate context, we can confirm that hypothesis. However, in a multivariate context, the confounding factor “prior mathematics track” pops up: female students more often take the social-science track in high school, male students more often take the science track, and social-science track students use worked-out solutions more often. Correcting for this confound, the gender effect completely disappears, and is even reversed in direction (but not statistically significant). Therefore, consider how inadequate an intervention could have been, derived from a simple, correlation-based LA prediction model.