A Multilevel Model Predicting Success In Online High School Courses From Students’ Days Active During The Pandemic Free

Historically, attendance has been a prerequisite of success in school and has been linked with multiple outcomes for students throughout their elementary, middle, and high school education. First, attendance strongly predicts overall grades, GPA, and test scores. Allensworth and Easton (2017) found that students who miss less than one week of school per semester have a GPA of 3.0 or higher at the end of their first year. Their results also showed that moderate absenteeism affected students' academic success, finding that even students with high test scores, who missed two or more weeks of school per semester, were more likely to fail than students with low test scores whose absences totaled one week or less (Allensworth & Easton, 2007). Further, missing two or more weeks of school negatively impacts students' math achievement (Balfanz & Byrnes, 2012; Gottfried, 2014; Gottfried & Kirksey, 2017). Absenteeism also hurts students' test performance (Balfanz & Byrnes, 2012; Ginsburg et al., 2014; Gottfried, 2014; Gottfried & Kirk-sey, 2017). Liu et al. (2021) found that missing 10 classes “reduces math or English Language Arts test scores by 3-4% of a standard deviation and course grades by 17-18% of a standard deviation” (p. 104441).

While most of the prior research on attendance and student outcomes was focused on brick-and-mortar schools, there needs to be more research in examining online schools. Online schooling was brought into the limelight with the COVID-19 pandemic as families across the United States became more familiar with online approaches to education. Many brick-and-mortar schools closed for a time and adapted their model to an online approach. However, online schools did not need to modify their academic programming during the pandemic. Having been in existence since 1997, they were able to continue providing quality educational experiences with curriculum made for online use, staff trained to teach online, and technology able to provide this mode of education. Therefore, exploring student outcomes and attendance data during the pandemic should produce results similar to those during non-pandemic times.

Using a generalized mixed-method approach to multilevel modeling, the researchers looked deeper at the implications of attendance in an online school and student grade outcomes. This research is essential to determine if attendance predicts academic success in the online school environment. This information would allow schools and districts to create educational support and policies grounded in scholarly research on online students' attendance and academic success.

The literature on attendance and student outcomes finds that, as students progress through middle and high school, there is a rise in the level of absenteeism (Balfanz & Byrnes, 2012; Bauer et al., 2018). Patterns of student absences can predict dropping out of high school (Al- lensworth & Easton, 2007; Balfanz & Byrnes, 2012; Kirksey, 2019; Schoeneberger, 2012). Roby (2004) explored the relationship between student attendance and achievement in Ohio at the 4th, 6th, 9th, and 12th-grade levels. School absences and their damaging impact on achievement rates were most significant in 9th grade. Transitioning from middle to high school is key for students (Welsh, 2018). Absences during this period are negatively associated with educational outcomes (Welsh, 2018). Ninth grade was also a critical year in analyses of student data in Chicago, where course performance at this grade level predicted the likelihood that students would graduate (Allensworth & Easton, 2007). In examining attendance data from a large, urban school district in California, Lui et al. (2021) found that 10 absences in 9th grade reduced both the probability of on-time graduation and subsequent enrollment in college. Similarly, Balfanz & Byrnes (2012) found that students with 10 absences or more during the 10th-grade year who graduated enrolled in post-secondary school at a rate of 25% less than their peers.

In 2015, when replacing federal No Child Left Behind legislation, the Every Student Succeeds Act (ESSA) added that states needed to adopt a school quality indicator and academic measures. The link between school absences and student outcomes is so vital that 36 states and the District of Columbia adopted chronic absenteeism as an indicator of school quality and student success (Bauer et al., 2018; Ginsburg et al., 2014).

In March 2020, many schools closed for some time while pivoting to online, synchronous, or asynchronous learning due to the pandemic. It was common for schools to be closed for one or more weeks during this transition. Missing school for that week or two placed students at an above-average rate in absenteeism levels in a regular school year. Prior literature on absenteeism and its significant impact on students' academic outcomes suggests that the pandemic and subsequent absences negatively impact students' academic achievement. Schools commonly report their academic and school quality indicator data annually. However, the pandemic impacted both the data collection and reliability of that data. Instead, researchers utilized literature on attendance and achievement to predict student outcomes. Their results indicated that students would end the 2019-2020 school year with decreased learning gains in ELA and math, with math being the most negatively affected (Kuhfeld et al., 2020; Liu et al., 2021; Santibanez & Guari- no, 2021). Further, Liu et al. (2021) projected a 3-to-4 decrease in high school graduation and college enrollment.

While brick-and-mortar schools were pivoting their model to provide education to students during a pandemic, virtual schools did not need to make such a shift. Virtual schools have existed since 1997, beginning with Florida Virtual School. As of 2022, four states offered a blended online learning model, and 35 states and the District of Columbia offered full-time virtual education (Gratz et al., 2022; Molnar, 2019). In virtual schooling, students receive their education remotely. Student attendance may be monitored from the Learning Management System (LMS) or through parent records (Nespor, 2019).

Online schooling can offer students flexibility and the ability to work at their own pace; therefore, the definition of attendance and data collected regarding attendance vary. Hawkins and Barbour (2010) found that this variation affected their ability to compare course completion rates and retention among virtual schools and between virtual and brick-and- mortar schools. We have included how each of the following studies determined absences.

Archambault et al. (2013) examined cyber-truancy at Minnesota Virtual High School (MVHS). MVHS’ funding is based on course completion and credits earned. Therefore, MVHS uses progress monitoring over the semester to determine absences. Student graduation rates were suggested to be jeopardized by not attending class. Liu and Cavanaugh (2012) used the number of times students logged into the LMS and how long they stayed in the course over a semester as their attendance measure. They found the time spent in the course was significant and positive for Algebra I and II. The number of times a student logged in to the LMS was significant and negative for the same courses. Dickson (2005) used clicks in the Michigan Virtual High School's LMS as a measure of attendance to determine student success. They found that the total number of clicks was strongly correlated with the student's grade in a course. Lowes et al. (2016) used a group of variables to measure attendance. These attendance variables were the number of logins, pages accessed, assignments submitted, and time spent in the LMS. They were investigating a connection between LMS use and course outcomes. Their results showed that the attendance variables positively correlated with a higher final grade, but there were gender differences. Specifically, there was no significant correlation between interactivity behaviors, defined as viewing and writing posts, and final grades for females.

Overall, the value of attending school as related to positive academic outcomes for students in K-12 has been researched and is a part of national data reporting for many brick- and-mortar schools. However, there needs to be more research exploring these variables in K-12 virtual schooling. More in-depth, scholarly research on a standard definition for attendance in virtual schooling and the connection between attendance and learning outcomes for virtual students is needed. We aim to fill that gap with the present study.

The theoretical framework for this study is based on the self-determination theory (SDT) developed by Deci and Ryan (1985, 2000, 2002). SDT is a framework that focuses on human motivation and personality development. It explains that individuals have three basic needs for their well-being and optimal functioning: autonomy, competence, and relatedness. According to SDT, when these three needs are fulfilled, individuals are more likely to experience intrinsic motivation, psychological well-being, and optimal development. Conversely, when these needs are not satisfied, individuals may experience diminished motivation, lower well-being, and difficulties in personal growth (Deci & Ryan, 1985, 2000, 2002). SDT has been applied in various domains, including education, to understand and promote optimal motivation, engagement, and well-being (Guay, 2022).

Engagement is a vital aspect of SDT, referring to an individual's active involvement and effort in their learning (Fredricks et al., 2004; Lu & Cutumisu, 2022; Wang et al., 2017). In a learning setting, student attendance is essential to engagement (Lenhoff & Pogodzinski, 2018). Attending school regularly allows students to actively participate in class discussions, group activities, and collaborative learning experiences, fostering engagement and enhancing their learning experience (Balfanz et al., 2007; Gottfried, 2009; Kearney, 2008). Studies have linked student engagement to academic success, increased retention, and higher graduation rates (Carini et al., 2006; Fredricks et al., 2004, 2016; Trowler, 2010; Wang et al., 2017). Other studies have shown that lack of engagement results in lower levels of academic success and learning (Carini et al., 2006; Handels- man et al., 2005; Kuh, 2009; Trowler, 2010; Wang et al., 2016). Learner attendance is one key indicator of a student's engagement in the learning process and thus is essential to student success (Fredricks et al., 2004; Gottfried, 2009; Kearney & Graczyk, 2014; Wang et al., 2017).

A common attribute among the many ways of measuring attendance in an online course is student activity within that course. For our research purposes, we used days of activity online, defined as the number of days a student showed activity in the course. Our research focuses on understanding how student course grades can be predicted from days of activity online in online high school classes.

Given the nested structure of our data, we examine students nested within classrooms nested within schools as a three-level hierarchical school-effect analysis. Individual differences in student activity and grades in a course depend on student characteristics and features of schools and classes. Conditions within online classrooms may influence the days a student is active, and the school structure may influence the classroom. In this study, we focus on the level-1 predictor, student days of activity online, and demographic variables of gender, ethnicity, and free and reduced lunch. The nested nature of our data within courses makes it meaningful to capture potential random effects and account for the unbalanced nature of students taking courses in a school. Level-2 predictors such as class size, grade level, and subject, and the level-3 predictor of school size were included.

Our research asks the question: Do days active and students' personal characteristics (gender, ethnicity, and free and reduced lunch) predict course grades when examining classroom characteristics (class size, grade level, and subject) and school size? The study's central hypothesis is that days of activity will be a meaningful predictor of improving course grades. The personal characteristics of students are explored in our central hypothesis.

Our data comes from online schools across 30 states during the pandemic school year 20202021. The data for this analysis contains information on 62,698 high school students (9th through 12th grade) across 660 online courses nested within 70 schools. Descriptive and demographic data are provided in Table 1 and Table 2.

Hierarchical linear models were used due to this study’s nested, multilevel nature. We designed a three-level multilevel analysis, common in educational studies, where students are grouped within classrooms and nested in schools. There are advantages to fitting multilevel linear models, but essential for our study is the unbalanced nature of students and accounting for the hierarchical nature of the data. We analyze the data in SAS (SAS, 2018) using the MIXED procedure and the default Restricted Maximum Likelihood REML to estimate models. We borrow our approach for multilevel modeling from sources such as (e.g., Bell et al., 2013; Dalal & Zickar, 2012; Fox, 2016 Raudenbush & Bryk, 2002).

Model 1 is the unconditional means model with no predictors but includes the random effect for the intercept. There were two intraclass correlation coefficient (ICC) values to calculate, one for courses and the other for schools.

These ICCs estimate the variation in course grades amongst our clusters of courses at level 2 and schools at level 3. In our first model, the default variance structure was used to estimate separate variances but no covariance. At each model-building stage, we build on the previous model.

Where:

In Model 2, we added fixed effects for the level-1 predictor, the days a student is active, and demographic variables (Gender, Ethnicity, Free and Reduced Lunch) to our model. Gender is coded as Male 1. Ethnicity (African American, Native American, Asian, Hispanic/ Latino, Multiracial, Native Hawaiian, Unre- ported/Other, White) and Free and Reduced Lunch (free, reduced, not eligible, and unknown) are dummy coded with unknown and White as 0. Preliminary analysis reduced the need to examine interaction effects amongst level-1 predictors.

In Model 3, we added the random effects of slope to determine if the relationships between our level-1 predictors and course grade vary amongst the course clusters at level 2 and the schools at level 3 of our model.

In Model 4, we add the fixed effects for courses at level 2: class size, grade level, and subject. These two categorical variables, grade level (9th, 10th, 11th, 12th) and subject (English, history, math, science), were dummy coded such that 12th grade for the grade level and science for the subject variable was coded as our reference group. Preliminary analysis indicated some value in examining interaction effects amongst level-2 categorical predictors. Because zeros are meaningful in our models, we choose not to center the model.

In Model 5, we add the random effects of slopes for our categorical level-2 predictors to determine if grade level and subject level-2 predictors vary amongst schools at level-3. Model 6, our final model, is based on our equations and retains significant effects from previous models with the level-3 predictor school size added to the model. Regardless of the significance of level-3 predictors, the clustering of schools may help account for nested effects.

The random intercept model, including all three levels, is expressed in Eq. (2). Note that categorical dummy-coded predictors do not provide a coefficient for the reference group. Eq. (2). Extend this model to include effects for random slopes in Equation (3).

The traditional standard variance component model is the SAS default covariance structure, Variance Components (VC). It was selected for parsimony after reviewing comparisons with the unstructured (UN) covariance structure and finding the VC acceptable (Eq. 3A).

The level-1 predictor of interest in this study was days reported active, the number of days a student showed activity in the course. In addition, we use several demographic characteristics of students at level 1 to predict course grades further. At level 2, class size, grade level, and subject were added, and finally, at level 3, school size was included in the final model. The dependent variable was course grade. Table 1 presents the descriptive statistics for continuous measures, and Table 2 is a frequency table for our variables used as categorical predictors.

The primary purpose of this paper was to determine if the level-1 predictor, the number of days a student was active in a course during the pandemic, influenced their outcome on the course grade and if the outcome was impacted by level-2 predictors of the subject area and grade level. The ICCs for course and school were calculated using the variance estimates from model 1 in Table 3. Associated ICC's:

We attribute 16.4% of the variation in course grade to the course, 15.9% to the school, and the remaining 67.7% variation of the course grade to the student. As is typical, the majority of the variance in the model is due to the student, but nearly a third of the model is explained by the variation amongst the courses and schools.

In model 2, random effects from model 1 remained significant when the level-1 predictor was added. Adjustments and decisions were made by developing the models based on significant effects. We present the results of random effects in Table 3. In model 3, the random slope was insignificant for level-1 predictors on level-2 courses, except for active days. Model 3 was rerun with only days active as a random slope on level 2 and level 3 (Table 3). The relationships between days active at level 1 and the outcome of course grades varied on courses and schools. Courses have various slopes supporting the random effect of days active predicting course grade. In model 4, random effects from Model 3 remained significant when the level-1 predictor was added. Model 5 with random slopes for level-2 variables did not converge correctly and were not positively definite, implying a slight variation accounted for when level-2 variables were added to the model. This was true regardless of freeing parameters with alternative covariance structures, such as unrestricted ones. In further exploring data, there appeared to be no reason to stress random slopes, and this aligns with practical considerations for grade level and subject. When course size was entered into the model as the only level-2 random slope, the model still did not converge. Further, we investigated this level-2 predictor visually to support the decision not to add random slopes for level-2 variables but to consider only fixed effects.

Although we did not successfully model level-2 random slopes, nor should we have given our visual inspection, the multilevel model helped model level-1 random slopes for days active, intercepts at levels 2 and 3, and for our design with unbalanced groups.

Model 6 added the level-3 fixed effect for school size; however, the effect, which is significant as a fixed effect, is small. Model 6 AIC and BIC were larger than model 4, indicating that model 4, without the fixed effect for school size, is the best model for final consideration. The model fit statistics for the models can be found in Table 2. This reduces our model from Eq. (3) to Eq. (4):

We have removed level-2 random slope and level-3 fixed effects from Eq. (3) but retain the random intercept.

The results of fixed effects for all models are presented in Table 4. The pattern of significance and effects is similar across models. When examining model 4, there are significant and meaningful patterns of effects across all level-1 predictors: Days active, Gender, Ethnicity (African American, Native American, Asian, Hispanic/Latino, Multiracial, Native Hawaiian, Unreported/Other, and White), and Free and Reduced Lunch Eligibility (Free Lunch Eligible, Not Eligible, Reduced Lunch Eligible, and Unknown Eligibility). Specific effects of note: Each day active increased the student's course grade by over 0.60%. Males, on average, performed worse, at 3.12%. When compared with the reference group White; African American (-1.25); Native American (-2.92); Hispanic/Latino (-2.05); Multiracial (-.90); Unreported/other (-.66) performed lower, while Asian (2.60), and Native Hawaiian (.90, but not significant), performed higher. Reduced Lunch Eligible and Unknown Eligibility have similar effects, while Free Lunch Eligible (-1.54) performed lower, and Not Eligible (1.77) performed higher on course grades.

Level-2 fixed effects (Table 5) were significant for course size (-.01) although the overall effect size was small; grade levels 9th, 10th, and 11th were all significantly lower than 12th grade; the subjects of English, more so for math, had lower associated grades than science. History appeared to have a similar main effect. In contrast, the interaction effects with grade and school were all positive. Overall math effects were -9.12, while the interaction effects with 9th (8.09) and 10th (2.89) grades were positive; 11th grade was not significant (.90) but still positive. This finding might be an issue of difficulty for those taking 12th-grade mathematics. History and English interacted with 11th grade for a positive effect, as well as 9th grade English. Finally, our level-3 predictor was significant, but there were better models than model 6.

Attendance and student academic success are connected, with student attendance as an essential sign of student engagement (Gottfried, 2009; Fredricks et al., 2004; Kearney & Graczyk, 2014; Wang et al., 2017). Most prior research has focused on brick-and-mortar schools rather than online ones. Online schools have been serving students for over 20 years. Research that offers insight into student success is critical in supporting online education.

The researchers sought to fill this gap in the literature.

First, the researchers hypothesized that days active would be a significant predictor of course grades. Attendance was determined by the indicator of students being present as in a course. The models indicate that a student's grade increased -by .60% units for every day they were active online. For example, an online school student who was active for 10 days increased their grades by 6.04% during the pandemic. This finding supports previous research findings that activity in an online course would increase student outcomes (Liu & Cavanaugh, 2012; Lowes et al., 2016). Depending on the variability due to classroom and schools, a 12th grader in a science course would need to be active online for 62 days to increase their grade from baseline to an A, depending on their ethnicity and gender. That same 12th grader must be active for 77 days to receive an A in math. Neither of these values seems unrealistic across an enrollment span, especially since days active can be concurrent, i.e., a student could log into their math and science courses on the same day. Having this information is impactful for online programming. Online schools should create goals using this information. Teachers could work with students to aim to be active in their classes with high achievement rates.

Next, the researchers examined whether student characteristics would affect course grades. Lowes et al. (2016) found gender differences with interactivity behaviors not correlated with female students' final grades.

While this study did not examine interactivity behaviors specifically, males, on average, performed worse with a -3.12% decrease in their overall grade. Gender disparities in attendance and final grades are an area for future research.

In exploring further demographic student data, attendance, and racial educational implications were found in this research supporting prior literature (Liu et al., 2021). African American, Native American, Hispanic/Latino, and Multiracial groups performed -.90 to -2.92 lower than White students, demonstrating that days of inactivity online are more significant for these groups in terms of educational outcomes. Asian and Native Hawaiian performed higher than the White student group. Knowing that days active have significant implications based on race, online schools must develop systems that support students of the racial groups who are impacted so that they are not perpetuating an achievement gap. The impact of race on student achievement in online schooling is an area for future research.

Balfanz and Byrnes (2012) found that the number of school absences in high-poverty areas was significant, with students missing six months to one year of school over five years. Similarly, Santibanez and Guarino (2021) found that some student groups, particularly low-income students, are more vulnerable to the adverse effects of school absences. This study found that Free Lunch Eligible students performed -1.54 lower than their peers, while Not Eligible students performed 1.77 higher on course grades. Adding to the body of research, SES impacts student success in school, specifically online school.

Further, classroom characteristics were examined to see if they affected course grades. It is important to note that class size for online schools should be considered different from that for brick-and-mortar schools. In online schools, students often do not work synchronously together simultaneously. The amount of asynchronous work in the class varies by course and school. Synchronous classes are also nuanced, depending on the school and class, with some classes offering breakout sessions within larger classrooms, causing a larger class to be comparable in size to smaller classes. There was a small negative effect (-0.012) as course size increased. The effects of limiting course sizes to small classroom sizes in units of 10 would not likely be meaningful; however, once the course size reaches several hundred (e.g., 300 plus students), the effect could be impactful. The lack of a stronger effect may be due to further complications in understanding the course size effects of asynchronous and synchronous learning opportunities. Being in a smaller class has advantages. Future research should combine course size with live teaching breakout opportunities to determine if indicators such as size and frequency of synchronous teaching opportunities impact outcomes. Additionally, we referred to course effect instead of teacher effect, as it was impossible in this investigation to separate courses from teachers. Given the nature and variety of courses, it might be helpful in the future to separate synchronous and asynchronous activities and track other specific teacher effects, such as engagement, which might add value to the student.

Finally, our level-3 predictor, school size, was significant, although model 4 without that predictor was selected based on mode fit. Extremely large schools might have a small impact on performance (0.0004 per student) above and beyond our other indicators, especially when interacting with online course size. Future research should focus on indicators that vary amongst schools, such as those associated with state- and district-level effects. It would also be worth investigating if the trends in larger courses and schools, when there is increased activity, are due to more highly developed courses and stability over time.

During the pandemic, brick-and-mortar schools struggled with pivoting to online education, with subsequent educational time missed and learning loss for students (Kuh- feld et al., 2020; Liu et al., 2021; Santibanez & Guarino, 2021). Students enrolled in online schooling may not have the same learning loss as those in brick-and-mortar schools, as there was no need for these schools to close or pivot their academic model. The data used was collected during the pandemic 2020-2021 year; however, due to the nature of online schools, the pandemic should not have impacted online student attendance as there was no need for these online schools to be closed or to have shifted their model for delivering education. Online school systems such as the ones we examined used previously existing online school structures and were not emergency schooling quickly pulled together to support brick-and- mortar students during the pandemic. We suspect established online schools could maintain student performance during the pandemic compared to brick-and-mortar schools. However, it would require further research with a comparative analysis across types of schools to state definitive conclusions.

A limitation was the inability to control attendance. In this study, attending or being absent from school is subject to variables such as student illness, technology issues, and intrinsic motivation, which the researchers could not control for in this research. Receiving information on why students were not active in their courses is an area that could be further explored.

The researchers were able to use a key indicator of student engagement for this study: attendance. However, a limitation is that they could only acquire engagement data within days of activity, such as clicks, viewed pages, or submitted assignments. In the future, it would be helpful to examine success in school with those additional measures of engagement (Archambault et al., 2013; Dickson, 2005; Liu & Cavanaugh, 2012; Lowes et al., 2016).

The researchers chose to define attendance as days of activity.A limitation of this definition is that it needs to provide information on engagement or time spent in the course. Further, days active, although an indirect indicator of engagement, can also mean a student logged on but completed no work. More in-depth, scholarly research is needed on a common definition of attendance in virtual schooling.

A generalized mixed method approach to multilevel modeling was used to find if days active for online students were meaningful and predicted student success. The researchers confirmed their hypothesis and found a significant relationship. Online student days active is an obvious and powerful predictor of student performance.

This finding is important because this study not only adds to scholarly research on online students' academic success but also offers valuable information to online schools concerning course planning and the ability to continue supporting student success. Ensuring that students remain active in their courses will produce positive grade outcomes in all courses. Brick-and-mortar schools implemented attendance procedures to help reduce student absences, and based on our findings, this is an area of recommendation for online schools to explore and implement.

Schools may want to incentivize attendance, rewarding students for regularly logging in to their classes. Online school districts could offer awards for attendance. Districts could also hold student competitions for the highest number of days active in courses. Teachers could also send supportive messages acknowledging and praising student activity in their respective course(s) throughout their time together.

Another recommendation would be a personalized approach to student success by having staff assigned to monitor student activity in courses. Outreach would be a vital component of monitoring student activity. If a student is inactive in their classes, school staff will immediately reach out and work to re-engage the student. This personalized approach would help keep the student connected to the school and build a positive, supportive climate amongst students and school staff.

Additionally, online schools should share information with caregivers on the importance of attendance for student success at online schools. Online schools should also engage and partner with caregivers to support active school participation. Outreach to students who are inactive in their courses should include outreach to the caregiver as well. Working together as a team to support the students in engaging in their courses will create an environment for student success.

Ensuring that every child has equitable educational opportunities and experiences is the foundation of our educational systems. Online schools offering attendance support for students may help close gaps related to race, gender, and SES. There is a significant difference in student achievement outcomes based on these variables. Knowing the role that attendance plays in exacerbating this gap, online schools need to take steps to ensure that these populations are offered additional support. Online schools may consider proactive strategies for the populations at more of a disadvantage when experiencing days of inactivity. Systems could be put in place with a team of staff working with these student groups to ensure that they do not begin to miss days of school. Each teacher, counselor, and administrator working with those respective students should have touch points to ensure students are connected and engaged with the school. Utilization of proactive strategies may be key in helping to close educational gaps and support successful academic outcomes for online students.

Additionally, this research lays the foundation for future research studying important engagement variables that may interplay with student attendance, such as intrinsic motivation, teacher effects, peer effects, engagement with course materials, and access to stable internet. Finally, researching additional student outcomes such as online student test scores, retention rates, and graduation rates would offer more insights into this work and support online school in ensuring all students are successful in their educational careers.