Structural variable validation of an Online Learning Response Behavior (OLRB) instrument: A comparison analysis of three extraction methods of Exploratory Factor Analysis

In a typical setting, universities and colleges are urged to incorporate blended learning for all academic programs. Blended learning involves a blend of traditional face-to-face training with digital learning. In blended learning classes, the learning materials were primarily provided through face-to-face interaction, but they were also accessible through a robust learning management system (LMS) to support and increase instructor–student and student–student connection after class. However, as of March 2020, all higher-education institutions in Malaysia have been shuttered owing to the COVID-19 epidemic. During the implementation of this measure, students will still be in the beginning of the semester and will be required to transition to an online learning mode. This has resulted in a rapid shift to online instructional delivery. To assist in limiting the spread of the virus that causes COVID-19, some schools have elected to cancel all face-to-face sessions, including laboratories and other learning opportunities, and have demanded that teachers transfer their courses online. Globally, the majority of colleges and institutions outside of Malaysia have used the same approach as one of the restraints to minimize the COVID-19 viral transmission process.

Due to this, digitalization has contributed to a surge in interest among educational scholars worldwide, particularly in developing nations and the Asian region. Responses to technology adoption are characterized by extremely disparate attitudes, digital skills and degrees of readiness (Yasin et al., 2020). The phenomenon has also led to an emphasis on behavioral intention in the context of individual components such as cognition and motivation (Costley and Lange, 2017; Oluwajana et al., 2019), in addition to learning tools (Wammes et al., 2019). In the context of this study, behavioral engagement is demonstrated by students' participation such as idea generation, participation in the classroom and extracurricular activities, concentration, assignment submission and adherence to the instructor's directions (Reschly and Christenson, 2012). All these variables are associated with a higher number of behavioral interventions and educational tools for students to increase their involvement in online classrooms for successful learning (Hughes et al., 2020). Consequently, it is vital to research how novice digital learners react to learning.

Although there are similarities in the research issues on behavioral response, the approaches used to investigate students' perspectives on learning behavior have not been consistently verified. Therefore, educational researchers utilizing digital platforms must have access to instruments or measures that have been cross-validated across varied groups (Guo and Liu, 2018; Hsiao et al., 2019). This would increase the reliability, generalizability and practicality of online learning studies.

Still, validated instruments are used rather infrequently in research on the response behavior of students to online learning (Yasin et al., 2020). As a result, the purpose of this study is to investigate the procedures that may be used to determine the validity and reliability of the instrument. Evaluation of students' abilities to comprehend, produce and evaluate information offered within the context of a Malaysian online educational setting is carried out. The same method is advantageous for designing evaluation tools to measure other aspects of a student's readiness, such as their perspectives and preferred learning style. Consequently, we established two aims for this study: (1) to identify significant factors for assessing the online learning response behavior in higher-education institutions and (2) to assess the structural variable validity and reliability of the online learning response behavior questionnaire.

In this study, both a cross-sectional applied survey and a combination of quantitative methodologies were utilized. According to Creswell (2014), the purpose of this study is to validate and investigate the structural aspects of the targeted structures, and hence, these two research methodologies were chosen. Moreover, the phrase of the operational items in the Likert scale that were adopted from the previous study was amended in accordance with the study context. Therefore, it is vital for the researchers to revalidate this instrument using a suitable quantitative technique (Hair et al., 2010).

This study is particularly interested in Malaysian public university students having at least one year of experience using an online learning platform during the Movement Control Order (MCO). In the lack of a sample frame for selecting respondents, this study used a combination of convenience and judgmental sampling methods, both of which are regarded appropriate (Creswell, 2014).

There were 410 Malaysian public university students who volunteered to participate in this study by responding to a social media–distributed questionnaire. The sample size for this study is sufficient since the minimal sample size for a large population (1m or more) is 384 (Krejcie and Morgan, 1970). In addition, Hair et al. (2010) and Tabachnick and Fidell (2007) recommend evaluating at least 350 exploratory factor analysis (EFA) samples. Consequently, a sample size of 410 respondents may be considered adequate for this study.

The process of constructing a questionnaire begins with the collection of accessible definitions, concepts and practicalities of the key notion of students' learning response behavior from the previous academic literature. This procedure begins with a search of the most cited sources pertaining to the assessment of learning response behavior in the Scopus database, which contains only peer-reviewed scientific publications, in an effort to exclude duplicate sources.

Using the Boolean search operator “TITLE ABS-KEY” (“learning response behavior”), a total of 167 relevant sites were retrieved (survey OR scale OR questionnaire OR model). From these, 108 relevant sources remained because they met the selection criteria for the inclusion of the studies, which are as follows:

1. The research is published in a scientific publication.

2. The study consists of empirical research with survey-based learning response behavior measures.

These 108 relevant sources were retrieved via the digital library service provided by the Universiti Teknologi MARA or via the open access option, with the material regarding survey-based assessment of learning response behavior being the only topic of interest. From a total of 108 suitable sources, 101 were eliminated from this procedure for the following reasons:

1. Title and abstract irrelevance (15 papers);

2. Conceptual papers (23 papers);

3. Papers that employ a qualitative technique (43 papers);

4. The articles' content did not address the optimal usage of questionnaire methodology (13 papers);

5. The papers were not authored in English (7 papers).

Following the screening procedure, seven papers that met the aforementioned selection criteria were included in the systematic review. Figure 1 depicts the flow diagram of the preferred reporting items for systematic reviews and meta-analyses (PRISMA)-based search procedure used in this research. We integrate two major types of methodological criteria in the process of framework or item assessment: evaluation of a survey-based operationalization measurement scale (Table 1) and evaluation of measurement quality (Table 2). These stages, according to Carmines and Zeller (1979), are required but not sufficient for obtaining a trustworthy and valid structural variable instrument. Based on these two methodological criteria, all seven of the selected studies fit the requirements, with 100% of the criteria met (Table 3).

To evaluate the learning response behavior of students, the researcher developed instruments based on a systematic review analysis that matched the PRISMA criteria and two major methodological criteria, as shown in Table 3. A 38-item survey was modified using a seven-point Likert scale, where 1 corresponds to strongly disagree and 7 corresponds to strongly agree, and was randomly distributed in the instrument setup. Adapted from Yasin et al. (2020), three elements of the student's learning readiness were used (i.e. Q1, Q5, Q6). Based on Gao et al. (2020), the student's learning engagement is defined as the reflection of cognitive and emotional feelings representing their behavior in response to an online learning environment consisting of seven modified items with two dimensions: cognitive engagement (i.e. Q2, Q7, Q9) and emotional engagement (i.e. Q2, Q7, Q9) (i.e. Q3, Q4, Q8, Q10). This instrument consists of eight items to assess the learning motivation construct referred to as the beliefs, intentions and feelings that motivate students to learn, modified from the study by Liu (2020): four items for intrinsic motivation (i.e. Q12, Q13, Q16 and Q19) and four items for extrinsic motivation (i.e. Q11, Q14, Q15 and Q17).

Alternative assessment, on the other hand, is the variable used to evaluate students' learning and support their cognitive development by regularly examining and providing feedback on their work. Thus, eight questions (i.e. Q21, Q24, Q25, Q27, Q30, Q31, Q33 and Q35) were adopted from the study by Leeuwenkamp et al. (2019). This study also modified the Awidi and Paynter (2019) student's learning experience, which is defined as the process of knowledge construction and converging ideas developed through social interactions, with four items (i.e. Q22, Q23, Q29 and Q32), whereas the remaining four indicators were adapted from the study by Utriainen et al. (2018) to evaluate the critical thinking learning experience (i.e. Q20, Q26, Q28 and Q37). This instrument also incorporated students' response behavior toward online learning, adapted from the research of Bui et al. (2020), with four items (i.e. Q34, Q36, Q38 and Q18). The questionnaire is supplied in  Appendix. Importantly, the premise of utilizing this questionnaire as an elicitation tool is that the unit of analysis must have online learning experience as the purpose of this questionnaire is to measure the respondent's online learning response behavior. In addition, the responder must be an adult learner in order to provide an accurate response to each questionnaire item as the items included in this questionnaire tend to be psychological measures that require exact evaluation.

The IBM SPSS version 21.0 statistical software was used for data input and analysis methods. The EFA and correlation analysis were used because the primary purpose of this study was to validate and investigate the structural variable validity and reliability of the instrument after undergoing the process of sentence modification and Likert-scale use by the original authors (Hair et al., 2010; Tabachnick and Fidell, 2007). As recommended by Hair et al. (2010) and William et al. (2010), this form of study enables the researcher to validate and modify the employed indicators (2010). In this work, three EFA extraction methods were utilized to compare the instrument's valid structure. The first extraction method that was used is a combination of principal component (PC) extraction method with a varimax rotation (PC + varimax), the second and third extraction methods were principal axis factoring (PAF) extraction method with a varimax rotation (PAF + varimax) and the final extraction method was maximum likelihood extraction method with a varimax rotation (ML + varimax).

Using a Kaiser–Meyer–Olkin (KMO) index, the existence of a nonidentical matrix and the sufficiency of the covariance matrix among the indicators must be determined at the initial phase of the study. As proposed by Field (2009) and Pallant (2010), this index value must exceed 0.60. As a support for this conclusion, Bartlett's test of sphericity must be significant (Pallant, 2010); hence, it confirms the existence of a nonidentical matrix and that the covariance matrix among the indicators is adequate for the EFA. Thompson and Daniel (1996) suggested that, in the second stage of this study, a researcher should use several criterion techniques to identify the number of dimensions that should be retrieved from the constructs and to check and validate the number of dimensions that exist in the constructs. Therefore, only dimensions with eigenvalues more than 1.00 are retrieved from the analysis, and the total percentage of variance extracted should be greater than 60%, as recommended by Hair et al. (2010), Field (2009), Tabachnick and Fidell (2007) and Thompson and Daniels (1996). In addition, Watkins (2000) recommends using the Monte Carlo PCA parallel analysis simulation eigenvalues to calculate the number of extracted variables. In this study, only extracted variables having an eigenvalue larger than the eigenvalue of the Monte Carlo simulation are considered extracted.

Examining the value of factor loading and communalities from each extraction technique (such as PC + varimax, PAF + varimax and ML + varimax) is the next step in validating the indicators under each dimension extract. According to Field (2009), for an exploratory study on refining and validating indicators using the EFA, the values of factor loading and communalities must be more than 0.40 and the sample size must exceed 350. This is a crucial step as it ensures that the retrieved dimensions are different, meaningful and valid with a sufficient degree of confidence. In the last step of the EFA methods, a Cronbach's alpha reliability test was used to evaluate the internal consistency of the indication group. Nunnally and Bernstein (1994) indicated that a threshold of 0.70 or above can be used to show the reliability of grouped elements.

As proposed by Creswell (2014), Hair et al. (2010) and Kaptein (2008), discriminant validity and nomological validity were assessed following the EFA and reliability analysis. Using correlation analysis, they describe nomological validity as the act of examining the logical and meaningful link between dimensions that follows the predicted direction of association (i.e. positive relationship or negative relationship). In addition, Hair et al. (2010) and Kaptein (2008) used the correlation coefficient to assess discriminant validity using correlation analysis. Observation suggests that when the correlation coefficient is less than 0.70, the extracted dimensions have discriminant validity.

The findings of multiple criterions for determining the number of variables to be retrieved from this instrument's 38 total items are summarized in Table 4. The study shown in Table 4 revealed that the EFA analysis for these three extraction techniques should extract nine variables from a total of 38 items since the first nine eigenvalues under Kaiser's criteria (e.g. range: 1.059 to 5.762) were greater than 1.00. Moreover, when comparing these Kaiser's eigenvalues with simulation eigenvalues from the Monte Carlo analysis, the same result occurs: only the first nine Kaiser's eigenvalues are greater than the first nine simulation eigenvalues from the Monte Carlo analysis, confirming that nine variables should be extracted from a total of 38 items. The cumulative percentage of variation explained for these nine extracted variables is 83.83%, indicating that the extraction of these nine variables from a total of 38 indicators is legitimate. Bartlett's test of sphericity for the items was likewise highly significant (Χ² = 20419.75, p < 0.01). On the other hand, the KMO index for these three extraction techniques of the EFA was 0.974. Therefore, the covariance matrix for these 38 items was not a matrix of identity. Consequently, all elements in the instruments can be used for the EFA if all three extraction techniques are included.

Table 5 displays the factor loading and communality value for each item based on the three extraction methods used in the EFA. As with the first EFA extraction approach (i.e. PCA + varimax), the analysis revealed that all items had loading (range: 0.504 to 0.903) and communality (range: 0.709 to 0.963) values greater than the minimal threshold value of 0.40. Alternatively, the second extraction technique of the EFA (i.e. PAF + varimax) yields the same result, with all 38 items having loading (range: 0.439 to 0.874) and communality (range: 0.508 to 0.915) values greater than 0.40. The loading (range: 0.387 to 0.948) and communality (range: 0.373 to 0.919) values were just below 0.40 for the third extraction technique of the EFA, ML + varimax. Since the sample size for this research (n = 410) is regarded to be substantial, the ML + varimax technique's findings can be considered reliable. Therefore, it is possible to infer that all 38 items were valid and could be utilized to measure the intended components in this study.

Based on the grouping pattern of indicators, these three EFA extraction techniques exhibit a similar pattern. Student's learning readiness, student's perception of alternative assessment provided and student's response behavior toward online learning factors were discovered to have maintained their one-dimensional structure. Based on these three extraction analysis approaches, all analyses demonstrate that the dimensions of the student's learning engagement, student's learning motivation and student's learning experience variables, each of which has a two-dimensional structure, were preserved. Cronbach's alpha scores for the grouped indicators ranged from 0.902 to 0.965, suggesting that all nine grouped indicators meet the minimal threshold value of 0.70. Therefore, it can be stated that all grouped indicators were legitimate and measured the variables consistently.

The nomological and discriminant validities of the extracted variables were examined using correlation analysis, and the results are presented in Table 6. At the 5% level of significance, the analysis revealed a substantial positive association between the extracted variables. All extracted variables may thus be confirmed to have good nomological validity since all bivariate correlations correspond to the predicted bivariate connection, which is a positive relationship. In addition, correlation coefficients for the analysis ranged from 0.410 to 0.667% when examined using the correlation coefficient. Since all correlation coefficients are less than 0.70, it can be concluded that the extracted variables provide a satisfactory level of discrimination between the variables.

This study illustrated the statistical processes for verifying, investigating and comparing the variable structures in the survey instrument utilizing three extraction techniques of the EFA and correlation analysis for the online learning response behavior instrument. Based on these three extraction techniques of the EFA, the validity of all items utilized to measure the targeted variables is satisfactory. This is due to the fact that all three different extraction methods for the EFA (i.e. PC + varimax, PAF + varimax and ML + varimax) show a similar pattern of item grouping after undergoing the process of item sentence modification and Likert-scale weighting at the start of instrument development. In addition, the examination of reliability revealed that all extracted variables had sufficient and satisfactory internal consistency, as all Cronbach's alpha values were greater than 0.70. The correlation analysis also confirmed that all extracted variables from the EFA have good nomological and discriminant validity as the bivariate relationship between the extracted variables was significant and followed the expected direction of the relationship, and all correlation coefficients were less than 0.70.

On the other hand, based on the grouped elements from the EFA, student's learning readiness may be viewed as a continuous process that shows the students' mental preparation to accept the diverse method of technology-based learning (Yasin et al., 2020). According to Gao et al. (2020), student's learning engagement may be characterized as a technique that guides students in cultivating the instructor-acquired information and can be classified as cognitive engagement and emotional engagement. The fourth and fifth grouped factors, internal and extrinsic motivation, respectively, can be considered the total learning motivation of a learner. In this study, extrinsic motivation is defined as motivating students to attain their objective by providing an external incentive throughout the learning process on an online platform in the hope that students will acquire a strong interest in learning via the online platform (Liu, 2020). In contrast to extrinsic motivation, the intrinsic motivation variable was measured by the internal needs of the students during the process of learning on an online platform, where internal incentives such as creating an enjoyable environment during the learning process are anticipated to bolster intrinsic motivation among the students (Liu, 2020).

On the basis of the quality of alternative assessments offered by instructors, students' perceptions of alternative assessments may be interpreted as their perceptions of their learning progress and information about their desired learning goals (Leeuwenkamp et al., 2019). Experience with competencies and critical thinking are two criteria that may be used to assess a student's learning experience. This variable represents the entire learning experience encountered by the learner during the learning process. According to Utriainen et al. (2018) and Awidi and Paynter (2019), a student's learning experience can also be viewed as the construction of knowledge by the student through interaction with others or as the development of competencies and the capacity to apply knowledge to other contexts, which is critical thinking experience. Finally, student's reaction behavior toward online learning variable may be described as the behavior of students continuing to use the online learning platform as the learning medium in the future and introducing the online learning platform to their social circle as the optimal means of learning.

Based on the findings of the EFA and correlation analysis, this online learning response behavior instrument building procedure is suitable for measuring the intended constructs. Due to the present COVID-19 epidemic, this study method may also be categorized as significant processes as this instrument can serve as an alternative beneficial tool for monitoring the entire online learning response behavior of students. This research also stressed the usefulness of the EFA and correlation analysis for verifying the survey instruments after the items have undergone a process of alteration from the perspectives of convergent validity, discriminant validity and nomological validity.

In addition, the outcomes of this study suggested that this questionnaire has the potential to be utilized internationally, not just in the Malaysian higher-education institutions setting, because most higher-education institutions worldwide have shifted to an online learning platform. In addition, all the factors used to quantify the online learning response behavior occur in the majority of higher-education institutions. This questionnaire is simple to administer because it consists of 38 items that may be categorized as small. Consequently, the likelihood of administrative data inaccuracy is likewise minimal. Since the structural themes of all variables revealed in this study lean toward the positive theme, the interpretation of the variables should likewise be positive. Using the average score approach, the score of each variable's structure must be high for a positive interpretation.

In reference to this research, one of the drawbacks of this study is the lack of a causal analysis to identify the global questionnaire fit indices. This study thus recommends a causal analysis to further explore the global fit indices of the instrument's validity by employing a covariance-based confirmatory factor analysis or partial least square consistency factor analysis. In addition, for the purpose of calculating an overall global fit index, this instrument may be administered to secondary students and private higher-education students in both urban and rural regions. Moreover, our systematic review approach is restricted to terms that assess just the learning response behavior, and only seven papers are best accessible for reviews. Therefore, it is advised that, for future study, a more comprehensive word, such as students' response behavior, and a greater number of publications should be chosen for examination.

Table A1