Construct Validation of Student Attitude Toward Science, Technology, Engineering, and Mathematics Project-Based Learning: The Case of Korean Middle Grade Students

Attitude was defined as “a cognitive, affective and behavioral reaction the individual organizes toward himself/herself or any object, subject or fact around him/her based on information, feelings, and motivation” (Aydin, 2012, p. 334). Student attitudes toward learning refers to an evaluative and emotional reaction comprising affect, cognition, and behavior to a complex learning environment (Finch, 2000; Rajecki, 1990; Zimbardo & Lieppe 1991). Attitudes emerge in students as an effect of learning environment, and are expressed as a shape of cognition, affect, and behavior. Therefore, student attitude toward a learning environment is a critical factor, which determines the effectiveness of instruction and learning outcome, in designing lessons.

Student attitude toward a specific instructional strategy is a critical factor in deciding the effective and efficient use of an instructional strategy. That is, students’ attitude shapes their learning style, which is how they adopt and react to instructional strategies in classrooms. For example, students’ perceptions and attitudes toward computers are important determiners of how effectively they use the technology instruments in learning procedure (Watson, 2007). When instructional strategy is well adopted or accommodated to student learning style, its impact on students can be more effective and efficient (Federico, 1999, 2000).

Student’s attitude profoundly influences learning behaviors and ultimately determines learning outcomes (Anderson & Maninger, 2007; Cotterall, 1995; Hughes, 2013; Ravitx, Bencekr, & Wong, 2000; Overbay, Patterson, Vasu, & Grable, 2010). This is why many studies have connected student attitude to academic achievement (Aydin, 2012). Moreover, many researchers have investigated student attitude toward a learning environment or instructional strategies when adopting them (Aydin, 2012; Federico, 1999).

STEM PBL is a relatively new instructional strategy for teachers and students. However, there have been very few studies that have investigated student attitude toward STEM PBL with a measurement instrument that has been shown to produce reliable and valid scores. The present study developed an instrument measuring student attitude toward STEM PBL having Korean students as participants.

STEM PBL is used to refer to an instructional strategy having five factors: self-regulated learning, interdisciplinary content, technology, collaboration, and hands-on activities. STEM PBL is grounded in Dewey (1938) teaching and learning theory, and consists of constructivism learning factors. Dewey (1938) described a learning environment where students drive their own learning and construct new knowledge based on prior knowledge. To encourage student learning, constructivists suggested preparing a collaborative, motivated, and student self-directed classroom.

A STEM PBL classroom is a student-driven learning environment. Students in STEM PBL classrooms explore project activities and construct knowledge with teachers facilitating and guiding the learning process (Barron et al., 1998). STEM PBL students leverage autonomy to direct their own learning. The teacher’s role in a STEM PBL classroom is more likely that of a facilitator, and students learn from exploration of the project and from others in groups (Johari & Bradshaw, 2008).

STEM PBL represents an interdisciplinary learning environment. “The study of technology and engineering is not possible without the study of the natural sciences. This in turn cannot be understood in depth without a fundamental understanding of mathematics” (Dugger, 1993, p. 10). STEM PBL is interdisciplinary in nature. PBL lessons for STEM education integrate multidisciplinary subject contents (Capraro, 2013; Holbrook, 2007). Students obtain either science or mathematics learning objectives based on the engineering contexts using technologies. Interdisciplinary learning enables students to obtain conceptual knowledge over procedural knowledge, which is the ultimate goal of learning (Cobb & Bowers, 1999). Student attitude toward an interdisciplinary learning environment has been connected to academic understanding (Little & Hoel, 2011). Students in the interdisciplinary learning environment showed statistically significant gains in linking interdisciplinary thinking and understanding across contents (Little & Hoel, 2011).

Technology is one of the critical factors in STEM PBL lessons, though not every PBL lesson includes a technology component. However, STEM PBL embeds technology factors in project activities. In STEM PBL classrooms, students search for information concerning project topics, test hypotheses, collect data for analysis, obtain results, and prepare presentations using technologies (e.g., computers, tablets, probes, graphing calculators) (Cifuentes & Ozel, 2008). Students’ use of technology is deeply related to their attitude or belief toward technology. Student attitude toward technology was the most critical predictor of their use in learning (Anderson & Maninger, 2007; Hughes, 2013; Overbay et al., 2010; Ravitx et al., 2000). For example, pre-service teachers’ perception toward integrating technology was the critical indicator of positioning toward technology integration (Hughes, 2013). In this sense, investigating student attitude toward technology is necessary to determine if the technology factor in STEM PBL is aligned to student learning styles.

STEM PBL activities are more often implemented in groups. Through group work in STEM PBL, students have more opportunities to collaborate with peers and teachers (Morgan & Slough, 2013). Ku, Tseng, and Akarasri-worn (2013) employed a survey consisting of 20 items to examine student attitude toward collaboration during the online course. The result from this study revealed that student attitude toward collaboration was directly connected to their satisfaction working in groups.

Hands-on activities provide contextual background, which helps students to connect mathematics and science content with real world problems (Woods & Morgan, 2008). The engineering component is another critical factor of STEM PBL. Science and mathematics content provide a realistic context for students, which in turn provides a richer contextual understanding for hands-on activities. Several studies have already investigated the effectiveness of hands-on activities on student conceptual understanding of abstract subjects such as science and mathematics (Holstermann, Grube, & Bogeholz, 2010; Johnson, Wardlow, & Franklin, 1997; E. J. Kim & Jang, 2009). However, E. J. Kim and Jang (2009) pointed out that student attitude toward those activities were important for the successful integration of hands-on activities in classrooms. Students value hands-on activities where they learn context and that mimic real world situations. However, students sometimes have fun in doing hands-on activities without obtaining any valuable knowledge embedded in activities (Barron et al., 1998). Therefore, investigating student attitude toward hands-on activity is important, because STEM PBL consists of diverse hands-on activities.

STEM education has been introduced and actively implemented in K-12 schools since 2010 in Korea. However, STEAM (science, technology, engineering, art, and mathematics) is a more common terminology indicating the interdisciplinary instructional approach. Since the Minister of Korea initiated the adoption of a STEM framework for all K-12 schools in 2011 (Yakman, 2013), there have been many efforts to develop curriculum and lesson plans integrating a STEAM framework (S. W. Kim, 2013; Shin, 2013). Unlike the educational system in the United States, Korea has a centralized curriculum and the government education office has led the development of curriculum and lesson materials.

Korean education, especially for science and mathematics, has worked for student academic achievement, not for their self-confidence or attitude toward STEM subjects. Even though Korean students have ranked top in science and mathematics in international academic assessments (e.g., Trends in International Mathematics and Science Study, Program for International Student Assessment), they showed low self-confidence and less positive affect in learning science and mathematics (Mullis, Martin, & Foy, 2008). The reason for this issue was due to the rote teaching methods that are rooted in the Korean education system (Korean Educational Development Institute, 1997).

The interdisciplinary STEAM education is an alternative educational framework to improve student interest in learning STEM subjects. PBL is a practical approach that has been and will be utilized in implementing STEM education in K-12 classrooms (Capraro & Slough, 2013). Therefore, elaborating student attitude toward STEM PBL is necessary in designing curriculum and lesson plans for teachers. The instrument developed in this study measures the extent factors of STEM PBL that are aligned to students’ preferred learning style.

We postulated that student attitude toward a specific instructional strategy is influenced by a student’s learning style. Every student has a preferred learning style, which is influenced by student achievement potential. In this sense, revealing student attitude toward STEM PBL is meaningful when considering its impact on student achievement. Based on the literature review, no study has investigated student attitude toward STEM PBL in considering the varied factors: interdisciplinary, collaborative, technology based, self-regulated, and hands-on activity based learning. Therefore, we adopted items from the surveys of other studies concerning these five factors and conducted EFA and CFA to answer the following research questions:

1. What are the factors that underlie STEM PBL components as measured by the student attitude survey?

2. Are the items of the survey reasonable indicators of the underlying construct of student attitude toward STEM PBL?

3. What is Korean student attitude toward STEM PBL?

Participants were 785 Korean middle grade students enrolled in five schools. The schools were scattered throughout a large city in Korea. Schools in the present study were randomly selected. Four hundred and fifty-six students were male (58.1%) and 321 were female (40.9%). Eight students did not respond to the gender item.

The survey employed in this study consisted of five sections: student centered learning, collaborative group work, interdisciplinary learning, technology based learning, and hands-on activity based learning. Figure 1 displays the model under consideration. The assumed structure of the survey having five sections was grounded on the literature review. Each section represented the features of STEM PBL. The 51 items were created by authors or adapted/adjusted from previous studies (Holstermann et al., 2010; E. J. Kim, 2011; Lim, Cha, & Noh, 2001; Little & Hoel, 2011; Miller, 1990; Najafi, Ebrahimitabass, Dehghani, & Rezaei, 2012; Saleh, 2011). The first section is about student-centered learning. Seven items were adapted from E. J. Kim (2011) and the authors created two items. Nine items of this section were used to examine whether students evaluated themselves as ready for a student-driven project. The second section consisting of 12 items was used to verify whether students had a positive perception of collaborative group work. Ten items were adapted from Lim et al. (2001) and Saleh’s (2011) surveys, and the authors created two items. The third section contained 11 items asking whether students believe in the positive impact of interdisciplinary approach for learning. Six items were adapted from Little and Hoel (2011) and Najafi et al.’s (2012) surveys, and five items were created by the authors. The fourth section consisted of 11 items about technology integration with learning. Nine items were adapted from Najafi et al.’s (2012) survey and the authors created two items. Finally, the last section was composed of 8 items to clarify students’ perception of hands-on activities in classrooms. Six items were adapted from Miller (1990) and Holstermann et al.’s (2010) surveys, and the authors created two items.

Finally, the paper based attitude survey was created, consisting of 51 Likert-formatted items to measure student attitude toward varied aspects of STEM PBL. Participants were required to use the entire extent of the 5-point scale, ranging from strongly agree, agree, neither agree or disagree, disagree, and strongly disagree, to show their response for each item. These response alternatives for each item were scored 5, 4, 3, 2, and 1, respectively. It took participants approximately 15 minutes to respond to all items.

To answer the first and second research questions, we conducted exploratory factor analysis and confirmatory factor analysis. Two types of factor analyses allowed constructing substantial validities of the survey concerning student attitude toward a STEM PBL learning environment. Mplus 6.0 was used for the data analysis (Muthén & Muthén, 2010). Exploratory factor analysis is one of multivariate statistics analysis, which aims to uncover the underlying structure (i.e., relationship) among the observed variables (Finch & West, 1997). Factor indicates the latent variable, which accounts for the variation and covariation among a set of indicators (i.e., observed variables). Through the exploratory factor analysis, we could specify the dimensional structures of five factors explaining student attitude toward STEM PBL (cf. Henson, Capraro, & Capraro, 2004). Using the results from the exploratory factor analysis and conceptual framework based on the literature review, the analysis design for confirmatory factor analysis was constructed. Factor loadings in a measurement model indicate the causal effects of the latent factor (i.e., construct) on the observed variables (i.e., items), and the higher factor loadings indicate the higher convergent validity (Brown, 2006). The reliabilities or the internal consistency (i.e., Chronbach’s alpha) and explained variations were calculated (cf. Capraro, 2004).

Several indicators of fit were utilized to evaluate the goodness of fit of the model. Five fit indices were reported (e.g., Tucker-Lewis index (TLI), comparative fit index (CFI), root mean square error of approximate (RMSEA), standardized root mean square residual (SRMR), and chi-square and degree of freedom (x /df)). Chi-squared value is usually adopted for comparison of nested models other than for an indicator of goodness of single model fit, because it is very sensitive to sample size. Therefore, studies having a large sample size can hardly obtain nonsignificance of null hypothesis (Brown, 2006; Marsh, Hau, Artelt, Baumert, & Peschar, 2006). The estimates of the parameters that were used to analyze the goodness of the model, TLI, CFI, RMSEA, SRMR were 0.90, 0.90, 0.08 and 0.08, respectively (Brown, 2006; Chang, 2013; Marsh et al., 2006).

To answer the third research question, the data collected from the student attitude survey on STEM PBL was calculated by using descriptive statistics mean, standard deviation, and frequency.

The exploratory factor analyses of the measurement model were conducted to specify the number of factors and the pattern of indicator-factor loadings (Brown, 2006). This study tests the extent to which the theoretical model, in this case the five factors model corresponding to the five subscales, adequately represents the covariance matrix of the data. The results of the exploratory factor analyses confirmed the theoretical grounded construct of STEM PBL and revealed that five factors were extracted. Five factors and 25 items with over a 0.30 factor loading were identified by exploratory factor analyses (see Table 1). The exploratory factor analyses provided support for the five factors model as shown by the pattern of good or acceptable of fit indices.

First, the six items (Item 1, Item 2, Item 3, Item 4, Item 5, and Item 6) were loaded onto Factor 1, which was named “Self-Regulated Learning.” The items of Factor 1 expressed student internal motivation to promote student-centered learning. Moreover, its strongest loading items expressed student willingness to find sources and pursue the learning by oneself. Second, the five items (Item 7, Item 8, Item 9, Item 10, and Item 11) loaded onto Factor 2 related to students’ interest in group work and the opinion of the benefits from collaborative group work. This factor was labeled as “Collaborative Learning Environment.” Third, four items (Item 12, Item 13, Item 14, and Item 15) were loaded onto Factor 3 related to the benefit of science, technology, engineering and mathematics integrating learning. This factor was labeled as “Interdisciplinary Learning Environment.” Fourth, five items (Item 16, Item 17, Item 18, Item 19, and Item 20) were loaded onto Factor 4 related to student interest and belief on technology instruments and technology integrated learning. This factor was labeled as “Technology-Based Learning.” Finally, five items (Item 21, Item 22, Item 23, Item 24, and Item 25) were loaded onto Factor 5 related to student interest and belief on the benefits from hands-on activities. This factor was labeled as “Hands-on Activity.” Of the original items of the survey, 26 items were deleted because their factor loading values were less than 0.3.

The results of exploratory factor analysis revealed that five STEM PBL factors contributed to explaining 56.967% of the total factor loading. The explained variances of five factors, Self-Regulated Learning, Collaborative Learning Environment, Interdisciplinary Learning Environment, Technology-Based Learning, and Hands-on Activity were 50.42%, 57.78%, 48.74%, 57.54%, and 64.04%, respectively.

Confirmatory maximum likelihood factor analysis was used to test the goodness of fit of competing four- and five-factor models of the structure of STEM PBL. Four models were tested. Model 1 tested a correlated five-factor model including all 51 items in the item pool. Model 2 also tested a correlated five-factor model structure but used only the 25 items that loaded above 0.30 for an EFA. Model 3 tested an uncorrelated five-factor model including 25 items, and Model 4 tests a correlated four-factor model including 25 items. These four models were all first-order models and consisted of uncorrelated factors.

A correlated five-factor model was concluded as the best fit model to the collected data. The fit indices for the four models were shown in Table 2. The indices for Model 1 indicated poor fit. Fit for Model 2 was much better than Model 1, and slightly better than Model 3. Model 2 showed overall acceptable fit whereas the fit indices of Model 3 represented slightly out of acceptable model. The indices for Model 4 again indicated poor fit. The confirmatory factor analyses provided support for the Model 3 as shown by the pattern of acceptable fit indices. Both TLI and CFI were 0.902 and 0.914, respectively, above the threshold of 0.90. The RMSEA and SRMR values were 0.067 and 0.049, respectively. The chi-square in relation to its degrees of freedom was less than 5.

The results of confirmatory factor analyses showed the convergences through high or acceptable factor loadings ranging from 0.538 to 0.931. The information of all factor loadings and reliability of internal consistency (i.e., Cronbach’s alpha) was reported (see Table 1). The internal consistency of each STEM PBL attitude factor was estimated by Chronbach’s reliability alpha using SPSS 20. The values of reliability of each factor were acceptable for five factors, having Chronbach’s alphas of 0.766 (Self-Regulated Learning), 0.861 (Collaborative Learning Environment), 0.780 (Interdisciplinary Learning Environment), 0.805 (Technology Based Learning), and 0.827 (Hands-on Activity). The overall internal consistency reliability (Chronbach’s alpha) coefficient was 0.871.

Correlations Among Factors. The correlation coefficient between factors was calculated. The factor correlation over 0.80 or 0.85 implies poor construct divergence (Brown, 2006). The results of the present analysis showed that correlations among factors ranged from 0.223 to 0.442. The highest correlation was found between Self-Regulated Learning and Collaborative Learning Environment (r = 0.442), and the lowest correlation was between Technology-Based Learning and Collaborative Learning Environment (r = 0.223). The remaining correlation coefficients were reported in Table 3.

Korean students’ attitudes toward STEM PBL was surveyed using the measurement instrument. Descriptive statistics including mean, standard deviation, and frequency were reported (see Table 4). The overall mean was 3.542. The highest mean among five factors was 4.072 (Interdisciplinary Learning Environment) and lowest mean was 3.328 (Collaborative Learning Environment). This means that Korean middle grade students had a positive attitude toward STEM PBL. Among six items of Factor 1 (Self-Regulated Learning), Item 1 presented the highest mean value (= 3.668), and 311 students (39.61%) students agreed or strongly agreed with this item. Within Factor 1, the most Korean students agreed or strongly agreed to Item 6 (59.49%), and disagreed or strongly disagreed to Item 3 (26.36%). Of the five items of Factor 2 (Collaborative Learning Environment), 541 students (68.92%) agreed or strongly agreed to Item 11, which showed the highest mean value (= 3.82). Within Factor 2, the most Korean students agree or strongly agree with Item 9 and Item 10 (68.92%), and disagree or strongly disagree with Item 7 (13.38%). Of five items of Factor 3 (Interdisciplinary Learning Environment), 633 students (80.64%) agreed or strongly agreed to the Item 13, which showed the highest mean value (= 4.12). Within Factor 3, the most Korean students agree or strongly agree with Item 13 (80.64%), and disagree or strongly disagree with Item 14 (26.36%). Of the five items of Factor 4 (Technology-Based Learning), 529 students (68.92%) agreed or strongly agreed with Item 16, which showed the highest mean value (= 3.98). Within Factor 4, the most Korean students agree or strongly agree with Item 18 (75.54%), and disagree or strongly disagree with Item 17 (12.48%). Of the five items of Factor 5 (Hands-on Activity), 477 students (68.92%) agreed or strongly agreed with Item 23, which showed the highest mean value (= 3.65). Within Factor 5, the most Korean students agree or strongly agree with Item 23 (60.76%), and disagree or strongly disagree with Item 21 (13.89%).

This study is the first study to suggest an integrated measurement instrument of student attitude toward STEM PBL. While the surveys of each factor of STEM PBL have been developed in the previous studies, there has not been a study providing a synthesis survey tool for measuring student attitude toward STEM PBL (Holstermann et al., 2010; E. J. Kim, 2011; Lim et al., 2001; Little & Hoel, 2011; Miller, 1990; Najafi et al., 2012; Saleh, 2011). Moreover, in this study, the measurement instrument was examined in terms of reliability and validity. Many previous studies dropped the process to verify the surveys’ reliability and/or validity before describing the results (Holster-mann et al., 2010; E. J. Kim, 2011). In the sense that results of a survey without a substantial reliability and validity are not valuable, the survey of this study will contribute to the reliable and valid results concerning student attitude toward STEM PBL.

These results suggest that student attitude toward STEM PBL is comprised of several independent factors. In this study, the 5-factor model was substantially improved across a range of fit statistics. This study’s factor decomposition was fairly consistent with the theoretical framework based on the literature review (Barron et al., 1998; Capraro, 2013; Cifuentes & Ozel, 2008; Woods & Morgan, 2008). The theoretical frameworks of this study were confined to the definition of STEM PBL and the functional factors that comprise the instructional strategy. Given the theoretical frameworks of this instrument and the application, it is logical to suggest that the factors studied have high validity and reliability and thus are the factors that underlie STEM PBL components as measured by the student attitude survey.

To answer the question, “are the items of the survey reasonable indicators of the underlying construct of student attitude toward STEM PBL?” it is important to note that the theoretical frameworks for this study were developed from several PBL models and instruments. The findings provide all but two items scored “good” to “excellent” for internal consistency in measuring STEM PBL factors and the relatively high degree of explained variance, and as such it is logical to suggest that the items of the survey are reasonable indicators. The factors that underlie STEM PBL construct the student attitude toward STEM PBL and are consistent with the reasonability of that construct.

Interesting to note is that the technology and hands-on activity factors had the highest internal consistency and explained variance whereas the interdisciplinary learning environment had the lowest, respectively. This could be explained by the students’ prior work with hands-on activities and interaction with technology, where most students have had high levels of interaction with both these terms and these pedagogical applications (Hughes, 2013; Woods & Morgan, 2008). Interdisciplinary learning is a relatively new concept with relatively new and underdeveloped pedagogical applications. Student perceptions, misconceptions, and lack of previous interactions with the word interdisciplinary and the conceptual application as a component of STEM PBL learning may both be confounding perceptual issues in student understanding of the terminology and/or its application to their learning.

From the findings, Korean middle grade students have positive perceptions of hands on, technology, interdisciplinary, collaborative, and self-regulated learning factors of STEM PBL. It is logical to suggest student perception of STEM PBL were impacted by their experiences in classrooms, interacting within each of these factors as part of their learning (Barron et al., 1998; Thomas, 2000). Given the sample size, the number of schools and classrooms surveyed, it is important to question what types of STEM PBL activities took place that motivated students to respond in a positive way about STEM PBL factors. Constructivist learning theory states that students are motivated to learn based on their construction of knowledge with respect to their learning style. It is beyond the scope of this study to suggest what types of activities took place at these schools and in these classrooms, though it is clear that student construction had a positive impact on their perception of their learning in STEM given the STEM PBL lessons in which they participated. For a deeper understanding of the impact of STEM education on students’ academic achievement see We, Robot: Using Robotics to Promote Collaborative and Mathematics Learning in a Middle School Classroom (Ardito, Mosley, & Scollins) in this special issue.

Development of the measurement instrument of the current study implies varied possible further studies. A research investigating the relationship between students’ attitudes toward STEM PBL and their academic achievements is necessary. Furthermore, it is critical to examine how each factor of STEM PBL influences student achievement depending on their individual factors such as ethnicity, gender, economic status, and academic achievement level. Revealing student attitude toward STEM PBL using the measurement instrument provides a venue to connect student individual factors to the STEM PBL factors. Moreover, the measurement instrument in this study may be used for comparative studies investigating student attitude toward STEM PBL across countries.

There are a few limitations of this study. First it is important to note that these constructs for STEM PBL investigated were extracted from several studies and instruments. Given the limitations of instrument construction it must be noted that other factors may underlie STEM PBL that were beyond the scope of the present study. It is also important to note that the findings of this study cannot be generalized to student actual ability and/or overall learning style preference. Student learning style preference may be due to other factors and to this end it is important to note there are potentially several methods to measure learning style preference given the STEM PBL environment, though the findings of this study point to positive perceptions of the factors investigated.

Finally, this study was an investigation of student self-reporting of perceptions of STEM PBL given their learning in classrooms in a large Korean city. The present study was a self-evaluation on student readiness for STEM PBL. For example, the interdisciplinary nature of the test was designed to measure student perception and belief of interdisciplinary teaching and learning. However, Jacobs and Borland (1986) pointed out that student readiness is the critical factor deciding the success of interdisciplinary studies. The developed survey instrument measuring student attitude toward STEM PBL is critical to understanding student learning style and readiness concerning the factors of STEM PBL.

The scores on the instrument indicated satisfactory parameters for valid and reliable scores; therefore, it is suitable for use in other studies. By employing the instrument, educators might have more knowledge of students regarding their attitude toward STEM PBL concerning the five factors revealed in this study. As the information on student attitude toward STEM PBL represents student learning style and readiness pertaining to STEM PBL, it will permit educators to approach the best practice in implementing STEM PBL in terms of the differentiated learning aspect.