A Factor Analysis of the Distance Education Surveys “is Online Learning Right for Me?” and “What Technical Skills Do I Need?”

There is a general consensus that the number of students receiving an education through distance education has continued to grow steadily (Instructional Technology Council, 2007; National Center for Education Statistics, 2004; Saba, 2005). However, some students have found the distance education environment less attractive than the on-campus environment (Lorenzetti, 2005a). This may have contributed to the lower retention rates reported for distance education students (Carr, 2000; Nash, 2005; O'Brien & Reneer, 2002; Scalese, 2001). The lower retention rates for distance education students has led several researchers to discuss the need for preassessments to identify students who might have difficulties in a distance learning environment (Biner, Bink, Huffman, & Dean, 1995; Buchanan, 1999; Lorenzetti, 2005b; Maki & Maki, 2003; Restauri, 2004; Valasek, 2001). However, commonly used readiness questionnaires given to students prior to the start of a course may not be useful in selecting students who will do well in and be satisfied with technology-mediated courses (Maki & Maki).

A study was conducted by Hall (2008) to determine the predictive validity of two survey instruments, “Is Online Learning Right for Me?” and “What Technical Skills Do I Need?” The survey “Is Online Learning Right for Me?” was developed by the Northern Virginia Community College Extended Learning Institute. The survey “What Technical Skills Do I Need?” was developed by Palm Beach Community College. Both surveys are 15-question, forced response, self-scoring guides that have face validity for individual traits and skills believed to contribute to potential success in an Internet-based course (Buchanan, 1999; Noah, 2001). Each response is assigned a numerical value. A total score is calculated by summing the individual question scores. The survey is designed such that the higher the total score, the greater the presumption for success in a distance education course. Copies of the survey items responses are provided in Tables 1 and 2. An abbreviated list of colleges and universities using one or both of these surveys include Ivy Tech Community College, Colorado Community College, Palm Beach Community College, University of Minnesota, Wisconsin Technical College, and the University of Maine.

In the predictive validity study by Hall (2008) copies of the surveys were distributed to 340 students enrolled in traditional on-campus courses enrolled on three different regional campuses of a Midwestern community college. Copies of the surveys were placed on web servers for 289 students enrolled in web-based distance education classes at the same community college campuses. Multiple regression models were constructed using the two scores from the survey instruments and categorical variables representing the type of course as the independent variables. The final semester grade percentage was used as the dependent variable. A total of 228 traditional on-campus students and 83 distance learning students completed the two surveys and received a final grade percentage in their courses.

The multiple regression model developed from the on-campus student data predicted 7% of the observed variance in final grade percentages. All of the observed variance was explained by the type of course (business, computer information services, criminal justice, or early childhood development) taken by the student.

The multiple regression model constructed from data obtained from the distance education students predicted 20% of the observed variance in the final semester grade percentage. However, Hall (2008) observed that scores from the two survey instruments accounted for only 8% of the total variance in the final semester grade percentage. Hall concluded that the two web-based surveys had little predictive validity.

The data used for the factor analysis were the survey answers obtained in the Hall, 2008 study of community college on-campus and distance education students. The survey items from both surveys were submitted to a principal component-components factor analysis with varimax rotation and an eigenvalue greater than 1. Individual survey items were associated with factors based on the highest factor loading.

Applied to the survey “Is Online Learning Right for Me?,” this analysis yielded four factors with eigenvalues ranging from 1.27 to 2.89. The four factors accounted for 18.1%, 11.0%, 9.1%, and 7.9% of the total variance, respectively. The resultant factors were labeled technology comfort, interaction, self-reliance, and general background. For purposes of this analysis, item Q1 was reversecoded. For the survey “What Technical Skills Do I Need?,” the analysis produced two factors with eigenvalues of 1.54 and 4.93. The two factors accounted for 28.7% and 8.4% of the total variance, respectively. The resultant factors were labeled skills and access. Tables 1 and 2 include the factors and loading values for the two survey items. Table 3 lists the factors, associated survey items, and the Cronbach alphas for each factor.

As shown in Table 3, the survey “Is Online Learning Right for Me?” appears to lack internal consistency. Only one factor, Technology Comfort, had a Cronbach alpha within the generally accepted range of 0.7 to 0.9 (McMillan & Schumacher, 2001). The low Cronbach alphas of the three remaining factors may be caused by several factors, including the low number of items associated with factor (Garson, 2007; McMillan & Schumacher, 2001) and low average correlation among items (Garson). Many items did not load strongly into any of the factors (Table 1).

An examination of Table 3 would appear to indicate that the survey “What Technical Skills Do I Need?” has high internal consistency. Both factors, Skills and Access, have Cronbach alphas of .84. However, the high Cronbach alpha associated with the skills factor may be more related to the relatively large number of items associated with this factor (13) than the individual loadings of these items to this single factor. The Cronbach alpha of .84 for the access factor is derived from just two items (Q13 & Q1) with relatively strong loadings (Table 2). A large number of moderately loaded items, or a few items with strong loadings, can generate relatively high Cronbach alphas, leading to a false conclusion of satisfactory internal reliability. The skills factor is an example of the former, the access factor an example of the latter.

The results from this study support the earlier conclusion by Hall (2008) that the two surveys “Is Online Learning Right for Me?” and “What Technical Skills Do I Need?” have low predictive validity. Although many of the items listed on these surveys have face validity with regard to traits and skills needed for success in distance education, the lack of internal reliability and predictive validity should be a consideration for institutions considering the use of these surveys for counseling or dispensing advice. However, the rising use of the Internet for instructional delivery, coupled with the desire to improve student retention, continues to generate a need for a viable prediction instrument for advising students considering distance education courses.

A potential assessment for further study is the Readiness for Online Learning questionnaire previously studied by Smith, Murphy, and Mahoney (2003) and Smith (2005). Smith suggests that further work to improve the two items that did not factor load distinctively may further enhance the value of this instrument. An attraction of the Readiness for Online Learning questionnaire is its brevity (13 items). Further items could be added to enhance its suitability to assess readiness and potentially predict performance.