This study compared the use of two alternatives for the development of a Web-based lesson by pre-service teachers. Six research questions were posed regarding the process of developing a Web-based lesson to examine efficiency, effectiveness, and appeal of the developmental process using the dependent variables; a) lesson creation time, b) perceived effort, c) inclusion of six specified instructional components, d) functionality of six specified technical components, e) teacher appeal, and f) likelihood of intended future use of the development alternative. The analysis suggested differences and non-differences between the developmental alternatives.
Web-based instruction is a powerful resource that can be used to capitalize on dynamic multimedia interactive technologies to promote rich environments for active learning. According to Carlson, Downs, Repman, and Clark (1998), Web-based instruction is at the cutting-edge of both today’s technology and instructional design methodology. There is an urgent need for integrating technology into the classroom; however, even the best technology is of little value if the classroom teacher cannot use it efficiently and effectively. A report from the National Center for Educational Statistics (NCES) (1999) details that although educational technology is considered a means for transforming education, only 20% of teachers feel well prepared to integrate technology into classroom instruction. Teachers must have access to technological innovations that help them meet the needs of a diverse audience, able to use it efficiently and quickly, and provide them with the ability to integrate best practices with the interface design and lesson content.
Purpose of the Study
This study focused on pre-service teacher skill acquisition and application as applied to the process of developing a Web-based lesson. The purpose of this study was to examine six research questions regarding the development process of a Web-based lesson by pre-service teachers. Each pre-service teacher created a Web-based lesson using one of two development alternatives, the process of developing the lesson was then evaluated on three criteria: efficiency, effectiveness, and appeal. Comparison of these three criteria (composed of 16 variables/sub-variables) were made between lessons created with a Web Editor (WE) only, and those created using a Web Editor in conjunction with a customized template/shell structure called “Teaching-Not-Teching” (T- N-T). The three criteria for usability (efficiency, effectiveness, and appeal) were applied to determine if one lesson development alternative provides better support for pre-service teachers compared to another alternative.
Sample
During the 1999-2000 Fall Semester, eight sections of Educational Technology Applications courses; four sections of ET 347 Educational Technology Applications for Elementary Teaching, one section of ET 348 Educational Technology Applications for Middle Grades Teaching, and three sections of ET 349 Educational Technology Applications for Secondary Teaching were taught. Of the 161 students solicited, a total of 144 students agreed to participate in this study.
The groups were selected by nonrandom methods; however, the intact class sections were randomly assigned to either the Control group or the Treatment group. Two of the four sections of ET 347 (elementary pre-service teachers) were randomly assigned to the Control group and the remaining two sections were assigned to the Treatment group. Two groups of ET 349 (secondary pre-service teachers) were drawn and assigned to the Control and one remaining section of ET 349 and one section of ET 348 (middle grades pre-service teachers) were assigned to the Treatment group.
Forty-one of the 144 students (28.47%) who had agreed to participate in the study either did not complete their Web-based lesson or other required instruments and subsequently were eliminated from the study. Thus, this study’s sample consisted of 103 students, that is, 64% of the pre-service teachers who initially agreed to participate. Final group size for the Control group was 52 and for the Treatment group was 51. Each student completed the Demographics Questionnaire and the Consent form during the first class session.
Research Questions
The research questions were centralized around the usability/evaluation criteria of efficiency, effectiveness, and appeal as follows:
Efficiency
RQ 1: Are pre-service teachers able to develop a Web-based lesson in less time when they use WE + T-N-T than when they develop a Web-based lesson using WE only?
RQ 2: Will pre-service teachers perceive that less effort is required by using WE + T-N-T than when they develop Web-based lessons using WE only?
Effectiveness
RQ 3: Are the six specified instructional components (lesson goal(s), objectives written in performance-based terms, student performance, student performance evaluation, student-to- teacher contact, and location cues in the site) present when pre-service teachers develop a Web-based lesson using WE + T-N-T and when using WE only?
RQ 4: If present, which of the six specified technical components (navigation, image presence, mailto links, interactive mechanism, audio and video) are functional when pre-service teachers develop a Web-based lesson using WE + T-N-T and when using WE only?
Appeal
RQ 5: What degree of teacher appeal is elicited from pre-service teachers’ interaction with the development process of a Web-based lesson using WE + T- N-T and from pre-service teachers who use WE only?
RQ 6: Do pre-service teachers intend to continue Web-based lesson development with the lesson development alternative they used in the study in their future instructional settings?
Table 1 visually depicts the relationship between the evaluation criteria, research questions, dependent variables, and instruments.
Design
This study is quasi-experimental (Smith & Glass, 1987). That is, the independent variable is an introduced treatment, although there will not be total control over which participants receive which treatment. Participants were selected by nonrandom methods and then the intact sections were randomly assigned to treatment groups. This research paradigm may be represented as follows:
R = Random assignment of treatment to intact classes, Consent Form, Demographic Questionnaire
O1 = Instruction on how to use the Web editor (Microsoft FrontPage 2000)
X = Intervention treatment (T-N-T)
O2 = Work Time-Log, Effort Questionnaire, Instructional Components Evaluation Checklist, Technical Components Evaluation Checklist, Computer Attitude Scale (CAS), and Intended Future Use Questionnaire, Summary Data Sheet.
Analysis
T-tests were used to analyze the data to determine if there was a significant difference between the Control and Treatment groups as measured by the 16 identified sub-variables. This study used six independent t -tests; therefore the probability of one or more Type-I errors was greater than the alpha set for any single t-test. Due to the multiple t-test error rates occurring from six t -tests, an alpha level of .0167 was set for each test result in an overall experiment-wise alpha of not more than .10. Given the exploratory nature of the study, this error range was judged appropriate.
Evaluation Criteria, Research Questions, Dependent Variables, and Instruments
| Evaluation Criteria | Research Questions | Dependent Variables | Instruments |
|---|---|---|---|
| Efficient | RQ1 | Lesson Creation Time | Self-Report Work Log |
| RQ2 | Effort | Effort Questionnaire |
| RQ3 | Six Specified Instructional Components | Instructional Components Evaluation Checklist | |
| Effective | RQ4 | Six Specified Technical Components | Technical Components Evaluation Checklist |
| Appealing | RQ5 | Teacher Appeal | Computer Attitude Scale |
| RQ6 | Likelihood of Future Use | Intended Future Use Questionnaire |
Based on the results of the independent t- test, there was no statistically significant difference between length of time required to develop a Web-based lesson using WE + T-N- T compared to using WE only (t = -0.9457, p = 0.1733). Data were compiled and analyzed from the Work Time-Log and the results are presented in Table 2.
Participants in the study used the Work Time-Log to record the number of minutes they spent designing and developing their Web-based lesson. The six specific tasks itemized on the Work Time-Log were:
Tinkering with the computer program,
Storyboarding,
Collecting images, audio, video, and other cool stuff,
Thinking about my Web-based lesson,
Collecting or creating lesson content, and
Developing Web-based lesson with the editor.
Although there was not statistical significance between the total numbers of minutes participants from each group took to complete their lesson, participants in the Treatment group spent, on average, 43 more minutes than did the participants from the Control group. See Figure 1 for a graphical display of the average time spent by each group on each of the six specific tasks itemized in the Work Time-Log.
Research Question 2
No statistically significant difference between perceived required effort by pre-service teachers when using WE + T-N-T compared to when WE only (t = 1.8673, p = 0.0324). This was substantiated by the scores on the effort Questionnaire completed by the pre-service teachers regarding the development of their lessons. The results of this analysis are found in Table 3.
The three questions on the Effort Questionnaire are as follows:
Using the Likert scale provided, how much effort do you believe was required by the process you used to develop your Web-based lesson?
I feel that the process of developing a Web-based lesson is a struggle, and
It does not take a great deal of effort to develop a Web-based lesson.
Although statistical significance was not found, the Treatment group scored higher overall on all three questions than did the Control group. This trend in the Treatment group members’ perception of less effort being required by the WE + T-N-T developmental process is graphically depicted in Figure 2.
Research Question 3
A significant statistical difference between the frequency of presence of the six specified instructional components:
Work Time-Log
| Group | Mean Time(minutes) | SD (minutes) | df | p |
|---|---|---|---|---|
| Treatment (n = 51) | 519.1764 | 215.3836 | 101 | 0.1733 |
| Control (n = 52) | 476.5384 | 241.1787 |
lesson goal(s),
objectives written in performance-based terms,
student performance activity,
student performance evaluation,
student-to-teacher contact,
and location cues in the site,
were found in the Web-based lessons developed by pre-service teachers utilizing WE + T- N-T than when developed using WE only (t = 18.5048, p = 0.0000). Scores for the presence of instructional components ranged from a low of zero to a high of six points. The data indicated a statistical favor for the WE + T-N-T group. See Table 4 to review the results.
The scoring protocol used for these three variables was to first look for objectives written in performance-based terms. The performance objectives are the foundation for the other two variables and if there were no performance objectives contained in the lesson there could be no points awarded for either “student performance activity” or “student performance evaluation.” As stated a-priori in the definition of student performance activity, performance was based upon stated objectives and if objectives did not exist in the lesson no points were awarded for student performance activity. Furthermore, no points could be awarded for student performance evaluation, because as stated a-priori in the definition, “student performance evaluation is to be based upon student performance activity.” Review of the operational definitions of
objectives,
activity, and
evaluation
in conjunction with the scoring protocol make it clear that participants could not receive points for “activity” and “evaluation” if “objectives” were not present.
Perceived Effort
| Group | Mean Effort | SD | df | p |
|---|---|---|---|---|
| Treatment (n = 51) | 7.0588 | 1.9226 | 101 | 0.0324 |
| Control (n = 52) | 6.3750 | 1.7928 |
Objectives written in performance-based terms: refers to a statement that describes what learners should be able to do when they have completed the lesson. According to Smith and Ragan (1993) “What learners ‘do’ must be observable so that the learners know that they have learned and what they have learned” (p. 91).
Instructional Components
| Group | Mean Score | SD | df | p |
|---|---|---|---|---|
| Treatment | 5.3333 | 0.9933 | 101 | 0.0000* |
| (n = 51) | ||||
| Control | 2.1153 | 0.7586 | ||
| (n = 52) |
Note: * = α < 0.0167
Student performance activity: refers to eliciting specific behavior from the student based on the performance objectives for the lesson.
Student performance evaluation: refers to supplying the student with feedback on his or her performance based upon student performance activity.
In order to examine the extreme scores of the Control group regarding
objectives,
activities, and
evaluation,
the researchers analyzed data beyond the scope of the study. One of the questions to be addressed by this ad hoc examination was to identify if the lack of scores was solely the product of the scoring protocol. Lessons created by the Control group did not include performance-based objectives, although some lessons included an activity and/or evaluations. Participants in the Control group earned points for the evaluation component even if they only included a single question. Figure 4 gives a graphic display of what the score comparison would look like if the lack of performance objectives did not negate points earned for the presence of the activity and evaluation.
Research Question 4
Statistically significant difference was found between the frequency of functionality of the six specified technical components:
Navigation,
Image presence,
Mailto links,
Interactive mechanism,
Audio, and
Video in Web-based lessons.
Technical Components
| Group | Mean Score | SD | df | p |
|---|---|---|---|---|
| Treatment | 3.4901 | 0.6744 | 101 | 0.0000* |
| (n = 51) | ||||
| Control | 2.0769 | 0.7883 | ||
| (n = 52) |
Note: * = α < 0.0167
When pre-service teachers utilized WE + T-N- T, the frequency of functionality of the technical components was statistically higher than when developed using WE only (t = 9.7680, p = 0.0000). For each component that was present and functional, the pre-service teachers received one point. This enabled them to earn a possible total of six points. The statistical preference was in favor of the WE + T-N-T group as can be seen in Table 5.
The scores on the Technical Components Evaluation Checklist were based on functionality of the six specified technical components. Not one from either the Treatment or the Control group attempted to include component (e) Audio or (f) Video in their lesson. Of the four remaining technical components, the Treatment group scored higher on average in every component. Figure 5 is a pictorial display of the analysis results of the technical components comparison between the Control group and the Treatment group.
Research Question 5
The degree of teacher appeal toward developing Web-based lessons was examined for the pre-service teachers using WE + T-N-T and the pre-service teachers using WE only. No statistically significant difference was found (t = -0.6403, p = 0.2617). Data were compiled and analyzed from the Computer Attitude Scale (CAS). When the four areas associated with the CAS;
Computer Anxiety Scale
| Group | Mean Degree of Appeal | SD | df | p |
|---|---|---|---|---|
| Treatment(n = 51) | 118.2450 | 16.7980 | 101 | 0.2617 |
| control (n=52) | 120.4807 | 18.5723 |
low anxiety,
high confidence,
liking, and
usefulness
Research Question 6
Data were compiled and analyzed from the Intended Future Use Questionnaire. T-tests analysis were used to find if there was a difference between the likelihood of pre-service teachers’ anticipation for developing Webbased lessons with either alternative in their future instructional settings. No statistical significance was found (t = -0.3946, p = 0.3470).See the results displayed in Table 7. Figure 7 graphically displays the responses of participants for the following questions.
Anecdotal Post-hoc Telephone Interviews
The post-hoc telephone interviews were completed with students from both the Control and Treatment groups as recommended by Babbie (1973). Two students were selected from each group based on most and least time recorded on the Work Time-Log. Two students from each group were also selected based on the highest and lowest compiled scores of the remaining five instruments (Effort Questionnaire, Instructional Components Evaluation Checklist, Technical Components Evaluation Checklist, Computer Attitude Scale, Intended Future Use Questionnaire). In theory, eight interviews would be performed; however, one of the individuals identified from the Control group fit in two categories (least time spent as recorded on the Work Time-Log and high composite score on the remaining five instruments), and therefore only seven participants were interviewed.
Data obtained from the anecdotal interviews revealed that one participant from the Treatment group had continued using the T-N-T template/shell after the study ended. He had recently completed his seventh lesson in only 20 minutes. His completion time for the initial use of T-N-T was 450 minutes. Although this finding is based upon a single data point, the substantial, reduction in development time needs to be explored further to determine if multiple use of the T-N-T template results in similar outcomes.
Many of the interviewees’ comments led the authors to “ex post facto hypothesizing” (Glock, 1967). The significance of reinforcing the attitude of professionalism and behavior of pre-service teachers through the use of a Webbased template definitely needs to be explored further. This apparent phenomenon is described by a female senior who reported the highest work time (1200 minutes) using WE + T-N-T. She reported with great enthusiasm how the template use promoted her to focus on pedagogical issues as follows,
I loved what I ended up with. I plan to teach elementary school kids, so this will be a great thing to have. I got so hooking into trying to figure out how to be sure that my activities actually taught what I said the goals and objectives of the lesson were that I just found myself feeling so completely like a teacher. I spent a lot of my time figuring out where to put things. I mean like what goals and objectives did I want to write? I played with it a lot up front before I did anything. Then, I just started after I was familiar with it and it didn’t take me very long to complete it because I had already thought through the goals and objectives and what would go with them as far as activities and test questions. I know it seems like it took me a long time, but I think it is about a much bigger issue than time.
The graduate-level female using T-N-T who spent the least time (160 minutes) to develop her Web-based lesson described a great deal of perceived growth on her part when asked to relate her comparisons of time input and lesson output.
I gained a strong sense of confidence in having lesson goals and objectives that are consistently tied throughout the lesson and assessed in either a performance-based activity or a Web-based quiz. I am really feeling that I am pretty competent at this and it makes me much clearer about what I am teaching, how I am teaching it and why I am teaching it.
Another T-N-T user affirmed these same issues,
There is going to be an inherent increase of effort that you need to put into a lesson format like this because it makes you keep goals and objectives in the forefront of your thought as you develop. I don’t think you do that when you don’t have the guide there in front of you. When you first start, it takes more time and effort, however after working with it, it gets easier—it is more routine and you can focus on creativity and putting energy into thinking up other things to add.
Consideration of the following issues should be kept in mind when attempting to generalize from the findings and/or to replicate the study. The failure of the Control group to develop Web-based lessons that included the lesson components as defined within this study was certainly the greatest limitation. According to the syllabus of the methods course these teacher preparation students had previously completed, lessons consisted of goals, objectives, activities and performance evaluation. The Control group did not include these lesson components in the Web sites they created. Without such instructional components, the nature of the sites created is much more informational rather than instructional/educational.
The university setting, while being the greatest strength of this study may also be the greatest weakness. The university setting offers the research opportunity to assess students’ perspective as they create and layout their lessons using the template format while limiting the generalizability due to the sample selection.
A third limitation is that this study used a self-report. Self-reports are susceptible to “error” through the difficulty of eliciting honest, accurate responses. In this study, the participants are not under pressure to produce in order to be paid; nor are they reporting on an issue that will influence their grade, although they may be under a certain amount of time constraint pressure because of their own organizational capabilities. The assignment will be graded “Pass” or “Fail” based upon criteria in the grading rubric; student input from the selfreport is not part of the grading criteria. These factors still do not guarantee the truthfulness of the responses. However, they do alleviate the pressure and stress to answer with “teacher pleasing” responses.
A final limitation of this study was control for experimenter contamination. The intervention, T-N-T Web-based lesson template/shell, was designed and developed by the authors and unquestionably there is researcher bias. The instruction for using the Web editor and the instruction for using T-N-T will be presented by the researcher. Maintaining fidelity during lesson presentation is a limitation. To minimize this limitation, scripted lessons were used. Another aspect of experimenter contamination is that the authors conducted the data collection and data analysis, therefore subject to researcher bias. To minimize this limitation, the pre-service teachers completed four instruments, while the authors scored component presence and functionality using the remaining two instruments.
Description of the T-N-T Template/Shell
The lesson template/shell is a complete Web-based lesson minus the content. The template/shell exists as a Web-based resource/tool for the teacher. This template/shell is designed to support a pedagogically sound linear presentation of materials and activities while concomitantly providing a multiplicity of instructional scaffolding cues designed to guide development of the content. Figures 8 and 9 are screen captures of two pages from the template.
The navigation bar includes seven links (Goals, Objectives, Readings, Activity, Quiz, Glossary and Teacher). Note that “Goals” is white (active page and inactive link) and the others are dark (active links). White text is a visual cue that the user is working in the Goals section. The title “Lesson Goals” directly below the navigation bar is another indicator of location and page purpose. Scrolling down the Lesson Goals page reveals a few lines of direct instruction to the developer.
Each page of the template (excluding Glossary and Quiz) includes such direct instruction and a brief rationale (theoretical foundation) for the suggested approach to content development within each part of the Web-based lesson. The “Glossary” and “Teacher” links are set apart within the navigation bar. This segregation is intentional as glossary and teacher sections are provided as Tools for the teacher/designer and not as having the same instructional rationale/purpose as the other five components of the lesson. The one link located on the navigation bar that does not open a Web page is titled “Teacher.” This link is an E-mail link to the instructor.
The lesson template also includes a ready- to-use ten question multiple-choice quiz. The interactive “Quiz” is a JavaScript program that solicits the user to answer questions relevant to the lesson content and receive immediate feedback on his or her performance.
When providing content for the Quiz section of the lesson, the teacher/designer is prompted to insert questions pertaining to the lesson and potential answers (correct and incorrect). The feedback section of the Quiz is a program designed to give students appropriate information about their performance. Note that in the following screen capture, answers to the various questions have been selected and the user has clicked on the “Calculate Results” button.
The student performance feedback section of the Quiz page provides the student two sets of information. First, it furnishes a text-based assessment of their performance and secondly it allows the student to see which questions they answer correctly and incorrectly. The unique aspect of this feature is that it does not provide the correct answer to incorrect responses.
Conclusion
This study examined the efficiency (time and effort), effectiveness (inclusion of specific instructional components and functionality of specific technical components), and appeal (anxiety, confidence, liking, usefulness and intended future use) of two lesson developmental alternatives (WE only and WE + T-N- T) by pre-service teachers. Statistical analysis of six hypotheses confirmed that developing Web-based lessons using WE + T-N-T was more effective than using WE only. In terms of efficiency or appeal, neither developmental alternative was statistically superior. Due to initial findings during the telephone interviews, further research would certainly be warranted regarding time reduction with repeated template use, impact of template use on student learning, and increased professional esteem as a result of technological skill acquisition and pedagogical skill application.
