Learning Best-Practice in Design and Problem-Solving Skill Development: MTutor—A Web-Based Distance Learning Tool

MTutor is a World Wide Web-based virtual learning environment (VLE) that supports problem-solving skill development in a distance-learning one-to-one tutorial environment. MTutor provides a complete web-based tutorial system offering the benefits of automatic, at-a-distance delivery of courseware (Culverhouse and Burton, 1998). It has as its basis the traditional one-to-one tutorial model that has proved so successful in education over the years. MTutor enables tutors to put up and run their own tutorials using their own materials. MTutorials are pre-compiled using a 'builder' software program and released onto a world wide web server. Distance learners interact via cgi-scripts, which handle user security, user navigation and tracking, questionnaire marking, and discussion group support. Each MTutorial takes learners serially through the instructional, problem-solving, and reflection activities that the tutorial comprises. Tutors use a word processor to produce the tutorial content, which is converted into HTML using automation tools. A technical assistant controls the format of the content to ensure a house-style, but tutors have editorial control. Pedagogic advice is also given to the tutor on distance learning issues. In this way tutors are allowed to focus on the pedagogy and content of the tutorial, which has direct benefit to the learner.

In 1994, the MTutor system was trialed in both the laboratory and classroom. MTutor is now being assessed in departments across the University of Plymouth in the UK. Since 1998 a design MTutorial has been part of the final year curriculum in the Department of Communications & Electronic Engineering (DCEE) at the University. All students take this MTutorial at the start of the October semester as part of their project preparations. It was developed to encourage and support problem solving in the context of product design and the students' final-year projects. It is made available to students during the first few weeks at the start of the autumn semester. The aims of the tutorial are to

1. improve final year Bachelor of Engineering (B.Eng.) project student’s problem solving abilities and to develop their use of best practices in design,

2. to make students realise that their project may be more complex than they thought, and

3. to motivate them to complete their projects early.

The task and its results are summarised below.

There is a lack of studies on domain specific problem solving and reasoning; research into this area only emerging in the late 1960's with work by Eastman (1969) on space-planning tasks common in engineering and architecture. Since this time there has been little investigation into cognitive processes in engineers working in the domain of electronics engineering, although Ullman et al. (1988) and Jefferies et al. (1981) have used protocol analysis to study mechanical engineering and software engineering respectively. The authors are aware of only Colgan and Spence, (1991) and Ball (reported in Ball 1990, Baker et al., 1991 and Ball et al. 1993) within the field of electronics engineering. Colgan, however, is more concerned with improving feedback to engineering designers who are involved in very specific technological design activities in analogue circuit designs.

Ball studied both expert electronics designers (professionals working in electronics companies) and novices (second year and final year undergraduate students). He generated problem behaviour graphs from Think Aloud (TA) traces gained during protocol analysis studies on these designers engaged in, constrained but real, design tasks. It seems that experts and novices alike design in an iterative goal-directed manner. They appear to follow a personal design schema that gives them a basic strategy for design. Within this schema problems are decomposed into sub-problems (sub-goals), then solutions sought and decisions made; this is applied iteratively until the problem is solved. Experts generally follow a top-down breadth-first route through this prob- lem/solution space, whereas novices follow a top-down depth-first route.

These observations are based on engineers problem solving in knowledge rich domains (i.e. real-world problem solving). They suggest that because experts search in a breadth-first manner they can attend to the linkages between problems and their respective solutions at each level within this hierarchy. Experts therefore generate a more optimal solution to the overall problem; whereas novices seek detailed technological answers to their high-level questions i.e. always seeking direct technological solutions to problems. Ullman et al. and Jefferies et al. support this in other domains.

This makes sense, for experts will already have an internalised knowledge (their personal experience) from which to draw pre-existing lower-level problem/solution schemas. Internalised knowledge may be applied to a design problem in much the same way as an expert car driver employs procedural knowledge. Figure 1 shows a trivial example of a possible problem/solution space for the design of a pencil. From this it can be seen that an expert might run through the design space breadth-first, enabling the trade-offs and constraints between the body, lead, shape, and lifetime of the pencil to be assessed without having to perform detailed technical searches and analyses first. This does not preclude depth-first forays in search of specific information or knowledge, but the predominant breadth-first approach of the expert minimises the chance of making errors in a design at points where the different sub-solutions meet within any layer in the hierarchical solution space. For example an expert in the field of pencil design would probably know the strengths of commonly used pencil body materials, in contrast to a novice who may have to derive the specific strengths of specific materials from first principles or by an information search.

Referring to Figure 1 again, an expert will be able to define satisfactory solutions for the pencil body, lead, shape, and lifetime given his or her prior knowledge of the strengths of suitable materials, possible colours, and the likely shapes of existing pencils for example. A novice, lacking this prior knowledge, will have to carry out detailed investigations into each of these matters. The attention to each of these sub-goals in isolation makes it more difficult for a novice designer to cope with the inter-dependencies between different parts of the hierarchy. So the chances of an expert designing a pencil with a very strong body but one that cannot be gripped in the hand are minimal, but the chances would be higher for a novice. This is domain dependent, so a novice may well be an engineer familiar with (and expert in) a different technology.

In summary, experts exhibit top-down breadth-first problem decomposition and synthesis strategies, novices prefer a top-down depth-first strategy. Experts solve problems in an ‘holistic’ manner, novices lacking schema, algorithms, and heuristics, look for an early reward by seeking detailed knowledge. This is supported by observations of students. For example: "Students often just pick up a pencil and paper and start solving the problem without serious consideration of the requirements and constraints" (Parkinson et al., 1997) and a personal observation "They just jump in and develop the first solution that comes into their heads."

How are novices turned into experts? It is clear that given sufficient time the process of turning a novice into an expert can succeed without specific training interventions. Rubinstein and Firstenberg (in Stice, 1987) note that progress toward being an expert is accompanied by a tolerance of uncertainty, a willingness to deal with complexity and confusion, an ability to accept dissonant information, and the courage to take reasonable risks. Greenfield (also in Stice, 1987) states that no matter how problem solving skills are taught, no matter what educational level at which they are taught, the problem-solving skills of the students, to whom those skills are taught, improve. So practice makes perfect. This suggests that increasing design or problem-solving opportunities will improve the performance of the novice. This is supported by Stoner (1992), who also gives a concise view of the steps involved with supporting problem-solving skill-development that is representative of other literatures. This can be summarised:

1. identify the problem,

2. bring to mind relevant concepts and principles,

3. require analysis of the task,

4. give prompted practice, and, finally

5. develop independent activity.

An MTutor tutorial assists in this process by defining a strategy for problem-solving, setting a problem and providing an information resource assisting with difficulties by providing hints to guide the learner, and discussing solutions and strategy, so defining expectations and allowing independent activity at any level. MTutor thus provides an opportunity to practice problem solving in a case-based manner. The key to the learning experience is the discussion of solutions that the tutor has developed, that crucially define the scope of each solution.

Since 1998, cohorts of final-year B.Eng. programme students have been required to undertake an MTutor tutorial (MTutorial) to design a Sound Level Meter. The problem was posed in a style similar to that found in industry, being part well defined but also open-ended. Students were introduced to the tutorial requirements in a lecture, given at the start of the semester, which provided details on how to access and use the MTutorial. They were given two weeks to complete the tutorial. A 30-computer open-access room was made available to them during term time, although they could complete the tutorial at home or from their place of work (for the part-time students). From previous laboratory and field trials, the problem was expected to take at least three hours to complete to detailed block diagram level. Immediately at the end of the tutorial period a face-to-face session was held with the students to solicit feedback on the value of MTutor in relation to their projects. A follow-up questionnaire was given to the students in the 1999 and 2000 cohorts one month after completion of the tutorial. This was to probe long term recall and application of their knowledge gained during the tutorial.

The Sound Level Meter problem was chosen to pose a taxing task for students from a variety of backgrounds. Plausible solutions could be generated using both digital and analogue electronic elements. The signal-processing path could be viewed as a classic receiver block to communications engineers and a classic transducer interface to systems and electronics engineers. Extensive on-line resources were provided to assist the problem-solving phase of the tutorial.¹

The tutorial has been assessed in a quasi-formative mode in 1998 and also in a standard summative mode in 1999 and 2000. In 1998 the tutorial students were told that marks would be deducted from their final project mark, up to a maximum of 5% if they did not attempt the tutorial. Students were also told that the marks awarded to their answers were formative and would be ignored by the assessing tutor. This method was selected to signal to the students that the work was an important formative task that they all must carry out. In the event, no marks were deducted from any student who started the tutorial. In 1999 and 2000 the assessment was changed to a more standard summative method, where the on-line questions contributed to a maximum of 5% of their project total, as part of the project management mark.

In 1998, the MTutorial had one problem embedded in it, which gave the customer requirements for the Sound Level Meter together with details on the Plan, Do, Review structure for design and information on problem-solving best practices. The MTutorial was revised in 1999 when the single problem was expanded to three, one for each of the process steps: Plan, Do, and Review. Thus, the implicit demands made on the learner to plan, do and review during their design activities in 1998 were made explicit in 1999.

During the introductory lecture learners were given an instruction sheet telling them how to gain access to the tutorial and what assessment they would be asked to do. This was supplemented by reprints of the on-line instruction pages, which gave essential MTutor system information, detailed the structure of the MTutorial, listed the on-line resources available and gave a short review of best practices in design and problem-solving.

The MTutor system tracks every learner interaction as a history trace. The resources and problems were labeled 1 to 5 corresponding to the expected levels of progress through problem solving; from the Problem Specification at level 1 through to the cost of equipment at level 5. This labeling reflects the preferred breadth-first search and synthesis approach that is thought to characterise expert behaviour during the early stages of the design process.

This tutorial had six components of data collection:

1. on-line history tracing,

2. on-line answers to multiple-choice questions,

3. on-line evaluation questionnaire responses,

4. a paper based feedback questionnaire given immediately at the end of the MTutorial,

5. interviews with a small random selection of students, and

6. a paper-based follow-up questionnaire given approximately one month later.

The follow-up questionnaire was introduced to the 1999 cohort for the first time. The year 2000 on-line evaluation questionnaire had been modified, from that given previously, to increase the amount of evaluation of the tutorial content.

The interviews were conducted by Burton, who had no prior involvement with the students and thus was seen as a non-threatening third party. The interviews were structured by a walk-through of the web-based tutorial in the presence of each interviewee. Each screen was presented and the learner prompted for questions or problems encountered during their live session. Comments and discussions arising from this were recorded and written as a diary. Each diary was later analysed for information concerning

1. the user interface,

2. the tutorial content,

3. the tutorial delivery system (the Web),

4. other problems.

Feedback from these analyses has since driven improvements to the MTutor system and the specific tutorial.

The only scheduled tutor contact with the learners was to introduce the MTutor system to the cohort and to hold the feedback sessions.

The results are presented in three parts: the evidence for user response to the MTutor system, the user response to the content of the tutorial and finally student performance during and after the tutorial.

Eighty-one students completed the 1998 tutorial in the allotted time (N = 81, 79 Males, 2 Females). Students took from 3 hours to over 12 hours to complete the tutorial. Some carried out the task in one sitting; others took two or more sessions over the two-week period. Many of the part-time students logged into MTutor from their place of work to do the tutorial exercise.

Overall, the students found the MTutor system easy to use (see Table 1 below). The improving student perception of the MTutor system over the three years is probably due to changes made to the system in the light of interviews with a small random selection of students and detailed review of the evaluation questionnaire responses.The users reported an increase in perceived system speed (72% in 1998 to 92% in 2000). This is due to general improvements in local area network and Web server quality during this period.

A comment field was also provided in the evaluation questionnaire. Several comments from the questionnaire are summarised below. These responses were made immediately at the end of the tutorial itself and record an instant reaction to the exercise. They ranged from being very positive to being very negative; the negative replies were mostly from students who were already experienced design engineers. These comments, drawn from all three years of the study, reflect the breadth of opinion expressed by the students whilst engaged in the tutorial.

This student found direct application of the content of the MTutorial.

This student recognized that the tutorial content can be re-applied in other areas of their studies.

This student did not need the tutorial, since they already had the requisite knowledge.

This student focused on access to information, rather than the innate value of the exercise.

The tutorial problem was designed to be complex and thus to mimic the decision making required during their project. This student has identified this similarity, but in terms to industrial practice.

Analysis of the on-line evaluation responses for the tutorial revealed that, as a cohort, students were generally pleased with the MTutor system. They liked the on-line theoretical and practical resources. They also found the hints moderately useful.

Learner responses to the problem-solving content and the best practices content have been analysed. The paper-based feedback questionnaire assessed the students' progress in planning their projects, as well as their view of MTutor in the week following the tutorial; Table 2 shows a summary of their responses (see below).

From the 1998 cohort, it is clear that 51% of the group valued the tutorial, but only 28% felt it contributed in some way to their planning of their project. In 1999, an increase in satisfaction with the tutorial content was noted with the changed problem format described in the methods section. This increase has been observed in the responses from the year 2000 cohort too. Learners in 1999 reported one month after completing the tutorial that 40% of them had applied the MTutorial-learnt skills to their project. This figure may increase through the tutorial period and points to the need for a further questionnaire at the end of the project.

History traces of two student interactions are shown below in Figures 2a and 2b. The first shows expert behaviour, being breadth-first in his information seeking (Culverhouse et al., 1992). This student (Figure 2a) also concentrates on theory information in the first third of the tutorial, a mixture of theory and practice in the middle third and a predominance of practice information in the last third. This student takes 3 hours to complete the tutorial. The second student (Figure 2b) is more characteristic of a novice, with a tendency to depth-first design (considering the detail of each functional block rather than focussing on the whole and its connectivity first). This student spends too much time reviewing practical data and not enough reviewing the theoretical issues. This student did the tutorial over a two-day period, taking about 12 hours in total.

Neither student attempted to check their solution with the original problem specification before finishing the tutorial. This shortcoming highlights one difference between expert and novice electronics engineers described in an earlier study by Culverhouse, Ball and Burton (1992). In this study experts reflected best practice by evaluating their designs against the original specification at the end of the tutorial.

Analysis of the feedback questionnaire (held one week after the tutorial) reveals the following. The student responses were, in general, favourable to their problem-solving experience. However it was clear that the responses could be split into three groups. Group one students felt that the exercise was trivial. Group two students were generally pleased with MTutor and valued the tutorial; this group found the task taxing. Group three comprised students who found the problem impossible to solve or did not do the tutorial at all.

Only four students failed to do the tutorial in 1998; in contrast all students in both 1999 and 2000 cohorts completed the tutorial. A scatter plot (shown in Figure 3) indicates a relationship between their MTutor score to the multi-choice questionnaire and their final project mark for the 1999 (Pearson's p = 0.386, N = 65 p < 0.01). This tutorial took students between 3 and 12 hours to complete. The trend line suggests a correlation between final project mark awarded to students and their MTutorial mark. This relationship will be explored in more detail with future cohorts.

The follow-up questionnaire, given to the students approximately one month after students completed the MTutorial, probed their ‘deep learning’ of the MTutorial topic. The results of this are summarised in Table 3 below for both 1999 and 2000 cohorts. It is interesting to note the correspondence between cohorts in the distributions of responses. The table has two parts, shown as unshaded and shaded rows. The unshaded rows reflect learner accuracy in using an engineering design term presented and discussed in the tutorial, the shaded rows report learner accuracy in recalling and writing the definition of a previously learnt engineering design term. The percent correct columns of both sections shows the percent correct student responses when asked to apply the best practices knowledge they acquired in the MTutorial one month previously. The cohorts are not exposed to any of the definitions used in the questionnaire prior to the MTutorial.

The responses indicate that the students are able to recall the definitions and apply them to new situations. The first (unshaded) section (category A questions) requires the students to apply definitions used one month previously, which they have not previously been exposed to. Students in the 2000 cohort responded at 70% and 90% (chance 33%) correct rates respectively. When asked to recall the definitions (category B questions), they responded at 66% and 61% (chance 50%) correct rates. It seems that the application of a definition is easier than to recall the definition and write it down. These results suggest a degree of “deep learning” being acquired by doing the MTutorial. To explore this more fully the year 2001 cohort will be asked to complete a pre-tutorial questionnaire to assess their background knowledge of Best Practices in design. These performances compare favourably to classroom lecture information retention, where a study by Bassy (in Bligh, 1971 p.39), exploring the value of rehearsal in information retention, showed an initial recall of 50% immediately following a lecture drops to 10% after one month if no rehearsal is prac- tised.Tutor contact with students outside of scheduled meetings was minimal. Each year some students could not gain access to the tutorial as they had lost their usernames or passwords to the MTutor system; these were emailed by return. No other contact was solicited by students or by the tutor.

The cohorts comprised B.Eng. students reading Electronics Engineering, Electrical and Electronics Engineering, Communications Engineering, Personal Computer Networks, and Robotics, and Automated Systems and Computer Engineering. Given the wide backgrounds, capabilities and interests of the students a variety of responses to the specific tutorial problem were expected.

Anecdotal evidence of previous final year project students suggests three categories of student: those who are keen and able to progress their project; those who are keen, but less able; and those who think that their project will be easy once they get into it but delay starting. It is hypothesised that Category 1 students commence organising their project at the start of the autumn semester, indeed some will have spent much time over the summer vacation thinking and planning what to do. They are already aware of some of the hurdles they will have to overcome and plan their work accordingly. Category 2 students do not carry out any planning prior to the semester start. They therefore spend much of the first half of the semester planning and identifying milestones and problems to overcome. This category of student will not really start their project until mid-semester. The third category of student will dally and not attempt to plan or identify issues. After much cajoling and guidance by their project tutor they will start their project in earnest late in the autumn semester. This group of students is most at risk of failure in their project assessment.

It is an aim of the DCEE to improve the performance of its project students. Progress could be achieved by focussing on moving all category 2 students into category 1 and moving category 3 students into category 2. The authors believe that exposing students to an MTutor problem-solving tutorial at the start of their project module can assist this process in several ways. Firstly by making students aware of the issues they must focus on. Secondly by giving them practice at solving difficult real-world problems. It is always nearly impossible to design an entity correctly first time if one is inexperienced, but the experience of design is an invaluable learning exercise. And finally by alerting tutors to those students who are experiencing difficulty with the tutorial and who may be vulnerable to falling behind in their own project.

In the future, it may be possible to design remedial action to assist the errant students. Category 3 students present a special challenge, as they are probably particularly weak in their mastery of design engineering. MTutor highlights these students, so they can benefit from additional attention by their project tutors. At the moment, remedial action is taken manually by tutors.

The literature on problem solving suggests that improving this skill is best done through practice at problem solving and being exposed to best practice techniques by the tutor (for example see Stoner, 1992). MTutor appears to provide this practice in a supportive, but in a non-contact, manner.

In conclusion, the overall aims of the tutorial exercise have been achieved for the majority of students. It is also encouraging to note there is strong evidence from the data in Table 3 data that by tackling and solving a difficult problem students are better able to focus and attend to their own difficulties early in their project. It has been demonstrated that

1. learners can recall and apply best practices in design,

2. they have a better appreciation of the complexity of their own projects, and

3. they have been motivated by their MTutor experience to alter the way in which they work on their projects.

These aims were achieved with no direct staff contact during the live tutorial.