Influential factors for technology-enhanced learning: professionals’ views Open Access

Different synonyms are used in research to talk about digital technologies that support human learning (e.g. computer-assisted instruction, educational technology, educational computing, e-learning, distributed learning and more) (Chan et al., 2006). One of these terms is technology-enhanced learning (TEL), which Goodyear and Retalis (2010) described as an attractive, broadly defined term because it includes all technologies that help make learning more effective, efficient and enjoyable. If this definition is accepted, the list of relevant technologies becomes quite long and includes technologies that have been developed for and are intentionally deployed in learning situations (e.g. mobile learning approaches, game-based learning approaches, interactive learning videos and more). Furthermore, even software that is needed to present information via WWW, e-mail and mobile phones can facilitate learning and are, therefore, part of TEL (Dror, 2008).

Goodyear and Retalis (2010) provide a clear overview that describes situations in which technologies can be used as part of TEL. Technologies can be:

• media for accessing and studying learning material;

• media for learning through inquiry;

• media for learning through communication and collaboration;

• media for learning through construction;

• used for learners’ assessments; and

• used to improve digital and multimedia literacy.

Currently, TEL approaches are widespread in higher education and the number of TEL approaches implemented is still increasing in German-speaking countries, in part because of the so-called digital natives. Many researchers claim that students born between 1980 and 1994, the digital natives (Prensky, 2001), demand technological support in their learning scenarios because they are used to using it in their daily lives (Pedró, 2006; Redecker, 2009; Noguera Fructuoso, 2015; Schweighofer et al., 2015). However, others have criticized this statement, pointing out the lack of convincing evidence that supports this claim (Bekebrede et al., 2011; Bullen and Morgan, 2011; Margaryan et al., 2011). Regardless of the reason for the spread of TEL approaches, it is indisputable that these approaches have become widespread in higher education. For instance, the results of a survey conducted in Austria showed that all 49 higher educational institutions that responded use some kind of TEL approach (Bratengeyer et al., 2016). Therefore, the implementation of TEL approaches needs to follow an instructional design (ID) (Bullen and Morgan, 2011) since using technology to help in learning processes is a complex process (Goodyear and Retalis, 2010).

ID or instructional system design (ISD) is a procedure used to develop learning experiences and environments, applying learning strategies to make the acquisition of knowledge and skill more efficient, effective and appealing (Merrill et al., 1996). The origin for ID is in behaviorism and system thinking (Rogers, 2002; Branch and Merrill, 2012). It is related to TEL in that professionals combine ID and instructional media, which includes various technologies (e.g. computer, internet, mobile devices and social media), to accomplish their goals (Reiser, 2001). By now, a wide variety of instructional design models are available (Branch and Merrill, 2012) that can provide assistance during this design process (e.g. a model created by Dick et al. (2015)).

According to Branch and Merrill (2012), practically all of these models are related to the ADDIE model, although the origin of this model is unknown (Molenda, 2003). ADDIE stands for analyze, design, develop, implement and evaluate, which are the process steps of a systematic product development concept. Therefore, ADDIE itself is not a specific, fully elaborated model but rather a structure used by taking different approaches (Branch and Merrill, 2012). These approaches define which influential factors must be considered during the process steps (cf. Ghirardini, 2011; Lohr, 1998).

However, it is not enough to merely consider relevant, influential factors that are related to ID; it is necessary to consider critical success factors (CSF) in order to be successful with a TEL approach. In general, CSF are characteristics, conditions or variables that can decide whether a company has success or not (Leidecker and Bruno, 1984). In addition, according to Pinto and Slevin (1987) considering such factors can also improve the chances of success of different project implementation whereas Cheawjindakarn et al. (2012) furthermore pointed out that identifying CSF is especially important in educational institutes. They defined CSF for online distance learning as factors that must be taken care of in order to be successful, more efficient and more effective. This study applies the same definition of CSF for TEL approaches.

In order to show the variety of CSF for TEL approaches and provide a selection of such CSF mentioned in literature, a search in Google Scholar using the search term “critical success factors” in combination with the terms “technology-enhanced learning,” “e-learning” and “online learning” was performed and a selection of studies identifying preferably different CSF was chosen. Table I contains this brief selection of studies and their identified CSF. The list is thereby not intended to be exhaustive.

In five of these studies (Frydenberg, 2002; Joo et al., 2011; McGill et al., 2014; Selim, 2007; Sun et al., 2008), CSF were identified for e-learning and online courses in general. Frydenberg (2002) presented a literature review and McGill et al. (2014) described the results of a survey of authors who had written articles about e-learning initiatives. The remaining three articles reported the results of surveys in which students identified CSF. The results of these five studies were quite diverse because there was no shared CSF occurring in all five studies and a greater consensus only existed among CSF related to acceptance. These CSF were perceived usefulness and ease of use (Joo et al., 2011), students like the innovation, innovation is easy for students to use, innovation is easy for teachers to use (McGill et al., 2014), perceived usefulness and perceived ease of use (Sun et al., 2008). Regional differences could be a possible reason that such diverse results were obtained. Richter and Pawlowski (2015) described differences related to e-learning between Germany and South Korea. For instance, they pointed out the differences between the teacher’s role, the value of errors and the preferred learning style. Furthermore, Paulsen (2003) also discussed regional differences in Europe with regard to the use of learning management systems.

In the remaining studies, CSF were identified that address more specific TEL approaches or specific aspects of TEL approaches such as blended learning, m-learning or technical infrastructure aspects in particular.

As mentioned before, this list of CSF is just a brief selection in order to show the variety of CSF for TEL approaches. However, prior to this study Schweighofer and Ebner (2015) conducted a vast literature review, analyzing 4,567 publications to identify factors that were potentially influential when implementing TEL approaches and clustered these factors into 20 categories with 76 subcategories. Table II shows how the discussed CSF fit into these categories. Additionally the table includes the subcategories of each category to further define the category as well as the number of articles in which influential factors of this category were identified.

The categorization revealed that, although the results stated in the publications were diverse, some similarities could also be identified. For example, most of the CSF identified were course-related, but other categories show also a high count of CSF. Furthermore, only two categories (demographic differences and requirements on teachers) did not include CSF with reference to the provided literature. Therefore, it seems like although different CSF are relevant for different TEL approaches, some CSF seem to be relevant for TEL approaches in general.

Finally, none of the selected studies investigated the teachers’ points of view, with the exception of two (Cochrane, 2010; Soong et al., 2001). However, in higher educational professional practice, the teaching staff or their management decide which TEL approaches will be implemented and which influential factors will be considered while doing so. Furthermore, the amount of time needed to consider these factors during the implementation is also important. If the teaching staff or their management do not believe that it is important to consider an influential factor or if they do not want to spend time considering such a factor, the fact that the students consider it a CSF will be irrelevant since the decision makers will not investigate or consider this factor.

In summary, the insights led to three assumptions and conclusions regarding the research question:

1. Regional differences determine the CSF for TEL approaches. Therefore, the research question should be regionally restricted.

2. Some CSF are relevant for TEL approaches in general. Therefore, in order to identify these CSF, the research question should not be restricted to specific TEL approaches.

3. Professionals in higher education determine which influential factors they will consider when implementing TEL approaches. Therefore, the research question should address the opinion of these professionals.

The purpose of this study is bipartite. First, the aim of this study is to broadly show which influential factors are important for professionals in Austrian and German institutions of higher education and, additionally, how much time these professionals would spend considering these factors. This information would support the implementation of TEL approaches, since it would allow to develop methods, which include important influential factors that professionals would actually consider during the development process. Additionally, the study should reveal influential factors that are important to professionals although they do not want to spend much time considering them. Methods that are easy to use and do not need much time must be developed for such factors.

Second, the study should identify influential factors that professionals do not consider as important. This information can help researchers identify fields in which more explanatory work is needed to emphasize the importance of these influential factors.

Finally, based on these conclusions and purposes, this study was conducted in an attempt to answer the following research question:

The remainder of this paper consists of a description of the research method used, followed by the survey results, a discussion of these results and, finally, a conclusion.

The overall research project applied a cross-sectional design, which is the most commonly-used survey design. This means that the units of analysis were studied either at one point in time or within a short time frame (Straits and Singleton, 2011). Furthermore, at this point it should be noted that the research project had two different target groups. For both the same survey was conducted. Although this paper focuses solely on the second group (institutions of higher education), in order to be accurate the following sections describe the research design for the overall research project.

The study population included: employees of Styrian enterprises operating in either the manufacturing or services industry (according to the classification of the Austrian statistical office, Statistik Austria, 2015) or schools (from secondary level upwards) and university extensions as well as institutions of higher education located in the German-speaking DACH region (Germany, Austria and Switzerland). Due to the broad nature of the study population, an online survey seemed to be an appropriate and resource-saving method that could be used to collect empirical data. An electronic mailing list and a suitable questionnaire were prepared. These were prepared with reference to a marketing database to obtain the Austrian data. This database includes information obtained from the Austrian commercial register and the association for the protection of creditors. To obtain data from Germany and Switzerland, publicly-available lists of German and Swiss institutions of higher education were examined and contact e-mail addresses were manually identified. These two data sets were supplemented by information from three of the authors’ personal author contact lists, which led to the creation of an electronic mailing list with approximately 19,000 entries. Thereby, approximately 1,300 entries regarded the group of institutions of higher education located in the German-speaking DACH region.

The elaboration of the questionnaire was an iterative process. A first draft of the questionnaire was prepared based on the 20 categories of influential factors identified by Schweighofer and Ebner (2015) (see Table II) and the question of interest. After conducting multiple cycles of linguistic refinement, an online form of the questionnaire (using EFS Survey software) was created. Pilot tests further improved the questionnaire, which resulted in the creation of an adapted and shortened version based on the testers’ comments. In summary, the final instrument can be described as follows: the questionnaire contains of three blocks using the 20 categories of influential factors identified by Schweighofer and Ebner (2015) as survey items, namely categories considered to date along with a satisfaction rating (or reasons for omission if not considered), a general assessment of each category’s importance and amount of time the respondents would spend to consider factors of these categories and the top five categories as ranked by each respondent, based on their considered importance and time investment (see  Appendix). In order to limit the scope of the interpretation, the labels of the 20 categories of influential factors included keyword descriptions in parentheses (see Table II).

The survey was available online as two versions: a personalized version and an anonymous version. In this way, it was possible to handle personal contacts individually and to use the system’s bulk mail function. Informative letters were directly mailed that contained a web link to the personalized version of the questionnaire, stated the research purpose and assuring the confidentiality of the personal contact information. In the remaining cases, the contacts were additionally asked to forward the message to teachers, lecturers and other interested parties. Furthermore, the web link to the anonymized version of the questionnaire was replaced, with the risk that individuals could respond multiple times and bias the results. Since it was possible to monitor the response behavior of each personal contact, reminders to non-respondents were sent one month after the initial mail was sent out. Of course, these measures could not be used in the other cases as it was not possible to either track who responded or determine whether the contacts forwarded the invitation.

Once the questionnaire responses had been collected, the data provided on 319 fully completed questionnaires between March and April 2015 was analyzed. The data were pre-processed during which incomplete data sets and data from schools, university extensions and Switzerland due to insufficient group sizes were excluded. The final data set included 276 records. As described, this paper emphasizes on the subpopulation from the field of higher education, which further reduced the sample set to 120 usable questionnaires. Table III shows the descriptive statistics for this data set; this information was entered at the top of the questionnaire by the respondents. These descriptive statistics and the following calculations were performed using IBM SPSS Statistics 22.0.

In the following section, the results of the analyses performed are presented based on the data obtained through the survey. The outcome shows which influential factors are considered to be important by professionals in higher education and how much time they would invest to consider these factors during the implementation. Therefore, the survey included the following six questions that directly addressed these issues (also see  Appendix).

Q1. How important (unrelated to your position) is the consideration of the following aspects during the development of TEL approaches or when using technology to support learning and teaching for you?

Q2. How much time would you spend (in your position) when considering the following aspects during the development of TEL approaches or when using technology to support learning and teaching?

Q3. Please choose those five aspects from the following list which are most important for you (unrelated to your position) during the development of TEL approaches or when using technology to support learning and teaching.

Q4. Now arrange these aspects in descending order based on their priority.

Q5. Please choose those five aspects from the following list on which you (in your position) would spend the most time during the development of TEL approaches or when using technology to support learning and teaching.

Q6. Now arrange these aspects in descending order based on the expenditure of time.

The combined results for all questions about the importance of the influential factors appear in Tables IV and V, and the results about the time invested to consider these factors appear in Tables VI and VII. The relevant frequencies and medians for the data are shown in these tables, because the median is considered the best measure of central tendency when using ordinal scaled variables and is also usually not as strongly affected by outliers and skewed data (Quatember, 2011).

As can be seen in Tables IV and V, the categories learning success and teachers’ teaching aspects appear to include the most important influential factors on the basis of the results of this study (median=4.0). In addition, more than 90 percent of the participants considered the categories motivational aspects, learners’ learning aspects, learners’ requirements, course-related aspects and requirements on teachers to be (very) important^[1]. The fact that these categories were most frequently ranked among the top five categories in terms of importance (with the exception of requirements on teachers) confirms these results. More than 70 percent of the respondents ranked the category learning success among the top five results. However, at the same time, at least 5 percent of the respondents considered these categories to be (very) unimportant. The category of influential factors with the least importance is social aspects (median=2.0) according to this survey. More than 50 percent of the respondents considered this category to be (very) unimportant. In addition, at least a third of the respondents deemed the categories business aspects, mind-set and feelings before TEL and demographic differences to be (very) unimportant. Nevertheless, at least some respondents ranked these four categories among the top five most important categories and, at least once, each was ranked first. More than 14 percent of the respondents thought that these categories were very important as well.

With regard to time investment, Tables VI and VII show that the respondents would spend the most time considering the influential factors of the category learning success. Specifically, 82.5 percent of the respondents thought they would spend much of even very much time considering influential factors in this category. At least two-thirds of the respondents also thought they would spend the same amount of time on the categories motivational aspects, support processes, learners’ learning aspects, course-related aspects, requirements on teachers and teachers’ teaching aspects. Again, the results showed that all these categories were ranked among the top five categories except requirements on teachers, which confirms these findings. The respondents indicated that they thought they would spend the most time considering factors in the category learning success (41.8 percent ranked this category in the first place). However, at least 5 percent of the respondents indicated they would spend very little time considering these categories. According to the data, the respondents stated they would spend the least amount of time (more than two-thirds said they would spend only (very) little time) considering factors in the categories mind-set and feelings before TEL and demographic differences. Despite these results, these categories were also ranked among the top five by a few participants, each was ranked in first place at least once and more than 6 percent of the respondents indicated they would spend very much time considering factors in these categories.

In general, the results indicate that all categories seem to be important for professionals in higher education since, for each category (except social aspects), more than 50 percent of the respondents – and, in part, much more than 50 percent – stated that the category was (very) important. In addition, more than 50 percent of the respondents indicated they would invest (a great deal of) time considering factors in most categories. However, less than a third would spend (a great deal of) time considering factors in the categories mind-set and feelings before TEL and demographic differences.

The results of the present study show that influential factors of the categories learning success, teachers’ teaching aspects, motivational aspects, learners’ learning aspects, course-related aspects and requirements on teachers were identified as the most important by the 120 respondents. The respondents tended to spend more time considering factors in these categories. CSF in five of these categories also appear in the literature investigated (see Table II). Thus, it can be concluded that professionals in higher education considered influential factors in the six categories mentioned as generally important when implementing TEL approaches, independent of the approach itself. Therefore, in general it can be suggested to consider the following questions when implementing TEL approaches:

• How does the approach increase learning success, effectiveness and efficiency?

• How does the approach fit to teachers’ teaching style?

• How does the approach increase learners’ motivation and engagement?

• How does the approach fit to learners’ learning behavior?

• How does the approach fit the purpose of the course and the teaching discipline?

• What are the requirements on teachers to use the approach?

It must be noted, however, that the reviewed literature did not mention influential factors in the category requirements on teachers and a factor in the category learning success appeared only once. Although the literature review was not exhaustive, especially with regard to factors in the category learning success, this result was surprising since this category was identified as the most important one in the present study. In addition, Schweighofer and Ebner (2015) emphasized the importance of this category because it contained by far the largest number of factors identified in their literature review.

Although the six categories mentioned above appear to be most important, in general, all categories were ranked as important by a majority of the respondents. Even categories like demographic differences or social aspects, which showed the weakest average results, were deemed very important by at least a small group of the respondents.

Bloom’s Taxonomy provides one possible explanation for the category social aspects (Bloom et al., 1956) in that it describes different levels of knowledge acquisition. Thereby, collaborative learning scenarios, like collaborative tagging, can be used to attain higher levels of applied and metacognitive knowledge in the hierarchy (Bateman et al., 2007). It seems like factors in the category social aspects that are especially relevant in collaborative learning scenarios (Schweighofer and Ebner, 2015) are, therefore, more important in courses that address these higher levels. The majority of the respondents may not have addressed these high levels because they do not use collaborative learning scenarios, leading to the result of the low (on average) importance assigned to the category social aspects. However, those who rated this category as very important might have addressed such levels and may use such scenarios. In order to verify this assumption, further research is necessary.

A possible explanation for the varying results in the category demographic differences is the fact that these factors are only relevant if there are any demographic differences to be considered. It is obviously not important to consider demographic differences in courses that have a homogeneous group of students (with similar age, socioeconomic status and cultural and/or ethnic backgrounds). Therefore, it can be assumed that these factors were relevant to only a minority of the respondents. To confirm this assertion, additional data needs to be gathered, and further research has to be conducted.

In summary, it can be concluded that, in addition to factors in the six most important categories, depending on varying circumstances such as the course, its goals and students, different influential factors are relevant and should be considered when implementing TEL approaches. This was also emphasized by White (2007), who claimed that CSF vary and local circumstances need to be identified in order to use existing strengths.

On the basis of the last conclusion, exploring which circumstances could affect the considerations of importance and time investment in the given data set was necessary. Consequently, the initially proposed analysis was extended and an additional aspect in order to consider potential moderators was added. It was hypothesized that the importance of the influential factors was ranked according to the personality of the respondent. This idea is based on that of Keller and Karau (2013), who conducted a survey with 250 students to investigate how the “Big Five” personality dimensions (extraversion, agreeableness, conscientiousness, emotional stability and openness to experience) influenced five specific types of online course impressions (engagement, value to career, overall evaluation, anxiety/frustration and preference for online courses).

Whether professionals in higher education consider a factor to be influential seems to be a personal decision. Several moderators affect such decisions. First, on the one hand, positive experiences based on decisions made in the past may have influenced subsequent decisions in similar situations (Juliusson et al., 2005). On the other hand, negative experiences can also affect future decisions, since people try to avoid making decisions that led to negative experiences (Sagi and Friedland, 2007).

Another moderator that potentially affects decisions is personal commitment, because people tend to invest more time and effort if they feel personally committed to a decision (Juliusson et al., 2005). Furthermore, according to Acevedo and Krueger (2004), feelings of personal relevance affects decisions as well. For example, if people believe something matters, they tend to invest more time on it.

Finally, demographic characteristics and environmental circumstances such as age or socioeconomic status can be important moderators of the decision process as well. This is in line with the influence of experiences because, for instance, people have had different experiences depending on their age and/or socioeconomic status which, as stated before, can affect their decisions (de Bruin et al., 2007; Finucane et al., 2005).

In conclusion, a personal decision such as whether an influential factor will be considered can be affected by experiences, personal commitment and relevance and demographic characteristics and environmental circumstances. Based on these deliberations, various correlation analyses were conducted. In order to determine whether a correlation was strong, the classification scheme of Brosius (1999) was used: 0 to 0.199 very weak, 0.2 to 0.399 weak, 0.4 to 0.599 substantial, 0.6 to 0.799 strong and 0.8 to 1 very strong. The following three sections show the results of these analyses. It should be noted that the use of a multiple regression would have been preferable but the (sub-)sample size and the conditions for the regression analysis with regard to predictors and the criteria prevented its use.

As part of the survey, the survey asked the respondents whether they had had experience with TEL in teaching. As can be seen in Table III, 97 respondents had had such experience. The survey also asked these 97 respondents whether they had already considered the factors belonging to the 20 categories. Based on the answers received, Cramér’s V was calculated to determine the influence on the importance of the categories and the time people would spend considering factors in these categories. Cramér’s V is a measure of the degree of association between two nominal variables that have two or more levels, which is equal to the Phi coefficient in the case of a 2×2 contingency table. The study used this in the analysis because it is robust where nominal and ordinal data are present.

The evidence indicates that, regardless of a few exceptions, significant weak or substantial correlations were observed between the importance of a category and whether someone had already considered the influential factors in this category. All these correlations indicate that someone who has already considered factors of a category believes it is still important to consider these influential factors. In this study, the highest correlations observed were: business aspects (r=0.520, p<0.01), social aspects (r=0.445, p<0.01) and cognitive aspects (r=0.440, p<0.01).

In addition, the calculation revealed many significant weak or substantial correlations between the amounts of time the respondents would currently spend considering factors according to their considerations of factors in the past. Again, the findings indicate that someone who had already considered factors of a category would spend more time considering factors in this category. The only exception was seen in the category mind-set and feelings before TEL (r=0.408, p<0.01). In this category, the findings revealed that respondents who had already considered the category in the past would spend less time to do so now.

Overall, the findings indicate that experiences affect human behavior such that the factors that have been considered previously are judged to be more important, and more time is invested during the consideration of these factors. Moreover, both positive (Juliusson et al., 2005) and negative (Sagi and Friedland, 2007) experiences in the past had the potential to determine personal decisions such as whether an influential factor would be important in the future. Based on these findings and the correlations described above, the respondents seem to have had predominantly positive experiences while considering influential factors in the past and, hence, indicated that they would be willing to invest time to consider these factors in the future. In contrast, with regard to the category mind-set and feelings before TEL, it could be inferred that the respondents could have had negative experiences in the past while considering these factors and, as a result, indicated that they did not want to invest much time to consider these either now or in the future. However, the strength of this claim requires further investigation. First, additional research on this topic could prove whether the assumptions are accurate. Second, if these assumptions are supported by evidence, additional research could identify positive and negative experiences and provide professionals in higher education with better methods that they can use to consider these factors in the future.

To determine whether personal commitment and judged relevance influenced the results, Spearman’s rank correlation was calculated between the importance of a category of influential factors and the time people would spend considering factors in this category. Furthermore, this analysis made it additionally possible to evaluate whether the importance of a category affected the time invested considering factors in other categories.

The results revealed that significant weak to strong positive correlations existed between the importance of a category and the time people were willing to spend considering factors in this category. This is valid for the entire set, meaning that the more important a category was considered to be, the more time the person was willing to spend considering factors in this category. In this context, the highest correlations observed were: demographic differences (r=0.646, p<0.01), mind-set and feelings before TEL (r=0.616, p<0.01) and social aspects (r=0.605, p<0.01).

These three categories also displayed significant and substantially strong correlations among each other. The more important the category demographic differences were ranked, the more time the respondents were willing to spend considering the categories mind-set and feelings before TEL (r=0.501, p<0.01) and social aspects (r=0.412, p<0.01). Also, the more important the category mind-set and feelings before TEL was ranked, the more time the respondents were willing to spend considering the categories demographic differences (r=0.499, p<0.01) and social aspects (r=0.448, p<0.01). Finally, the more important social aspects were ranked, the more time the respondents were willing to spend considering the category demographic differences (r=0.448, p<0.01).

Overall, many of the substantial to strong correlations that were identified reveal that the more important a category was ranked by the respondent, the more time the respondent was willing to spend considering factors in this category. Therefore, it can be inferred that personal commitment and judged relevance influence the results.

Furthermore, the evidence indicated that the categories that were considered to be important only to a small group seem to be interrelated. For instance, respondents who thought that it was important to consider factors in the category demographic differences also tended to spend more time considering factors in the category social aspects. This supposition needed to be investigated; however, it can be assumed that collaborative learning scenarios are often used in courses with demographic differences, whereby the factors in the category social aspects would become more relevant.

Finally, Spearman’s rank correlation (only for ordinal scales) and Cramér’s V were used to analyze whether the descriptive data (see Table III) affect the importance of each category and the time respondents would spend considering factors in each category.

Concerning the importance, correlations between the descriptive variables and the importance of each of the categories (Q1), the choice of the top five categories (Q3) and the ranking within the top five categories (Q4) were calculated. The analysis results showed that the first two areas of interest were related by only a few significant but weak to very weak correlations. With regard to the ranking within the top five, the analysis allowed the identification of stronger correlations but none were significant because the relevant data set for these high correlations was too small (n=2–22).

With regard to the time invested, the calculation revealed similar correlations. It can be concluded that the tested demographic characteristics and environmental circumstances did not affect the ranked importance of categories and the time respondents would be willing to spend considering factors in these categories. It seems as though other characteristics (e.g. the course subject, the level of knowledge acquisition and the heterogeneity of the students) should be tested in the future to verify whether these circumstances result in an influence on relevance and, therefore, inferred importance.

This contribution has several recognizable limitations. The subsample of 120 respondents included in this study represented only a small fraction of the defined subpopulation. High non-response rates can bias the data set (Frohlich, 2002) which can jeopardize the generalizability and validity of the results of the study. However, it is difficult to test for the presence of this bias, as this would have required the collection of a comprehensive data set from all professionals from all Austrian and German institutions of higher education. Such information theoretically might have been obtained from marketing databases or through manual searches. In the present context, however, commercial databases display certain weaknesses: they cannot guarantee the absolute accuracy of the data because complete coverage of the institutions defined in the population does not exist and the number of employees indicated includes administrative, scientific and teaching staff but frequently does not include lecturers that give classes and are employed on a part-time basis. For these reasons, it was not economically possible to precisely calculate the size of the (sub)population or to compare the basic characteristics of the populations with those of the present sample in this study. Because an electronic mailing strategy to disseminate the questionnaires was used, the composition of the sample may have shifted in favor of the paper authors’ contact coverage. Moreover, because the survey was carried out in part anonymously, the possibility that certain respondents responded multiple times cannot completely be rule out, although no apparent abnormalities in the data set were detected. Finally, surveys are typical types of studies that rely, according to their definition, on the participant’s ability to read questions and select responses on the questionnaire by themselves without interference from the researcher. To that effect, the empirical data may have suffered from different cognitive biases such as acquiescence or social desirability. This especially concerns the interpretation of the 20 categories, which is why the questionnaire provided additional keyword descriptions for them.

The purpose of this study was to identify which influential factors professionals in Austrian and German institutions of higher education considered important and how much time these professionals would invest to consider these factors when implementing TEL approaches. Thereby, intentionally the broadly defined term TEL was used to target influential factors that were considered important, independent of the approach taken.

The results revealed that influential factors in the six categories learning success, teachers’ teaching aspects, motivational aspects, learners’ learning aspects, course-related aspects and requirements on teachers were generally considered as important by 120 professionals. In addition, respectively, at least a small group of these respondents considered the remaining factors as important too, and different circumstances seemed to affect this importance.

Therefore, the given data set was analyzed again to identify potential moderating effects. The findings revealed that experiences and personal relevance seemed to strongly affect how professionals in higher education chose which influential factors they were willing to consider. However, the tested descriptive variables (i.e. age, type of organization, country, function in organization, time in position, field of work and experience with TEL in general) did not influence these choices.

For this reason, future research should focus on the relationships between different moderating variables and relevant influential factors. The research should identify which factors are relevant under given circumstances. A qualitative research approach using interviews with several experts as well as teachers seems to be an appropriate research method for this purpose. This approach can also address some limitations of the given study like the possibility of different interpretations regarding the survey items. Revealed relevant circumstances with an impact on the relevance of influential factors could include the course subject, the addressed level of Bloom’s Taxonomy, the available technical infrastructure, experience and knowledge with TEL approaches, the number of students and much more.

Based on this research, the development of a methodology that better support the implementation of TEL approaches is possible. This methodology should include two steps: the identification of relevant circumstances and the identification of relevant influential factors that should be considered under the given circumstances. Using such a methodology should be more efficient, since only relevant influential factors will be considered during the implementation of TEL approaches. Additionally, the implemented TEL approaches should be more successful too, since CSF would have been considered. For example, the results of this study indicated it is generally not very important to consider demographic differences in German or Austrian institutions of higher education, but if a course would involve a highly heterogeneous group of learners, this factor might become a CSF and should be considered.

Finally, with several case studies the new methodology should be tested at different higher educational institutions in Germany and Austria in order to verify the value of this approach.

The authors declare that they have no conflict of interest.