This research aims to investigate the use of conversational artificial intelligence (CAI) in academic practice through the lens of activity theory, which emphasises the mediation of human actions by tools within a social context. Additionally, it seeks to determine if and how the results of qualitative analysis differ when using traditional qualitative analysis software tools compared to using artificial intelligence tools.
A pragmatic approach to the research design was used. The data collection phase included a survey, with open- and closed-ended questions and was distributed to academics in four countries (South Africa, Hungary, Lebanon and Wales). The data analysis phase included a mixed-methods approach integrating and interpreting both types of data to leverage the strengths of both qualitative and quantitative insights. Furthermore, traditional qualitative analysis methods and artificial intelligence tools were used for the analysis phase, allowing for a comprehensive understanding of the interactions between academics and these tools.
Younger academics used CAI more for research than teaching, with academics from the science faculty using it more for teaching, and business management lecturers using it more for research. While viewed positively, concerns arose about ethics and educational alignment. This research shows how CAI supports qualitative analysis by saving time and suggesting new directions.
Using an “activity theory” theoretical lens, with a pragmatic approach, the research explores how CAI tools impact academic practices. The study enriches theoretical discourse and offers practical recommendations for education.
1. Introduction
Higher education has changed significantly over the last few years. Its transformation began with the COVID-19 pandemic, which necessitated the worldwide introduction of online learning, and has gained further momentum with the recent uptake of conversational artificial intelligence (CAI) agents such as ChatGPT (OpenAI, 2023). ChatGPT—developed by OpenAI and introduced on 21 November 2022—interrogates the large language model (LLM) generative pre-trained transformer (GPT) and is pushing the boundaries of the capabilities of artificial intelligence (AI) in comprehending and creating text that resembles human language. ChatGPT’s popularity has spawned several similar conversational agents—such as Claude (Anthropic, 2024), Gemini (Google, 2023) and Microsoft CoPilot (MicroSoft, 2023) to name a few.
Even though traditional teaching methods are not passé, it will need to change, and these changes will require the large-scale integration of technology into education. AI has already impacted teaching, learning and research in higher education institutions and although this adaption might be challenging for academics, some have embraced it. For example, Stephen Atlas (2023) commented in his book that ChatGPT can serve as a writing partner by aiding the author with word generation; act as a reliable research assistant by swiftly retrieving information and responding to inquiries; and further serve as an administrative helper for drafting reports.
Our study explored the views and practices of academic staff to assess how CAI was being used in their academic practice in teaching and research in higher education. Conducted in 2023, the research involved an online survey, directed at academic staff at four universities in Hungary, Lebanon, South Africa and Wales. The data were analysed using quantitative methods, a traditional qualitative method and CAI tools.
The researchers took a pragmatic approach to the research, using activity theory as a theoretical lens to understand the interactions between educators, students and AI tools. This research not only highlights how CAI is used within higher education but also addresses the ethical considerations that arise from its use, reflecting on the complexities outlined in activity theory regarding the alignment of tools with educational goals and practices. Furthermore, the research suggests that CAI can be effectively used for the identification of themes in qualitative analysis. In general, the results were very similar to that found by a researcher using a traditional software tool, NVivo (Lumivero, 2023), with efficiency gains noted. The criticality of researchers using CAI, is an important skill that was used during this process.
The next sections consider the related literature, introduce the methodology, followed by the results and conclude with some summarising remarks.
2. Related literature
2.1 CAI use in higher education
AI is transforming higher education with a variety of applications that improve learning experiences, optimise administrative processes and enhance research. According to Rienties et al. (2020), AI is employed in several key areas in higher education: to support personalised learning through adaptive platforms and intelligent tutoring systems; to automate administrative tasks like admissions and student services; to boost student engagement and support through chatbots and predictive analytics for student success; and to automate grading and feedback.
Chinedu and Abejide (2021) found that chatbots are being used in educational settings for teaching and learning, administrative support, assessments, advisory roles, and research and development. Most commonly, they found that it is being used for teaching and learning (66%), followed by research and development (19%) and in a lesser degree it is used for assessment (6%), administration (5%) and for advice (4%). AI technology, its application and acceptance in higher education is expected to grow, which will introduce creative methods to improve the educational experience. Concerns about data privacy, bias in AI algorithms and the digital divide, require thoughtful consideration. As such, several studies have highlighted the potential impact of AI on teaching and learning in higher education, reflecting on both the positive as well as the negative perceptions of AI (Bozkurt et al., 2021; Cain et al., 2023; Chocarro et al., 2021; Kumar, 2023; Zawacki-Richter et al., 2019).
Kumar (2023), in a study exploring how ChatGPT could aid academic writing, found that ChatGPT had an exceptionally fast response time, delivering up to 500 words in less than 2 min. This ability of ChatGPT and other AI-driven conversational agents to summarise information quickly can greatly decrease the time needed to gather information, removing the need to search across multiple search engines (Farrokhnia et al., 2023). Cranfield et al. discovered that the references generated by the free version of ChatGPT (ChatGPT 3.5) are often incorrect, highlighting that academic users of this and other AI tools should always verify the references and information they produce (Cranfield et al., 2023). Chocarro et al. indicated that chatbots could boost efficiency, suggesting that academic institutions should promote their use as supportive tools (Chocarro et al., 2021).
In a comprehensive review, Zawacki-Richter et al. highlighted the necessity for educators to critically examine the challenges and risks of AI in education, advocating for the incorporation of pedagogical insights and ethical considerations (Zawacki-Richter et al., 2019). According to Han et al. AI-based chatbot programs are employed to deliver educational content and support remote learning while also aiming to stimulate interest in education, encourage self-directed learning and offer supplementary educational resources (Han et al., 2022).
Despite concerns and negative views regarding AI’s role in higher education, there is a growing consensus that AI tools should be seen in positive light (Rudolph et al., 2023; Atlas, 2023; Alenezi, 2023; Akinwalere and Ivanov, 2022). Crompton and Song (2021) highlight AI’s significant potential to enrich teaching and learning in higher education. Rudolph et al. argue that AI can offer valuable opportunities when both students and academics are trained to use the technology with integrity. They suggest that attempts to strictly regulate the use of these tools are counterproductive, recommending instead that educators conduct assessments in class or create assignments requiring student presentations, rather than relying solely on traditional testing methods. For students, they suggest writing assignments with AI as a supporting tool to enhance writing skills and spark new ideas, rather than merely copying and pasting text (Rudolph et al., 2023).
AI applications are increasingly being utilised in higher education to improve learning outcomes and offer personalised assistance thereby advancing our understanding of their influence on academic practices (Tick, 2024).
2.2 Comparing traditional and AI-driven research methods
2.2.1 Traditional qualitative analysis methods–strengths and limitations
Traditional qualitative analysis methods, like thematic analysis, content analysis and grounded theory, are essential in social science research. They capture the nuances of human behaviour and social interactions, uncovering themes and patterns that quantitative methods might miss. These methods provide a clear framework for analysing qualitative data and understanding participants' experiences (Braun and Clarke, 2006).
However, these methods have limitations; they can be time-consuming and labour-intensive, requiring extensive manual coding and interpretation, which may introduce biases from subjective interpretations (Balsam et al., 2011). Reyes et al. contends: “It is also a matter of the coding process being both an iterative process … and a tedious task where simple mistakes can be made” (Reyes et al., 2024, p. 112). Additionally, traditional methods may struggle with large datasets, limiting their effectiveness in studies involving extensive qualitative data (Devonald and Jones, 2023).
2.2.2 AI-driven qualitative analysis tools–strengths and limitations
AI-driven qualitative analysis tools offer significant advantages in speed and data processing capabilities (Sripathi et al., 2023), being able to analyse vast amounts of qualitative data much faster than human researchers, allowing for efficient data handling and quicker insights (Beaulieu-Jones et al., 2019). AI tools can automatically identify themes, sentiment and patterns within text data, providing a comprehensive overview of the dataset (Mahadevkar et al., 2024), and “ […]leading to a better understanding of the text and, consequently, the formation of new research directions[...]” (Lixandru, 2024, p. 65).
Moreover, AI-driven tools enhance the rigour of qualitative analysis by minimising human biases and increasing coding consistency. They also facilitate the integration of qualitative and quantitative data, enabling mixed methods approaches that leverage both paradigms' strengths (Carter et al., 2016). However, while AI tools can augment qualitative analysis, they may not fully replace the nuanced understanding that human researchers bring to qualitative data interpretation.
2.3 Insights into integrating these approaches
One significant contribution to this field is the work by Hitch (2024), which discusses how CAI can augment qualitative analysis. Hitch suggests that combining CAI with traditional qualitative methods significantly enhances and transforms research by streamlining data processing and theme identification while preserving in-depth analysis. Furthermore, the researcher and the CAI technology could be viewed as multiple coders which could increase the “reliability” of the analysis.
A study by Li et al. (2024) explores the comparative capabilities of CAI and human researchers in qualitative data analysis, revealing that CAI can detect subtle nuances and emotional contexts that may be overlooked by traditional methods. This finding suggests that CAI can serve as a valuable complement to human analysis, enhancing the overall quality of qualitative research.
Finally, the work of Christou (2023) highlights the potential of CAI in theory development within qualitative research, suggesting that AI can facilitate the generation of new theoretical insights by analysing existing qualitative data in innovative ways. This perspective reinforces the idea that CAI is not merely a tool for efficiency but can also contribute to the theoretical advancement of qualitative research methodologies.
3. Methodology
The objectives of this study were to investigate the use of CAI tools from the academic’s perspective in higher education. Furthermore, to examine the traditional qualitative thematic analysis methods compared to using an AI approach. These objectives translated into the following research questions:
RQ1: How is CAI tools used in academia?
RQ2: How does traditional thematic analysis compare with qualitative analysis using CAI?
3.1 Philosophical approach
This study adopted a pragmatic approach, focusing on the practical implications of research findings and the integration of diverse methods to tackle complex research questions. In higher education, particularly regarding AI, pragmatism provides a framework for educators and institutions to navigate technology-enhanced learning environments. This aligns with John Dewey’s educational theories, which emphasise that knowledge should be grounded in real-world experiences and that education should prepare individuals for practical life (Fang et al., 2023).
3.2 Theoretical lens – activity theory and CAI
The integration of AI in higher education is a rapidly evolving field that necessitates a comprehensive understanding of its implications. Activity theory provided a useful framework for analysing interactions between educators and AI tools (Sengul, 2024). Developed from Vygotsky’s (1978) work and expanded by Engeström (1987), activity theory emphasises the interplay between individuals and their environment, considering factors such as tools, community and rules, emphasising how technological tools, shape interactions and influence educational dynamics (Alhusaiyan, 2024).
Yang and Kyun (2022) conducted a systematic review categorising studies on human–AI interaction using activity theory, revealing AI’s potential to enhance foreign language learning by mediating educational activities. Kou and Gui (2020) examined community–AI interactions, emphasising how AI explanations can mediate user engagement. Their findings highlight the versatility of activity theory in analysing human–AI relationships.
3.3 Data collection
In 2023, a Qualtrics survey with open and closed-ended questions was distributed using e-mail to academics across four countries (Qualtrics, 2002). The study is exploratory in nature and the first phase of a larger study.
A convenient sample of 144 participants responded, though not all participants answered every question. Ethical assurances were provided, including anonymisation and secure data storage. The research protocol received approval from ethics committees in Wales, Lebanon and South Africa.
3.4 Data analysis
A mixed-methods approach was employed to analyse the data to leverage the strengths of both qualitative and quantitative insights, providing a richer understanding of the research. Quantitative data were analysed using Statistical Analysis Software (SAS, 1972). To analyse the qualitative data, two methods were used, 1) traditional qualitative analysis methods, specifically using thematic analysis, and 2) CAI tools were used with prompt engineering and comparative analysis. One researcher used NVivo and thematic analysis while two further researchers used the CAI tools Claude, Gemini and ChatGPT 4.0 (see Figure 1).
The flow diagram begins at the top, with the text box labeled “Philosophical Approach” with the description “Pragmatism”. From this, a directional arrow leads down to “Mixed Research Methods”, which includes two parts: “Quantitative” and “Qualitative” methods. It leads to the next step labeled “Data Collection”, with two types labeled: “Survey, closed questions, data”, and “Survey, open ended”. From here, an arrow leads to “Data Analysis”, which is divided into two sections: “Traditional - Researcher 1” and “A I tools" as separate paths. Under “Traditional - Researcher 1”, there is a box labeled “Thematic Analysis” followed by “Open coding, using N Vivo”. Under “AI tools,” the diagram includes “Prompt engineering”, followed by “Claude”, “Gemini”, and “ChatGPT”. On the left side, the entire flow diagram is vertically labeled “Literature Review”. From Mixed Research Methods, the low diagram is vertically labeled “Activity Review”.The research approach (Created by the authors)
The flow diagram begins at the top, with the text box labeled “Philosophical Approach” with the description “Pragmatism”. From this, a directional arrow leads down to “Mixed Research Methods”, which includes two parts: “Quantitative” and “Qualitative” methods. It leads to the next step labeled “Data Collection”, with two types labeled: “Survey, closed questions, data”, and “Survey, open ended”. From here, an arrow leads to “Data Analysis”, which is divided into two sections: “Traditional - Researcher 1” and “A I tools" as separate paths. Under “Traditional - Researcher 1”, there is a box labeled “Thematic Analysis” followed by “Open coding, using N Vivo”. Under “AI tools,” the diagram includes “Prompt engineering”, followed by “Claude”, “Gemini”, and “ChatGPT”. On the left side, the entire flow diagram is vertically labeled “Literature Review”. From Mixed Research Methods, the low diagram is vertically labeled “Activity Review”.The research approach (Created by the authors)
3.4.1 Traditional qualitative thematic data analysis
Guided by Braun and Clarke’s (2006) framework, for thematic analysis, data were uploaded to NVivo. Codes generated by reviewing the data, highlighting relevant words, sentences or paragraphs were assigned to NVivo nodes (codes). These nodes represent the initial themes or concepts that emerged from the data, which were then reviewed and grouped, categorising similar codes together and identifying patterns in the data. Only one coder conducted this analysis, and therefore coding inconsistencies were avoided.
3.4.2 Qualitative analysis using CAI tools
Two researchers used different CAI tools (Claude, Gemini, and ChatGPT 4.0) to interrogate the data. The identified themes were then analysed and checked for correctness, and alignment with the data.
3.4.3 Combining traditional and AI approaches
The CAI tools’ themes and the NVivo codes and themes were then compared. Further revision of the combined themes was critically reviewed and simplified thereafter.
4. Results
The use of AI in higher education was explored using both quantitative and qualitative analysis methods.
4.1 Quantitative analysis
4.1.1 Demographics
Males (54%) and females (46%) were close to equally represented. No participants indicated that they belonged to the category non-binary or indicated that they preferred not to disclose their gender.
Thirty-nine percent of the staff were either associate lecturers or lecturers, while 46% were senior lecturers, associate professors or professors. Fifteen percent of the respondents came from the “other” category (see Figure 2).
The pie chart is titled “Seniority of Staff”, and the data from the pie chart in the clockwise sense are as follows: Junior staff: 39 percent. Senior staff: 46 percent. Other: 15 percent.Respondents by seniority (Created by the authors)
The pie chart is titled “Seniority of Staff”, and the data from the pie chart in the clockwise sense are as follows: Junior staff: 39 percent. Senior staff: 46 percent. Other: 15 percent.Respondents by seniority (Created by the authors)
Most of the 144 participants (63%) were older than 40 years – 37% were between 40 and 49 years and 26% were 50 years and older. Only 5% of the respondents were between 20 and 29 years of age, whereas 32% were in their thirties (see Figure 3).
The pie chart is titled “Age”, and the data from the pie chart in the clockwise sense are as follows: 20 to 29: 5 percent. 30 to 39: 32 percent. 40 to 49: 37 percent. 50 plus: 26 percent.Age of respondents (Created by the authors)
The pie chart is titled “Age”, and the data from the pie chart in the clockwise sense are as follows: 20 to 29: 5 percent. 30 to 39: 32 percent. 40 to 49: 37 percent. 50 plus: 26 percent.Age of respondents (Created by the authors)
Most of the respondents were from business management (50%), while 16% of the respondents came from science and 10% from social sciences (see Figure 4).
The pie chart is titled “Discipline”, and the data from the pie chart in the clockwise sense are as follows: Business Management: 50 percent. Social Science: 10 percent. Science: 16 percent. Other: 24 percent.Respondents by discipline (Created by the authors)
The pie chart is titled “Discipline”, and the data from the pie chart in the clockwise sense are as follows: Business Management: 50 percent. Social Science: 10 percent. Science: 16 percent. Other: 24 percent.Respondents by discipline (Created by the authors)
4.1.2 Descriptive analysis
Of the total, 49% of the respondents indicated that they use CAIs for their teaching practices, while 64% indicated that they used it for research purposes. The most popular CAI used was ChatGPT 3.5 (see Figure 5).
The horizontal axis of the bar chart shows markings “Chat G P T 3.5”, “Chat G P T 4”, “Gopher”, “Bing A I”, “Lamda” and “Other” from left to right. The vertical axis of the bar graph ranges from 0 to 50 percent in increments of 5 percent. The data in the bars is as follows: Chat G P T 3.5: 41 percent. Chat G P T 4: 3 percent. Gopher: 1 percent. Bing A I: 2 percent. Lamda: 6 percent. Other: 42 percent. Note: All numerical data values are approximated.Types of CAIs used (Created by the authors)
The horizontal axis of the bar chart shows markings “Chat G P T 3.5”, “Chat G P T 4”, “Gopher”, “Bing A I”, “Lamda” and “Other” from left to right. The vertical axis of the bar graph ranges from 0 to 50 percent in increments of 5 percent. The data in the bars is as follows: Chat G P T 3.5: 41 percent. Chat G P T 4: 3 percent. Gopher: 1 percent. Bing A I: 2 percent. Lamda: 6 percent. Other: 42 percent. Note: All numerical data values are approximated.Types of CAIs used (Created by the authors)
4.1.3 Comparative analysis
Comparisons were made based on the respondents' gender, field of study and academic roles. It was observed that the 40–49 age group used CAIs to support their teaching the most, with 59% reporting its usage. The Science Faculty were the most active users of CAIs (64%) for teaching. When considering academic positions, lecturers were the group most likely to employ CAI s in their teaching, with 67% using it.
In total, 77% of the 30–39 age group used CAI tools for their research. Among academic roles, associate lecturers were the top users of CAI tools for research, with 83% reporting that they used it. The Business Management Faculty were the most inclined to use CAI tools for research, with 69% employing it when doing research.
4.2 Qualitative analysis
4.2.1 Teaching practice
Twelve themes were identified for “How did you use AI such as ChatGPT to support your teaching practice?” With the first submission to ChatGPT, ChatGPT identified ten themes, Claude identified eight, Gemini four and using NVivo, the researcher identified nine themes. Four themes were identified by all the analysis tools used (see entries in blue italics in Table 1).
Themes identified for AI support for teaching practice
| Themes | ChatGPT | Claude | Gemini | NVivo |
|---|---|---|---|---|
| Educational support | X | X | X | X |
| Administrative efficiency | X | X | X | X |
| Research and writing | X | X | X | X |
| Language and communication | X | X | X | |
| Teaching and learning enhancement | X | X | X | X |
| Coding and technical support | X | X | X | |
| Exam and assignment preparation | X | X | X | |
| Collaboration and group work | X | |||
| Critical thinking and evaluation | X | X | ||
| Project compilation | X | |||
| Idea generation and problem solving | X | |||
| Improve general knowledge | X |
Source(s): Table created by authors
A further submission was made to ChatGPT, requesting it to indicate the frequency with which each theme was mentioned. A slightly different set of themes were then identified (see Figure 6).
The horizontal axis is labeled “Count” and ranges from 0 to 30 in increments of 5 units. The vertical axis shows markings “Educational tasks”, “Research support”, “Training and presentation preparation”, “Content generation and writing support”, “Administrative and planning tasks”, “Problem solving and idea organisation”, “Specific technical tasks”, “Learning and language improvement”, and “Testing and understanding A I tools”from top to bottom. The data in the bars are as follows: Educational tasks: 27. Research support: 13. Training and presentation preparation: 10. Content generation and writing support: 8. Administrative and planning tasks: 7. Problem solving and idea organisation: 4. Specific technical tasks: 3. Learning and language improvement: 3. Testing and understanding A I tools: 2. Note: All numerical data values are approximated.Frequency of AI support for teaching practice themes identified by ChatGPT’s second response (Created by the authors)
The horizontal axis is labeled “Count” and ranges from 0 to 30 in increments of 5 units. The vertical axis shows markings “Educational tasks”, “Research support”, “Training and presentation preparation”, “Content generation and writing support”, “Administrative and planning tasks”, “Problem solving and idea organisation”, “Specific technical tasks”, “Learning and language improvement”, and “Testing and understanding A I tools”from top to bottom. The data in the bars are as follows: Educational tasks: 27. Research support: 13. Training and presentation preparation: 10. Content generation and writing support: 8. Administrative and planning tasks: 7. Problem solving and idea organisation: 4. Specific technical tasks: 3. Learning and language improvement: 3. Testing and understanding A I tools: 2. Note: All numerical data values are approximated.Frequency of AI support for teaching practice themes identified by ChatGPT’s second response (Created by the authors)
When ChatGPT was asked to revisit its response, it responded with six main themes divided into specific subthemes (see Table 2 and Figure 7).
The horizontal axis ranges from 0 to 25 in increments of 5 units. The vertical axis shows markings “Educational Tools and Assessment Design”, “Content Development and Summarisation”, “Information Retrieval and Resource Sourcing”, “Research Assistance and Analysis”, “Language Support and Translation”, and “Routine Task Automation” from top to bottom. The data in the bars are as follows: Educational Tools and Assessment Design: 20. Content Development and Summarisation: 17. Information Retrieval and Resource Sourcing: 8. Research Assistance and Analysis: 8. Language Support and Translation: 8. Routine Task Automation: 3. Note: All numerical data values are approximated.Frequency of AI support for teaching practice themes identified by ChatGPT response 3 (Created by the authors)
The horizontal axis ranges from 0 to 25 in increments of 5 units. The vertical axis shows markings “Educational Tools and Assessment Design”, “Content Development and Summarisation”, “Information Retrieval and Resource Sourcing”, “Research Assistance and Analysis”, “Language Support and Translation”, and “Routine Task Automation” from top to bottom. The data in the bars are as follows: Educational Tools and Assessment Design: 20. Content Development and Summarisation: 17. Information Retrieval and Resource Sourcing: 8. Research Assistance and Analysis: 8. Language Support and Translation: 8. Routine Task Automation: 3. Note: All numerical data values are approximated.Frequency of AI support for teaching practice themes identified by ChatGPT response 3 (Created by the authors)
AI support for teaching practice themes with subthemes as defined in a further iteration by ChatGPT 4, the third response
| Themes | Count |
|---|---|
| Educational tools and assessment design | 20 |
| 11 |
| 5 |
| 4 |
| Content development and summarisation | 17 |
| 8 |
| 5 |
| 4 |
| Research assistance and analysis | 8 |
| 7 |
| 1 |
| Information retrieval and resource sourcing | 8 |
| 6 |
| 2 |
| Language support and translation | 6 |
| 3 |
| 3 |
| Routine task automation | 3 |
| 3 |
Source(s): Table created by authors
As can be seen, the wording of the themes—identified by ChatGPT 4 in three responses (see Table 1, Figures 6 and 7)—is slightly different.
The four similar themes (see themes in italics in Table 1)— identified by the researcher using NVivo and identified by the three CAI Tools—were considered main themes, and the other themes collated under these main themes (see Table 3).
Summary of NVivo and the AI tools’ responses to the question “How did you use AI tools such as ChatGPT to support your teaching practice?”
| Main themes | Themes identified by | |||||
|---|---|---|---|---|---|---|
| ChatGPT response 1 | ChatGPT response 2 | ChatGPT response 3 | Claude | Gemini | NVivo | |
| Educational support | Coding and technical support Critical thinking and evaluation Language and communication | Educational tasks Content generation and writing support Specific technical tasks | Educational tools and assessment design Language support and translation | Coding and technical support Critical thinking and evaluation Language and communication | Educational support | Coding and technical support Language and communication Improve general knowledge |
| Administrative efficiency | Administrative efficiency | Administrative planning and tasks | Routine task automation | Administrative efficiency | Administrative efficiency | Administrative efficiency |
| Research and writing | Research and writing Project compilation | Research support Problem solving and idea organisation Testing and understanding AI tools | Research assistance and summarisation Information retrieval and resource sourcing | Research and writing | Research and writing | Research and writing Idea generation and problem solving |
| Teaching and learning enhancement | Teaching and learning enhancement Exam and assignment preparation Collaboration and group work | Training and presentation preparation Learning and language improvement | Content development and summarisation | Teaching and learning enhancement Exam and assignment preparation | Teaching and learning enhancement | Teaching and learning enhancement Exam and assignment preparation |
Source(s): Table created by authors
It is a question of semantics how the meaning of the words, used by the respondents, is interpreted by the researchers.
To summarise, the combination showed how academics perceived the usefulness of LLMs when using it to support their teaching: it provides educational support, assists with administrative tasks, supports research and enhances the teaching and learning experience of students.
4.2.2 Research practice
Participants were asked how they used CAIs to support their research practice. Out of the combined 15 themes, ChatGPT and the researcher using NVivo, identified nine, Claude identified seven and Gemini identified six themes. Only one theme was identified by all, namely “Social media mining” (see Table 4).
Themes identified for AI usage in research
| Themes | ChatGPT | Claude | Gemini | NVivo |
|---|---|---|---|---|
| Efficiency and time saving | X | |||
| Social media mining | X | X | X | X |
| Literature review and reference materials | X | X | X | |
| Rephrasing and proofreading | X | X | X | |
| Research support and citation | X | X | X | |
| Language editing and translation | X | X | ||
| Information retrieval and search | X | X | X | |
| Student engagement and teaching support | X | X | ||
| Results analysis and idea generation | X | X | X | |
| Sentiment analysis and thematic identification | X | |||
| Platform and tool usage | X | |||
| Code assistance | X | X | ||
| Bibliographic references and documentation | X | |||
| Academic integrity | X | |||
| Planning, generate and organise ideas and tasks | X |
Source(s): Table created by authors
However, when ChatGPT was asked to revisit the analysis and to indicate the number of times, the identified themes were mentioned by the participants in their responses; it responded as follows:
Research assistance – 14 mentions. Mining social media for insights, finding and summarising literature, assisting with literature reviews, finding recent research topics, summarising research.
Writing and editing support – 12 mentions. Rephrasing sentences, proofreading, editing texts, translating and checking grammar, summarising texts.
Academic enhancements – 8 mentions. Finding reference materials, supporting citation processes, formulating research questions, structuring content, paraphrasing content, generating ideas.
Educational tools – 6 mentions. Creating quizzes, managing educational materials, assisting in student research.
Information retrieval – 4 mentions. Searching for specific research papers, seeking bibliographic references.
Communication and collaboration – 3 mentions. Facilitating communication with colleagues and students, discussing research findings.
When considering these themes and the meaning of each, it can be concluded that Information retrieval (four mentions) can be included under Research assistance (14 mentions) and Communication and collaboration (three mentions) can be considered Academic enhancements (eight mentions). When combined, it results in Research assistance (now 18), Academic enhancements (now 11) (see Figure 8).
The horizontal axis is labeled “Count” and ranges from 0 to 20 in increments of 2 units. The vertical axis shows markings “Research assistance”, “Writing and editing support”, “Academic enhancements”, and “Educational tools” from top to bottom. The data in the bars are as follows: Research assistance: 18. Writing and editing support: 12. Academic enhancements:11. Educational tools: 6. Note: All numerical data values are approximated.Themes identified by ChatGPT for AI usage in research (Created by the authors)
The horizontal axis is labeled “Count” and ranges from 0 to 20 in increments of 2 units. The vertical axis shows markings “Research assistance”, “Writing and editing support”, “Academic enhancements”, and “Educational tools” from top to bottom. The data in the bars are as follows: Research assistance: 18. Writing and editing support: 12. Academic enhancements:11. Educational tools: 6. Note: All numerical data values are approximated.Themes identified by ChatGPT for AI usage in research (Created by the authors)
When summarising the initial ChatGPT response as well as the revised ChatGPT response together with the responses of the other tools (refer to Table 4), it can be seen that the participants used AI to support their research, their writing, to enhance their analysis but it also provided educational tools for teaching.
4.2.3 Challenges
For the question “What do you think are the challenges of integrating AI into your academic practise (teaching and research)?” it can be seen in Table 5, that ChatGPT 4 identified 11 of the 13 identified themes, Claude identified eight themes and Gemini only four themes. The researcher using NVivo identified eight themes. Only one of the themes were identified by all, namely “Ethical issues and plagiarism” (see theme in italics in Table 5).
Themes identified for AI integration challenges in academic practise
| Themes | ChatGPT | Claude | Gemini | NVivo |
|---|---|---|---|---|
| Accuracy and reliability | X | X | X | |
| Ethical issues and plagiarism | X | X | X | X |
| Technical problems | X | X | X | |
| Dependency and lack of effort | X | X | X | |
| Teaching and learning impact | X | X | X | |
| Integration challenges | X | X | ||
| Access and cost | X | X | ||
| Educational paradigm shift | X | X | ||
| Creativity and self-reliance | X | X | ||
| Student and academic dependence | X | X | X | |
| Contextual limitation | X | X | ||
| Learning focus and adaptation | X | |||
| Training and familiarity | X | X | X | |
| Redefinition of roles and responsibilities for teachers and students | X |
Source(s): Table created by authors
As with the previous results, some of these could be combined. Nonetheless, the participant responses revealed a variety of concerns. These include issues related to the accuracy of information provided by CAIs, the use of incorrect references and the possibility of erroneous answers. In addition, ethical challenges such as plagiarism and over-reliance on CAIs were highlighted. Participants felt that the use of CAIs, with students seeking quick results, could hinder their creativity and critical thinking. Technical problems like biases, miscommunication and the expense of using CAIs were also noted. These concerns encompass a range of issues related to changes in educational approaches.
5. Discussion
In this discussion, the researchers build upon the foundations established in the literature review regarding activity theory as a framework for understanding the integration of CAI in higher education. To visualise these interactions, a theoretical framework (see Figure 9) that presents the interrelationships among key components: subjects (users of AI tools), objects (education and research goals), tools (AI and traditional), community (academic), rules (ethical and institutional) and division of labour (roles in academia) are presented. These components reflect the diverse ways in which CAI can reshape educational practices and outcomes, while ethical considerations are examined reinforcing the necessity for responsible integration of AI technologies in academia.
The diagram is titled “ACTIVITY THEORY UNDERPINNED”, and it shows “Conversational A I in Higher Education” at the center. From Conversational A I in Higher Education, different arrows arise in all directions and lead to individual sections. The content in each section is as follows: “COMMUNITY”: “Academic Community”, Collaborative network of educators, researchers, and students. “RULES”: “Ethical Guidelines and Institutional Policies”, Frameworks and guidelines governing the use of A I tools. “DIVISION OF LABOUR”: “Roles in Academia”, Different responsibilities of faculty, students, and administrative staff. “ETHICS”: “Ethical Considerations”, AI bias, academic integrity, and responsible technology use. “TOOLS”: Includes “Conversational AI Tools” such as Claude, Gemini, and ChatGPT, as well as “Traditional Research Tools” like Nvivo. “SUBJECT”: Represents the “Academic Staff” and “Students,” users of AI tools. Three arrows from community, ethics, and tiles lead to the subject. An arrow from the subject leads to “OBJECT”, which includes “Education and research goals,” aiming to improve teaching efficacy and research quality.Activity framework for CAI (Created by the authors)
The diagram is titled “ACTIVITY THEORY UNDERPINNED”, and it shows “Conversational A I in Higher Education” at the center. From Conversational A I in Higher Education, different arrows arise in all directions and lead to individual sections. The content in each section is as follows: “COMMUNITY”: “Academic Community”, Collaborative network of educators, researchers, and students. “RULES”: “Ethical Guidelines and Institutional Policies”, Frameworks and guidelines governing the use of A I tools. “DIVISION OF LABOUR”: “Roles in Academia”, Different responsibilities of faculty, students, and administrative staff. “ETHICS”: “Ethical Considerations”, AI bias, academic integrity, and responsible technology use. “TOOLS”: Includes “Conversational AI Tools” such as Claude, Gemini, and ChatGPT, as well as “Traditional Research Tools” like Nvivo. “SUBJECT”: Represents the “Academic Staff” and “Students,” users of AI tools. Three arrows from community, ethics, and tiles lead to the subject. An arrow from the subject leads to “OBJECT”, which includes “Education and research goals,” aiming to improve teaching efficacy and research quality.Activity framework for CAI (Created by the authors)
This research therefore provides novel insights into the impact of AI—particularly CAIs and more specifically its use—in higher education. The results are discussed in terms of the objectives set for the study:
How are CAI tools used in academia?
The use of CAIs was found to be more prevalent for research, which concurs with the findings of Chinedu and Abejide (2021), suggesting a trend where academic research is more quickly integrating AI technologies. The demographic and quantitative comparative analyses offer deeper insights into who is using CAIs and whether they are utilising them for research or teaching. The observed preference for AI tools in research over teaching, especially among younger academics indicates that AI may become more central to academic research than traditional teaching. This variability can be understood through activity theory, which highlights how these tools mediate activities. The integration of AI in research practices could enhance efficiency and reshape traditional academic interactions, leading to a more dynamic and collaborative research environment (Yu and Guo, 2023).
The widespread use of CAIs emphasises its growing importance in academic environments. The participants’ feedback on the various uses of CAIs in both teaching and research indicates a notable shift in conventional academic practices. These applications range from boosting administrative efficiency to promoting greater student engagement and comprehension.
How does the traditional thematic analysis compare with qualitative analysis using CAI?
This research suggests that using CAI tools for qualitative analysis can effectively enable the initial identification of key concepts within the research data, which confers with the study conducted by Hamilton et al. (2023). The research also posits that the use of CAI is a game changer for researchers in terms of being able to be more efficient in the use of their time during the data analysis stage, being able to therefore spend more focused time in the criticality of the research.
Several of the themes identified by the various CAI tools, as well as the traditional qualitative analysis tool NVivo, were similar. This suggests that utilising both is valuable for identifying key concepts within the data, providing researchers with a structured approach to organise a coherent interpretation of participants' experiences and viewpoints (Hamilton et al., 2023). The CAI tools considered, differed considerably in the number of themes identified, which might indicate that a variety of these tools could greatly benefit a more comprehensive view of the survey responses.
There is a clear need for both—the researcher and CAI tools. The researcher cannot be replaced—despite the invaluable contribution CAI tools make only the researcher is able to critically evaluate the context of the research findings. Chubb (2023, p. 13) agrees with this statement “… AI tools for researchers are not something to replace me or others like me. Instead, it is a tool to make space to think more critically about the situatedness of research findings, freeing up more time to prepare results for impact.”
The research findings underscore the pivotal role of CAI as an activity mediator in higher education. Drawing on activity theory, it is evident that the adoption of these technologies leads to significant shifts in teaching and research practices. CAI tools can enhance academic productivity and engagement but also raises challenges concerning academic integrity and reliance on technology (Yu and Guo, 2023).
6. Conclusion
The growing use of CAI in higher education has several benefits but also introduces challenges that require careful thought (Venter et al., 2024). It is essential to collaboratively develop a shared understanding of how to ethically integrate these tools into academia. The integration of CAI with traditional qualitative methods presents a promising frontier for researchers. By leveraging CAI’s capabilities, researchers can enhance the depth, efficiency, and rigour of qualitative analysis, paving the way for innovative approaches to understanding complex social phenomena.
The researchers agree with Hitch (2024), and contend that the question of CAI augmented thematic analysis is not about whether it should be adopted but a question of the best way to adopt it. Critical analysis remains a fundamental skill required to do high quality research, and therefore “AI can augment but not adequately replace human researchers” (Hitch, 2024, p. 606). This study provides a foundation for future research and policy initiatives to harness the advantages of AI in higher education while minimising its potential downsides.
6.1 Limitations
Since this field of study is relatively new, there is a lack of seminal literature. The results discussed in this paper represent the outcomes from the first phase of an ongoing research project, highlighting the initial themes that emerged from the study. In future studies more participants from various disciplines will be included.
6.2 Future research considerations
For future research it might be useful to consider how these CAI tools identify themes and how often the theme must appear in the participant responses to be listed as a theme. Additionally, how CAI tools can be used for theory generation and not only for the identification of themes.
6.3 Contribution to knowledge
Integration of AI and Traditional Methods: The paper discusses combining qualitative analysis methods with AI tools, which is a relatively new area of research, highlighting efficiency and enhanced data analysis capabilities.
Comparative Analysis: It contrasts traditional qualitative analysis with AI-driven techniques, providing insights into their respective strengths and limitations.
Practical Application: Real-world applications of CAI in academic practices and research support are investigated, adding practical significance to the research.
Empirical evidence provided how qualitative analysis by CAI tools compared to practices using NVivo.
