Artificial intelligence-based assessment in ELT exam creation: a case study of Van Lang University lecturers Open Access

Assessment plays a pivotal role in the educational training process, offering essential insights into the effectiveness of language instruction by identifying challenging areas and gauging learner progress in language courses (Pearson and Murphy-Judy, 2020; Voss, 2018). It also enables lecturers to assess their students’ language abilities, pinpoint strengths, and weaknesses, and propose targeted improvements (Brown and Abeywickrama, 2010; Purpura, 2016). Commonly used tools such as multiple-choice questions, reading comprehension tasks, and oral language assessments (Hughes and Hughes, 2020) require significant time and effort to develop, as they must align with learners’ current needs and desired outcomes. This time-intensive process is crucial to ensuring precise and effective language proficiency evaluations at the university level.

Despite the recognized value of language assessment, lecturers often face the burden of developing language test items that align with institutional or national curriculum standards. The time and energy invested in these efforts are essential for fostering future student success and ensuring quality education. However, recent technological advances, particularly in artificial intelligence (AI), have transformed the educational landscape by streamlining time-consuming tasks (Çakmak, 2019; Kukulska-Hulme and Morgana, 2021; Selwyn et al., 2021). In language education, technology has significantly enhanced teaching and learning practices, offering tools to create assessments automatically and adaptively (Nazaretsky et al., 2022). Although AI presents substantial opportunities for improving assessment efficiency and adaptability (García-Peñalvo et al., 2020; Gardner et al., 2021), research exploring the potential risks, hidden errors, and limitations of AI-generated language tests is limited (Van Moere and Downey, 2016). This study seeks to address this gap by examining the unique challenges and opportunities faced by language institute lecturers when using AI to generate test questions and banks for English Language Teaching (ELT) exams. A key contribution of this research is the identification of AI’s dual role – not only as a tool for automating test creation but also as a partner in enhancing test content, reducing human biases, and personalizing assessments to meet diverse learner needs. The study provides novel empirical insights into both the practical benefits and the potential pitfalls of AI in ELT assessment, offering a balanced perspective on its integration into the educational process.

Language Testing and Assessment (LTA) is an integral part of educational systems, aimed at enhancing the quality and efficacy of language teaching and learning (Heaton, 1988). LTA encompasses designing and evaluating assessment procedures to monitor, grade, and assess learners’ knowledge and progress (Heaton, 1988; Vogt and Tsagari, 2014; Lan and Fan, 2019). Lan and Fan (2019) trace the origins of LTA to its role in identifying and bridging knowledge gaps among students, thus guiding them toward language acquisition. In contemporary education, it is essential for lecturers to align their assessments not only with theoretical standards but also with practical criteria to ensure that they accurately reflect learners’ capabilities and areas for improvement (Stiggins, 2006; Tsagari and Vogt, 2017). Despite the significance of LTA, challenges persist in aligning assessments with teaching due to unpredictable outcomes and various influencing factors, including resources, materials, and lecturer expertise (Bachman and Adrian, 2022; Lam, 2015). Recent empirical studies reveal gaps in the adequacy of test questions and their alignment with students’ language levels (Lam, 2015; Tsagari and Vogt, 2023). To address these challenges, LTA needs to be updated and innovated by integrating new factors and methodologies that reflect real-world complexities rather than relying solely on theoretical models (Bachman and Adrian, 2022). This evolution will lead to more timely and accurate assessments that address the gaps between learners’ abilities and instructional intent.

Artificial Intelligence (AI), as defined by McCarthy et al. (2006), involves machines created through scientific and engineering processes that can deliver valuable results across various fields, including education (Helm et al., 2020). AI systems are designed to simulate human thought and action within a virtual scope, enhancing various educational applications (Eaton et al., 2021; McCarthy et al., 2006). In education, AI has introduced innovative tools that significantly impact language teaching and assessment. For example, AI-powered tools like Grammarly and QuillBot assist in grammar checking and paraphrasing, respectively (Godwin-Jones, 2022; Zhang and Zou, 2022). Moreover, AI facilitates automatic grading and feedback on student work, streamlining the assessment process (Gardner et al., 2021; Borade and Netak, 2021; Yu et al., 2022). AI’s ability to generate a large volume of diverse test questions quickly improves the efficiency and effectiveness of assessment creation (Hinkelman, 2018). Research indicates that AI algorithms can produce varied question formats, such as multiple-choice questions, that cater to different testing needs (Killawala et al., 2018). Yunjiu et al. (2022) highlight that AI can create diverse test formats, which contrasts with human test creators’ tendency to produce repetitive questions based on memory. This capability underscores the potential of combining AI’s generalization with human expertise to enhance test content and format diversity (Chen et al., 2023).

At Van Lang University, the use of AI in test creation has become increasingly prevalent, offering numerous benefits in terms of speed and quality. AI systems at the university generate customized test items based on extensive educational resources, syllabi, and learning goals, thereby streamlining the test development process (Lee and Kim, 2020; Binh et al., 2024). AI- driven platforms enable lecturers to create diverse and high-quality testing items, aligning with curricular objectives and saving valuable time (Johnson et al., 2019). Furthermore, AI can tailor test questions based on student performance data, providing personalized assessments that enhance learning outcomes (Alshammari and Al-Enezi, 2024), at the same time, noting that Algorithmic biases and data privacy issues must be addressed to ensure equitable and secure assessments. Despite these advantages, several challenges accompany the use of AI in test creation. Concerns about the fairness and validity of AI-generated test questions, lecturers may need training to effectively use AI-driven tools (Rodriguez et al., 2022). The potential for adaptive assessments, which adjust difficulty based on student performance, represents a promising area for future development (Wang et al., 2021). The existing literature on AI in language assessment often highlights the technology’s potential to enhance the quality and efficiency of test creation. However, there remains a significant gap in empirical research concerning its practical applications and associated challenges. Current studies have not thoroughly explored how AI can address issues such as human-related biases, improve the richness of test content, and accelerate the test creation process.

In line with this aim, the following research questions were proposed:

• (1)

  How might artificial intelligence be used to develop English Language Teaching (ELT) test questions in terms of quality, content richness, and efficiency?

• (2)

  What challenges do lecturers encounter when using artificial intelligence to generate ELT test questions?

The present work is a case study, a kind of qualitative research, in the form of semi-structured interviews to explore the answers in open-ended to explore valuable and unexpected data from the participants. The purpose is to investigate the lecturers’ experiences, opinions, and attitudes about using AI-based evaluation in the context of creating exams. Through an in-depth examination of an individual’s experience, emotions, and perceptions, complicated social phenomena were to be explored, analyzed, and explained. Because case studies gather and analyze data from many sources, they enable researchers to comprehend a particular individual, group, or event. The careful examination of the data gathered and the researcher’s participation in group projects throughout the study was essential to the design.

The study included seven lecturer groups working at the Institute of Language, Van Lang University, which are located at Campus 3 (Known as Main Campus) and are ranked their English courses from Level 1 to Level 7; this data is presented in Table 1 below as well. A purposive sample procedure was utilized in selecting participants so that the objectives of the study were met. For each group, three full-time lecturers were assigned to teach courses number 27, 28, and 29 in the fall semester of the academic year 2023–2024.

To conduct this, strict inclusion criteria were applied to enhance the validity and reliability of the study outcome. It was an obligation to have qualified lecturers with a master’s degree or more, over 5 years of teaching experience, and be active members of the English Language Teaching (ELT) examination board. These criteria were designed on the assumption that well-experienced and trained teachers would provide a better and wider view of modern teaching methods and assessment methods according to the literature reviewed (Smith and Fletcher, 2020; Johnson and Johnson, 2020).

The purposive sampling method was considered suitable for this study, as it facilitated the selection of participants with specialized knowledge pertinent to the research objective (Palinkas et al., 2015). Such a sampling strategy was used because, in this study, its only function is to pick individuals who have expertise that is relevant to the focus of the research (Palinkas et al., 2015).

This was crucial for examining AI-assisted learning and assessment techniques, as well as assessing conventional and alternative testing methods within the educational framework. To safeguard participant anonymity, all personal data gathered during surveys and interviews were anonymized in accordance with ethical research standards (Bryman, 2016). Thus, this was very important when addressing AI enabled learning and assessment strategies, as well as testing conventional and alternative approaches for evaluating and administering.

During the research process, the researcher assures strict adherence to the study design requirements, which include the following:

Connectivity: The researcher constructs the survey questions and questionnaires in alignment with the study goals. The questionnaire primarily addresses the variables of ELT’s assessment process among participants.

Generalization: The study’s results should apply to a larger population because of the research’s extensive breadth. To sum up, this study’s results include broader demographic characteristics.

Feasibility: Every investigation is constrained by restricted resources. If the research design is beyond the available resources and data accessibility, it is impractical to conduct.

The data-collecting procedure for this research included many essential stages to obtain thorough insights into the use of artificial intelligence (AI) in the design of English Language Teaching (ELT) exams at Van Lang University. The interview questions asked are listed in  Appendix 2, depending on the interview, the order may be different. Most interviews were recorded on tape. Then, it was transcribed and checked by the authors. After that, a summary of each interview with each participant organized by themes and sub-themes was drafted. This study adapted a structure proposed by Gioia et al. (2013) and Jaskiewicz et al. (2015) to synthesize interview transcripts and generate main themes and sub-themes. Summaries of the individual interviews with informed consent forms were then sent to each participant for verification ( Appendix 1).

Semi-structured interviews were conducted with lecturers from the Institute of Language at Van Lang University. Each interview, lasting between 45 and 80 min, was recorded, transcribed, and then summarized to capture key themes and insights. The analysis process adhered to the framework proposed by Gioia et al. (2013) and Jaskiewicz et al. (2015), which involves a rigorous coding procedure to identify themes and sub-themes within the interview data. This approach facilitated a detailed exploration of lecturers’ experiences and perceptions regarding AI-assisted exam development.

To ensure consistency and reliability, interview guidelines were developed with open-ended questions designed to elicit comprehensive responses on AI’s role in test creation, including its impact on test item generation, grading, and feedback provision. After transcribing the interviews, the data was coded using an iterative approach: familiarization with the data, initial coding, theme identification, and synthesis (Braun and Clarke, 2006; Saldana, 2015). This process involved generating themes and sub-themes to reflect the lecturers’ viewpoints on the use of AI in English Language Teaching (ELT) assessments.

The thematic analysis was complemented by data triangulation, where examples of both traditional and AI-generated test materials were analyzed to provide a broader understanding of AI’s impact (Noble and Heale, 2019). Summaries of individual interviews were sent to participants for verification, ensuring the accuracy of the data representation and enhancing the credibility of the findings (Lincoln and Guba, 1988). This comprehensive approach not only illuminated recurring patterns and trends in the lecturers’ feedback but also provided actionable insights into the integration of AI in language assessment practices.

The data analysis for this study employed a rigorous qualitative approach, focusing on thematic analysis to uncover insights from the semi-structured interviews. The analysis adapted a structured process as outlined by Braun and Clarke (2006), which involved several key stages:

The initial step involved thorough reading and re-reading of the interview transcripts to become intimately acquainted with the content. This phase allowed the researchers to immerse themselves in the data and gain an initial understanding of the prevalent issues and patterns.

• (1)

  Initial coding: Next, the data were systematically coded. This involved identifying and labeling segments of the text that were relevant to the research questions. Codes were applied to pieces of data that represented significant concepts or ideas related to AI in exam creation. Focusing on specific items such as “AI efficiency,” “bias reduction,” and “content richness” which were used to tag relevant excerpts from the transcripts.

• (2)

  Themes searching: After coding, the researchers aggregated similar codes to form potential themes. This step involved collating all the data relevant to each code and examining how they could be grouped into broader themes. Meaning that, the codes related to “AI efficiency” and “time-saving” were combined into a theme reflecting the efficiency benefits of AI.

• (3)

  Themes reviewing: The identified themes were then reviewed and refined. This involved checking whether the themes accurately represented the data and whether they provided a coherent narrative. The researchers iteratively adjusted the themes and subthemes to ensure they captured the essence of the data.

• (4)

  Themes identification: Each theme was clearly defined and named to reflect its content and relevance to the research questions. Subthemes were also identified to provide a more nuanced understanding of the data. For instance, within the theme of “AI Efficiency,” subthemes such as “automated grading” and “test generation speed” were established to detail specific aspects of AI’s impact.

• (5)

  Interpretation and description: Finally, the findings were synthesized into a comprehensive report. The themes and subthemes were discussed in relation to the research objectives, and illustrative quotes from the interviews were used to support the analysis. This approach ensured that the interpretation of the data was grounded in the participants’ experiences and perceptions.

By following these steps, the study ensured a systematic and transparent approach to analyzing the interview data, which provided rich insights into the role of AI in language assessment and addressed the research questions effectively.

After the interviews, most interviewees agreed that Artificial intelligence should be used in ELT exam creations because of its overt advantages.

Some of the key features that the authors have listed in Table 2 above clarify the creative characteristics of AI in language testing illustrate the crucial benefits that AI provides are much higher than its limitations. Particularly, the data has elucidated the two sides of AI-based test creation:

The results indicate several ways that AI might be useful for the education sector’s English Language Teaching (ELT) exam question preparation process:

The process of creating test questions for English Language Teaching (ELT) examinations can be automated with artificial intelligence (AI).

In supporting this, Johnson and Lee (2022) give opinions that thanks to Natural Language Processing (NLP) algorithms, AI is able to evaluate vast corpora of text in order to recognize pertinent linguistic patterns and create a wide variety of test questions that are contextually acceptable across a range of language skill levels.

The adaptive testing systems driven by AI can modify test materials, serving individual in practicing skills to upgrade their learning process with appropriate levels.

To create a tailored testing experience that effectively tests learners’ language abilities and delivers targeted feedback, artificial intelligence may continually analyze learner replies and modify the difficulty level of questions (Garcia et al., 2023). This allows AI to provide personalized testing questions on student capacity.

Assessment systems that are based on artificial intelligence can provide learners with rapid and thorough feedback, which assists them in understanding their areas of strength and improvement.

Group lecturers A, C, and G expressed clearly that “the feedback and correction sometimes provided by the AI is incredibly detailed, highlighting not just the correct answers but also explaining why certain responses are incorrect”.

Therefore, AI is able to give insights into learners’ language skill levels, grammatical faults, and vocabulary use via automated grading and analysis of replies (Chen and Smith, 2021). This enables more effective learning tactics to be implemented.

By identifying inconsistencies, biases, or ambiguities in item development, artificial intelligence has the potential to contribute to implementing quality assurance measures for English Language Testing (ELT) questions.

Artificial intelligence, therefore, has the ability to detect problematic items and advise adjustments to improve the validity and fairness of assessments (Brown and Nguyen, 2020). This is accomplished by employing machine learning algorithms to examine test item data and discover learner responses. According to Taylor et al. (2019), these dynamic assessment formats have the potential to engage learners effectively and give a more genuine measurement of language competency.

Using AI to create ELT test questions presented several obstacles, which the participants pointed out and be summarized in these key viewpoints below:

Participants were worried that AI algorithms would not be able to handle creative or contextually sensitive item-generating activities because of the richness and subtlety of language. Artificial intelligence systems may generate artificial or incorrect test items due to their inability to comprehend idiomatic terms, cultural subtleties, and nuanced language nuances. Additionally, as the notes indicated, sometimes the questions were found to be too basic or complex for the intended audience

Problems with Access to AI Tools, Inadequate Training, and the Integration of AI Systems into Current Evaluation Frameworks are Examples of Technical Limitations. Participants emphasized the need for specialized expertise and technical support to effectively utilize AI in ELT exam creation, highlighting resource constraints and infrastructure limitations as key concerns (Chen and Smith, 2021)

Participants who took the survey were worried about the possible biases and ethical ramifications of the AI-generated evaluation materials. Concerned about the potential for AI- AI-generated material to reinforce cultural prejudices and stereotypes, they stressed the need for inclusive, fair, and culturally sensitive test item development. In other words, AI systems may be unable to consider the cultural and linguistic context of the questions they generate, and they can often produce questions that might be biased towards specific groups, which require intervention by teachers or human experts.

This can be particularly problematic if the questions generated are used in an assessment context, as the bias can lead to unfair results, especially in high-stakes nationwide or worldwide examinations.

Although AI (OpenAI) assigns all the rights regarding the use of the output generated by the AI to the user and states that it will not claim copyright over content generated by the API (OpenAI, 2023), participants emphasized the need for human validation and control. Their main point was that AI-generated test items still need human evaluation and feedback to ensure they follow pedagogical principles, learning outcomes, and curricular goals. Further discussions and ethical issues might also exist even when human experts revise or alter these questions, which was also voiced in other studies (e.g. Adams et al., 2022).

Considering technical, linguistic, ethical, and pedagogical factors is crucial for maximizing the benefits of AI while minimizing its drawbacks, as the results show that the challenges of using AI to create ELT exams are complex and multi-faceted.

The research examined seven cohorts from the Institute of Language at Van Lang University, corresponding to English competence levels 1 through 7. Participants were chosen by purposive sampling to guarantee agreement with the study’s aims. Each group had three full-time lecturers tasked with teaching courses 27, 28, and 29 during the Fall semester of the 2023–2024 academic year. Stringent inclusion criteria were used to augment the validity and reliability of the study results. Qualified lecturers were mandated to have a master’s degree or higher, possess at least five years of teaching experience, and be active members of the English Language Teaching (ELT) test board. The criteria were formulated on the premise that seasoned and highly trained educators are more inclined to provide nuanced perspectives on advanced educational techniques and assessment frameworks, as shown by previous research (Smith and Craig, 2022; Johnson et al., 2021).

From these reliable standards, this research looks at language evaluation which provides educators and students the opportunity to examine pedagogical methods, identify challenges, and modify instruction and learning to accommodate the distinct requirements of individual learners. Another aim of this research is to investigate the opportunities and challenges associated with the use of AI for the development of an English Language Test. The findings indicated that artificial intelligence facilitates the development of ELT assessments, specifically for groups of lecturers at the Institute of Languages at Van Lang University, which was the primary objective.

The extensive data and description-gathering in findings are expected to provide significant discoveries, enhancing the knowledge of how seasoned instructors use AI-driven technologies in their teaching and evaluation methodologies. The implementation of artificial intelligence faces many challenges, including linguistic complexity and subtlety, technological limits, ethical and prejudice issues, the need for human oversight, and the need for validation. These difficulties culminated in the determination that human intervention remained essential to guarantee the reliability and quality of the AI-assisted English Language Test. According to Ji et al. (2023), educators use AI to enhance their instructional methods, but AI does not entirely supplant instructors in guiding students throughout class, irrespective of the classroom’s conventional or contemporary nature. Simultaneously, issues like over-dependence on AI and less human connection in education must be addressed to preserve a balanced and comprehensive learning experience.

Nonetheless, it is an undeniable fact that any research has inherent limitations, and the authors’ work is no exception. Interview groups may be susceptible to researcher bias because of the limited breadth of lecturer connections. Moreover, the study was limited to the data provided by participants and available publications, which may not represent the whole of the situation. This is because the study was constrained by its temporal scope, which may not have adequately represented the intricate interactions among various participants, including the application of AI in formulating questions for reading assignments, such as the automated generation of pre-reading inquiries and corresponding answers (Attali et al., 2022). Future research can shed light on AI-generated content in any aspects that AI can interfere with, especially in language teaching and testing, and raises significant legal issues such as transparency when using AI, particularly in the context of developing ELT examinations. This has emerged as the subsequent issue that authors must focus on.

To sum up, this research highlights the possible long-term effects of AI on instructors and students. AI offers educators the potential to improve their teaching effectiveness by automating repetitive duties like grading and providing feedback, so enabling them to concentrate on innovative and engaging instructional strategies. It underscores the need for ongoing professional development to properly incorporate emerging technology.

The growing dependence on AI prompts ethical issues, especially about transparency and prejudice, necessitating that educators rigorously assess AI systems. AI has the potential to transform education by providing customized learning experiences that cater to individual requirements, hence enhancing engagement and accessibility, particularly for language learners with special needs. However, to assess the opportunities and challenges of AI systems in the new area, it needs more analysis in universities to extend the scope of participants in contributing and implementing how to use AI more effectively.

Table A2