Instructional technology in a time of crisis: Cases of The Center for Pedagogical Measurements

The students’ selection is one of the high-stake procedures, which is focused on the identification of students’ abilities to study science and mathematics by assessing subject knowledge, functional literacy, and language competencies.

The government of the country took the decision to lockdown in Kazakhstan the day before students’ selection process launch. By this time, all the materials, schools’ staff involved in test administration were ready and CPM test coordinators had already arrived at the testing venues. As a result of lockdown, students’ selection procedures had undergone significant changes.

First, considering the number of applicants (20,084) and in order to build social cohesion among all stakeholders, we started discussions with the government of the country about actions to be taken. In this circumstance of uncertainty, when the duration of such emergency measures is likely to continue for a certain time, decision-making was difficult. The only decision that was appropriate was to postpone the dates of the test to an uncertain period.

Furthermore, we understood that cancellation of this procedure would lead to the loss of opportunity for those applicants to get access to one of the best schools in the country. Namely, for those who are from economically or socially disadvantaged families “such loss may have negative effects on productivity growth, innovation and employment, including future lower earnings for the student cohorts directly affected by the lockdown” (Di Pietro et al., 2020).

Secondly, communication plan had to be revised within short time frames to address to all stakeholders. This included posts in NIS’ official website and all official social network accounts, telephone calls to all the applicants’ legal representatives; appointing the responsible staff to invigilate at schools at the scheduled day to inform those who could come.

Rescheduling the students’ selection to later dates could bring such risks, as the beginning of the school year behind time and psychological issues for applicants, which might cause burning out of their motivation. In this regard, there was a need to rearrange the forthcoming students’ selection procedure to reflect new challenges. CPM had to ensure the quality, validity, transparency, and safety of all procedures in the students’ selection.

In March, the revision of the stages of students’ selection process began. All concerned stakeholders who were involved in the follow-up selection process should have been aware of their functionality during the testing in new conditions. An update of the instruction for the test coordinators and for all school staff who were involved, identifying their roles and responsibilities in the selection process were required.

As the World Health Organization recommended to maintain a physical distancing between people in the room, conditions of the test were changed. Namely, the sitting plan in testing rooms had to be changed. Thus, it was necessary to recollate materials of the test, since each material was intended for a specific applicant. This was mostly done in April.

The following stage in May involved dispatch of test materials to 20 schools. In this process, there were difficulties with the delivery of materials to the cities. This challenge was overcome by well-coordinated work between CPM employees and the logistics company. As a result, exam materials were delivered on time to the cities in a secure condition.

One of the main stages was conducting the selection of students in 20 schools in June in compliance with sanitary and antiepidemio-logical standards by all participants. In this regard, rules for ensuring sanitary and epidemiological measures were developed in accordance with the recommendations of the World Health Organization. 20 test coordinators, and 31 assistants were involved in administering the selection process. There was also an online instruction for the Intellectual Schools’ staff. As a result, the students’ selection was successfully conducted in compliance with all sanitary and anti-epidemiological standards.

Statistical and psychometric processing of the test results were carried out online in July together with the strategic partner CITO, Netherlands.

All the above-mentioned measures did not have any negative effect both on applicants’ performance on the test and their health state. As a result, in August 2, 952 applicants were admitted to Intellectual Schools to study in the seventh grade of Nazarbayev Intellectual Schools.

According to Di Pietro et al. (2020), COVID-19 may affect students negatively both in acquisition of cognitive and non-cognitive skills, and may have important long-term consequences in addition to the short-term ones.

We understood that ongoing assessment of students’ learning achievements contributed to the development and maintenance of students’ skills. However, during the time of lockdown, students are likely to suffer from the lack of ongoing feedback from their teachers, and this can result in learning loss. Studies show that school closures could result in significantly lower achievement for students. However, Organisation for Economic Co-operation and Development research shows that through continuous assessment, it is possible to maintain acquisition of skills by students as well as their interactions with teachers (Organisation for Economic Co-operation and Development, 2020). Thus, several actions were taken to respond to this challenge.

In March, the holidays at schools started a week earlier than usual. As a response to this measure, we had to invest our time in contingency planning, namely, a great deal of effort was put forth to develop documents and guidelines for schools, to implement technologies and put them in place, and adapt to the new learning environment.

The ongoing assessment of students’ learning achievements was adapted to the conditions of new learning environment in Term 4, as “the landscape of classrooms changed significantly” (Tarasawa, 2020). Furthermore, digitalization of everyday classroom instructions changed the ways of delivering formative and summative assessments. In this regard, teachers were informed and instructed via live webinars about the actions taken by CPM, and they received explicit guidance on how to assess in new context.

In April we released recommendations for teachers on how to make assessment virtually available and how to organize and deliver formative and summative assessments during the period of distance learning. Policies and protocols for organizing and conducting ongoing assessment of students’ learning achievements were revised as well. It was decided that several subjects with less contact hours should not be assessed summatively, to balance students’ study load. Also, the time allocated for summative assessment was reduced across all subjects and grades. Consequently, these affected the content of 90 test specifications for Term 4 summative assessments.

Furthermore, we had to determine the needs of the schools in terms of ongoing assessment. In this regard, research on distance learning was carried out in April by the research department with participation of teachers, students, and their parents from 20 Intellectual Schools. The results and feedback received on assessment showed general satisfaction of the respondents. In their feedback, some students asked to increase the time for completion of summative assessment assignments or weaken its requirements due to the current circumstances. Most teachers were concerned about the issue of external summative assessment and what the procedures and the format would look like. Teachers were also interested in how the principles of academic honesty could be monitored during the assessments.

Reflecting students’ and teachers’ needs, a series of subject-specific webinars were conducted as a support for NIS teachers on how to organize summative assessment, to develop test items aligning with distance learning, to use data from assessment to improve students’ skills to ensure academic honesty in a remote environment.

All Term 4 summative assessments were conducted in online format in May. To ensure the quality of these assessments, CPM externally reviewed summative assessment papers set by Intellectual Schools’ teachers. Microsoft Teams platform together with Learning apps, Google Forms, Microsoft Forms, as well as other technical solutions like scanned copies, photo sent via email and other messengers were and could be used to carry out the assessments.

Analysis of progress and quality of knowledge done in June at the completion of Term 4 and academic year showed that there was no decrease in performance results of students. This was mostly achieved because of reinforcing the role of the formative assessment practice in schools and incorporating to the timetable additional consultations for small groups of students or individual ones.

July and August were the time for creating “restarting plan for schools” (Tarasawa, 2020), preparing documents and procedures for the new academic year. Summative assessment specifications, Rules and Guidelines for assessment were revised taking into account the assessment experience from distance learning in Term 4.

Schools did not open in the new academic year, so we continued Term 1 in distance format in September. In this regard, CPM is implementing coordination of formative and summative assessment under the conditions of distance learning.

We understood that COVID-19 would probably last a while, thus it was very important for NIS to find the tool to remotely identify students’ learning achievements and the gaps in order to incorporate more scaffolding. Studies show that less time spent in learning can lead to learning loss (Kuhfeld & Tarasawa, 2020). Notwithstanding the shift to online learning and teaching, monitoring of students’ learning still needs to be conducted (UNESCO, 2020). Empirical studies showed the effectiveness of curriculum-based measurement as a reliable indicator of student progress in reading and math, and it can have a positive impact on further students’ learning achievement (Foegen et al., 2007; McMaster & Espin, 2007).

It was difficult to understand what virtual instruction and missing contact hours of teaching and learning may mean in terms of students’ achievements, and it was more difficult to identify learning loss of students without appropriate tools to measure. According the reports of the United Nations (2020), learning loss will have long-term effect in students. We instantly reacted to this situation and at the end of March, we began internal conversation with the information technology department, schools, and managing team about how to transform current tools of monitoring that are in place to virtual ones.

The following two measures were designed to help all stakeholders to identify learning trajectory of each student during the time of lockdown:

• monitoring of students’ level of functional literacy within three domains: mathematics, science, and language, as a prerequisite for the secondary school external summative assessment;

• monitoring of Grade 7–12 students’ learning achievements on the topics that were taught in Terms 3 and 4 of 2019–2020 academic year.

Both measures were taken to support schools in filling the gaps in students’ learning experienced during the lockdown. Afterwards, we came to a decision that for monitoring both we needed an item bank that would be friendly with technologies, thus in March we immediately started to develop test items as well as online platforms to deliver monitoring for students.

For the first monitoring, 500 items were developed, and the online testing was launched at the end of April for 2,070 students. Schools were provided with 6,208 reports that comprise detailed information on students’ achievements across all items tested.

Additionally, schools were provided with online simulator where students and teachers could work together, and teachers could scaffold students by exercising with released items.

For the second monitoring of students learning achievements toward expected outcomes of the curriculum, namely during the period of lockdown, 2,720 new items were developed and reviewed by CPM experts. In addition, they were reviewed in terms of language equivalency and migrated into online platform. At the beginning of September, logins and passwords for 14,929 accounts were generated and sent to schools; students were able to take the tests from their home using their laptops or personal computers. After the test, students’ results were processed statistically and individual reports and reports at the class/school level were generated. These reports were distributed to all stakeholders (school administration, teachers, students, and parents).

Furthermore, NIS teachers were provided with explicit guidelines and trained remotely how to interpret and use data given in the reports to integrate them with teaching and learning. Also, data from these reports allowed teachers to use a variety of instructional techniques to move students progressively toward expected outcomes of the curriculum, to reach higher levels of comprehension and skill acquisition, and to bridge learning gaps.

This emergency action in getting schools to maintain the standards of learning showed that both monitorings were tremendously helpful for schools and could be implemented several times to provide individualized support helping teachers and understand why students may not be progressing. In addition, this approach could be considered as one of the ways of parents’ engagement as it sought to support not only students and teachers, but also parents by informing them via reports, as parents play a significant role in the success of students learning at home.

Over the last 6 years, Grade 10 and 12 students of Nazarbayev Intellectual Schools annually take an external summative assessment (ESA).

Grade 10 ESA is administered by CPM, and students are awarded Secondary school certificate (attestat).

Grade 12 ESA is administered jointly with Cambridge Assessment International Education (Cambridge International) and is equivalent in standards to International AS/A Level. After passing examinations, graduates of the Nazarbayev Intellectual schools are awarded a certificate that is recognized by 18 Kazakhstani universities, as well as by international universities in the United Kingdom, Korea, China, and international organizations (UK National Recognition Information Center, Universities and Colleges Admissions Service, Netherlands Education Support Office).

However, due to the pandemic, in order to ensure the safety of students, Grade 10 ESA was canceled by internal decision, and Cambridge International also decided not to hold AS/A Level examinations in May–June 2020 in all countries, including Grade 12 ESA in Kazakhstan.

This unexpected decision triggered misunderstanding among our stakeholders, thus NIS leaders swiftly designed a response to this challenge to protect young graduates and give them an opportunity to be admitted to universities. In the report of the World Bank, 2020 it is also noted that matriculation exams are critical to the future of students, especially those from disadvantaged groups. Thus, NIS in cooperation with Cambridge International were working hard and scrambling since April and started to invest time to the development of the method of calculation graduates’ grades. Starting from April both teams put all the effort to create a robust statistical method to standardize the award of ESA certificates based on the students’ performance throughout their period of study as well as teachers’ professional judgment. This was done to maintain the standard of the ESA and ensure the comparability of the current year certificates with the certificate of previous years’ graduates.

To implement this change in policy, NIS formed contingency plan with detailed actions to ensure common vision and understanding across all Intellectual schools’ network. Schools were provided with instructional materials on how to collect data and rank students. During April and May, CPM staff had supported schools remotely, even after the release of the ESA grades. Within 2 months, data were gathered and analyzed afterwards, and within another month, grades were awarded and released.

Fortunately, this approach showed its effectiveness in times of crisis. The method that was used to standardize grades was the most robust system for producing grades that could have been implemented in this circumstance. The method has ensured that the standard of the Grade 12 ESA has been maintained between years and is equivalent to the standard of International AS/A Level from other years.

This method also helped to maintain comparability with international examinations and ensure the possibility of entering higher education institutions in the country and abroad. Generally, 2,044 students were graded according to this method, and 417 of them were admitted to international universities abroad, 1,143 students were admitted to Kazakhstani universities, and 602 were admitted to Nazarbayev University, and eight students decided to take a gap year.

In regard of examinations for national graduation certificates, there was a shift to technologies, and they were held online by using computer-based testing with proctoring system, which allowed recording students by a video camera and observing them throughout the examination session. This system also allowed live communication with students if any issue arose during the exam via instant messenger tool (more than 200 messages were sent to students). It worked well and seems to be very good alternative for inspecting students during tests and both cost-effective and time efficient.

There is saying that “disaster is a laboratory for innovations.” Therefore, in this unprecedented case, NIS sought to establish policies and support students and teachers. Also, as result of these changes now a new procedure for grade prediction is in place and has been implemented for the new academic year graduates since September.

Teacher appraisal is also one of the high-stake assessment procedures that helps to identify the capability and effectiveness of the teacher in delivering classroom instructions or work-related needs, and it is critical in documenting teacher performance. This procedure seeks to improve teacher’s own practice by identifying strengths and weaknesses for further professional development and ensure that teachers perform at their best to enhance student learning (Organisation for Economic Co-operation and Development, 2009). Thus, teacher appraisal in NIS is a judgmental process, which comprises three stages: evaluation of teacher performance at school by school administration; independent assessment by CPM experts of teacher’s reflective report based on the delivered lesson; appraisal committee meeting where teachers present results of their experience during appraisal cycle. Afterwards, the appraisal committee makes a final decision by triangulating the results of all stages.

COVID-19 caused cancellations and suspensions of in country business travels, that made it impossible to hold face-to-face appraisal committee meeting and manage other processes. Whereas, CPM had to face the specific issues, as to administer the procedure according to established policies and time frame, and to keep the previously set standards. Therefore, there was a need to rethink teacher appraisal procedure to meet the needs of the new context, which resulted in their partial transition to the remote mode.

In late March and early April, we put our efforts on revision of procedures and rebuilding of schedules. The key to effective appraisal is getting the balance right between assessing performance and assisting personal development (Elliott, 2015); therefore, lesson observation sessions were transformed into the remote methodical support followed by webinars with question-and-answer sessions. Webinars were delivered for teachers on writing reflective reports focused on the content of subject and complying with ethical code.

In addition to other supports, teachers’ reflective reports are formatively assessed by school trainers, who have special expertise in this area and are capable to conduct lesson observation and provide professional feedback to teachers. The process of formative assessment of teachers’ reports had to be carried out remotely, using shared online platforms. The final reports had been uploaded to CPM platform to be assessed summatively.

In May, the second stage of appraisal was delivered remotely in a regular way, thus, it did not cause any additional challenges. All the essential processes ran smoothly; 406 teachers uploaded their reports to CPM, which were checked using Turnitin service to prevent plagiarism. Whereafter the reports were downloaded, encoded, and distributed among the experts for independent assessment. By encoding author’s personal data, we maintained confidentiality and objectivity of the assessment at the appropriate level. From that moment until the final decision was made, the experts had worked only with encoded materials.

To ensure reliability and validity of the scores of assessed reports, they were moderated by the group of experts. The online implementation of this procedure caused certain challenges in communication among experts in making a group decision. In this regard, the team leader of the group of experts was appointed, who coordinated the distribution of workload and responsibilities of experts and recorded the decisions made. The results of the independent assessment and feedback for further improvement of teachers’ practice were provided timely.

Also, in May CPM held standardization meeting via Microsoft Teams platform for the members of appraisal committee, on approaches and criteria to assess teachers’ presentations. The members of the appraisal committee were familiarized with the key aspects of teachers’ presentation that were worth focusing on. As a result, there were eliminated risks that could negatively affect the adequate perception of the teachers’ speeches and the objective assessment of their practice.

In June appraisal committee meetings were first-ever held in distant mode, so more than 300 teachers could make their presentation either from home or school.

Analyzing of all the stages of teacher appraisal, it can be concluded that all the assessment and methodological support procedures were transferred to a remote format without a loss in quality and objectivity. Despite the new conditions of holding the appraisal committee meetings, more than 80% of teachers have successfully completed appraisal. The results of the appraisal in current year are comparable to the results of previous years. This formed an understanding that the further teacher appraisal procedure could be partially delivered remotely.

In July–August, CPM experts in cooperation with teachers, who completed appraisal with distinction, prepared video a collection to support teachers in organizing teaching and learning practice in virtual environment. As stated in a Organisation for Economic Cooperation and Development report (2012), to improve teachers’ practice, it is important that feedback is followed up with appropriate professional development. Hence, more than 200 trainers were instructed via online courses to provide teachers with professional support in terms of new cycle of teacher appraisal.

Based on the results of the work carried out during the last half of the year under COVID-19 conditions, the following conclusions can be derived:

• CPM possessed enough technical, technological, expert, and human resources to redesign the main assessment procedures in the new mode at a sufficient level. That, in fact, would not have been possible in and of itself if the NIS was not sufficiently flexible and open to making innovative decisions in new realities. The decisions were diverse and were driven by different contexts, and there wasn’t a single universal response across all the cases: canceling procedures and grading students based on existing data; postponing and adapting assessment procedures; and delivering procedures in a remote environment.

• Comparability studies of current and previous years’ assessment procedures show that the results are comparable, fairly objective, and the procedures align with the set standards. We may conclude that the rapid and large-scale transition to distance learning has identified not only the challenges for educational organization, but also perspectives for further development, as well as opportunities for rethinking educational practices and policies.