Library assessment and the COVID-19 pandemic: a practical exemplar

The University of Illinois Urbana-Champaign (UIUC) is a public research university in two adjacent cities, Urbana and Champaign, in Illinois, 140 miles south of the city of Chicago. Established in 1867, it currently has 14 colleges and schools and offers more than 250 undergraduate, graduate or professional degree programs. Based on the data from spring semester of 2025, the enrollment was 46,984 full-time and 9,635 part-time undergraduate, graduate or professional students. The UIUC also offers a variety of fully online programs, certificate programs (e.g. business analytics, civil engineering or communication and health management) and non-degree programs. UIUC has 7,640 faculty and academic staff, of which 3,946 are instructional faculty (including tenure-system faculty and specialized faculty).

In the field of science and information technology, UIUC is best known for its top-ranking engineering and computer science programs, the #1 Library and Information Science master program (MSLIS) and distinguished researchers and alumni who invented LED, the first computer-composed music, the first graphical web browser Mosaic and YouTube.

The University Library at UIUC (UIUC Library) is a campus-wide network of physical and virtual libraries and services that support learning and research in many disciplines. The Library houses more than 15 million volumes (physical and electronic) with millions of users passing through library buildings and services each year. The Library is considered to be one of the top 5 academic research libraries in the United States in terms of the collection size. As a public university library and a pioneer in open access, the Library not only provides access to its print and electronic collections to the University students, faculty and staff, but also to users and libraries around the world ^[1]. The Library also has in-house digitization services that focus on reformatting paper and print collections for preservation and access. During the fiscal year 2024 (July 1, 2023 to June 30, 2024), the Library made over one million books publicly available from Google Books, Internet Archive or HathiTrust.

On March 11, 2020, the World Health Organization (WHO) assessed the COVID-19 outbreak as pandemic. Universities and libraries had to face many new challenges brought on by the pandemic – economical, social, health and mental health related; and people continue to face many challenges as the pandemic recedes. UIUC and the UIUC Library are no exceptions.

In Illinois, all public and non-public schools were ordered to close on March 17, 2020 (initially through March 30), and the statewide stay at home order took effect on March 21. At the University of Illinois Urbana-Champaign, all students were asked to move to off-campus housing after the school closure. On March 23, the University began alternative online course delivery. “Alternative” course delivery meant no face-to-face instruction could be offered, and all instructors were expected to teach scheduled courses online, synchronously or asynchronously.

The university campus was quiet without students. Many planning and resources went into the planning of the 2020–2021 academic year. The University decided to provide instruction via a mix of in-person and remote learning, with a free, twice-a-week testing (SHIELD Illinois ^[2]), masking mandates, social distancing measures and limited indoor capacity. However, less than two weeks into the Fall 2020 semester, an unexpected upswing of COVID cases prompted the University to order a brief lockdown during September. Students who lived were required to stay in their university or local off-campus housing. In December 2020, vaccines started to become available. For the 2021–2022 academic year, all students, faculty and staff at the University were required to be vaccinated, or they would be required to complete testing on-campus. On February 28, 2022, the indoor masking mandate in the State of Illinois ended. And on April 10, 2023, the United States federal government’s COVID-19 national emergency declaration also ended.

There has been an increased interest in examining pandemic responses from academic and academic libraries. For example, The HathiTrust ^[3] Emergency Access Digitization service (hereinafter referred to as ETAS), supported eligible member institutions by providing reading access to in-copyright work held in library print collections was launched on March 31, 2020 (Fulkerson, McIntyre, & Stewart, 2020). JSTOR ^[4] expanded access for subscribing institutions; users who were not affiliated with any subscribing institutions were also allowed access to JSTOR public health journals, COVID-19 literature and other resources for teaching and learning.

Individual libraries’ pandemic responses and lessons learned have been presented in the literature. Mehta and Wang (2020) and Michalak and Rysavy (2022) described the shift to digital services and collections during the pandemic. Munip et al. (2022) performed a meta-synthesis of nine studies conducted across all Pennsylvania State University campuses. It presented the usage of space, services and resources; and employee and student perspectives during Fall 2020 and Spring 2021. In addition, assessments of academic library resource use across institutions of various sizes (Goldberg, 2024; Connell, Wallis, & Comeaux, 2021), the difficulties faced by library workers (Bury, 2024; Medina-Hernandez, 2024; Shin, Pine, Martin, & Bardyn, 2022; Carter, Andersen, Turner, & Gaunt, 2023) and the changes that occurred within specific library types (Biddle & Gowen, 2023; Zaugg et al., 2021) represent not only the impacts of the pandemic, but also the responses. In addition, discourse by academic and research libraries present reports and research to understand strategies to support student wellness (ACRL Research Planning and Review Committee, 2024; Johnson, 2023; Brewster & Cox, 2023).

The UIUC Library swiftly mobilized after the State’s stay at home order and the University’s announcement of alternative course delivery. A response team was formed to oversee planning and communication to library staff. Library facilities were closed to all users on March 20, 2020. Remote work for all employees was encouraged when job duties allowed and with supervisor approval. For employees whose work could not be done remotely, the Library supported accommodation for those who were willing to work on site as well as for those who chose not to report to work.

On April 1, 2020, only a few days after the closure of the campus, the UIUC Library enabled HathiTrust ETAS. In May, the Library shared a “Return to On-site” Work Plan with the University. The plan established how the Library should support the following services: fulfillment; reference, research consultation and instruction and acquisition of library materials. The UIUC Library established an “e-first” strategy, though not “e-only” (Prom, 2020). The Library implemented a contactless print delivery system that used lockers and followed social distancing guidelines in order to allow the patron to arrive in person and obtain their materials (Weible & Sander, 2020). Rimkus et al. (2021), colleagues who created the Emergency Access Digitization service at the UIUC Library, described how the Library “used existing staff roles and the existing digital library infrastructure to offer on-demand digitization and limited online access to patrons working remotely.” From July 20, 2020 to April 20, 2021, the service digitized items not available in the HathiTrust ETAS upon faculty or students’ requests and made the items available for online viewing in a restricted manner.

The Association of Research Libraries (ARL) ^[5] defined “library assessment” as “Describe and measure the performance of research libraries and their contribution to teaching, research, scholarship, and community service” (Kyrillidou, 2005). This definition has been widely recognized since the early 1990s by library and information professionals working at different types of libraries.

In some ways, library assessment work since the COVID-19 pandemic has not changed. The goal of library assessment is still to “describe and measure the performance.” Libraries continue to use quantitative (e.g. statistics) and qualitative (e.g. focus groups) methods to assess the usage of collections and services as well as user needs and perceptions. Table 1 shows UIUC Library statistics two fiscal years before and after the pandemic started (FY21). It can be observed that the usage of library spaces (Visitors) and some library services (Items Circulated and Reference Questions Answered) decreased significantly during FY21, the use and maintenance of electronic resources (Books Digitized and Electronic Downloads) remained stable.

However, library assessment faced new challenges arising from the pandemic. First, libraries have faced fiscal challenges since the pandemic started. Frederick and Wolff-Eisenberg (2020) found that 75% of library directors who participated in the Ithaka S + R US Library Survey have experienced reduced budgets during 2020. The UIUC Library was no exception. University faculty and administrators challenged the UIUC Library with inquiries like “Students are all online. They shouldn’t pay for the Library/IT fee” ^[6]. As a result, the demand to collect data and evidence to “describe and measure the performance” heightened.

Another challenge centered on library workers’ needs, the individuals who were working on outreach and engagement and affected by the changing pandemic responses. Some of the important questions we heard from UIUC Library staff were “How are we able to provide library services in a climate where everything is changing regardless of if we can meet in person or not?” “How has COVID-19 impacted in person academic library engagement programs and events?” “How has COVID-19 impacted library workers who provide these services?”

To respond to the challenges listed above, many assessment activities took place across the UIUC Library. Some activities were designed to address immediate or short-term needs related to the pandemic. For example, like many private and public organizations, the Library conducted a pulse survey, “Library Stay-at-Home Period Survey,” in order to understand staff’s well-being and needs during the beginning of the pandemic. The survey was conducted only once in summer 2020.

On the other hand, some assessment projects have continued to be valuable four years after the pandemic started. We present three such assessment projects in this case study paper. Each one was designed to respond to the pandemic challenges from a different perspective:

1. The user perspective: Library Service Satisfaction Survey,

2. The library worker perspective: library engagement during COVID-19, and

3. The data perspective: EZproxy logs and Wi-Fi logs.

At the height of the pandemic, students at the University of Illinois Urbana-Champaign had to adapt to alternative research, teaching and learning methods. Compounding the challenge, these alternative methods could change abruptly if coronavirus cases surge locally. Initially, the goal of the Library Service Satisfaction Survey was to use a short online poll to assess the students’ usage and attitudes towards library services and resources during campus closures and remote learning. Although it was designed with users during the pandemic in mind, it has become a standard semi-annual assessment project that focuses on how students engage with library services, resources and spaces.

The Library Service Satisfaction Survey was first administered in fall 2020, the semester after the COVID-19 pandemic began. At the time of the publication of this article, the survey has been conducted nine times. All survey administrations completed the university’s human subject research review and received institutional review board (IRB) approval (exempt status). Due to a University of Illinois Urbana-Champaign policy and processes governing electronic surveys and questionnaires, we were permitted to distribute the survey to a randomly selected sample of students for each administration. The first two administrations of the survey were distributed to 10% of the undergraduate students (over 18 years old) of each level (freshman, sophomore, junior and senior); and 10% of the graduate or professional students.

As the pandemic continued on, two issues emerged in higher education. First, many universities adopted remote or other forms of non in-person learning. The number of emails from schools and professors that students received increased. However, student engagement in academic work decreased due to issues related to mental health, social isolation, remote learning and other obstacles. Second, observations and reports that the pandemic has impacted college students of certain demographic groups differently have caught our attention. For example, a UIUC Student Affairs study found that “Latinx students reported more obstacles to transitioning to a new education context during COVID-19” and “international students had more concerns about access to food, housing, medical care…” ^[7]. This led to a modification in the sampling strategy for some administrations of the survey with the goal of collecting information from the demographic groups that have been most affected by the pandemic. The updated sample strategy (2,500 participants for each semester) is explained in Table 2.

The survey questionnaire (Appendix I) was developed by the Library’s Library Assessment Team with input from library administrators and an internal Library Assessment Committee. The questionnaire has been updated slightly over the last four years to reflect different stages of the pandemic, however, the structure of the questionnaire remains the same. It has three parts: environment (where participants live and how they take classes), use of library services and collection and questions that can collect data about ongoing library services.

The survey is administered on a UIUC online survey system, Webtools Forms ^[8]. The system automatically records any participant’s NetID (unique usernames assigned to those who affiliate with the University), college and degree information (all participants were informed of this data collection). Microsoft Excel is the primary tool used for survey results analysis. Table 2 compares the first (October 13–20, 2020) and the most recent administration (March 18–29, 2024). We can see from the first administration, which was during the beginning of the pandemic and before vaccines were available, 95% of participants had all or most classes online. And that number was down to 18% as of March of 2024.

It is important to note that while there were campus building closures and social distancing guidelines during the first academic year (2020–2021) after the pandemic started, many participants (70%) chose to live on campus (in residence halls) or in a local community (in houses or apartments that were close to the University). The impact was that many students looked to the Library to support their needs for a study space. One survey participant said: “I would love [it] if the library would open up more study spaces … it is extremely frustrating that there is literally nowhere to study on campus.”

In the beginning of the pandemic, there was a worry across the academic library community: Once students left college campuses and started remote or hybrid learning, would they forget the services and collections that libraries offer virtually? The Library Service Satisfaction Survey project was initiated after the pandemic started and the results showed that students continued to use the Library. E-books or e-journals as well as physical library items such as books and laptops were utilized by survey participants. In addition, reference and consultation with librarians was one of the most used services early in the pandemic. Now four years after the pandemic started, another question we have been hearing in the academic library community is: have student behaviors and perceptions with regards to academic libraries changed? While the Library Service Satisfaction Survey project does not have data from before the pandemic, the survey results show that the usage pattern has been steady. Table 2 shows that the percentages of participants who used e-books or e-journals and library items during October 2020 and March 2024 have been consistent.

The results from the Library Service Satisfaction Survey have been used in the UIUC Library’s annual reports and budget requests submitted to the University. We also examined the results to identify the services that participants had most difficulty using. For example, many participants were dissatisfied with the online system for reserving study spaces, specifically on how often they could make a reservation. As a result, the Library relaxed the restriction after evaluating the survey results and actual reservation data.

While user feedback leads to positive changes, frequent surveying can have its drawbacks. One such issue is “survey fatigue,” which occurs when the same populations are repeatedly asked to participate in surveys, leading to a decline in their willingness to complete them. Survey fatigue contributes to low response rate and poor data quality; it is a common and known challenge among assessment professionals working in higher education. The pandemic has complicated survey work just like it had affected other aspects of life. The response rates for all nine administrations of the survey had been from 3 to 5%. The survey has not offered incentives (e.g. cash incentives or prizes) to those who take the survey. While a higher response rate would have been preferable, a 3–5% response rate from a student survey at the University of Illinois Urbana-Champaign is understandable.

As scholarly work emerges to discuss libraries responses to the COVID-19 pandemic, there is little focus on how it has affected academic library engagement via programs or events. While general information has been collected, what is missing are the specific impacts to these programs, events and the staff who do this work. The “Library Engagement During COVID-19” project utilized a human-centered approach to study library workers, aiming to understand their methods for conducting and evaluating library engagement programs/events during the COVID pandemic (Shearer & Yu, 2021).

Invitations to interview were disseminated through seven library listservs affiliated with the American Library Association. The project conducted one-on-one interviews (see Appendix II Library Engagement during COVID-19 Interview Guide) with 28 library workers in the United States or Canada who are responsible for engagement events or programs at academic libraries. The project completed the University’s human subject research review and received an institutional review board (IRB) approval (exempt status). The interview questions focused on three key aspects of library worker engagement during COVID-19: research services and instruction, programming and events and developing or maintaining non-library partnerships. Motivation driving these workers included creating and promoting programs tailored to student needs and diverse campus environments, fostering outward engagement to encourage participation campus-wide and cultivating inclusive environments where people feel welcome. This project also focused on the importance of building connections and partnerships within communities and beyond the library.

The project uncovered the library workers’ needs and challenges in engaging user groups, specifically students, during the constant shifts brought on by the pandemic and beyond. First, library workers that prioritize delivering comprehensive research services and instructional support recognized the importance of catering to various learning styles during COVID-19. Emphasizing continuous improvement, these workers relied on familiar assessment and feedback mechanisms to refine their services and instructional programs effectively. Second, during COVID-19, library workers also emphasized programming and events as vital for engaging and connecting with campus communities. Creativity and innovation became crucial, along with effective marketing strategies to maximize attendance and engagement despite challenges posed by the pandemic. Finally, there was a heightened desire to form mutually beneficial partnerships and collaborations with groups external to the library. These partnerships became crucial for extending the reach of their programs during limitations imposed by social distancing, in addition to strengthening necessary skills for engagement. Partnerships and collaborations bolstered the impact and effectiveness of library initiatives and helped to address emerging needs and challenges during the pandemic. Given the challenges faced by these library workers during the pandemic, we compiled the library worker experiences and needed to better understand the difficulties they encountered and the support required below.

Amidst these collaborations, partnerships played a pivotal role in enhancing the impact and effectiveness of library initiatives, addressing emerging needs and challenges during the pandemic. One challenge that came to the forefront included a difficulty of assessing programs/events. Evaluating library engagement programs and events posed challenges for several reasons, primarily due to a multitude of complex challenges representing a distinct core need. We identified five core needs as (1) a lack of familiarity with assessment techniques, (2) a sudden shift in what to assess, (3) an unfamiliarity with community needs, (4) a dependence on physical presence to apply techniques and (5) limited assessment options.

A second challenge included meeting professional development needs. These needs encompassed areas such as assessment, facilitation and partnering; which is essential for workers who conduct events and programs. This professional development not only equips them with necessary skills and knowledge to excel but enables them to adapt to changing needs, enhance effectiveness and advance their careers. To address this, it is necessary that libraries address worker needs for connection and support. This is another way to invest in professional development because it fosters teamwork and cultivating partnerships that are essential strategies for enhancing the effectiveness and sustainability of library events and programs. These efforts ultimately contribute to the overall growth and success of library workers, libraries and the institutions they serve.

EDUCAUSE, a nonprofit association that focuses on the advancement of information technology in higher education, ranked “Driving to Better Decisions” (2024 EDUCAUSE Top 10 #2, 2024) as the second most important topic in its 2024 EDUCAUSE Top 10 trend report. Furthermore, EDUCAUSE called for “Improving Data Quality and Governance” as the strategy in this matter. At the University of Illinois Urbana-Champaign, a vast amount of data is being collected and managed based on security and privacy guidelines. The Library has used data from the EZproxy service and the Wi-Fi network prior to the pandemic. However, since many library services and collections were changed during the pandemic, more attention has been given to the EZproxy and Wi-Fi data and how they were analyzed, reported and communicated for decision making.

The Library subscribes to numerous electronic resources with Internet Protocol (IP)-based access control. When a user is not connected to the University network, EZproxy is the web proxy server that can authenticate users and allows them to access the electronic resources as if they were on the University campus. A user might encounter troubles using EZproxy based on network settings on the user’s end. In those situations, a user can use Virtual Private Network (VPN) to access the subscription electronic resources from off-campus locations.

The University provides Wi-Fi services to students, faculty, staff and guests. Wireless access point (WAP) is a networking device that allows wireless-capable laptops, cell phones, game consoles, etc. to connect to a wired network and the WAP automatically produces and collects logs.

EZproxy and Wi-Fi at the University both produce log data that record “events.” An EZproxy event can be a remote user downloading a journal article or performing a search in a scholarly database. A Wi-Fi event can be a laptop connecting or disconnecting to a WAP. Log data have a great level of detail, but they are machine-generated and hard to use in the original format. Thus, the Library has been creating summary reports (aggregated statistics) from the EZproxy and W-Fi log data before the pandemic. Since both EZproxy and Wi-Fi events are recorded in real time, they were great data sources for monitoring rapid changes in user activities during the height of the pandemic and providing insights on current user needs.

Data from EZproxy logs and Wi-Fi logs can be summarized or analyzed in different ways, depending on the UIUC Library’s data needs. Below we discuss two charts and two infographics that have been used to demonstrate how the Library supports students and faculty during the last few years.

Figure 1 shows that the usage of library electronic resources has significantly increased after the University began alternative online course delivery on March 22, 2020. Since then, the highest transactions by students were recorded on November 9, 2020; 10,305 students accessed EZproxy that day. The highest daily transactions by faculty and staff were recorded on February 8, 2021 with 1,811 faculty or staff accessing the EZproxy. It is important to note that since pandemic restrictions have been lifted, EZproxy usage has shown a steady decline since 2022. However, this does not mean that faculty, staff and students have used library electronic resources less. The decline is a result of most teaching and research activities having resumed on campus (within the University IP range), and faculty, staff and students on campus do not need to access the library resources via EZproxy.

The UIUC Library supports online students who are enrolled in fully online programs that are degree-seeking, certification or massive open online courses (MOOCs) as well as on-campus students who are taking a few online courses. Since many online students do not live on campus, they rely on using the EZproxy to access library subscribed e-books or e-journals. Figure 2 shows the ten online programs with the most students accessing library electronic resources off-campus (based on the EZproxy data) during the fiscal year 2024 (July 2023–June 2024). It should be noted that the iMBA Online and the Master of Computer Science Online are both in high demand and provide additional revenue for the University.

In contrast to the EZproxy data showing remote usage of the Library, Wi-Fi data shows the physical, in person activities in the Library. In fact, Wi-Fi data is often used to detect real-time occupancy of buildings and spaces. At the UIUC Library, the EZproxy data and the Wi-Fi data showed diverging trends since the beginning of the pandemic. Figures 3 and 4 show the dramatic decrease in Wi-Fi sessions in 2020 in four of the major library buildings on campus because only a small number of people were allowed to be on campus during the first few months of the pandemic. It also reveals that while indoor restrictions were eased in 2021 and the libraries were open for limited indoor capacity, the Wi-Fi usage did not bounce back to the pre-pandemic level until 2022.

There are many data analytics and business intelligence (BI) tools that can collect and visualize real-time and large quantities of data effectively. We used Splunk Cloud ^[9] and Tableau Desktop ^[10] to compile the data presented in the charts and infographics. However, to use the data to tell compelling stories that communicate impact and resonate with people, showing numbers is not enough. Figures 3 and 4 are parts of a series of Star Wars themed infographics that the Library created. The inspiration for the theme came from what we heard from students, faculty and staff, some through the Library Service Satisfaction Survey results – they wanted to “return” to in-person and “back to normal.” That was how we landed on the Star Wars theme and the key concepts such as “return”, “hope” and “strike back.”

Three different assessment projects used different qualitative or quantitative methods. First, the Library Service Satisfaction Survey project used an online survey to collect data from users’ perspective. Next, the “Library Engagement through COVID-19” project conducted interviews with library workers. And the last project used quantitative data and analytics tools to analyze the EZproxy logs and the Wi-Fi logs. The projects also focused on different populations. The Library Service Satisfaction Survey was designed for the University students while the EZproxy/Wi-Fi project analyzed data collected from all user groups at the University. The “Library Engagement through COVID-19” sought transferable knowledge from outside of the University through conducting in-depth interviews with library workers at other academic libraries.

There are common themes emerging from the three assessment projects despite methodological differences. First, all data and findings underscore the demand for academic libraries and library workers to navigate the unprecedented challenges presented by the COVID-19 Pandemic, particularly in the realm of remote engagement. Their proficiency in leveraging existing expertise, fostering innovation and pursuing collaborative initiatives reinforce the critical role of libraries as dynamic and responsive institutions, even amid a crisis. This adaptability needs to be assessed and documented, not only to ensure the continuity of library services but also to strengthen the libraries’ connection with users, ultimately affirming the libraries integral support systems during periods of uncertainty.

The second theme the present case study has identified is that the pandemic has reshaped the landscape of higher education, revealing significant challenges related to online learning and student engagement. On the other hand, demand for core library services or resources have either reverted to the pre-pandemic level or increased. Data from the Library Service Satisfaction Survey (see Table 2) shows that e-books and e-journals have become the most used library resources. The EZproxy data from the last few years confirms that trend (see Figure 1). The Library Service Satisfaction Survey data also indicates that while there were measures such as building closures and limited indoor capacity in 2020 and 2021, the use of library physical spaces remains high. The Wi-Fi data (see Figures 3 and 4) further illustrates the scale of sharp decrease as well as an increase in physical space usage during the last few years. Library workers are at the frontline dealing with these ongoing and new demands and they need professional development, peer connection and support as identified through the “Library Engagement through COVID-19” project.

Looking back on the three academic library initiatives and actions during the pandemic, it’s clear that they were more than just reactions; these projects illustrate how the library’s response to the pandemic have deepened its understanding of student needs, service effectiveness and engagement strategies. Insights from each initiative demonstrate that the challenges of COVID-19 were not isolated, but rather catalysts for ongoing, adaptive change. The Library Service Satisfaction Survey now serves as a continuous feedback mechanism, while the study of library engagement programs illuminates the importance of human-centered approaches to fostering connections. Data from EZproxy and Wi-Fi logs reveal evolving usage patterns, underscoring the return to in-person services and the growing demand for online support. Collectively these projects inform the library’s future strategies by emphasizing the need for flexible, evidence-based approaches to meet the changing needs of students, staff and faculty.

The authors gratefully acknowledge the study participants of the “Library Engagement during the COVID-19 Pandemic,” who shared experiences with us during one of the most difficult times in modern history. They also thank Belinda Bolivar, Lindsay Taylor as well as IT staff at University of Illinois Urbana-Champaign and the Library.

The supplementary material for this article can be found online.