This study describes how librarians at an R1 institution developed a rapid process to understand the information literacy skills and concepts covered in a series of asynchronous online information literacy workshops. Because the workshops were originally designed and offered in an ad-hoc manner, this project was part of a larger assessment and revision effort to improve quality and make the workshops more cohesive.
Workshop reviewers streamlined mapping by focusing on knowledge practices from the ACRL Framework, providing the basis to understand frame coverage and gaps across all workshops. The reviewers strengthened map quality by including two Framework companion documents into the process to help fill in conceptual gaps in the original Framework.
The authors describe the mapping method developed and share recommended best practices for librarians who may need to assess rapidly the coverage of information literacy skills and concepts across multiple digital learning objects. The librarians learned that the Framework companion documents hold value to general information literacy work despite their implied disciplinary nature.
This study provides librarians the means to understand the coverage of information literacy concepts across asynchronous digital learning objects which may have been created individually and without a specific plan. The authors make a case for including Framework companion documents into the process and focusing on Framework knowledge practices to create the map, bringing more originality to the paper.
Introduction
The growth of online learning modules and tutorials accompanied the growth of distance education programs (Gonzales, 2014; Watts, 2018), flipped library instruction (Arnold-Garza, 2014; Obradovich et al., 2015; Boucher et al., 2013), the need to address capacity issues (Stiwinter, 2013; Tooman and Sibthorpe, 2012; Walters et al., 2015), and the arrival of the COVID-19 pandemic (Ghazi-Saidi et al., 2020; Olsen and Harlow, 2022; Pinto et al., 2022; Mery et al., 2022). Thus, the trend in library instruction for at least the past 20 years has been to mitigate the limitations and time constraints of the traditional one-shot, in-person instruction model by incorporating new types of programs such as embedding library instruction content into course management systems to extend and flip instruction or to replace all basic instruction classes with asynchronous online learning modules or workshops. As Watts notes:
Online instruction, whether semester-length, one-shot tutorials, point of need instruction or instruction for flipped classrooms—is becoming a primary means for delivering library instruction. The trend to provide more instruction online will only grow as more colleges experience an increase in online and hybrid courses and shifts in library staffing, both of which necessitate delivering instruction online and asynchronously (2018, p. 50).
The subsequent assessment and reworking of the online material often came later – or sometimes not at all – even though it's clear that instructional programs, regardless of format, should constantly undergo assessment in order to keep material current, impactful, and relevant to the curriculum. This is a common and recursive process, and best practice is to repeat it often (Eastman et al., 2018; Oakleaf, 2009; Woitte and McCay, 2019). This case study at a Research I institution documents the process of assessment and revision of an asynchronous online badging workshop program through mapping those workshops to the ACRL Framework and Framework companion documents. This work of explicitly connecting workshop content and learning outcomes to knowledge practices and their associated frames within the Framework demonstrates how academic libraries can assess and improve their online curricula as well as make it more inclusive.
Assessment of asynchronous library instruction
A significant difference in assessment methods between online, asynchronous content and the synchronous one-shot instruction session is that the one-shot is traditionally delivered by a subject librarian on a class-by-class basis, in cooperation with that course's professor. Because this traditional faculty partnership-based instruction is so driven by individual classes or curricula, and so rooted in faculty requests, the learning goals and structures can vary widely. Variables leading to some inconsistencies include: (1) discipline faculty and librarians may retire or otherwise separate from an institution, (2) new faculty not yet aware or interested in partnering with librarians may change the direction of the curriculum, or (3) courses may change or simply not be offered one semester or year to the next. Because of these inconsistencies, librarian-created, asynchronous online badging workshops provide a better opportunity to incorporate Framework mapping as an assessment tool. These types of learning objects can be more consistently and uniformly offered, thus making the resulting maps more meaningful. Loftis and Wormser (2016) describe a process of developing an online information literacy (IL) curriculum aimed at undergraduate art students using the Framework as a roadmap. The results of this project provided a “simplified approach to information literacy” while still covering all six of the primary frames (2016). Using the ACRL Framework as a reference point for creating and improving instructional materials is a way to encourage a shift in thinking, to provide opportunities for incorporation of threshold concepts, and to form a basis for assessing instruction programmatically (Eastman et al., 2018; Ward and Kmetz, 2023; Ziegler, 2018). Digital learning objects for teaching IL are frequently created to fill an immediate or discrete need, and without being part of an overall IL plan. A structured review of a collection of learning objects can provide an opportunity to evaluate with a more cohesive programmatic direction (Olsen and Harlow, 2022; Bryan et al., 2018). This case study introduces a novel method for mapping online IL modules by focusing on knowledge practices from the ACRL Framework and Framework companion documents to better understand concept coverage and identify potential gaps. This approach may be applied retroactively, providing librarians a model to evaluate digital learning objects that were created ad-hoc to fill discrete instructional needs.
Case study background
The University of South Florida (USF) is an R1 university which serves over 50,000 students on three campuses located in Tampa, Sarasota, and St. Petersburg, Florida. Prior to 2020, these three campuses operated independently; in 2020 the three USF campuses went through a state-mandated consolidation into a single entity, thus unifying the three libraries. There are nearly 40 faculty librarians across the three campus libraries, and 23 librarians in the intra-campus Research and Instruction (R&I) unit. While the majority of the work described in this case study was done by a focused working group made up of some of the R&I librarians, all R&I librarians participated in the development and assessment of the library learning materials in some way. Inclusion of all the librarians in the work of creating and assessing the workshops helped to foster a sense of collaboration and buy-in within the entire R&I unit (Jacobs et al., 2023).
The USF Libraries' slate of online, asynchronous badging workshops was developed formally in 2019, to align with the University of South Florida's Enhanced General Education Curriculum (University of South Florida, 2018). Prior to that initial project, each library had created their own collection of online materials on a case-by-case basis, in partnerships between individual librarians and teaching faculty members (Jacobs et al., 2023). In 2020, the combined factors of the COVID-19 pandemic and consolidation created a need to merge three disparate collections of instructional content into one coherent, organized, and effective curriculum. The USF Libraries did not design this slate of asynchronous online modules as a rejection of the face-to-face one shot, but rather in response to a variety of factors: a then-reduced number of librarians, increased demands on librarian time, and later, the COVID-19 pandemic (Jacobs et al., 2023). A diminished ability to provide face-to-face instruction at a high enough volume spurred the creation of the online badging workshops to take care of teaching the basics to large groups.
The first large-scale peer review of these workshops took place concurrent with consolidation efforts in 2020. This effort resulted in improved quality, scope, and cohesiveness of the interactive library learning modules. That peer review project hinged on a rubric the librarians created, that was based on the Quality Matters Course Design Standards (Jacobs et al., 2023). The use of the rubric in the 2020 peer review project helped librarians be methodical in their assessment of these workshops and, in turn, helped the workshop creators bring increased standardization to their revised structure, design, and content (Jacobs et al., 2023). Two other outcomes of the 2020 assessment included some modules being retired due to lack of use and others consolidated to streamline redundant content. This resulted in a suite of cohesive but focused modules that were easier to maintain.
A second assessment iteration occurred in 2022–2023 in response to (1) retirements and new hires within R&I, (2) a need to update content, and (3) the knowledge that the foundation built could be improved further by including alignment with the ACRL Framework and associated companion documents. While similar to the 2020 process, librarians made two key changes: revising the module peer-review rubric and mapping module content to the ACRL Framework and two of the ACRL companion documents. Reviewing, mapping, and revising the modules with both the Framework and companion documents in mind provided a more comprehensive look at the IL concept coverage and provided insight for future improvements.
Developing a mapping process with the Framework
The initial assessment of each workshop in 2022–2023 was to be based on a variety of factors including usage rate, student and faculty feedback, and whether the learning outcomes were aligned with workshop content. Librarians also wanted to better understand where these workshops connected to the concepts in the Framework and where there were gaps or areas of overlap. They anticipated being able to use this information to strengthen individual workshops, and by extension the whole of what had become a foundational piece of the information and data literacy program.
While librarians looked to some Framework mapping efforts in the literature (Condon and Pothier, 2022; Renirie, 2020; Willson and Angell, 2017), those studies did not fit with the needs of the project. Renirie's work (2020) focused on reference interactions, Willson and Angell (2017) mapped the Framework to established standards in a specific discipline (nursing), and Condon and Pothier (2022) mapped Framework concepts to a set of business data literacy competencies they had proposed in an earlier article (Pothier and Condon, 2019). As this project was not directly mapping the Framework to disciplinary standards or university curricula, the working group needed a new approach. Combining methods from established curriculum mapping strategies (Archambault and Masunaga, 2015; Charles, 2015) these sources, each provided ideas to develop a mapping process to understand the gaps and overlap of learning module content.
Developing a mapping methodology proved to be a recursive process. Finding a balance between a meaningfully comprehensive map and doing so within a short timeframe meant establishing an appropriate level of content mapping detail. After drafting and proposing an initial strategy, a group of 4 R&I librarians tested the strategy. Each worked independently to map the content of a single workshop to concepts from the Framework and selected Framework companion documents. The group then reconvened to compare results and reconcile any differences in their findings. This test enabled the group to learn strengths and weaknesses to the approach and revise the strategy accordingly to focus on what mattered most, to improve the quality of mapping, and to streamline the workflow. Central to the process was the decision to look for strong connections in the workshops with knowledge practices listed in the Framework and companion documents, a process partially informed by Renirie's work (2020).
Renirie (2020) leveraged both knowledge practices and dispositions from the Framework to inform mapping to reference questions. However, the Framework treats knowledge practices as “demonstrations of ways in which learners can increase their understanding” and dispositions as “a tendency to act or think in a particular way” (ACRL, 2016). Given the scope, purpose, and self-contained nature of the workshops, it was impractical to assess whether a learner would completely alter their thinking and behavior in light of new knowledge. Knowledge practices tend to be more straightforward concepts to assess and were more appropriate to use as connection points for a map. In practice, this meant if a concept from a frame's knowledge practices seemed well-represented in a particular workshop, that knowledge practice was marked as relating to the workshop. Once this was done, frames identified as having the most knowledge practices representing them were mapped to the workshop. One example of this: when mapping to the APA Citation workshop, librarians were able to see if module content and summative assessment included elements of the knowledge practice “give credit to the original ideas of others through proper attribution and citation” (ACRL, 2016). If so, the workshop would be mapped to the Information Has Value frame. If a knowledge practice seemed like it should be included in a workshop's content, but was not present, this was also noted as a potential gap and area for later improvement. Table A1 ( Appendix 1) illustrates a map of knowledge practice connections and gaps observed using the “Data Literacy 2” workshop as an example.
Including Framework companion documents
The working group believed it could strengthen the mapping process and the workshops by including concepts from the Framework companion documents. The ACRL Framework companion documents not only contextualize the Framework, but also address some of the general IL concepts that have been further developed with the benefit of scholarly discourse on the Framework (DiNardo and Ariew, 2024; Rose-Wiles, 2024; Roth-Katz, 2022; Schlesselman-Tarango, 2022). After nearly ten years, some elements of the Framework are arguably outdated (Sweet, 2021). Concepts related to authority, diversity, equity, inclusion, social justice, and open access are generally more developed and have more explicit treatment in many of the companion documents (DiNardo and Ariew, 2024). Given the short timeframe to complete mapping and the relatively large number of companion documents, the group intentionally limited the number of companion documents they used to two: ACRL's Companion Document to the ACRL Framework for Information Literacy for Higher Education: Women's and Gender Studies (2021) (WGS companion) and the Companion Document to the Framework for Information Literacy for Higher Education: Science, Technology, Engineering, and Mathematics (2022) (STEM companion). While content from many of the other ACRL Framework companions would have been useful to their effort, they felt that these two contained concepts that were important to their program goals and were broadly applicable across learning modules.
Using a similar process, the working group mapped concepts from these two companion documents to each of the workshops. Note, however, that the WGS companion eschews “knowledge practices” and “dispositions” terminology in favor of “Actions and Attitudes” respectively as a better representation of the ethos of the related disciplines (ACRL, 2021). This required a bit of interpretation and flexibility when applying the mapping process to bring the value of this companion to the work, and in practice was easily incorporated.
The influence of these Framework companions can be illustrated by two examples. The Avoiding Plagiarism learning module included the concept articulated in the WGS companion that learners “actively listen to the conversation through research and reading” (ACRL, 2021), so this connection was mapped. In contrast, the knowledge practice in the STEM companion stating that learners “recognize that a given scholarly work may not represent the only or even the majority perspective on the issue” (ACRL, 2022) was not found in evidence and therefore was not mapped. However, given that this knowledge practice seemed to have direct and important application to the workshop content, it was noted as a gap and could potentially be included for future workshop improvement. In this way the project could produce a map of existing Framework connections, as well as uncover gaps in instructional content.
Implementing the mapping process
Once all of the connections to knowledge practices in the Framework and the companion documents had been noted by individual librarians, they brought their findings together to discuss, reconcile differences, and reach consensus. Then, using the knowledge practices as a guide, librarians determined which three frames were most strongly represented in a learning module, and these became the frames mapped to that module. Given the interconnected nature of concepts throughout the Framework, the effort to limit the map to the three best represented frames helped avoid a result where nearly every frame could be found in evidence in nearly every workshop. Exceptions to this guideline were made for two workshops that had a more comprehensive scope. The final map took the form of a matrix of workshops and their associated frames. For example, the most strongly represented frames in both the APA citation and avoiding plagiarism workshops were Information Has Value and Scholarship as Conversation. Wider gaps in skills and concepts were noted, such as the workshop about primary sources in the humanities (Table 1). A separate spreadsheet record was kept of the associated knowledge practices identified as connected or as gaps for every module. This provided a more granular look at Framework and companion document concepts that were found to be present or missing for future workshop improvement. Finally, a summary document was created for each workshop. As part of the larger refresh project, the working group provided these summaries to librarians responsible for revising each workshop to help guide their work. An example summary of the Data Literacy 1 workshop can be found in Appendix 2. All of the workshop summaries included the following information:
Workshop title
Workshop level (target audience)
Stated workshop outcomes (listed within the workshop)
A list of skills the workshop seemed designed to teach
Frames noted as being represented or as a gap
Knowledge practices noted as being represented or as a gap
Other notes relevant to workshop revision
Results
The primary goal of this mapping project was to better understand the overall IL coverage of the ACRL Framework across the slate of online workshops and reveal potential gaps. In addition to achieving this goal, the project yielded additional outcomes including more specific recommendations for workshop revisions, and a better understanding of Framework companion documents and their potential applications. The resulting map ( Appendix 3, Table A2) reveals that, in total, these workshops offer points of coverage across all six Framework frames. A simplified summary of these results (Table 2) shows the frame Searching as Strategic Exploration had the fewest points of coverage, while the frame Scholarship as Conversation held the most. Four workshops also included observations that certain frames needed stronger representation in future workshop revisions. Two of these observations connected to Searching as Strategic Exploration. The working group's discussion of these results noted that while the highest and lowest represented frames did not automatically indicate a problem, two observed gaps in the workshops corresponded to the lowest represented frame. As such, this mapping approach facilitated understanding which specific workshops could improve overall coverage of Framework-related skills and concepts.
Two additional project-related outcomes emerged through 1) using knowledge practices to create the map and 2) bringing ACRL Framework companion documents into the process. As illustrated in Table A1 ( Appendix 1), noting observed connections or gaps vis-à-vis knowledge practices from the Framework and companion documents enabled reviewers to provide more detailed and concrete recommendations for future revision than the broader frames connection map. Including companion documents into this process also opened the door for recommendations to incorporate IL concepts that have been further developed subsequent to adoption of the Framework, such as improved treatments of authority, diversity, equity, inclusion, social justice, open access, and emerging technologies as they pertain to IL.
An unanticipated outcome not directly related to the mapping project was that the working group raised awareness within the entire department about the value and utility of the ACRL Framework companion documents. This prompted one of the education librarians to read the Companion Document to the ACRL Framework for Information Literacy for Higher Education: Instruction for Educators (ACRL, 2023) and incorporate more explicit open access instruction for students who would soon be practitioners cut off from university access. One of the working group librarians has begun pursuing a gap in the literature surrounding the Framework companion documents, perhaps indicating they are underused resources.
Future mapping efforts at USF will include additional ACRL companion documents (Education, Sociology, Social Work, etc.). Further, to facilitate these efforts, authors of new workshops will be asked to consider which knowledge practices and frames will correlate to their learning outcomes. Other planned assessments include faculty and student focus groups, satisfaction surveys, and continued monitoring of enrollment numbers for each workshop (Figure 1). The working group knows through prior assessment that most students enroll in these workshops because a professor requires or recommends them. After rolling out the revised modules, observational evidence suggests that faculty feel the changes made to workshops are positive. Enrollment data from the most recent academic year shows a dramatic spike for two workshops: Creating Manageable Topics & Research Questions and Conducting a Literature Review. Analysis shows that the higher enrollment for these two workshops comes from students enrolled in English composition courses, indicating a wider adoption by the faculty in that department. The working group speculates that with no change in marketing strategy, this may be due to faculty word-of-mouth. Three other consistently popular workshops have been: (1) Finding, Evaluating, and Using Information, (2) Avoiding Plagiarism, and (3) APA Citation. While a few workshop enrollments leveled off or decreased slightly following the project, most saw increased enrollment.
Recommendations
The effort and collaborative work that went into this process showed some clear best practices when mapping ACRL Framework concepts to asynchronous online badging workshops. Those who want to pursue this should:
Bring ACRL Framework companion documents into consideration during the mapping process in addition to the core concepts in the original ACRL Framework.
The companions facilitate the incorporation of more current thinking related to authority, diversity, equity, inclusion, social justice, open access, and emerging technologies.
Use knowledge practices as a basis for connecting ACRL Framework and companion document concepts to content.
This provides a roadmap to more assessable learning outcomes within a self-contained learning module.
Involve more than one person in mapping each module.
This helps in avoiding bias, provides multiple perspectives, and promotes standardization rather than personal styles.
Limit connections to three frames.
Doing so allows the team to highlight stronger connections to the Framework and companion documents, making for a more meaningful map.
Use the map to help:
Guide decisions related to content that may be missing or redundant.
Include better treatment regarding equity, inclusivity, and accessibility.
Assess an information and data literacy instruction module program.
Use reflections from a mapping project to plan for future assessment, revision, and refinement of the mapping process.
Conclusion
This case study introduces new and innovative approaches to mapping the ACRL Framework and companion documents to online learning modules. Mapping facilitates informed decision making through better understanding of overall coverage of IL skills and concepts within this component of an IL program. Including the Framework companion documents in mapping not only improves the process by introducing more current IL understandings, it also raises awareness of what the companion documents offer librarians. By applying the ACRL Framework and companion documents to the mapping project, the R&I librarians at USF have learned new ways to incorporate concepts from the documents to their practice. For example, the emphasis on the need for quality open access research materials in the field of social work and education has started a new initiative to teach open access research more intensively to students about to graduate and become practitioners. The companion documents update technology components of information literacy such as including digital humanities for literature and composition teaching and introducing open educational resources for preservice teachers. With the advent of AI tools in all areas of research and instruction, the need to continually review and revise online and in-person pedagogical practices is essential. Mapping the ACRL Framework and companion documents to online asynchronous workshops results in effective pathways to improving an IL program and raises librarian awareness and knowledge of more current IL understandings.
The authors would like to thank Matthew Weirick Johnson for their ongoing support and critical eye toward structure and content.
Appendix 1
Data Literacy 2 workshop – mapped knowledge practices
| Short frame title | ACRL framework knowledge practice mapped or noted as gap |
|---|---|
| Authority | N/A |
| Creation | X – assess the fit between an information product's creation process and a particular information need X – recognize that information may be perceived differently based on the format in which it is packaged X – develop, in their own creation processes, an understanding that their choices impact the purposes for which the information product will be used and the message it conveys O – recognize the implications of information formats that contain static or dynamic information |
| Value | N/A |
| Inquiry | O – synthesize ideas gathered from multiple sources; whether that be by synthesizing published research in new ways (literature reviews, meta-analysis), reusing data to build new models or answer new questions, or conducting empirical or observational research (primary research articles) O – reconcile information drawn at various stages of the information creation process such as raw data, pre-prints, and published research |
| Scholarship | O – cite the contributing work of others in their own information production O – critically evaluate contributions made by others in participatory information environments |
| Searching | N/A |
Note(s): An “X” indicates a strongly observed connection. An “O” indicates an observed gap of content that the working group felt should be filled
Appendix 2
Example Workshop Report Including Frame and Knowledge Practice Mapping Observations.
Workshop Title: Data Literacy 1: An Introduction to Finding and Evaluating Data
Workshop Level: Early Undergraduate
Stated Workshop Outcomes:
Critically evaluate data and data visualizations
Find data and data visualizations
Cite data when using it in your research
Skills Addressed:
Locating data
Data presentation
Evaluating data representation
Mapped Frames and Knowledge Practices*:
Authority is Constructed and Contextual
Define different types of authority, such as subject expertise (e.g., scholarship), societal position (e.g. public office or title), or special experience (e.g. participating in a historic event);
Recognize that authoritative content may be packaged formally or informally and may include sources of all media types;
Use research tools and indicators of authority to determine the credibility of sources, understanding the elements that might temper this credibility;
From the Women and Gender Studies Companion Document
Seeks marginalized voices in respectful ways; acknowledges, sheds light on, gives voice to, and holds a space for those voices and the platforms where they can be discovered.
Understands that the formal structures of academic scholarship privilege particular types of knowledge production and dissemination, which do not necessarily reflect the experiences, and norms of marginalized groups.
Searching as Strategic Exploration
Determine the initial scope of the task required to meet their information needs;
Identify interested parties, such as scholars, organizations, governments, and industries, who might produce information about a topic and then determine how to access that information;
Utilize divergent (e.g. brainstorming) and convergent (e.g. selecting the best source) thinking when searching;
Match information needs and search strategies to appropriate search tools;
Design and refine needs and search strategies as necessary, based on search results;
Understand how information systems (i.e., collections of recorded information) are organized in order to access relevant information; y use different types of searching language (e.g. controlled vocabulary, keywords, natural language) appropriately;
Manage searching processes and results effectively.
From the Women and Gender Studies Companion Document
Identifies avenues for seeking out the scholarship of underrepresented or marginalized communities, and knows where to gather such “hidden” information.
*An “X” indicates a frame or knowledge practice is represented. An “O” indicates a frame or knowledge practice is missing, but is recommended for consideration in revision.
Notes:
Should there be a learning outcome that addresses the item “what is data literacy”? – e.g. “students will recognize__”
We feel this workshop should cover the methodology of searching for data in more detail.
There is a list of resources, with little to no direction on how students actually go about finding the datasets/data visualizations
This would help differentiate Data Literacy 1 from Data Literacy 2
Appendix 3
Map connecting ACRL framework frames to content within online workshops
| Workshop title | ACRL framework shortened frame | |||||
|---|---|---|---|---|---|---|
| Authority | Creation | Value | Inquiry | Scholarship | Searching | |
| APA Citation | X | X | ||||
| Avoiding Plagiarism | X | X | ||||
| Changes in Perspective | X | X | X | |||
| Chicago Citation | X | X | ||||
| Conducting Lit Reviews | X | X | ||||
| Copyright | X | X | X | |||
| Manageable Topics | X | X | ||||
| Poster Presentations | X | X | X | |||
| Virtual Presentations | X | X | X | |||
| Data Literacy 1 | X | O | ||||
| Data Literacy 2 | X | O | O | |||
| Find, Eval, and Use Info | X | X | X | X | X | |
| Story Maps | X | X | ||||
| Evaluating Info | X | X | X | |||
| Resources for Marketing | X | O | X | |||
| MLA Citation | X | X | ||||
| Opposing Claims | X | X | ||||
| Primary – Humanities | O | |||||
| Primary – Sciences | X | X | X | |||
| Using Images Effectively | X | X | X | X | ||
Note(s): An “X” indicates a strongly observed connection. An “O” indicates an observed gap of content that the working group felt should be filled
