Association of College and Research Libraries 10th National Conference
Kate Manuel and Judy Horn
Kate Manuel
Assessment of Information Literacy
The focus of the ACRL 10th National Conference and its attendees upon information literacy comes as no surprise, given that January 2001 brought the ACRL Board's approval of"Objectives for Information Literacy Instruction: A Model Statement for Academic Librarians" (http://www.ala.org/acrl/guides/objinfolit.html). More than three dozen of ACRL's preconferences, panels, contributed papers, and poster sessions addressed information literacy in some way, and few programs were more eagerly attended than those on assessment, whose audiences regularly exceeded the seating capacity of conference rooms.
Kenneth Smith of the University of Arizona delivered an invited paper on "Assessment of Student Learning", in which he articulated reasons for assessment beyond the commonly mentioned need for educational institutions to be "accountable" in terms of student learning outcomes to taxpayers and the government for the resources invested in them. Smith instead situated the real value of assessment in its capacity to create and focus faculty attention upon a "shared culture."That is, faculty who must generate criteria with which to assess students are forced to communicate and reach a certain level of consensus upon what really matters in their fields and, indeed, in higher education generally why it is students need certain knowledge and skills, what students really need to know, and how students will be served by knowing something. Without assessment,many faculty would continue to teach discrete content within their individual classrooms without any attention to such broader topics.
As the value of assessment accrues when the conversations begin, libraries and librarians are especially well situated according to Smith to use assessment as a way of changing their "'business' to fit the new realities" of education. Academic disciplines are becoming less important as the ability to acquire knowledge becomes less important than the ability to evaluate and use knowledge. Smith also characterized assessment as a way of looking at academic programs from students' points-of-view and shifting the focus from teachers' knowledge to what students need to know. He also made it clear that he believes that assessment is not an end in itself; rather, the findings of assessment should be used to document, explain and improve the performance of academic programs. In practical terms, everything cannot be assessed all at once. Ideally, those interested in assessment should use it to gain insights into the things with which they are least comfortable and into critical points within students' educational experiences.
Debra Gilchrist, one of the reactors to Smith's paper, described quite concretely how assessment at Pierce College has helped librarians and faculty to focus upon what they want students to be able to do and to focus more upon teaching. To those who would characterize assessment as "smoke and mirrors," Gilchrist replied that assessment is "clearing out the smoke and getting accurate reflections from the mirrors."
Many other sessions focused less upon assessment in a theoretical sense and more upon findings of assessments within local contexts. In "An Online Competency Test for Information Literacy: Development, Implementation, and Results," Lynn Cameron and Rebecca Feind of James Madison University (JMU) discussed results of the Information Seeking Skills Test (ISST) that all first-year students at JMU must pass before being allowed to enroll in sophomore-level classes. JMU's situation is somewhat unusual in that the campus has long placed strong emphasis on assessment even holding an "assessment day" each year, when students are required to show up for measurement of their attainment of student learning outcomes, and students' scores on the ISST are reflected on their transcripts.
In the most recent administration of the ISST, 2,671 of 3,277 first-year students took the test. Some 70 percent of them 2,468 students passed the test, with 22 percent of these passing at an "advanced" level. The ISST required students to demonstrate understanding of concepts as well as to perform information-retrieval tasks, and results showed that students did better on the applied parts of the test than on the knowledge parts. Students performed worst on the questions about periodicals, but their knowledge of reference sources was another weak area. Students also expressed more comfort with the Web than with traditional library resources. Results of the ISST found no significant difference between the scores of students who learned information-seeking skills from JMU's Go for the Gold tutorial (http://library.jmu.edu/library/gold/modules.htm)and those who learned from sources other than the tutorial.
"Assessing the Information Literacy of Undergraduates: Reports from the University of California, Los Angeles (UCLA) Library's Information Competencies Survey Project" (see http://www.bol.ucla.edu/~jherschm/project/),presented by Eleanor Mitchell, Patti Schifter Caravello, and Eloisa Gomez Borah,was the outgrowth of a project begun in 1998 and offered another institution's take-up on assessment. In implementing the ISST, JMU had specifically wondered whether a competency test, where students' answers impacted them in some way,would yield different results than a traditional assessment test. UCLA, in contrast, opted for a traditional assessment test, but used it to test their hypotheses that exposure to libraries over an academic career would increase students' information competence, that students majoring in disciplines that required significant library research would score higher, and that students who received large quantities of library instruction sessions would score higher. To test these hypotheses, a survey with 25 questions 11 of which were demographic and 14 competency-based was administered to 453 students. These 453 represented a fraction of the 3,180 students to whom the UCLA Registrar's office sent e-mail notifications about the survey, and each student was paid $10 for completing the survey.
Overall, the average score on the competency-based questions was 61.7 percent, which was considered as failing, and only about 1 percent of students evidenced mastery of information literacy skills. Other findings were that, while seniors scored highest, there was no progressive improvement in students' scores between their first and final years as undergraduates; that, while humanities majors did score somewhat higher, the scores of other majors were not significantly different; and that lots of library instruction did not correlate with a higher score (indeed,respondents indicated that most of their library instruction occurred in high school). Only a high rate of library use was found to correlate with high student scores. Students did not know how to evaluate Web sites, when to cite sources, the components of book and article citations, the use of Boolean operators, or the meaning of call numbers. In fact, some students were unaware that the call numbers were related to library materials topics one focus group even expressed the desire for libraries to organize their books in some way like Borders does!
Lisa O'Connor, Carolyn Radcliff, and Julie Gedeon of Kent State University (KSU) presented yet another assessment instrument and set of findings in "Assessing information-literacy skills: developing a standardized instrument for institutional and longitudinal measurement." KSU's 43 percent graduation rate was a stimulus behind this assessment effort, which sought to determine whether information-literate students are more likely than their non-information-literate counterparts to complete their degrees. Work was inspired by the Wisconsin-Ohio Reference Evaluation Program (WOREP) and sought to create an instrument that was standardized, rather than specific to a particular library. The KSU presentation was especially helpful to attendees by detailing their procedures for pre-testing the instrument (with one-on-one, small groups, and field trials) and for identifying "misfitting" questions and populations. For example, a question on what students should do first when going to the library for a research assignment was found to be a "problem", because too many students found appealing items other than the correct one(s). Similarly, means of determining whether students answering incorrectly correspond to particular population groups and thus whether questions need to be re-written to give valid results were also helpful information.
Finally, "Crossing to Web-based Instruction: The Importance of Instructional Assessment" by Ann Roselle and Rachel Fenske of Eastern Washington University (EWU) offered yet another impetus for and model of assessment. EWU sought to assess Web-based instruction using a Web-based instrument. Basic library instruction at EWU is presented in the context of a computer literacy course required of all students and enrolling, at any one time, some 350 students. Initially librarians had taught students to choose an appropriate database for information needs, to interpret bibliographic information, and to evaluate information within the context of this class by giving between five and seven 50-minute lectures to classes of 60-100 students.
Restructuring of the computer literacy course notably smaller class sizes necessitated changes in the provision of library instruction for it, and an online tutorial was substituted for the in-person lectures. The online tutorial included exercises that were computer-graded and furnished students with immediate feedback on their progress. Unfortunately, software grading errors plagued the fill-in-the-blank questions. Later revisions transformed these fill-in-the-blank questions to multiple choice ones; librarians thought, for example, that they had included all possible variants of answers, but this proved untrue.
Librarians then engaged in a study of 283 students to assess whether in-person or Web-based instruction better facilitated student learning. Students were divided into two groups,given instruction, and then completed online exercises. In-class lecture students were more likely to agree that online exercises helped reinforce concepts and that information was clear and well-organized, but there were no differences in students' mastery of learning outcomes.
These ACRL presentations about assessment of information literacy thus embodied some of the commonly cited reasons for assessment demonstration of students' attainment of learning outcomes (James Madison University), determination of students'competency levels (UCLA), analysis of the degree to which academic programs or skills serve broader institutional or social goals (e.g. raising graduation rates) (Kent State), and comparison of the efficacy of different teaching methods (Eastern Washington University). A range of assessment methods was also highlighted mandatory competency testing, surveys, pre- and post-testing,and Web-based testing.
Kate Manuel(kmanuel@csuhayward.edu) is Physical Sciences Librarian, California State University, Hayward.
Judy Horn
Electronic Reference
Electronic Reference is an emerging new field and represents the first significant shift in the way we provide reference service. At the ACRL 10th National Conference held in Denver,March 15-18, 2001 there were invited speakers, contributed papers, panel presentations and poster sessions on this topic. All were among the most heavily attended sessions of the conference.
Electronic or digital reference involves reference service done online whether it is via e-mail, the Web, chat, or Web contact center software. Nearly all academic libraries offer e-mail reference service but chat and Web contact software services are still emerging. The sessions at ACRL focused on the planning, the length of time it takes to bring up a service, training, and new staffing patterns rather than on a discussion or identification of individual software packages. It has been recognized that planning and evaluation are key components in developing a successful electronic reference service.
Invited Speaker Panel
That electronic reference service has many challenges and possibilities for academic libraries was evident in all of the various presentations on this topic. David Lankes, Director of the Information Institute and Director of the Virtual Reference Desk Project,Syracuse University, was one of the invited speakers who led a discussion on"Digital Reference: The Future of Academic Reference?" A panel composed of Paul Constantine, Cornell University, Steve Coffman, Library Systems and Services,and Linda White from Digital Library Learning Center at the Library of Congress each presented a different aspect of electronic/ digital reference.
David Lankes set the stage with an introductory speech on the need for us to consciously shape digital reference rather than just letting it evolve. With digital reference, we are moving from digitizing and networking reference sources to networking reference staff and expertise. Additionally, with digital reference, we are disconnecting reference from time, place and collection. Digital reference provides academic libraries with the opportunity to build stronger bridges throughout academia,with faculty, students and others. Digital reference has the potential to renew and re-energize reference in academic libraries. Linda White described the ambitious Collaborative Digital Reference Service (CDRS) project at the Library of Congress that is currently in the pilot 3 final testing stage. More information on this project can be found at http://www.loc.gov/rr/digirefPaul Constantine from Cornell University titled his presentation, "The Times, They are a-Changing" in which he discussed some of the challenges being faced by academic libraries, including students and other users whose life has been shaped by technological innovations, new clienteles, and changing emphases within research libraries. The challenge for academic libraries is to use traditional strengths and values to build new programs and services. He said that it was important for libraries to create a new paradigm in which we no longer define reference service in terms of "the desk." If we don't make these changes, we will become increasingly irrelevant to our users. Steve Coffman echoed these sentiments of Paul Constantine when he concluded his presentation by asking what would happen if we don't change the face of reference or change it fast enough? His response was that, if we don't do it, others will do it for us! Steve Coffman also presented various cost-saving scenarios, which he said would allow libraries to answer more questions at less cost. He also discussed various solutions to questions that local staff are unable to answer by giving some examples of various syndicated reference services.
Contributed Paper
"The Future is Now:Reference Service for the Electronic Era" was the title of a contributed paper presented by Judy Horn, University of California, Irvine. Demonstrating how Web contact center software could be used to share Web pages and to chat with users,the presentation focused on the planning that is necessary for the successful implementation of electronic reference service. The University of California,Irvine is currently participating in a Metropolitan Cooperative Library System(MCLS) test of eGainLive+ Hipbone to assess the applications of this software to academic libraries. Using such software efficiently takes training and practice,an increase in multi-tasking proficiency (chatting and locating Web pages at the same time), and a change in the way that reference librarians normally answer reference questions. The test so far has demonstrated that this type of software has the potential to revolutionize reference service by providing for a new way of organizing services, providing an extension of traditional services, and leading us to take a fresh look at reference standards. The presentation is available at http://sun3.lib.uci.edu/~jkhorn/acrl/acrlx.htm
Panel Discussion
A panel discussion with librarians from Cornell using LivePerson and MIT, using LSSI, discussed their efforts at implementing electronic/digital reference. Entitled "Digital Dominoes: The Impact of Digital Reference on the Traditional Reference Model,"members of the panel looked first at what has been done at these two institutions to implement electronic/ digital reference and then discussed such issues as the time that it takes to implement a new service, human resources needed, staff buy-in, training, making referrals, and collaboration. Their conclusion was that, for digital reference to be effective, staffing models for reference must change significantly. Serving on the panel from MIT were Pat Flanagan, Deborah Helman, Lisa Horowitz, and Sarah Wenzel. The Cornell perspective was presented by Paul Constantine and Nancy Skipper.
Poster Sessions
Additionally, there were three poster sessions, highlighting work done at other academic institutions on electronic reference. "Repackaging Reference: Ask Us Digital Reference Service at the University of Michigan" provided tips and information on accessing,managing and planning a virtual reference service. This poster session was compiled and presented by Suzanne Grey (sgray@umich.edu)and Susan Hollar (shollar@umich.edu)of the University of Michigan. The University of Minnesota Twin Cities poster session on "Current Dimensions of Digital Reference Services" provided a multifaceted evaluation of Info Point, their digital reference service, which answersover 3,000 questions annually. Presented by John Butler, Project Director, Distance Learning Development (j-butl@tc.umn.edu),a series of charts provided detailed evaluation of their service. It was interesting to note that their evaluation showed, as have similar evaluations of electronic/digital reference services, that the bulk of the questions are asked in the middle of the day, i.e. 12-2, when the reference desks are staffed,rather than during late night or early morning hours, when librarians are not normally available. The third poster session on electronic/digital reference topics was "Why Ask Jeeves? Comparing Library Reference Services and Online'AskA' Dot Coms." Presented by Thomas Turner, Metadata Librarian and Philip Herold, Information Services Coordinator at Cornell University, this poster session examined a project designed to compare traditional and digital reference service in libraries with search services such as "Ask Jeeves." The project will evaluate the services by looking at accuracy of results, user satisfaction, and willingness to return to the site. They are testing the survey on a small group of users to make sure that it measures what they expect it to measure and then to conduct the survey on a large group of users from various population groups. They are looking for partners to help them conduct the surveys outside Cornell. If interested in participating, contact the authors at ph31@cornell.edu or tpt2@cornell.edu From the various aspects of electronic/ digital reference presented at the ACRL conference, it is obvious that it is a hot topic in academic libraries. Libraries all are struggling in this pioneer stage with identifying the best tools, new methods of doing traditional reference, new staffing configurations, and new paradigms for reference service and standards. We do not know what will emerge from this early stage, but it is obvious, as Paul Constantine strongly stressed, that times are changing and to retain our relevance in the academic community we need to quickly seek ways to meet these new challenges. Those who are interested in pursuing the topic of electronic/digital reference further may be interested in attending the 3rd Annual Virtual Reference Desk Conference to be held in Orlando, November 12-13, 2001.
Judy Horn (jkhorn@lib.uci.edu) is Head, Government Information Department, University of California, Irvine.
