In the fall of 2008, Savage Library set out to create its student IL assessment. Assessment questions from several IL assessments formulated by colleges and universities throughout the country provided excellent models from which the library created its assessment questions. Each question was tied to one or more learning outcomes from ACRL’s list of five IL standards and eighty-seven learning outcomes.1 The assessment comprised fifty questions worth ten points each. The majority of the questions were from ACRL Standards 1–3. Relatively few questions were linked to Standards 4 and 5 because those standards are difficult or impossible to operationalize and assess quantitatively. It was up to the faculty to assess those standards by grading assignments according to rubrics established by the COTH program. It was our goal to have an assessment that would be a tool to improve IL instruction from semester to semester and a measure of how well the program achieved national standards by mapping each question to ACRL’s IL competencies.

Each question was labeled to show which standard, indicator, and learning outcome was being assessed. Some questions were relevant to multiple learning outcomes. The following question assessed IL Standard 2, Performance Indicator 3, and Learning Outcome b:

The assessment was piloted during spring 2009. During the pilot phase, the assessment was administered twice in most classes, once as a pre–test and later in the semester as a post–test. Some negative aspects came to light. Many students wanted to get a good score, but pre–testing meant that students would probably miss at least half of the questions. When plus deltas were conducted, several students complained that the assessment made them feel stupid. Some classes that were supposed to be post–tested were not because the post–test conflicted too much with the academic demands on students late in the semester. Some faculty chose not to post–test so not to over-burden their students if they were working on research papers or capstone projects. In some cases, students who had completed the pre–test were upset about being tested twice. They did not see the point in taking it twice. Several didn’t bother, or didn’t have time, to take the post–test. Because there was not sufficient faculty and student buy-in to the idea of being tested twice during the semester, the validity of the data generated by the post–test were in doubt.

To mitigate these problems, the following changes were made:

• To deal with the reticence of students to take the assessment twice, some classes were given only a pre–test while others received only a post–test. More accurately, instead of a pre–test/post–test model, we moved to an assessment model where students were assessed only once, either pre–instruction or post–instruction. Although the usefulness of this research method is a bit weaker than the traditional pre–test/post–test model due to its inability to account for variables, it was reasonable to suppose that if IL skills were increasing from year to year, this would be revealed through positive longitudinal data derived from post–instruction scores.
• So as not to take away from class time, most assessments were delivered online as classroom assignments. One exception to this was the baseline score, which was determined almost exclusively by using data from pre–instruction freshmen tested in class in a controlled computer lab setting.
• To mitigate complaints that the assessment made students feel stupid, those who received the pre–instruction test were told before taking it that any score over 40 percent was considered to be good. It was explained as similar to taking a chemistry final on the second or third day of class. It wasn’t to be expected that they would do well. When they understood the purpose of the pre–instruction test, to assess areas of strengths and weaknesses, most were satisfied, but not all.
• To satisfy students who wanted to achieve a high score, the assessment could be taken as many times as the students wished. Most questions provided feedback that students could read after the assessment had been submitted. Students achieved much higher scores the second or third time around. Nevertheless, only the first attempt score was used for data analysis.

The official assessment was launched in fall 2009. All students at WSC are required to take COTH 202 to graduate. Because COTH had decided to invite librarians to work with every section, almost every student was assured of receiving at least one IL instruction session, with most receiving at least two sessions with the librarian. Because the course was required for every student, there were freshmen, sophomores, juniors and seniors in almost every class, though most of students were sophomores. Assessing COTH 202 classes turned out to be an excellent way to sample every grade level.

After each class was assessed, the results were emailed to the faculty so they could adjust their instruction to address areas of weakness. The grade level of each student in each class was determined to aggregate the statistics not only by pre– and post–instruction, but by freshman, sophomore, junior and senior.

A freshmen baseline score of 42 percent was established. The baseline was derived from incoming freshmen who were tested before receiving IL instruction. Most of these freshmen were in 099 remedial English classes, but some of them were in 100-level English and Psychology classes. In most freshman classes, there were one or two students who scored zero on the assessment because they were unfamiliar with Blackboard and would select the “finish” button prematurely, thinking that it meant they were finished with the question, when it actually meant they were finished taking the assessment. These scores were thrown out. After eliminating these outliers, the average score was used rather than the median score. Some students insisted on having the opportunity to achieve a high score on the assessment, so multiple attempts were allowed, but all assessment score data that was used for analysis and included in the longitudinal analysis was taken from the first attempt. Every class had at least two students who took the assessment two or three times until they achieved a score of 90 percent or better. It was an unexpected surprise to see how many students used the assessment to study the feedback section and increase their IL skills.

Useful formative and summative assessment data was obtained by giving some classes a pre–instruction assessment and other classes a post–instruction assessment. Although this lacked the ability to control for variables, it provided credible data that the information literacy program was effective as evidenced by positive trend lines that spanned semesters.

Table 1 and figure 1 record and plot the results of the assessment. Note the remarkable positive slope of the solid red line. Since most of the data for this line are derived from sophomores in an embedded IL class that chose to receive the post–instruction assessment, it is clear that embedding has a noticeable positive effect on student learning outcomes. The post–instruction assessments were administered after only the second library classroom session.

Offering post–instruction assessment to every student in a few classes appears to provide adequate data to document that learning outcomes are being achieved on a program and institutional level.

The dotted red line (pre–instructed sophomores) reveals improvements in IL skill levels with sophomores from one semester to the next. This is likely attributable to the fact that some sophomores received IL instruction during their freshman year or in other sophomore classes. A cumulative benefit derives from exposing students to IL concepts in a variety of classes from year to year. The positive slope of this line also shows the formative benefits of the assessment. When the data are analyzed, areas of greatest weakness were easily identified. This information was shared with the faculty so that both they and the librarian could modify IL instruction and focus on areas of greatest need (for an example of the question scores and how they are displayed by Blackboard, see the third column under “Whole Group” in appendix A).

Even though the pre– and post–instruction results for juniors and seniors generated a positive trend line, the embedded COTH 202 class surpassed these results. Juniors and seniors improved assessment scores at a rate of about 3 percent each semester. The capstone score for seniors four semesters after the assessment was launched was 58 percent. The trend-line predicts that after an additional four semesters, the capstone score for seniors would be about 70 percent.

An interesting result appeared in the post–instruction scores of freshmen in fall 2010, and spring 2011, with 2011 scores averaging 59.2 percent compared with 57.8 percent for seniors. There are at least two explanations for this. First, most of the freshmen who were tested post–instruction were in John Steele’s embedded class, and they scored almost as high as the sophomores in his class. The reason seniors didn’t score higher is likely due, in part, to the fact that the librarians never teach to the test. The instruction is geared toward the class assignment. IL instruction for COTH 202 covers many IL concepts, but the IL instruction in capstone classes rarely covers more than a handful of IL concepts (focusing perhaps on the use of a specialized database such as eHRAF). It is clear that if more embedded classes were taught throughout the college, senior capstone scores would exceed the 70 percent result currently predicted.

Another score should be noted. The single ENG 099 pre–instruction entry of 42 percent is identical to the trend line for freshmen pre–instruction scores, most of which were derived from COTH 202 assessment. This indicates that students who did not qualify to take COTH 202 in their freshmen year performed at the same IL skill level as their peers in COTH 202. This is surprising, and it suggests that although these ENG 099 freshmen may not have been able to draft a proper sentence or write an essay, they understood IL concepts as well as their freshmen peers in COTH 202, according to data generated by this assessment.

Stagnation of scores for upper-division students may also be because the greatest positive impact on post-instruction scores came from better learning outcomes in ACRL Standards 1 and 2 for sophomores, which are the standards that librarians tend to focus on. Standards 3, 4, and 5 are more difficult for librarians to have an impact on, or assess, because they are taught by classroom faculty and require qualitative assessments that are unable to be achieved by an IL assessment tool such as the one we developed.

From a faculty perspective, embedding information literacy into a research writing course makes perfect sense. At WSC, COTH 202: Academic Writing and Inquiry, the second-year writing requirement of all students, addresses many facets of Western’s institutional mission to provide students with “a solid foundation in written and spoken communication, problem solving, critical thinking, and creativity.”² While focusing primarily on research writing and advocacy, the COTH 202 program manual also states as a common goal “a consistent focus on information literacy.” The COTH 202 program manual further states (see appendix B):

• COTH 202 recognizes that information literacy is an essential component of written and spoken advocacy. Instructors are highly encouraged to incorporate library sessions that extend classroom discussion of information literacy. The staff at Savage Library are well aware of the heavy research component to this course and are very helpful with orienting the students to the variety of academic research avenues in today’s world. Our current Director of Library Services, Nancy Gauss, provided the following information to further clarify the role the library plays in assisting our instructors as we guide the students to acquire a collegiate level of information literacy:
  • Information literacy is a set of abilities requiring students to “recognize when information is needed and have the ability to locate, evaluate, and use effectively the needed information.” Information Literacy Competency Standards for Higher Education. American Library Association. 2006. http://www.ala.org/acrl/ilcomstan.html (Accessed 1 May, 2008).
  • The development of information literacy abilities is necessary for students to:
    • “learn to evaluate sources for accuracy, relevance, credibility, reliability, and bias” as required for a state-guaranteed communication course
    • “select, analyze and employ primary and secondary sources in order to write and present evidence-based arguments” as referenced in the suggestions for coursework that will aid in assessing student learning
  • During one or more sessions, each instructor collaborates with a librarian on providing instruction and exercises that will help students develop and enhance their information literacy abilities. It is our intention that students apply the skills and techniques learned during these sessions to other classes and situations.
  • Specific topics addressed during sessions may include:
    • Distinguishing different sources of information and identifying library resources and databases appropriate for student needs
    • Database searching strategies
    • General orientation to finding materials in Savage Library and through electronic resources
    • Strategies for evaluating information from a variety of sources
    • Guidelines for citing information sources³

Information literacy dovetails nicely with the learning outcomes for COTH 202; in fact, students would not be able to adequately meet the standard learning outcomes for COTH 202 without IL skills. To ensure that COTH 202 does in fact meet these learning outcomes, WSC employs a rigorous series of assessments of all its courses.

The policy paper students must write is the primary assignment that utilizes information literacy and is the assignment that requires embedding information literacy. The policy paper is subsequently adapted into a policy speech, in which students must persuade their fellow classmates and instructor that their policy is the best policy for a given issue that each individual student has chosen to explore through research, reasoning, and argument. In COTH 202, the policy paper is framed in the following manner: (1) a current problem(s) exists because of a failed policy or a lack of a policy; (2) there are alternative policies that could solve the problem; (3) there is an optimal policy that would best solve the problem; (4) the optimal policy is both practical and beneficial and should be adopted as the new policy. Since most students are beginning scholars just entering the ongoing conversations related to their particular policy issue, research is required to familiarize themselves with what has already been said and done in relation to the policy. They must contextualize the issue through their own research. To build a persuasive argument, students must develop an ethos, pathos, and logos to be used in their arguments, which is where IL becomes vital to their success.

By working with the librarians, scheduling times for the entire class to meet in the library, and co-teaching with the IL librarian, the COTH 202 instructor is able to embed IL in such a way that IL skills are developed through the application of those skills to a specific assignment. Theory and practice meet in a practical application; therefore including the IL assessment has proved to be a vital and useful tool to teach information literacy by providing a feedback loop for the instructor to identify problem areas and adapt on-the-fly during the course of the semester to address and correct any deficiencies. This process has greatly improved the quality of student policy papers and policy speeches by strengthening the evidence used to support student claims. Students build a much stronger ethos through the credible research they do during classes scheduled in the library with the information literacy librarian, who co-teaches with the instructor to meet the needs of the assignment.

Having students receive information literacy instruction before taking the assessment, which occurs before their specific research for their policy paper, allows the instructor to better prepare the students to meet the challenges they may encounter while doing their research. One practice that has been particularly beneficial to students is asking students to volunteer a topic during instruction, which the information literacy librarian can use in class as an example inquiry to begin the research process. Modeling the research process, incorporating all of the information literacy standards, gives students a concrete example for how to conduct their own research. Following this instructional “walkthrough” students are “turned loose” to do their research in class, which allows both the COTH 202 instructor and the librarian to give individualized help to students with specific research problems as they encounter them.

It has been the experience of this COTH 202 instructor that students sincerely and greatly appreciate this opportunity. They discover they can find a wealth of information on their policy topics, and they are not overwhelmed by this wealth of information because the instruction given to them teaches them how to evaluate and use the information effectively. It is an empowering experience for both the instructor and the students. The information literacy assessment is the cornerstone of this experience. When it is used as a teaching tool, IL skills are grounded in a praxis where their immediate application to objectives set forth by the instructor helps students to internalize the skills they will need for the rest of their lives as information literature scholars and citizens. For the COTH 202 instructor, embedding information literacy and the information literacy assessment into their class is an invaluable resource. The collaboration between faculty and librarian enriches the classroom and makes IL skills come alive.

In many educational establishments, students are bribed to take assessments by offering them free pizza, gift certificates, or the chance to win a significant prize through a raffle. WSC achieved randomness by finding faculty who were interested in assessment and then assessing every student who happened to be in those classes. This approach provided a way to achieve random selection, and it had the advantage of working with faculty who were interested in the assessment. Nor was the experience grievous to the students because it integrated IL assessment with the courses that the students were taking. Instruction was delivered that supported the needs of the class and any current assignment rather than instructing to the assessment. The assessment had a context that gave it academic meaning and therefore more value to the students who took it—as long as they only had to take it once. The complete online assessment is in appendix C.⁴