rusq: Vol. 52 Issue 3: p. 208
Information Literacy Assessment: Keep It Simple, Keep It Going
Lisa O’ Connor, Larry Sheret, John A. Steele

Larry R. Sheret (sheret@marshall.edu) is Instruction and Emerging Technologies Librarian, Marshall University, Huntington, West Virginia
John A. Steele (john.steele@coyotes.usd.edu) is English Instructor, University of South Dakota, Vermillion, South Dakota
Correspondence: Correspondence concerning this column should be addressed to Lisa G. O’Connor, Associate Professor School of Library & Information Science University of Kentucky, 327 Little Library Building Lexington KY 40506-0224; email: Loconnor@uky.edu.

Is it possible to administer an information literacy assessment in only a few classes that would provide essential, adequate, data from semester to semester? In a college with a student body of about 2,000, would it be possible to obtain actionable assessment results if only 150 to 200 students were assessed each semester?

This article is the result of the creation and implementation of the information literacy assessment that was launched in the fall of 2009 by Savage Library at Western State College of Colorado (WSC). WSC changed its name to Western State Colorado University on August 1, 2012. The authors of this article, a librarian and a lecturer in English, collaborated closely to embed the information literacy assessment into multiple sections of the second-year writing class required for all Western students. This article presents an overview of the information literacy (IL) assessment and an analysis of the data obtained from the assessment. The article also provides an overview of how to embed IL instruction and IL assessment into the classroom to improve student skills in critical thinking, IL, public speaking, and research and persuasive writing.

Through a specific case study in which IL instruction and assessment was used in multiple sections of the same second-year required writing course (COTH 202: Academic Writing and Inquiry), a broader set of implications is suggested for the usefulness and relevancy of the IL assessment in almost any academic course, regardless of the discipline.

One model for how librarians and faculty might collaborate by incorporating IL instruction and assessment in a classroom setting is presented. This particular collaboration resulted in a substantial improvement in student learning outcomes as well as an easy-to-use formative method of assessing and then adjusting IL instruction as it is situated in a required writing course.


OBJECTIVES

Because the college had established IL as an essential general education requirement to ensure that students were achieving a proficient level of IL skills, the Communications and Theater program (COTH) committed itself to embedding library instruction into a class that was required for graduation. COTH 202: Academic Writing and Inquiry is a research writing and public speaking class where students learned scholarly research skills, wrote research papers, and communicated the results of their research in front of the class using a variety of media. IL was one of the learning outcomes specified for this class as well as for the COTH program.

By expanding the number of classes receiving IL instruction, by ensuring that every student received information literacy by embedding this instruction in the required COTH 202 classes, and by assessing IL skills and making changes to the IL program based upon data from the assessments, it was expected that IL skills would increase from semester to semester.

In the fall of 2008, Savage Library set out to create its student IL assessment. Assessment questions from several IL assessments formulated by colleges and universities throughout the country provided excellent models from which the library created its assessment questions. Each question was tied to one or more learning outcomes from ACRL’s list of five IL standards and eighty-seven learning outcomes.1 The assessment comprised fifty questions worth ten points each. The majority of the questions were from ACRL Standards 1–3. Relatively few questions were linked to Standards 4 and 5 because those standards are difficult or impossible to operationalize and assess quantitatively. It was up to the faculty to assess those standards by grading assignments according to rubrics established by the COTH program. It was our goal to have an assessment that would be a tool to improve IL instruction from semester to semester and a measure of how well the program achieved national standards by mapping each question to ACRL’s IL competencies.

Each question was labeled to show which standard, indicator, and learning outcome was being assessed. Some questions were relevant to multiple learning outcomes. The following question assessed IL Standard 2, Performance Indicator 3, and Learning Outcome b:

2.3.b. Library of Congress Classification

When using books in the library, we request that you not reshelve them when finished. Suppose that, “wishing to be helpful,” you decide to reshelve the books you have used. Place the following books back on the shelf in their proper order.

(Note: The 2.3.b. label that precedes the question heading, Library of Congress Classification, assesses IL Standard 2, Performance Indicator 3, and learning outcome/learning activity b.)

Standard Two:

The information literate student accesses needed information effectively and efficiently.

Performance Indicators:

3. The information literate student constructs and implements effectively designed search strategies.

Outcomes Include:

b. Uses various classification schemes and other systems (e.g., call number systems or indexes) to locate information resources within the library or to identify specific sites for physical exploration.

Once the initial set of questions was created, the assessment was distributed to a dozen librarians throughout the country, to the librarians and library work-studies at WSC, and to four WSC faculty members to review and provide feedback. Extensive revisions were made until everyone agreed that the questions were pertinent and easy to understand.

Once the questions were ready, they were placed onto Blackboard to allow the assessment to be taken as a homework assignment. The librarian added students into Blackboard and ran the assessment to make it easier and more likely that more faculty members would have their students participate in the assessment and to make data analysis easier by ensuring that the librarian had access to the results.


THE ASSESSMENT PILOT

The assessment was piloted during spring 2009. During the pilot phase, the assessment was administered twice in most classes, once as a pre–test and later in the semester as a post–test. Some negative aspects came to light. Many students wanted to get a good score, but pre–testing meant that students would probably miss at least half of the questions. When plus deltas were conducted, several students complained that the assessment made them feel stupid. Some classes that were supposed to be post–tested were not because the post–test conflicted too much with the academic demands on students late in the semester. Some faculty chose not to post–test so not to over-burden their students if they were working on research papers or capstone projects. In some cases, students who had completed the pre–test were upset about being tested twice. They did not see the point in taking it twice. Several didn’t bother, or didn’t have time, to take the post–test. Because there was not sufficient faculty and student buy-in to the idea of being tested twice during the semester, the validity of the data generated by the post–test were in doubt.

To mitigate these problems, the following changes were made:

  • To deal with the reticence of students to take the assessment twice, some classes were given only a pre–test while others received only a post–test. More accurately, instead of a pre–test/post–test model, we moved to an assessment model where students were assessed only once, either pre–instruction or post–instruction. Although the usefulness of this research method is a bit weaker than the traditional pre–test/post–test model due to its inability to account for variables, it was reasonable to suppose that if IL skills were increasing from year to year, this would be revealed through positive longitudinal data derived from post–instruction scores.
  • So as not to take away from class time, most assessments were delivered online as classroom assignments. One exception to this was the baseline score, which was determined almost exclusively by using data from pre–instruction freshmen tested in class in a controlled computer lab setting.
  • To mitigate complaints that the assessment made students feel stupid, those who received the pre–instruction test were told before taking it that any score over 40 percent was considered to be good. It was explained as similar to taking a chemistry final on the second or third day of class. It wasn’t to be expected that they would do well. When they understood the purpose of the pre–instruction test, to assess areas of strengths and weaknesses, most were satisfied, but not all.
  • To satisfy students who wanted to achieve a high score, the assessment could be taken as many times as the students wished. Most questions provided feedback that students could read after the assessment had been submitted. Students achieved much higher scores the second or third time around. Nevertheless, only the first attempt score was used for data analysis.


ASSESSMENT LAUNCH

The official assessment was launched in fall 2009. All students at WSC are required to take COTH 202 to graduate. Because COTH had decided to invite librarians to work with every section, almost every student was assured of receiving at least one IL instruction session, with most receiving at least two sessions with the librarian. Because the course was required for every student, there were freshmen, sophomores, juniors and seniors in almost every class, though most of students were sophomores. Assessing COTH 202 classes turned out to be an excellent way to sample every grade level.

After each class was assessed, the results were emailed to the faculty so they could adjust their instruction to address areas of weakness. The grade level of each student in each class was determined to aggregate the statistics not only by pre– and post–instruction, but by freshman, sophomore, junior and senior.


RESULTS

A freshmen baseline score of 42 percent was established. The baseline was derived from incoming freshmen who were tested before receiving IL instruction. Most of these freshmen were in 099 remedial English classes, but some of them were in 100-level English and Psychology classes. In most freshman classes, there were one or two students who scored zero on the assessment because they were unfamiliar with Blackboard and would select the “finish” button prematurely, thinking that it meant they were finished with the question, when it actually meant they were finished taking the assessment. These scores were thrown out. After eliminating these outliers, the average score was used rather than the median score. Some students insisted on having the opportunity to achieve a high score on the assessment, so multiple attempts were allowed, but all assessment score data that was used for analysis and included in the longitudinal analysis was taken from the first attempt. Every class had at least two students who took the assessment two or three times until they achieved a score of 90 percent or better. It was an unexpected surprise to see how many students used the assessment to study the feedback section and increase their IL skills.

Useful formative and summative assessment data was obtained by giving some classes a pre–instruction assessment and other classes a post–instruction assessment. Although this lacked the ability to control for variables, it provided credible data that the information literacy program was effective as evidenced by positive trend lines that spanned semesters.

Table 1 and figure 1 record and plot the results of the assessment. Note the remarkable positive slope of the solid red line. Since most of the data for this line are derived from sophomores in an embedded IL class that chose to receive the post–instruction assessment, it is clear that embedding has a noticeable positive effect on student learning outcomes. The post–instruction assessments were administered after only the second library classroom session.

Offering post–instruction assessment to every student in a few classes appears to provide adequate data to document that learning outcomes are being achieved on a program and institutional level.

The dotted red line (pre–instructed sophomores) reveals improvements in IL skill levels with sophomores from one semester to the next. This is likely attributable to the fact that some sophomores received IL instruction during their freshman year or in other sophomore classes. A cumulative benefit derives from exposing students to IL concepts in a variety of classes from year to year. The positive slope of this line also shows the formative benefits of the assessment. When the data are analyzed, areas of greatest weakness were easily identified. This information was shared with the faculty so that both they and the librarian could modify IL instruction and focus on areas of greatest need (for an example of the question scores and how they are displayed by Blackboard, see the third column under “Whole Group” in appendix A).

Even though the pre– and post–instruction results for juniors and seniors generated a positive trend line, the embedded COTH 202 class surpassed these results. Juniors and seniors improved assessment scores at a rate of about 3 percent each semester. The capstone score for seniors four semesters after the assessment was launched was 58 percent. The trend-line predicts that after an additional four semesters, the capstone score for seniors would be about 70 percent.

An interesting result appeared in the post–instruction scores of freshmen in fall 2010, and spring 2011, with 2011 scores averaging 59.2 percent compared with 57.8 percent for seniors. There are at least two explanations for this. First, most of the freshmen who were tested post–instruction were in John Steele’s embedded class, and they scored almost as high as the sophomores in his class. The reason seniors didn’t score higher is likely due, in part, to the fact that the librarians never teach to the test. The instruction is geared toward the class assignment. IL instruction for COTH 202 covers many IL concepts, but the IL instruction in capstone classes rarely covers more than a handful of IL concepts (focusing perhaps on the use of a specialized database such as eHRAF). It is clear that if more embedded classes were taught throughout the college, senior capstone scores would exceed the 70 percent result currently predicted.

Another score should be noted. The single ENG 099 pre–instruction entry of 42 percent is identical to the trend line for freshmen pre–instruction scores, most of which were derived from COTH 202 assessment. This indicates that students who did not qualify to take COTH 202 in their freshmen year performed at the same IL skill level as their peers in COTH 202. This is surprising, and it suggests that although these ENG 099 freshmen may not have been able to draft a proper sentence or write an essay, they understood IL concepts as well as their freshmen peers in COTH 202, according to data generated by this assessment.

Stagnation of scores for upper-division students may also be because the greatest positive impact on post-instruction scores came from better learning outcomes in ACRL Standards 1 and 2 for sophomores, which are the standards that librarians tend to focus on. Standards 3, 4, and 5 are more difficult for librarians to have an impact on, or assess, because they are taught by classroom faculty and require qualitative assessments that are unable to be achieved by an IL assessment tool such as the one we developed.


FACULTY PERSPECTIVE ON EMBEDDING INFORMATION LITERACY IN THE CLASSROOM

From a faculty perspective, embedding information literacy into a research writing course makes perfect sense. At WSC, COTH 202: Academic Writing and Inquiry, the second-year writing requirement of all students, addresses many facets of Western’s institutional mission to provide students with “a solid foundation in written and spoken communication, problem solving, critical thinking, and creativity.”2 While focusing primarily on research writing and advocacy, the COTH 202 program manual also states as a common goal “a consistent focus on information literacy.” The COTH 202 program manual further states (see appendix B):

  • COTH 202 recognizes that information literacy is an essential component of written and spoken advocacy. Instructors are highly encouraged to incorporate library sessions that extend classroom discussion of information literacy. The staff at Savage Library are well aware of the heavy research component to this course and are very helpful with orienting the students to the variety of academic research avenues in today’s world. Our current Director of Library Services, Nancy Gauss, provided the following information to further clarify the role the library plays in assisting our instructors as we guide the students to acquire a collegiate level of information literacy:
    • Information literacy is a set of abilities requiring students to “recognize when information is needed and have the ability to locate, evaluate, and use effectively the needed information.” Information Literacy Competency Standards for Higher Education. American Library Association. 2006. http://www.ala.org/acrl/ilcomstan.html (Accessed 1 May, 2008).
    • The development of information literacy abilities is necessary for students to:
      • “learn to evaluate sources for accuracy, relevance, credibility, reliability, and bias” as required for a state-guaranteed communication course
      • “select, analyze and employ primary and secondary sources in order to write and present evidence-based arguments” as referenced in the suggestions for coursework that will aid in assessing student learning
    • During one or more sessions, each instructor collaborates with a librarian on providing instruction and exercises that will help students develop and enhance their information literacy abilities. It is our intention that students apply the skills and techniques learned during these sessions to other classes and situations.
    • Specific topics addressed during sessions may include:
      • Distinguishing different sources of information and identifying library resources and databases appropriate for student needs
      • Database searching strategies
      • General orientation to finding materials in Savage Library and through electronic resources
      • Strategies for evaluating information from a variety of sources
      • Guidelines for citing information sources3

Information literacy dovetails nicely with the learning outcomes for COTH 202; in fact, students would not be able to adequately meet the standard learning outcomes for COTH 202 without IL skills. To ensure that COTH 202 does in fact meet these learning outcomes, WSC employs a rigorous series of assessments of all its courses.

The policy paper students must write is the primary assignment that utilizes information literacy and is the assignment that requires embedding information literacy. The policy paper is subsequently adapted into a policy speech, in which students must persuade their fellow classmates and instructor that their policy is the best policy for a given issue that each individual student has chosen to explore through research, reasoning, and argument. In COTH 202, the policy paper is framed in the following manner: (1) a current problem(s) exists because of a failed policy or a lack of a policy; (2) there are alternative policies that could solve the problem; (3) there is an optimal policy that would best solve the problem; (4) the optimal policy is both practical and beneficial and should be adopted as the new policy. Since most students are beginning scholars just entering the ongoing conversations related to their particular policy issue, research is required to familiarize themselves with what has already been said and done in relation to the policy. They must contextualize the issue through their own research. To build a persuasive argument, students must develop an ethos, pathos, and logos to be used in their arguments, which is where IL becomes vital to their success.

By working with the librarians, scheduling times for the entire class to meet in the library, and co-teaching with the IL librarian, the COTH 202 instructor is able to embed IL in such a way that IL skills are developed through the application of those skills to a specific assignment. Theory and practice meet in a practical application; therefore including the IL assessment has proved to be a vital and useful tool to teach information literacy by providing a feedback loop for the instructor to identify problem areas and adapt on-the-fly during the course of the semester to address and correct any deficiencies. This process has greatly improved the quality of student policy papers and policy speeches by strengthening the evidence used to support student claims. Students build a much stronger ethos through the credible research they do during classes scheduled in the library with the information literacy librarian, who co-teaches with the instructor to meet the needs of the assignment.

Having students receive information literacy instruction before taking the assessment, which occurs before their specific research for their policy paper, allows the instructor to better prepare the students to meet the challenges they may encounter while doing their research. One practice that has been particularly beneficial to students is asking students to volunteer a topic during instruction, which the information literacy librarian can use in class as an example inquiry to begin the research process. Modeling the research process, incorporating all of the information literacy standards, gives students a concrete example for how to conduct their own research. Following this instructional “walkthrough” students are “turned loose” to do their research in class, which allows both the COTH 202 instructor and the librarian to give individualized help to students with specific research problems as they encounter them.

It has been the experience of this COTH 202 instructor that students sincerely and greatly appreciate this opportunity. They discover they can find a wealth of information on their policy topics, and they are not overwhelmed by this wealth of information because the instruction given to them teaches them how to evaluate and use the information effectively. It is an empowering experience for both the instructor and the students. The information literacy assessment is the cornerstone of this experience. When it is used as a teaching tool, IL skills are grounded in a praxis where their immediate application to objectives set forth by the instructor helps students to internalize the skills they will need for the rest of their lives as information literature scholars and citizens. For the COTH 202 instructor, embedding information literacy and the information literacy assessment into their class is an invaluable resource. The collaboration between faculty and librarian enriches the classroom and makes IL skills come alive.


CONCLUSION

In many educational establishments, students are bribed to take assessments by offering them free pizza, gift certificates, or the chance to win a significant prize through a raffle. WSC achieved randomness by finding faculty who were interested in assessment and then assessing every student who happened to be in those classes. This approach provided a way to achieve random selection, and it had the advantage of working with faculty who were interested in the assessment. Nor was the experience grievous to the students because it integrated IL assessment with the courses that the students were taking. Instruction was delivered that supported the needs of the class and any current assignment rather than instructing to the assessment. The assessment had a context that gave it academic meaning and therefore more value to the students who took it—as long as they only had to take it once. The complete online assessment is in appendix C.4


References
1. American Library Association, "ACRL Standards Committee, Information Literacy Competency Standards for Higher Education"January 18, 2000, www.ala.org/acrl/standards/informationliteracycompetency#stan
2. Western State College of Colorado COTH 202 Academic Writing & Inquiry Program Manual (Gunnison, CO:  Communications and Theatre Department, Western State College of Colorado, 2008):
3. Ibid., 3
4. Western State College of Colorado (Gunnison, CO:  Savage Library, Western State College of Colorado, 2010): "Information Literacy Assessment: Questions with Feedback. "
Appendix A. Assessment Manager Assessment Reports

Appendix B

The learning outcomes detailed in the COTH 202 Program Manual comply with state guidelines, and as such students who complete COTH 202, regardless of the instructor’s academic freedom to tailor the class to “their knowledge, experience and strengths” to meet these learning outcomes, will be able to exhibit the following:

  • an understanding of the writing and speaking processes as applied to the rhetorical tasks of analysis and advocacy
  • an ability to apply rhetorical strategies of informing, persuading, and arguing
  • an understanding of the characteristics and challenges of written, oral, and mediated messages
  • an understanding of how to employ multiple strategies for generating, revising, and completing an extensive analytical essay
  • the nature of critical thinking through advocacy
  • an ability to demonstrate thoughtful engagement with complex readings through written and oral expression. (COTH 202 Program Manual)

Students meet these criteria through the following common practices in all COTH 202 classes:

  • lessons that require the design of persuasive messages for various, specific audiences
  • exercises that clarify the different modes of preparation according to purpose
  • assignments that develop communication competence, both written and oral
  • a writing process that requires multiple strategies for writing including drafting, revising and editing
  • curriculum that requires they learn and employ the necessary elements of advocacy
  • coursework in which they select, analyze and employ primary and secondary sources to write and present evidence-based arguments. (COTH 202 Program Manual)


Appendix C. Information Literacy Assessment

See supplemental material


Figures

Figure 1

Standards 1–5



Tables
Statement Response Value Correct Match
E 415.9 F79 H5 not answered 0.0% 1st
E 415.9 G19 E33 not answered 0.0% 3rd
E 415.9 F8 B7 not answered 0.0% 2nd
Score: 0/10

Table 1

Standards 1–5


99 pre 99 post 100 pre 100 post 200 pre 200 post 300 pre 300 post 400 pre 400 post
fall 2009 42.3% 52.8%
spring 2010 33.0% 42.0% 51.0% 55.0%
fall 2010 42.0% 48.0% 57.0% 48.0% 52.2% 56.0% 51.2% 59.0% 57.0%
spring 2011 43.0% 59.2% 49.0% 60.0% 54.0% 57.8%

Information Literacy Skills Assessment COTH202 Spring 09: Download Records


Percent Answering Correctly
Question Title N Whole Group Upper 25% Lower 25% Discrimination Mean Standard Deviation Actions?
1.2.a. Speed of information 35 11.43 11.11 11.11 0.00 31.43% 33.00%
3.2.a. Evaluating the W3 35 80.00 100.00 55.56 44.44 80.00% 40.58%
2.5.c. What citation style is this? 35 25.71 55.56 0.00 55.56 25.71% 44.34%
2.5.c. How many pages are in this article? 35 14.29 33.33 11.11 22.22 14.29% 35.50%
2.5.c. Who is the publisher? 35 22.86 66.67 0.00 66.67 22.86% 42.60%
2.5.c. Identify the volume number. 35 8.57 11.11 0.00 11.11 8.57% 28.40%
2.5.c. Identify the author. 35 65.71 77.78 33.33 44.44 65.71% 48.16%
2.5.c. Identify the title of this article. 35 17.14 44.44 0.00 44.44 17.14% 38.24%
2.2.c.f. Controlled vocabulary search for a journal article 35 0.00 0.00 0.00 0.00 25.00% 17.15%

The following chart is used to assess COTH 202:


Written Communication Skills Student Learning Experiences and Alignment to Goals Tools used or to be used asEvidence of Student Learning
Students will exhibit:1. an understanding of writing and speaking process as applied to the rhetorical tasks or analysis and advocacy. Students will provide a written and oral challenge to a problem-solution message using language effectively and persuasively. To develop their argument, students will write and read texts appropriate for the issue, employ multiple drafts, and use full sentence outlines for their speeches. Analytical EssayPolicy Speech
2. an ability to apply rhetorical strategies of informing, persuading, and arguing. Students will be instructed on the importance of credibility, form and logic as they complete informative and persuasive assignments. Policy Speech
3. an understanding of the characteristics and challenges of written, oral, and mediated messages. Both writing assignments and speeches will be assessed on the student’s rhetorical ability to compose and communicate messages as appropriate for their audiences. Students will be instructed in the necessary differences between preparing written and oral assignment, as well as the vital connections between the two. Critical thinking and the connection between form and content are emphasized throughout. Policy SpeechAnalytical Essay
4. an understanding of how to employ multiple strategies for generating, revising, and completing an extensive analytical essay. Students will complete and extended research essay which works to support a factual value or policy claim. Students will use multiple drafts and critique the work of others within their essay. Important to the process will be learning to evaluate sources. Documentation of sources will be emphasize,d and the final writing project will be assessed on form and grammatical content as well. Analytical Essay
5. an ability to assess the nature of critical thinking through advocacy. Students will be introduced to various forms of logic and rhetorical processes with which to create and analyze advocacy. In analyzing current public issues students will employ critical thinking to dissect the arguments presented. Students will also be asked to critically assess which forms of rhetorical theory are most pertinent as they develop advocacy for their chosen topics. Analytical EssayPolicy Speech
6. an ability to demonstrate thoughtful engagement with complex readings through written and oral expression. Students will be asked to research, evaluate, and critique a variety of sources to support their processes of informing, persuading, and arguing. Inherent in this process is the discussion of the variety of ways to evaluate evidence as well as the types of evidence best suited for particular rhetorical contexts. Policy SpeechAnalytical Essay


Article Categories:
  • Library Reference and User Services
    • Columns

Refbacks

  • There are currently no refbacks.


ALA Privacy Policy

© 2019 RUSA