This study seeks to determine the effectiveness of the BiblioBouts information literacy game for improving the quality of the sources undergraduate students cite in their written papers. It answers these specific research questions:

1. Is the quality of the sources undergraduate students choose for their in-game bibliographies better than the quality of the sources they originally contributed to the game?
2. Is the quality of the sources students cite in their final papers better than the quality of the sources they choose for their in-game bibliographies?
3. Do BiblioBouts players cite more scholarly sources in their final-paper bibliographies than nonplayers?
4. Are students’ source credibility assessments comparable to the quality ratings of expert coders?

Answers to these research questions will help determine the effectiveness of BiblioBouts for teaching students information literacy skills and concepts and guide the future design and development of the BiblioBouts game.

BiblioBouts has been conceived as a web-based game with social networking features to give students guidance while conducting library research to complete a written paper. BiblioBouts’ design, development, and evaluation has been supported with funds from the Institute of Museum and Library Services (IMLS). This paper’s first author is the principal investigator of the IMLS-sponsored research project, and although she is ultimately responsible for all aspects of BiblioBouts’ design, development, and evaluation, she is assisted by project staff charged with graphic design, programming, information literacy training, user support, and data collection and analysis. Librarians at four participating institutions also assist, some recruiting instructors at their universities to incorporate BiblioBouts into their academic courses and others deploying BiblioBouts in their library’s information literacy instruction program. This paper’s second and third authors are project staff and an instructor, respectively, whose class played BiblioBouts in winter 2011. Between September 2010 and December 2013, BiblioBouts has been available for free to instructors at colleges and universities in North America and abroad; however, evaluation activities are restricted to participating institutions where librarians assist project staff with human subjects review board business.

BiblioBouts is an online tournament made up of a series of mini-games or bouts, each of which introduces students to a specific subset of information literacy skills within the overall research process. Instructors choose a broad-based topic for students to research, then schedule the game’s four bouts over a three-week period. Table 1 describes the game’s bouts, their suggested duration, and summarizes the information literacy skills, concepts, and tools students encounter when playing each bout.

In the initial Donor bout, students search the web and library databases for sources and save them using the Zotero citation management tool. In the Closer bout, they choose their best sources to “do battle” in the game. In the Tagging and Rating bout, students evaluate the content and quality of their opponents’ sources including rating their credibility and relevance on a 100-point scale. Lastly, students specify a research topic and choose the best sources from the pool of everyone’s sources for a best bibliography on this topic. The objective of BiblioBouts is to donate the very best sources that one’s opponents rate highly and choose for their paper’s best bibliography.

The evolution of academic librarianship is now being driven by a paradigm shift occurring in society as a whole as the result of the rapid development of new information technologies.³ One outcome of this paradigm change has been the creation of the “blended librarianship” model and its vision of integrating library services and practices into the teaching and learning process.⁴ Blended librarians embrace educational technology and design thinking in creating innovative approaches to their educational role.⁵ The Association of College and Research Libraries’ Information Competency Standards for Higher Education have been mapped to progressively advancing levels of cognitive development.⁶ In response, new methods of delivering information literacy instruction are being explored in which information technology is utilized to help support student learning.⁷

Educational games are well suited to scaffold students’ information literacy skills acquisition. The use of games for learning has been widely researched, and the literature on “serious” or educational games is extensive, covering many educational disciplines and game genres. Several authors have discussed the elements of traditional games that incorporate principles of good learning, such as developing organizational and problem-solving skills and engagement through goals, feedback, interaction, and achievements.⁸ In his foundational work “What Video Games Have to Teach Us about Learning and Literacy,” Gee presents a list of thirty-six learning principles embodied by video games, including active critical learning, meta-level thinking, experimental probing, and the on-demand and just-in-time presentation of needed information.⁹ Games enable students to learn by doing, undertake purposeful and meaningful tasks, reflect on their experiences and work with others to achieve learning goals, while also helping create engagement.¹⁰ Ultimately, effective game design can create personalized learning experiences that motivate players to learn new skills without realizing they are in the midst of the learning process.¹¹

Educational games can incorporate these principles of good learning into information literacy instruction as a structure for learning and practicing the information-gathering process.¹² As students search for and evaluate information, they have the opportunity for repeated practice and reinforcement, which can be particularly effective for skills-based learning such as information literacy, where logical thinking and problem solving are crucial components.¹³

To measure the effect of the BiblioBouts game on student learning of information literacy skills, an evaluation instrument was developed to evaluate the quality of sources chosen during the game process. Citation analysis of student bibliographies is frequently used to measure students’ information literacy knowledge generally or the effectiveness of information literacy instruction specifically. In fact, a meta-analysis of 91 case studies of information literacy assessment found that citation analysis of bibliographies was used in 17 percent of studies, the second-most common technique behind multiple-choice questionnaires that were used in 34 percent of the studies.¹⁴ Several early studies set the groundwork for this method.¹⁵ Kirk scored bibliographies according to the variety, relevance, and scholarliness of cited sources, then compared resulting scores from two types of bibliographic instruction.¹⁶ Dykeman and King expanded the evaluation criteria to writing, organization, and content, and compared the performance of a treatment group whose members received bibliographic instruction with a control group that did not.¹⁷

Measuring the quality of students’ cited sources has continued to be used in more recent studies.¹⁸ In many of these studies, quality is measured by scoring the students’ use of scholarly sources against a standard rubric. Long and Shrikhande used an “information literacy grading scale,”¹⁹ while Tuñón & Brydges’s rubric scored a citation on a scale of 1 to 4 for five criteria (breadth of resources, depth of understanding, level of scholarliness or “quality,” currency, and relevance).²⁰ Middleton developed a “scholarly index” (SI), a rating the proportion of scholarly vs. nonscholarly sources utilized in the student’s bibliography.²¹ A limitation of such systems is that the binary choice of “scholarly versus non-scholarly” does not leave room for more fine-grained analysis of the level of scholarliness in different types of sources, especially web-based sources that do not always fit conventional criteria.

To score students’ bibliographies in this study, the authors reviewed other researchers’ scoring methods but ultimately chose not adopt them because they uniformly assigned low scores to websites as a genre. When students play BiblioBouts, all the sources in the game come from the web, thus, a scoring method was needed that was able to distinguish sources that come from credible vs. non-credible websites, rewarding the former with high scores and penalizing the latter with low scores. The researchers developed a format-neutral taxonomy that can be applied to both online and offline sources. Its design was inspired partly by the quantitative rating scale of Middleton’s SI,²² and partly by Crowston and Kwasnik’s faceted classification system for categorizing online genres.²³ The taxonomy bears five facets: Information Format, Literary Content, Author Identity, Editorial Process, and Publication Purpose. Each facet is subdivided by categories that describe attributes of sources. Academic faculty who teach undergraduate students participated in the development of the scheme’s scoring system, assigning numerical values to each category based on the desirability of the attribute in a source. Because the taxonomy is faceted, scoring is multidimensional, considering several factors. Tests of the taxonomy reveal that quality scores are consistently higher for scholarly, academic, and peer-reviewed sources, and consistently lower for anonymous, self-published, and nonreviewed sources, thus demonstrating the taxonomy’s usefulness as a measurement tool for citation analysis.²⁴ The taxonomy is used in this study to answer its research questions about the quality of students’ sources.

The researchers sought a medium-size undergraduate course at a participating institution where students played BiblioBouts while they completed a research-and-writing assignment. Fitting these specifications was a highly recommended but not required second-year English course named “Academic Argumentation” in which forty-five students were enrolled in winter 2011. The course is an immediate follow-up to the institution’s first-year composition class. It assists students in advancing from basic writing competence to engagement with more challenging paper topics, and with more refined argument techniques. The course represents many students’ first experience with a requirement to incorporate properly cited supporting materials into their written arguments. The instructor assigned students an essay on the topic “How climate change has brought about a particular adaptation in a particular human population.” Essays were expected to advance a causal argument bringing about a particular effect that was supported by cited references to research. The syllabus gave these instructions regarding cited sources:

• Use and cite at least 5 outside sources.
• No more than 2 of the 5 sources can come from the open web.
• Citations to Wikipedia are not acceptable.

The game’s Tagging & Rating (T&R) bout randomly selected an opponent’s source and displayed it to at least eight different players who tagged its Information Format (IF) and Publisher (PUB), rated its credibility, and commented on the credibility ratings they gave to it. The display included a full citation, a link to the source’s full-text, and, when available, an abstract. To assign IFs, players moused down on a pulldown menu that grouped IFs by their purpose and moused up to select one. Here are the IFs grouped according to purpose:

1. to inform and/or facilitate learning: Consumer Magazine, Consumer Newspaper, Trade Magazine, Trade Newspaper, Research Report, Conference Proceedings, Course Material, Encyclopedia, Scholarly Journal, Dissertation or Thesis, Public Affairs Information, Book
2. to promote or persuade: Blog, Promotional Material, Policy Statement, Public Sharing Site, Informational Video
3. to catalog or list: Database, Directory, Online Repository

Next, players tagged the source’s Publisher (PUB), listing these options in a pulldown menu: Individual Person, Commercial Business, Nonprofit Organization, K–12 Education Institution, Government Organization, and Higher Education Institution.

Next, players answered three questions about the source’s credibility. Students could display a tool tip that gave more explanation about the task by hovering over the underlined word (that also was highlighted in red). Questions and tips were the following:

• To what extent do you believe that this source is written by an expert? (Tool tip: The source provides evidence that its author has expert knowledge, skills, and competence in the subject.)
• To what extent do you believe that this source is trustworthy? (Tool tip: The source provides evidence that the information is truthful, fair, and reliable.)
• To what extent do you believe that this source is scholarly? (Tool tip: The source is a product of serious academic study, and, possibly, original research.)

To register their credibility rating, students pulled sliders beginning with 0 percent (Not at all) and ending with 100 percent (To a great deal). When players’ credibility ratings are discussed in the sections that follow, average credibility ratings are given for students’ three separate ratings above.

BiblioBouts logged students’ game play data. Logged data included citations and full-texts for the sources students contributed to the game in the Donor bout, their IFs, PUBs, credibility ratings, and comments for opponents’ sources in the T&R bout, and the sources they chose for their in-game bibliographies in the Best Bibliography bout.

The researchers anonymized students’ final papers, extracted cited sources from them, and gave sources to coders. Coders were second-year master’s students in University of Michigan’s School of Information and trained by the researchers to apply the taxonomy. Coders then retrieved the full-texts of both BiblioBouts sources and cited sources from students’ final papers and applied the taxonomy, scoring sources along these five attributes: (1) Information Format (IF), (2) Literary Content, (3) Author Identity, (4) Editorial Process, and (5) Publisher (PUB).

For each attribute, coders chose one element that best described the cited source. For IF and PUB attributes, taxonomy elements were the same as the ones players used to tag sources during game play. Thus, we can compare players’ tags with coders’ assigned elements to determine whether players’ ratings and coders’ taxonomy scores co-vary.

The taxonomy associated a numerical value from 1 (low) to 4 (high) with each element that represented how useful and appropriate the element would be to an undergraduate student researching sources for a class assignment. These values were determined based on consensus of the researchers who developed the taxonomy and instructors whose classes played BiblioBouts.²⁵Table 2 gives examples of two sources, a scholarly and a non-scholarly source, and how coders would score them using the taxonomy.

When coders assigned scored elements to a state-of-the-art literature review published in a Scholarly Journal, the source’s taxonomy score was close to the maximum of 20.0 points. A blog written by a college professor on an academic topic was substantially less at 12.9. Note that values for all five elements describing the literature review were high whereas values for the blog were a combination of high and low scores.

An intercoder reliability study resulted in overall mean agreement of 0.80, or 80 percent agreement between coders. Given the large number of possible elements for each of the five facets, the researchers felt confident in the overall level of agreement between the coders.²⁶

At the end of the game, a dozen players volunteered to participate in a focus group interview. Project staff recruited students by sending them email messages and offering a $25 VISA gift card for their participation. The interview was held over the lunch hour and participants were treated to pizza and soda pop. Researchers also conducted a post-game personal interview via phone with the instructor after papers were graded and returned to students; instructors were given questions in advance to prepare their answers. Thus the researchers enlisted several data collection methods to evaluate BiblioBouts’ effectiveness.

When playing BiblioBouts, players played it safe, knowing that they would score high for the scholarly sources they put into play and chose for their best bibliographies. Thus they limited their BiblioBouts sources almost exclusively to scholarly sources. That players’ credibility ratings for scholarly and nonscholarly sources were roughly equivalent is troubling; however, the credibility ratings for the latter were based on so few sources that it is difficult to generalize based on data from this one analysis. Subsequent analyses of BiblioBouts game-play data will enable the researchers to thoroughly investigate students’ source credibility assessments to determine whether they are comparable to the quality ratings of expert coders.

Writing their final papers, players went farther afield, citing more non-scholarly than scholarly sources and hardly using the sources they and others found while playing BiblioBouts. That players’ final-paper sources were comparable to non-players’ final-paper sources in terms of IFs, PUBs, and taxonomy scores (see tables 5 and 8) is evidence of players’ reversion back to their habitual patterns of source selection. Especially troublesome was that this reversion occurred despite their experience of playing the game with its focus on source evaluation and the availability of the BiblioBouts Post–Game Library of mostly scholarly sources on the broad-based topic in play that players built as a result of game play.

It has been suggested that topic specification played a role, that is, forcing players to seek entirely different sources because their topics evolved during game play and afterward while writing their papers (see table 6). The broad topic in play—how climate change has brought about a particular adaptation in a particular human population—also contributed. For example, table 4’s nonscholarly IFs such as policy statements, public sharing sites, blogs, and promotional material that communicated the opinions, viewpoints, and policies of governments, NGOs (nongovernmental organizations), nonprofits, and individual people about climate change were likely to have relevant content for the arguments players made in their final papers.

The researchers will monitor future classes whose instructors choose different broad-based topics to determine whether large proportions of BiblioBouts sources make up the sources of players’ final papers. If and when they do, we will scrutinize the characteristics of these broad-based topics to advise subsequent instructors about topic selection so that their students benefit from BiblioBouts’ Post-Game Library.

Scholarly sources especially scholarly journals dominated BiblioBouts sources; however, players chose few (12.0 percent) BiblioBouts sources for their final papers. This suggests that undergraduate students may not understand scholarly sources well enough to integrate them into their papers. When playing BiblioBouts, players were probably able to recognize, retrieve, and contribute scholarly sources on the broad-based topic in play; however, integrating them into their final papers to support their claims might have been too much of a stretch. As a result, players as well as nonplayers turned to nonscholarly sources that discussed global warming in terms that the average layperson could understand. It may take several years of coursework in a discipline for students to develop competence and facility with its scholarly literature. Our speculation in this regard is echoed by Rebecca Jackson, who, as a result of mapping information literacy competencies to cognitive development models, asserts that successfully meeting information literacy standards that include using information to generate new concepts can only be accomplished by graduates of higher education institutions.²⁷

Players’ failure to cite BiblioBouts sources in their papers came to the researchers’ attention after the class ended so they had no opportunity to query students about this. The researchers sought explanations from students and instructors who played BiblioBouts after the class that is described in this study. Interviewees agreed that game-play occurred too early in the research process for them to have settled on their papers’ topics and cited sources; however, playing BiblioBouts exposed them to information that had an impact on their final papers. Here is what students from other classes said in this regard:

• “When I did BiblioBouts, I didn’t have my thesis all the way constructed… . After I did BiblioBouts, I needed a lot of sources that were similar to the topic that we used but they weren’t exactly the same thing. Like I had to do a lot of other research after the fact to find information that was similar to [but not the same as] the [broad-based] topic.”
• “Because I feel like we just kind of hopped into research without really formulating our thesis completely.”
• “Because it’s a research paper and so you have to look at what sources you find and then make some kind of thesis out of that.”

Here is an instructor who lauds BiblioBouts for its ability to expose students to more information than they would have found on their own. Because of this exposure, students were inspired to think in new and different ways about their topics and about the need to find more information after the game ended to support their ideas.

• “A lot of [students] found it was very useful to have references that they all pooled together and found and rated and that was good. But … the students … said, ‘Initially I started with an idea of what I wanted to write on this theme. But after all this reading and so on, my ideas have formed in slightly different ways from what I had initially thought of and that was too simple. So can I not use any of those [BiblioBouts] sources because I have formed my idea about what I want and I need to look for other sources? I might use one [BiblioBouts source] but I want to use something else as well.’ I let them do that.”

This paper’s research questions regarding the quality of players’ chosen sources increasing as they progressed from bout to bout and culminating with writing their final papers were not supported by analyses involving IFs, PUBs, and taxonomy scores; however, the instructor who graded students’ final papers observed better-quality arguments and sources. Here is what he said in this regard.

• “Their understanding of how sources applied to their argument went up as a result of looking at a number of different sources all talking about the same thing. Even if they looked at the sources only briefly, they were able to see how the sources that they did choose compared to other available sources. That forced some of them to have a greater understanding of how the information that you’re citing impacts the information that you’re putting into the paper as part of the argument. The quality of argument went up and the quality of sources overall went up.”

Focus group interviews revealed other ways in which students benefited from game play. Before the game’s start, a librarian visited the class to demonstrate the university library’s database portal and relevant databases. The instructor also profiled BiblioBouts with a list of relevant databases. Some students acknowledged the game as being their first exposure to these databases.

• “For me it was the first time I have ever used the university library system and so it was a really interesting experience for me just because like I have searched Google, Google Scholar, and other things … but I have never used databases before.”

Students added that playing BiblioBouts emphasized the importance of evaluating the sources they find.

• “The biggest thing for me is after playing the game is getting used to finding sources, legitimate sources, rather than just typing something into Google and bringing up whatever websites came. [Playing the game] definitely helped me reevaluate what sources I want to use and at a university level rather than just what I’ve been doing thus far.”
• “[Playing BiblioBouts] did change my mentality on what a good source is and what isn’t and a lot of it was the repetition of constant tagging and looking at different sources and being able to evaluate them in those terms.”

Another benefit the instructor cited was keeping sources on hand after a course ended to use at a later time.

• “Creating a database that they might use in the future is something that they gained from this. In the past, sources were disposable. You … used them for a paper or project and then discarded them… . In the future, you keep a running database … especially within your major. Some of them have already told me that they are going to do exactly that… . They would use Zotero to do it… . They were very fond of Zotero.”

Not long after the game ended, students described how they were already using Zotero for assignments in other courses. Game play also increased their use of the university library.

• “Now I use Zotero for all of my projects and all of my papers when I need to find different articles.”
• “I actually used the university library and services a lot more, which I like never had and like they were really helpful.”

Because of playing BiblioBouts, students made a habit of collecting more sources than their instructors required, picking and choosing the best ones to make their argument rather than limiting themselves to the first ones retrieved on the topic.

• “Whenever I did research I used to just find however many citations needed or however many the professor had required for the particular assignment. [Since playing] the game, I now [collect] a huge amount of [sources], then I narrow it down, much in the same way that we did in the game.”

Students also acknowledged that playing a game motivated them because it was an open competition with their fellow students.

• “The leader board, to know who was on top and who was behind and how far you were from the top five, I liked that because it pushed you and it gave you like the motivation to be, ‘Oh, I’ll do more.’”
• “I think the game is an effective way [to learn about library research], especially if it’s with others because a lot of people are competitive.”
• “[Because the game] involves others … with each other trying to reach the same goal, it’s able to motivate you more to do what you need to do and so I think a game is a good way to learn.”

Playing BiblioBouts did not immediately produce the desired behavior, that is, students citing increasingly more scholarly sources in their final papers. It did, however, expose them to more sources than they would have found on their own, to the importance of evaluating the sources they find, and to a new software tool for organizing the sources they find online, including using this tool to keep sources on hand just in case they need them in their future coursework. Their instructor also observed that the benefits of playing BiblioBouts went beyond simply allowing the students to support their arguments more effectively; it enhanced their understanding of the relationship between researched sources and argument planning, as well. They began to see that engaging with higher-quality sources early in the preparatory stages for their papers helped them to recognize avenues of investigation that might not otherwise have occurred to them, resulting in more thoughtful theses and stronger subtopics.

Students from a second-year English course at a research university played the BiblioBouts information literacy game. To determine whether game play improved the quality of the sources students cited in their final papers, several methods were used—analyses of game logs, cited sources in players’ and nonplayers’ final papers, a personal interview with the instructor, and focus group interviews with students.

As a result of playing the game, the players in this one particular class did not progress from lower-quality to higher-quality sources; however, scholarly sources dominated the sources they submitted to BiblioBouts, demonstrating that players were able to recognize quality sources for the purpose of playing the game, and they gave their highest credibility ratings to sources from scholarly journals. Although players used few BiblioBouts sources for their final papers, they now know that they can find scholarly sources by searching relevant databases available through the university library’s database portal and use Zotero to automatically generate citations and save digital full-texts. Of the several factors contributing to players’ failure to cite the game’s scholarly sources in their papers, most important was the evolutionary nature of their final-paper topics, becoming more specialized than the original broad-based topic that was the impetus for the sources they submitted to the game, and the game’s exposure of players to many more sources than they would have found on their own. By the time the game ended, some players were already using Zotero to organize their sources for assignments in other classes. BiblioBouts players were partial to learning how to conduct library research by playing a game because the game situated them in an open competition with fellow students, stimulating them to go above and beyond what they would have normally done because of the immediate rewards they reaped as a result of their game-play activity.

Because this paper’s analysis was limited in scope to a single class of second-year students playing the BiblioBouts information literacy game, results may not be statistically valid or generalizable. However, results are important because they expose the “black box” events of research-and-writing assignments such as the sources students and their classmates find and prefer, the topics they have in mind for their papers, and the best sources for their topics, and thus provide researchers with opportunities to analyze the decisions that students make in the process of achieving their objectives. This paper provides evidence that despite encountering scholarly sources several times during the research process, students depended on nonscholarly sources for their final written papers. The authors speculate that underclassmen are not intellectually ready to synthesize the scholarly sources they find. Follow-up studies are needed to confirm this speculation. A number of different scenarios are possible including comparing students’ game-play performances under different conditions: (1) incentives, for example, requiring students to play the game versus playing it for extra credit, (2) class level, for example, underclassmen versus upperclassmen, master’s students, and/or doctoral students, and (3) level of instructor involvement, for example, no instructor involvement versus instructors who play the game and give their students previews of the information literacy skills and concepts that needed to effectively and efficiently play each bout.