Conducting usability studies of a library’s web interface provides concrete evidence about user behavior and preferences. The literature documents the benefits of usability studies along with basic principles and practices.³ While there is little evidence that systems that incorporate user input in the development stage are more efficient or effective end products, users excel at determining whether an interface is intuitive and able to be efficiently navigated. Bailey notes that users provide much-needed insight into novice user behavior and are better at defining the parameters of the system rather than contributing to design of the infrastructure.⁴ Usability studies provide guidance by gathering information from an end user perspective.

Specific usability methods have been developed for libraries,⁵ and Letnikova provided a summary of academic library usability case studies and compiled a standard list of questions for testing.⁶ Most library usability studies are qualitative in nature, using as few as five test subjects to inform design characteristics.⁷ For a quantitative usability study that allows the results to be generalized to broader user behavior, twenty users need to be observed.⁸

The JMU studies touch on several relevant areas of information-retrieval and search behavior. In a seminal article, Kuhlthau urged researchers to add user-oriented approaches to information-seeking studies as opposed to solely focusing on systems.⁹ In their 2004 article, Järvelin and Ingwersen suggested system efficiency can be assessed along several dimensions, including not only the quality of documents retrieved but also the searcher’s effort (time), satisfaction, and the tactics employed. “The real issue in information retrieval systems design,” say Järvelin and Ingwersen, “is not whether its recall-precision performance goes up by a statistically significant percentage. Rather, it is whether it helps the actor solve the search task more effectively or efficiently.”¹⁰ It is within Kuhlthau’s and Järvelin and Ingwersen’s context that the present article’s study is situated. Rather than study a statistical sample of citations in the three systems examined, this study focused on how effective the interfaces were at helping users complete the tasks.

Lookup tasks, or known-item searches, have been studied repeatedly by information scientists in the context of the library catalog.¹¹ Known-item searches, wrote Marchionini, begin with “carefully specified queries” that should “yield precise results with minimal need for result set examination and item comparison.”¹² This article examines known-item searching for article citations, which is a common physical and virtual reference need.¹³

When usability studies at libraries have concentrated on known-item searching, these studies have involved locating specific journals or books, rather than articles.¹⁴ Letnikova noted finding journals proved to be one of the most difficult tasks on academic library homepages.¹⁵ Ipri, Yunkin, and Brown included a journal title task on a fourteen-task test with five graduate and five undergraduate students. They found many users had trouble distinguishing “Journals” tabs from “Articles and Databases” tabs and combined article and journal searching on one tab.¹⁶ Mvungi, de Jager, and Underwood found confusion among their five participants over the difference between electronic journals and print journals.¹⁷ In contrast, Whang and Ring tested twenty undergraduate and thirteen graduate students and found that 90 percent of undergraduates and 100 percent of graduates were able to find a specific journal title using either the library catalog or the library’s SFX journal finder.¹⁸ These studies show differences depending on local context.

Fewer studies have examined the task of finding an article by citation. Ascher, Lougee-Heimer, and Cunningham had eight participants perform five tasks at a health sciences library, one of which was finding an article given the citation.¹⁹ In this study the participants were instructed to find the article from the library homepage rather than from a particular interface. Most of their participants used PubMed, and all users successfully found the journal’s page. However, they experienced problems related to local authentication.

Terminology is also a major challenge to finding articles: library jargon, nonspecific terms, or variant terms (e.g., serials, journals, periodicals) are barriers, especially when used inconsistently throughout the library website.²⁰ Kupersmith’s website notes terms like “journal article” or “find article” are cited as being more helpful than “databases,”²¹ but choosing specific words to further distinguish between article- and journal-related tasks is still challenging. McHale used a card sort to help choose new language for her library’s website redesign, and “find a journal by title” and “find articles and more” both ended up in the “search” category.²²

Very few library usability studies have focused on evaluating the interface of journal portals or link resolver forms with known-item citations. The University of North Carolina at Greensboro (UNCG) developed a journal portal and studied its use.²³ Because the portal was locally developed, developers were able to respond to barriers such as retrieving no hits when a user enters an ampersand or colon. Ellington conducted a usability study on UNCG’s journal portal with forty participants and two of the questions related to finding known citations in the journal portal. Her users commented that they liked finding a link that allowed them to enter article citation information, and they performed better on the task with complete citation information, including volume and issue, rather than just article title, journal title, and date.²⁴ Jayaraman and Harker studied what makes link resolver software effective, but they focused on the linking action rather than the web entry form.²⁵

Library studies using Google Scholar have focused on searching for topics or general search terms and comparing results with subscription sources. Callicott and Vaughn highlighted differences in content and the user experience between traditional library resources and Google Scholar.²⁶ They found that, although Google Scholar guarantees results, constructing complicated queries or limiting the results retrieved is difficult. Lee found that while users prefer the simplicity of the Internet search box, they readily admit they are trading quality for speed and ease of use.²⁷ Donlan and Cooke noted that Google Scholar provides a helpful filter for the web, but it is still unclear what this search engine indexes.²⁸ In addition, users might be prompted to pay for access to journal articles when attempting to access library resources remotely.²⁹ Studies were not found that evaluate user success with Google Scholar for finding journal article citations.

This article, which analyzes results from two usability studies in an academic library, adds to existing research on finding journal titles and attempts to fill gaps in the research relating to finding articles by citation and the relative usability of journal portals, link resolver web forms, and Google Scholar.

JMU is an undergraduate-focused institution with approximately eighteen thousand students. The JMU Libraries’ usability lab features one workstation with two pieces of usability software: Techsmith’s Morae (version 2) (www.techsmith.com/morae.asp), which records participant actions during the usability studies, and Bailey’s Usability Testing Environment (UTE) (version 2) (www.mindd.com/Content.aspx?pid=UTEStandard), which presents participants with tasks in the web browser environment. The UTE also presents end-of-task questions to measure time on task and task success.

Two studies conducted in April 2009 were covered by an institutional review board-approved protocol. The authors recruited participants for both studies through a blast e-mail sent to all students and faculty, and interested respondents were randomly selected to include a variety of grades and majors. There was no overlap in the two studies’ participants. Both studies began with several pre-study questions and ended with the System Usability Scale (SUS). The SUS is a ten-item scale using statements of subjective assessment and covering a variety of aspects of system usability.³⁰

The first study, the “homepage study,” had twelve tasks chosen to measure library homepage navigation for central functions. Only two of these tasks will be discussed in this article:

• 1.Does the library have access to the journal Brain Research?
• 2.Find the full text for the journal article “Anxiety in High-Functioning Children with Autism” by Alinda Gillott, Fredd Furniss, and Ann Walter that was published in a 2001 issue of the journal Autism.

The homepage study included twenty-one participants with a range of experience: eight freshmen, five sophomores, three upper-classmen, one graduate student, and four faculty. Twelve of the participants were from the arts and humanities, seven were from the sciences, and two were from the school of business. Sixty-two percent of the homepage study participants said they visited the library website at least once per month.

The “find-a-citation” study consisted of twelve tasks that asked participants to find four citations in three web interfaces: a simplified version of the library’s journal portal, Periodical Locator (PL), which isolated the journal search function; Google Scholar (GS); and the library’s link resolver form, known as Check for Full Text (CFFT). Both CFFT and PL are Serials Solutions products. Twenty participants were chosen to make our findings generalizable to the JMU student population.³¹ However, as a rule of thumb, Manning, Raghavan, and Schütze have suggested that fifty information needs are necessary to evaluate an information-retrieval system, so the twelve tasks used in this study was not a sufficient number to make generalizations about the tested systems.³²

The find-a-citation study had nineteen students and one faculty member. Fifteen of the twenty participants had been at JMU for just one or two years. Ten of the participants were from the arts and humanities, seven were from the sciences, and three were from the school of business. Eleven participants indicated they needed full-text journal articles on a monthly basis or more, while nine said they needed full-text articles “a few times per semester” or less. No one thought it was very difficult to “find full text for journal articles.” The authors asked participants if they had previously seen the “Check for Full Text” link resolver button in research databases or the link in GS; more than half had used the button in research databases, and one-third had used the link resolver from GS.

While many readers will be familiar with these interfaces, it is important to consider how each one operates when trying to find an article by citation. To use PL, the user must identify and search on the journal title. The results show which information providers offer access to each journal title and the various dates of coverage for each provider. A user looking for an article must click on the information provider having the appropriate dates of coverage, then conduct a search for the article citation on that provider’s site. If PL does not have any matches by journal title, the user sees a “no results were found” screen, which offers tips for conducting additional searches.

To use GS, the user should enter at least the article title from the citation. Entering only the journal title will often return too many results to effectively find a specific citation. If GS finds a match, the user can click directly on the result (often the article title) and get to the full text. Since the library’s link resolver software is enabled in GS, participants had the additional option to click on “Check for Full Text @ JMU,” which would take them to the link resolver results screen. While clicking on the result itself is the fastest way to get to full text, the “Check for Full Text @ JMU” link offers the most options. If GS does not find a match on the user’s query, it generally still has enough information to display some results, however irrelevant they may be.

CFFT requires the user to identify an article citation’s parts and enter each part into the correct field. The user submits the form and must then choose the correct link on the results screen. Depending on the completeness of the user’s entry and the accuracy of the software, the results screen might have article links or journal links, or it might indicate no full text was found. Clicking on the article links delivers the user to full text. Clicking on the journal links requires the user to perform an additional search on the publisher’s website, similarly to PL. Finally, if there is no full text found, the user is offered options to search the library catalog or the library’s journal portal.

The find-a-citation study began with a practice task and seven pre-study questions (available from the authors on request). The user then completed four tasks using the modified journal portal, PL, followed by a post–interface questionnaire, the SUS. Next, the user completed four tasks using Google Scholar (GS) and again completed an SUS. Finally, the user completed four new tasks in the link resolver form (CFFT) and an SUS. Each task asked the user to find full text for a given citation. All participants took the test in the same order: PL, GS, and finally CFFT. To measure whether they had found full text, each of the twelve tasks offered the participants a multiple-choice question when they clicked “Answer,” which required them to choose the correct first three words of the article full text. “No full text is available” was always an option. In each set of tasks, one task asked the participant to find full text for a citation for which JMU had no full text access. Participants timed out if they failed to provide an answer after five minutes, and for each question, users had the option to skip the task. At the end of the study, participants answered three post–study survey questions about interface preferences.

The authors address limits of this method in the discussion section.

This study approached known-item information-seeking from the user-oriented, task-focused approach recommended by Kuhlthau and Järvelin and Ingwersen, and it focused on the user’s effort, satisfaction, and tactics.³³

The first research question was, “What difficulties do users encounter when trying to find a journal title from the JMU Libraries’ homepage?” In agreement with Letnikova’s findings at other libraries,³⁴ the results of the “homepage study” revealed that locating a journal title was one of the most difficult tasks to perform on the JMU libraries’ website. Terminology was a major issue because the links did not clearly differentiate where to begin searching for this task. For finding a journal, most participants either entered the journal title in the front-and-center search box, which only targets the library catalog, or clicked the journal portal link. Participants who had difficulty primarily struggled interpreting the results from both the catalog and the journal portal. This supports Marchionini’s assertion that known-item searches should produce precise results sets with little need for item comparison.³⁵ It also corroborates Mvungi, de Jager, and Underwood’s findings, which suggest that results obfuscate whether a journal is online or in print.³⁶ A dropdown menu within the catalog search box included the term “periodicals,” which some participants failed to equate with “journals.” Although these findings may not seem surprising in the context of other libraries’ studies, it was important to determine what the specific problems were at JMU in order to make effective changes.

The second research question was, “What difficulties do users encounter when trying to find an article by citation from the JMU Libraries’ homepage?” Finding a known article was the other one of the two most difficult tasks to perform on the library homepage. Study participants most often tried to use the journal portal by clicking on the “Periodical Locator” link. More than half eventually resorted to using the library catalog search box on the homepage, which does not support citation searching. A common problem involved entering the article title into a search box, whether using the library catalog or the journal portal. Four of the participants chose the link “Find Articles” from the library homepage, which led to a list of the libraries’ general research databases and other resources, such as PL and GS.

The final research question was, “Of the three interfaces easily available to library users, which web interface supports finding a citation effectively: the journal portal (PL), the link resolver form (CFFT), or Google Scholar (GS)?” GS was the users’ favorite tool for finding citations in the find-a-citation study, as determined by the SUS and the post–study questions. Participants were also the most successful and completed tasks more quickly when using GS. Although the link resolver was enabled in GS, most participants clicked on the article title for the full text rather than the link resolver. In GS, it is more effective to search on the article title, but several participants still searched first on the journal title when trying to find a citation. Since the study was conducted on campus, users were automatically authenticated for access to JMU resources. However, off-campus users would have to enable the link resolver in GS to find JMU full-text subscription resources. This finding points to the need for additional instruction about GS on the library website.

The CFFT interface posed some significant challenges for users, but employing the form saved many participants time. The most common problem observed in the study was the failure to complete the most important fields of the form. Participants often omitted important elements resulting in journal-level, rather than article-level, links. Some did not notice the date ranges on the link resolver results screen. Other participants clicked on journal-level or database-level links, even when an article link was available. Yet, if participants completed the CFFT form correctly and found the article link, they completed the task quickly. Participants seemed to have more confidence that full text was not available when using the CFFT form than with PL, perhaps because the form appeared to be a more robust tool with clearly labeled data entry fields.

The PL interface slowed users down. Even the task where all participants were successful took longer than a similar task in the CFFT interface. If users entered the article title instead of a journal title, they retrieved no results, which increased the time on task. When there were no results, videos showed that users were not reading the textual suggestions on the “no results” screen in PL, but instead would try to take immediate action. They would change the dropdown menus, click the browser’s back button, or select a link from the “no results” page header. Even for successful tasks, using PL to find an article involved interpretation of the results screen and navigation of an additional website (e.g., the journal publisher’s website), introducing an automatic extra step.

The observations regarding the question of which interface most effectively allows searching for articles by citation would suggest that librarians should direct users to GS as a first choice and that it should be featured most prominently for finding articles by citation. However, in addition to this study’s limitations, discussed below, there are several other important considerations. Since Google does not reveal what GS indexes, it is difficult to determine appropriateness and completeness of coverage.³⁷ When JMU librarians were presented with the results of this study, some thought GS should be the top-recommended tool, but others had understandable concerns about using a commercial tool with unpublished policies rather than licensed vendor software. The study team encouraged JMU librarians to test GS for their disciplines and to recommend its use and instruction at their discretion.

This study revealed several additional trends in user behavior that did not fall within the purview of any of the three research questions.

First, none of the interfaces gave the user confidence that no full text was available. Figure 5 shows an example of the circuitous pathway taken by one participant that illustrates the general pattern of “trying everything” when full text was not available.

Second, contrary to anecdotal evidence, users attempted to use the dropdown menus in the journal portal (PL) and on the library website. Unfortunately, the dropdown menu options were not helpful for these tasks. For example, the users who tried to use the library homepage search box to find a journal title did not interpret the option “periodicals” as a way to find journals, and the library homepage search box contained no dropdown options that would support finding an article by citation. On the journal portal (PL), the dropdown menu includes options such as “title equals” and “title contains all words,” and although participants did sometimes try these options, this did not address the most common problems of entering the article title rather than the journal title.

Another interesting behavior observed was that users clicked on “Refine/Alter Search” when their initial attempt at using the CFFT link resolver form failed, often because they failed to fill out only one or two fields. At the time of the study, the form provided no guidance about which fields were most important.

Finding journal titles and finding articles by citation will remain challenging tasks for users. The principle of least effort, also known as Zipf’s law, suggests that individuals “will adopt a course of action that will expend the probable least average of their work.”⁴⁰

Librarians can change the information architecture of their homepages to respond to this principle and to make the pathways more intuitive. They may also be able to make some customizations in third-party software, such as journal portals and link resolver forms. In addition, librarians should remain open-minded toward commercial tools, like GS, that have the potential to increase success and save time for users. Subject experts should use GS when searching within their discipline until they have an intuitive sense of whether it is a good option for users. Finally, librarians need to remember that many users do not begin their search on the library website. If the top tools offered on the library website are user-friendly and effective rather than frustrating and time-consuming, users will have a reason to begin their search there.