ltr: Vol. 44 Issue 6: p. 17
Chapter 3: User Experience, Feedback, and Testing
Jennifer L Ward
Steve Shadle
Pam Mofjeld


The University of Washington Libraries, in collaboration with OCLC, began a pilot project that resulted in the creation and launch of a new discovery and delivery platform called WorldCat Local. It enables users to search what were previously three separate catalogs as well as five article databases. The search interface is simple, which research has shown that users prefer.1 WorldCat Local also integrates what were previously separate delivery systems, resulting in a seamless, easier-to-use service. WorldCat Local debuted on April 30, 2007, when the University of Washington Libraries made it the default search box on its home page. The impact on delivery has been impressive, with users requesting and getting far more materials than when we had separate “silos” for discovery and delivery. WorldCat Local provides a new service environment to the library user. Because WorldCat Local integrates a number of data sources (local library catalog, consortial library catalog, digital collections, article databases) and library services (local and consortial catalog requesting, interlibrary loan, link resolving, direct full-text linking, and online reference services), it is important to have the functional experts in all of these areas involved in planning and implementation. This article explores why such a tool was desirable; describes various aspects of the implementation, the effect of usability on development, and the impact on discovery and delivery; and discusses future plans.

From the start, WorldCat Local was designed to be an end-user tool, one that would seamlessly integrate our myriad discovery and delivery systems into one interface, and ideally an interface with as few dead ends as possible. Its design was based on knowledge about users' information-seeking behaviors and on past usability testing conducted by OCLC staff. As part of the implementation, usability testing was conducted at the University of Washington in May 2007, shortly after WorldCat Local went live at the first pilot site.

The UW Libraries has had a program to evaluate the usability and quality of the online user experience since 2001, when it was established as part of our strategic planning process.1 We believe that we need to talk to users often to seek their input to inform our system and service design. We also test these designs to make sure that we have correctly understood this input and that our implementations are effective.

Based in the Information Technology Services (ITS) unit, the usability program is managed by the head of Web Services, and additional staff support for the program is provided by a .50 FTE graduate staff assistant. In 2002, a small study room was dedicated to the usability effort and turned into a usability-testing lab.

The lab is equipped simply—a whiteboard, phone, office desk and chair for the participant, smaller table and chair for the observer, and paper file storage. The computer is a current-generation, standard issue Dell desktop system with a sound card that supports audio input. The only special software on this machine is the Camtasia Studio screen recorder application by TechSmith. With the microphone and software, we are able to capture on-screen activities such as typing and mouse and cursor movement, as well as the participant's and facilitator's voices. These captured sessions are later reviewed by the development teams to corroborate information in our notes.

For the purposes of testing WorldCat Local, a slightly more elaborate setup was used so that interested parties on the east and west coasts could observe all sessions in real time. From the usability lab, the test facilitator connected to a LiveMeeting Web conferencing service. OCLC staff in Dublin, Ohio, joined this session from a conference room, as did UW WorldCat Local team members. Audio of the session was transmitted via telephone to Dublin, and then retransmitted to the conference room at the University of Washington. The test participant and facilitator heard no audio, while the observers could hear what was going on in the usability lab as well as communicate with one another. Sessions were recorded in Dublin from video input and in Seattle using Camtasia.

If you have never had the opportunity to observe a usability test, it is a humbling, sometimes frustrating, and always enlightening experience. What system designers or experts think is totally obvious can easily be lost on the user.

WorldCat Local Usability Testing, Round One

The initial round of usability testing was developed based on a prototype and an understanding of how WorldCat Local would work. The purpose of this evaluation was to determine how successful UW students would be in using WorldCat Local to discover and obtain books and journal articles (in both print and electronic form) from the UW collection, from the Summit consortium, and from other WorldCat libraries.

Long before the interface was complete, staff from OCLC and the University of Washington began to map out use cases and find representative examples to test. Because we wanted to test the integration of discovery and delivery, users were asked to locate and/or access online a number of known items (that is, they had a citation and just needed to find the items). A round of pilot tests were conducted with participants in Ohio to ensure the script and protocols were well-defined and the technology worked as anticipated.

UW students were targeted for the first round, and we worked with seven undergraduate and three graduate students. Our overall user profile was based on the demographics of the University of Washington's student population, so we had a diverse mix of discipline, gender, and year in school. Announcements of the sessions were posted on the UW Libraries' home page and on the start screen on all computers in campus computer labs. Participants were offered a gift card to the University Bookstore as an incentive and token of our appreciation for their time.

With two exceptions, all tests were conducted in the UW Libraries' usability lab, facilitated by an OCLC staff member, and observed in real time by OCLC staff in Dublin, Ohio, and UW WorldCat Local team members. Two participants worked from an off-campus location so we could test remote access to electronic content. Throughout the testing, users were asked to think aloud as they conducted the tasks so that we could better know what they were thinking as they worked with the system.


While users were generally successful finding materials, barriers connecting the user to the item sought were still sufficiently high that they needed additional attention by the system designers. Based on this feedback, the following changes were introduced to WorldCat Local.

On the brief results list, users were frequently perplexed by reviews that appeared higher in the list than the item itself. This hindered their ability to locate the item, so changes were made to the ranking algorithm so that reviews wouldn't necessarily float higher than the item in question.

The detailed record screens clearly presented more challenges for our users, so more attention was paid to them. Figure 12 shows a screen shot of what early users of WorldCat Local would see if they looked at a detailed record for a book:

  • Note the Request Item button in the upper right corner of the screen. The button was orange, and it was thought during development that the prominent placement and bright color would draw the user's attention. Clicking on this button would lead the user to the appropriate request system: in this case, the Summit consortial system. However, many users did not see this button, and at least one commented that it appeared to be an ad of some sort since that's where ads are usually placed on screen. It was obvious to those observing that the controls were not well placed.
  • The block headed “Location and Availability” was the second area that received attention. While it was easy to see the location of materials available at the University of Washington, most users didn't care which Summit libraries held an item and how many copies they had. As we'll see in just a moment, this expanded view could be problematic if there were many holdings for a given item.
    In addition, messages were added to the system that provided users a better idea of next steps if the item was not available or requestable. These changes were intended to reduce the number of dead ends experienced by users.
  • Just beneath the Locations and Availability block were the two “do it” features available at first release: “Add This Page to Favorites” and “Link to this Page.” As more features and functionality were added to WorldCat Local, these items were moved off into their own area with a bit more prominence.
  • OCLC can map an IP address to a geographic region and present customized information based on this. Just above the tabs is a line of relevant services offered to users based on their computer's IP address. For the most part, it made sense that WorldCat Local inherit the features offered by WorldCat, but this is one very clear case in which it does not (WorldCat no longer displays these options, but did at the time of testing). The services available in that line were generally misleading to the users. In the case of a monograph, a user would not want to choose “Check for UW journal holdings.” “Request item through Interlibrary Loan” would not be appropriate in this case. “Search the UW Libraries catalog” is redundant (Didn't the user just search the catalog? Most thought so.) and takes the user into a totally different interface with no clear path for how to get back. And users had no idea why they'd need to choose “OCLC FirstSearch.” Of all the services presented, only “Ask a Librarian” was appropriate.
  • The second biggest problem with this screen—and one of the ways in which WorldCat Local still deviates from—was the default tab. At the time of testing, the Libraries tab was the default and was what users saw when they first came to the screen. This was problematic for two reasons. For one thing, in the context of a local system, you expect that the UW Libraries would have the item, so you wouldn't need that information. Also, most users didn't care who else had something as long as they could request or otherwise obtain it through this interface. Probably more important, though, was what happened when a user first loaded this page.
    Once a user selected an item from the brief results list and wanted to see the details, WorldCat Local began the process of fetching detailed holdings and circulation data. While the detailed record screen “built,” the user saw all the information in the bibliographic block first. The availability block was mostly blank and had a spinning green circle (indicating that the system was busy doing something). Then the tabs were displayed, with the Libraries tab showing as the default. Consistently, users wouldn't wait for the availability block to build, but would click straight into the UW Libraries link in the Libraries tab. After selecting that link, they would be taken to that record in the UW local catalog.
    This was not really an issue for monographs, as there was generally a good record-to-record match between the two systems and users were able to successfully complete the task (albeit not through the interface we were testing). However, much more is findable in WorldCat Local than just monographs. When these steps were repeated by users trying to find an article in our catalog (for which there was no metadata), it was immediately obvious that this design decision needed to be changed and quickly. If users were lucky, they would find themselves at the record for the journal title, but very few knew how to make the leap from the journal title to the article content. This was a path from which it was easy to stray and almost impossible to recover.

Figure 13 shows an example of a detailed record screen for an item with many holdings. Note that the holdings in Summit (not clearly identified as Summit here, but under the expanded UW holdings) took up virtually the entire screen. With this arrangement, a lot of white space was wasted and the remaining relevant information was pushed below the fold.

Figure 14 shows a detailed record screen for a journal: The Economist. As with the monograph example above, we had the problem of the action button—in this case Access Online—being hidden in the upper right corner and easily being mistaken for an advertisement. We also had the expanded Summit holdings taking up much screen real estate.

More troubling in this example, though, was that if the user missed the Access Online button, then there was no easy way to discover that this title was available online. WorldCat Local knew that electronic content was available at the University of Washington, yet forced users to click the button to find specifically what was available. “Why not show electronic information up front?” we wondered. This dead end was fixed as a result of this testing.

One other issue with this display (not shown in figure 14) was that links for electronic materials available through Summit would sometimes also appear along with links to UW items. This was problematic because our users couldn't access electronic resources licensed by other Summit libraries. Users suggested that these irrelevant links not even be shown in the interface, and OCLC listened.

Sometimes the trigger for displaying the fulfillment buttons needed some refinement, as in the Ocean's 11 example in figure 15. You can see that both the Request Item and Check for Online Access buttons display. The former is correct (users can request this video be sent to a library close to them), but the latter is not. The presence of a URL in the 856 field triggered the appearance of the Check for Online Access button, even though the link was just to a review of this movie. Now the system correctly displays only the Request Item button and correctly labels the link to the review.

Changes to Local Systems

As a result of this round of usability testing, in addition to changes made to WorldCat Local, the UW interlibrary loan office made changes to the look and feel of the ILLiad request screens. Throughout the testing, users made various negative comments about the out-of-the-box interface and past experiences with the system. Now the screens look like any other Web page on the UW Libraries site, and user reaction has improved.

Interestingly, now that WorldCat Local has been live for over a year, many users (students especially) feel the “Interlibrary Loan” link on our home page is redundant and should be taken away. Their rationale is that you can get to interlibrary loan through integration in our other various systems, so why do we need a link to it on the home page?

Usability Testing, Round Two

Although many improvements were made to WorldCat Local throughout the year, UW and OCLC staff were still concerned about some aspects of the system. In the second round of testing, we wanted to examine two areas: WorldCat Local's effectiveness in topical discovery and the effectiveness of WorldCat Local's FRBR displays (in which all manifestations of a work were aggregated behind the display of a single edition).

This time our subjects were all undergraduates, to test an audience different from that of the University of California system, which would soon be testing its version of WorldCat Local with graduate researchers. As before, we sought a diverse user group that was representative of our campus population and offered a gift card to the University Bookstore as a token of our thanks.

For this round of tests, we asked users to do a combination of known item searches and some general topic searches in areas where they already had some subject expertise. We wanted to know how well users would do when they were looking for materials on a given subject. Did they use facets or scoping more or less? Were they satisfied with their results?

At this point, the data from this round of testing is still being analyzed. We heard that users started their search for information elsewhere—mostly Google or Wikipedia—and come to the library usually when they have a need for known item searches. This has been said before, but hearing it from users firsthand is very powerful.

Other Feedback

In addition to formal usability testing, we gathered user feedback in other ways. From the day WorldCat Local went live, both the UW Libraries online catalog and WorldCat Local had links to an online survey. Initial feedback from that tool indicated issues with the system, but feedback began to improve once the system improved.

All sites that run WorldCat Local are able to access detailed Web statistics that count a large number of parameters about what users are clicking on from what screen, and the number of scoped searches that are conducted. These reports continue to improve and provide useful information about the system's use.

More powerful than a Likert scale or raw count of Web hits are the users' own words. As we were deciding which links to include in the WorldCat Local user interface, we all felt strongly that a link back to our online reference service must be present. We participate in the QuestionPoint 24/7 service, and thus have someone who can “answer the phone” any time, day or night, to help users. Users were not shy about letting us know how they felt about the system. Some liked it:

This is not a question, it's feedback. The new search catalog, WorldCat, is AMAZING compared to the old University of Washington library search. It's one of those things I wished we'd had, and then, all of a sudden it was there. Bravo!

The new service is a great resource for finding material in multiple locations at once.

Some didn't:

I just wanted to say I thought the old University of Washington library catalog system functioned MUCH better, and I continue to use it. At a minimum, please do not phase it out; if possible, please use it as the primary system on the front library webpage.

The following e-mail was from a graduate student in the biology department who was writing at a time that WorldCat Local was down for maintenance, and he let us know how much he relied on the system to get his work done:

I've become accustomed lately to accessing online journal in full text from my own home in the evenings. It seems to be the time I get the best work done.

My best source so far has been the main keyword search from the library homepage. The articles covering this material show up in equal amounts across the chemistry, engineering, health and biology stacks. But for the most part, every single one in the last ten years is also listed as an online source with direct access to full text

The breadth of this searching option has made it the most useful. Is there another source to search that broadly among articles that the University of Washington has a subscription to?

1. University of Washington Libraries, “UW Libraries Usability,” (accessed May 30, 2008).


[Figure ID: fig1]
Figure 12 

Screen shot of detailed item record from the first round of usability testing of WorldCat Local at the University of Washington. Accessed september 2007.

[Figure ID: fig2]
Figure 13 

Screen shot of a detailed record for Guns, Germs and Steel from the first round of usability testing of WorldCat Local at the University of Washington. Accessed september 2007.

[Figure ID: fig3]
Figure 14 

Screen shot of a detailed record for The Economist from the first round of usability testing of WorldCat Local at the University of Washington. Accessed september 2007.

[Figure ID: fig4]
Figure 15 

Screen shot for Ocean's 11 from the first round of usability testing of WorldCat Local at the University of Washington, showing two fulfillment options. Accessed september 2007.

Article Categories:
  • Information Science
  • Library Science


  • There are currently no refbacks.

Published by ALA TechSource, an imprint of the American Library Association.
Copyright Statement | ALA Privacy Policy