The study sought to answer two research questions:

• Question 1: Are library patrons carrying out complex searches employing advanced search features such as limits and Boolean operators?
• Question 2: What is the reaction of library users when they launch an OPAC search that gets zero hits?

Question 1 grows out of an awareness that librarians usually have been trained in search strategies and information retrieval in the OPAC environment, and that undergraduate users have not received that same degree of training and do not have the same depth of experience. OPACs are populated, however, with carefully created MARC records that contain numerous fields that allow for limits to be enacted or that permit other advanced searches.

Question 2 begins with an understanding that undergraduates are more at home using Google and other search engines, which index whole documents along with metadata and information about linking sites. As Marcum noted, full-text indexing provides considerably more keyword access to full-text results in the search engine than to the surrogate in the OPAC.¹⁴ The number of hits returned in a Web search can be staggering. OPACs are not meant to work in the same way as Web-based search tools. In OPACs, especially those of medium-sized academic libraries such as the one investigated, receiving zero hits in response to a search is common.

Question 1: Are library patrons carrying out complex searches employing advanced search features such as limits and boolean operators? For the three days analyzed in-depth, the OPAC was queried 3,698 times from off-campus and library personnel computers. The largest percentage of queries (31.7 percent) was the default “Title (omit initial article).” Because this is the default search, it requires the least amount of effort to initiate on the part of the user. The second most common query was the “Keyword—Boolean” search (15.6 percent) and the third most common query was the “Keyword—Relevance” search (12.2 percent). These latter two searches were, as mentioned earlier, hidden from view on the drop-down menu (see figure 1 for the five visible index choices). Users had to intentionally seek out the keyword searches to use them. In table 1, the queries are listed in the order that they appear in the drop-down box visible to patrons. The results listed in table 1 only include queries keyed in by patrons. They do not include browses that were launched by clicking a hyperlinked access point in an opened bibliographic record. Such browses do not require patrons to select indexes, enter terms, or decide on a query strategy; for this reason, they have not been included. Noteworthy is that 1,071 (28.96 percent) of the 3,698 searches conducted during the three days initially failed.

For the purpose of this paper, the “Keyword—Boolean” and “Keyword—Relevance” were the only two queries considered in the discussion of keyword searching. “Title keyword” was ignored because it was initially assumed to be much less popular. In retrospect, including it would have been worthwhile because it was more frequently used than expected. Future iterations of this research will therefore consider “Title keyword” along with the other two keyword search queries possible.

Measures of search complexity, such as the number of terms included in each search and the number of limits used, are a way of documenting the sophistication of the queries. More than one-third of the three days’ keyword searches used two terms in the query. The average number of search terms was 2.6. Fewer than 2 percent of queries entered included either “6 terms” or “7 or more terms.” See table 2 for further details.

Another measure of search complexity is the use of limits and advanced operators such as Boolean operators. Both were observed being used in the searches studied. In TCNJ library’s OPAC, a searcher can post different kinds of limits, including date of publication, type of material, and location in the library. In the keyword searches studied over the three-day period, 10 percent (365) had these kinds of limits applied. The addition of limits can be an effective strategy for restricting to the information when used correctly. The number of searches with limits is much higher than anticipated, and implications will be discussed below. Table 3 shows the advanced and special operators used in the searches over the three days. The advanced search technique most commonly used employed Boolean operators. Operators may not have been used intentionally especially if, for example, they were simply words entered as part of a title. Intentionality on the part of the user is impossible to ascertain in a study such as this where users are not able to explain their actions. Although not advanced operators per se, periods appeared in several of the keyword searches (2.3 percent). “Not” and “&” were never used over the three-day period, while quotation marks were used in almost 10 percent of keyword searches.

Question 2: What is the reaction of library users when they launch an OPAC search that gets zero hits? During the three days of the study, almost 30 percent of all searches failed overall. For this study, failure is defined as a query that results in zero hits. The default left-anchored title browse led to the most failures. In the drop-down box, this query is labeled “Title (omit initial article).” Despite the instructions about the initial article, a full 5 percent of these queries began with either “A” or “The.” All of these queries failed because of the incorrect use of the initial article whether or not the material was used. Despite the fact that this query is a browse of the title index, if the character string does not make an exact match with entries in the title index, the system declares zero corresponding results.

When faced with failure, users had to devise a strategy for how to proceed. The most common reaction (52.3 percent) to a failed OPAC query was to relaunch it using different or modified terms at least once in the session. More than one-third (35.9 percent) of the searches opted to change the index, either with or without changing the search terms. Nine percent of users simply stopped searching after getting zero hits. More than 12 percent ran the same failed search again (either immediately or later in the session) without any changes. This percentage is lower than the 18.9 percent of re-run failed searches reported by Connaway, Budd, and Kochtanek before the advent of the Web.¹⁵ Some users tried adding search limits or other advanced search strategies when relaunching a search; in doing so, they did not always rekey the search terms. Although the addition of limits can be a good strategy when narrowing a large result set, it is not an effective strategy if the basic search is yielding no hits. Table 4 presents the user responses to a failed search.

This study examined the transaction logs from queries carried out by students, faculty, and off-campus users at a medium-sized academic library. Librarian searches were excluded from this study. Queries considered in this research likely were initiated by undergraduates, as this is the largest population served by the TCNJ library. Transaction logs give no indication of the status, experience, mindset, or goals of the user, and these aspects of the user experience cannot be evaluated in this study.

The answer to the first research question—whether users employ advanced operators such as Boolean operators and search limits—was seen as affirmative. For this element of searching, users understand that there are ways to limit within the library catalog in order to refine their search. The use of limits was especially noteworthy because patrons had to apply them manually from the search screen. The prevalence of using Boolean operators is less certain, because the conjunction “AND” may have simply been a word that was part of the search string being queried.

Almost 30 percent of searches resulted in failure (zero hits) with the current OPAC system. Despite different types of queries, use of limits, and browses that could potentially get users the information they require, the TCNJ OPAC produced failure nearly one third of the time. A medium-sized academic library does not have every piece of material that its users may want. The OPAC, however, should be designed in a way that is intuitive for patrons.

Because the default search and most-used query was the “Title (omit initial article)” and because this search was responsible for the most failures, it deserves special consideration. Users are supplying initial articles with this query despite instructions. At this selective college with top-ranked students, users are not reading instructions and are approaching the default query assuming they know how to use it. Libraries need to be sure that the default search made available to patrons is one that does not need explanation or instructions.

The way the ILS is constructed influences some of the failure that patrons experience. Patrons are forced to supply Boolean operators or quotation marks when doing a “Keyword—Boolean” search. Users who do not add these advanced operators generate failed searches, no matter how correct their search terms are. Of the failed searches in the study, Boolean searches accounted for 16.3 percent of the total (see table 1). Users had fewer instances of failure when carrying out the “Keyword—Relevance” search. Relevance searches failed less than 2 percent of the time. However, when users opted for the relevancy ranking afforded by the “Keyword—Relevance” search, their terms combined in a way that made multi-term queries or nonunique terms ineffective.

Other failures or problems came from the patrons themselves. Users did not necessarily choose the most logical index or query type for the terms that they entered. In some instances, users did not read the instructions. This is clear in the way that they did not follow the examples, especially concerning the use of initial articles in the default search. Typographical errors also kept patrons from finding materials (see figure 1 for an example of the basic search interface result for no hits due to a typographical error). Interestingly, patrons often re-ran a failed search, exactly as typed, before quitting the session.

The strategy of applying limits to searches, especially failed ones, demonstrated that users were willing to make use of advanced operators and special OPAC-only capabilities. Users were aware that these features were available and were willing to try them. For this reason, despite the current ubiquity of Web search engines as gatekeepers to digitized information, suggestions about abandoning some of the rigorous work that goes into cataloging library materials would be premature. Users do understand that library OPACs have unique features, and are forming their search strategies accordingly. Librarians will also want to continue studying the pros and cons of providing only a simple “Google-like” search box as the primary OPAC interface if the system provides no way for patron-supplied limits to be incorporated in the search.

Two primary recommendations for improving the OPAC search experience could be considered based on this study. The first recommendations include adding elements to the ILS to enhance the user experience. The second set of recommendations assumes that ILS technology cannot be modified for the moment and that a reconfiguration of the local decisions that underlie the OPAC setup must be carefully undertaken.

To alleviate the issue with failed searches due to typographical errors, automatic spell-check functionality should be added to the system. The spell-check feature should be based on the live local OPAC, and should only suggest spelling suggestions that are actual terms in active OPAC records. Search engine spell-check features or generic dictionary-based approaches are discouraged. They may suggest terms that are spelled correctly, but that do not correspond with the content of bibliographic records in the local OPAC. Correcting the misspelled word is desirable; being able to suggest relevant library holdings after the correct spelling is identified is more desirable.

As an added feature to the ILS, a separate indexing of titles that includes initial articles could be generated to complement the regular index. If a system had titles indexed both with and without initial articles, regardless of the coding in the MARC 245 field, left-anchored title searches beginning with initial articles would no longer fail. Left-anchored title searches would succeed even if patrons misunderstand the instructions for title entry. From a cataloging point of view, this solution is not perfect. It would, however, get patrons closer to the material they seek and solve the problem of failed title searches that include initial articles.

The second set of recommendations focuses on the in-house setup of the database. Even among TCNJ librarians, confusion existed between the two types of keyword searching. For the public search experience, one type of keyword search needs to be chosen and perfected within the limitations represented by the system. The search terms ideally would be “AND”ed together automatically and should not require patrons to enter Boolean operators in order to run a basic search. After terms are searched using an automatic “AND,” a way to “OR” together the terms and to continue to populate the list of hits should be possible. Because the default left-anchored title search resulted in so many failures, the library should make the new keyword search the default search for users.

Finally, the OPAC interface needs to be intuitive enough to use without reading instructions, tips, or help screens. Web usability is studied in the creation of Web pages; it should also be a major factor in the creation of a design layout for the OPAC. Patrons are major stakeholders in the look and feel of the OPAC, and they need to be consulted when the OPAC interface is being studied. User-centered design principles should be employed. When possible, the OPAC should be customized for the patrons, keeping in mind their expectations and the popular online information systems they use.

Like the search interfaces and online services (e.g., book, video, and music merchants) that are ubiquitous on the Web, library OPACs should not be change-adverse. Evaluating and updating interfaces should be continuous. Continued and repeated research, such as the study described in this paper, should to be conducted to ensure that the library Web site continues to serve user needs. OPAC design and evaluation is an ongoing process and through proactive initiatives such as OPAC redesign projects, libraries are uniquely positioned to work with patron-stakeholders to meet their evolving information needs in the Web environment.