This study is based on the “real-life” approach of Wakimoto and others (2006) to allow a historical comparison with their 2004 SFX testing results.² Resolver results from likely keyword searches for a number of popular databases were tested from September 2009 through June 2010. Stratification by document type was added to increase exposure of non-journal resources. Each author tested seven databases, collecting results for journal articles (10), book chapters (5), books (5), dissertations (5), and newspaper articles (5) whenever citations to those document types were available in the source database (table 3). Citations that included native full text were avoided, as well as those from journals or books that had been tested previously.

Overall, 351 source URLs were tested in this study. About half of the resulting resolver menus offered one or more online full text links (n = 169 [48%]; average full text link number = 2.01). The other half of the menus indicated that no full text was available, offering links to search the catalog, populate an ILL request, and search Google Scholar instead (table 4). Every full text link was checked for access (n = 343), and Google Scholar and Google were searched for each result with no full text available (n = 182). The results were then coded into six categories, mirroring Wakimoto, Walker, and Dabbour's designations.³ Their results are included for comparison (table 5).

Wakimoto and others (2006) reported that about 20 percent of their resolver results were erroneous. Roughly half of the errors incorrectly indicated availability (false positives), while the other half incorrectly failed to indicate availability (false negatives). Our result rates for these errors were similar. For this study, however, the category “Required search or browse for full text” was reassigned from the Correct group to the Error group to reflect reduced user willingness or ability to further navigate to the full text. When the target full text item or abstract with full text links is not presented on the target page, most users and even many librarians perceive the resolver as having failed. This category increases the total error rate by nearly 70 percent, averaged across both datasets. This results in total error rates of 35 percent for the Wakimoto and others dataset and 29 percent for our dataset (table 5).

The error rates increase further when freely available content is taken into account. All “no FT available” items were searched in Google Scholar and Google, using links provided from the resolver window or with the LibX browser add-on. Twenty-one of 138 (15 percent) were available via the Web. Tapping into this content is equivalent to increasing our budgets by 15 percent. Furthermore, the percentage of “externally available” items is likely to be higher in an article-heavy dataset and will increase over time as authors continue to post their own content on personal webpages and in institutional repositories. This additional category of false negatives increases the overall error rate to 33 percent. While expanding resolver knowledge bases to enable direct retrieval of “external” items may not currently be possible, we can accomplish improved access to them from our resolver windows. As a first step, links to extend full text retrieval to Google should be made more prominent in resolver menus. It should be our eventual goal to fetch the full text link (or even the document⁴) from the Web and present it in the resolver window.

To be fair, there is a less critical way to measure resolver success: how many resolver menus that offer full text contain at least one link that leads directly to accessible full text? By this definition, the CUC resolver was successful 93 percent of the time (in 86 of 93 menus), and the EKU resolver was successful 70 percent of the time (in 54 of 77 menus). Thus, by this measure, the resolvers were successful approximately eight out of ten times for the combined dataset.

The opposite of the resolver error rate is the accuracy rate: 71 percent overall for the citations tested. Book chapters and book menus were far more accurate than those for other document types (0.98 and 0.95, respectively, table 6). Unfortunately, the vast majority of these successes (101 of 105) reported negative results, reflecting small e-book collections or their absence from the knowledge base. In addition, because the study was designed to emphasize book content (40 percent of the source URLs tested), the overall accuracy rate is probably an overestimate of what most users experience. Indeed, when book results are excluded, the overall accuracy rate is reduced to 64 percent (270 of 420 results). With this in mind, our results show that only about two out of three non-book resolver results are accurate.

In contrast to book content, newspaper and dissertation results had much lower accuracy rates than average (0.38 and 0.30, respectively, table 6). Newspaper article citations occurred in only two of the databases and yielded contrasting accuracy rates. Ebsco's Academic Search Premier citations had many more bad links than Serials Solutions' Summon. This is probably at least partly due to the restricted newspaper content in ASP: the Wall Street Journal and New York Times are notoriously hard to link to. It is also possible that Summon's unified index has improved the success rate for this document type. More data is necessary to distinguish among these alternatives. In contrast, the data for dissertations were quite consistent. Accuracy rates were very low across the board, with most of the successes attributable to specialized indexing (as in Summon and ERIC) or to older results that were correct by default because full text is not available online. We further address the poor accuracy rates for newspaper and dissertation content in the section on causes of failure, below.

Nearly two thirds of the results were for journal articles, so perhaps not surprisingly, their accuracy rate most closely mirrored the overall results (75 percent, table 6). America: History and Life (AH&L) had an unusually high success rate (0.97), while the National Criminal Justice Reference Service (NCJRS) database was on the very low end (0.18). The high error rate for NCJRS is attributable mostly to the limited metadata sent in its source URLs. They include only journal title, date, and article title. The reason for AH&L's high success rate is less clear.

Although it is tempting to further analyze our accuracy results by source database, we deliberately chose not to do so, for three reasons. First and foremost, although source URL quality can influence linking accuracy, they are the furthest from the final result, being dependent on the “downstream” resolver and target database. Secondly, only journal articles could be tested across all citation databases, and half of the database/vendor combinations were tested at only one institution. Finally, the IOTA project (Improving OpenURLs through Analytics) is focused on assessing source URL quality for large OpenURL datasets and is better positioned to do so. Instead, we present an analysis of the causes of failure recorded in our study. To our knowledge, this is the first systematic attempt to categorize the causes of a set of OpenURL failures and determine their relative frequencies. It is our hope that these results will help determine which aspects of the resolution chain need the most attention and identify solutions that will address the most common failures.

IOTA (Improving OpenURLs through Analytics)

www.openurlquality.org

Librarians and OpenURL aficionados alike often disagree as to who or what is at fault for link resolution failure. Some say it is poor standards implementation or metadata quality in source databases. Others blame their link resolver vendor and advocate for switching to a different supplier. Still others claim that it is poor holdings data in the library's knowledge base. The final scapegoat is the full text provider, which may fail to resolve perfectly formed (and standardized) target URLs. In one sense, the answer is simple: each component contributes to the problem at least some of the time. But this simple answer obscures a key question: which component or components are most commonly at fault in any given library? It remains to be seen whether generalizations can be made. It is certainly true, however, that for particular combinations of source, resolver, knowledge base, and target, some components are more at fault than others. Libraries should evaluate and improve these components for their most important sources and targets. This section presents the framework of a rubric which can be used to do so.

Our study also provided a great deal of insight into the effectiveness of our resolver menus that is not reflected in the data presented above. As active users of the product, we noticed a number of aspects of the front-end functionality that need improvement. These observations pertain to the specifics of OpenURL functionality, providing a complement to the application of general Web usability principles to resolver menus in chapter 2. We present them there as specific constructive criticism of our own systems, but most will apply to resolver implementations at other libraries.

The primary user expectation when clicking the resolver button is that it will lead them to full text. Given that about that about half of the requests sent to our resolvers do not match full text covered in our knowledge bases, it is important to make these results as clear and effective as possible. At EKU, the notification states, “This item is not available online” (figure 5). Although the statement is clear and simple, it is false for items that are accessible on the Web but not represented in the knowledge base (as in this example). At Claremont (CUC), the phrase is “No full text for this citation was found in the online collections of the library.” Although technically correct in all cases except for knowledge base errors, this text is wordy and is not the most important information for the user at that point of need. Put another way, users generally do not care whether the item is in the library's collection: they clicked the resolver button because they want to know whether the item is immediately accessible to them. This principle calls for an interface improvement that is far more important than the terminology. We need to restructure our resolver menus so that additional instantaneous paths to the full text are colocated with the results from the knowledge base. Thus we recommend that the links to extend the full text search to Google Scholar be moved up to the second position in the resolver result menu rather than being placed near the bottom as a solution of last resort. This is a particularly important improvement for CUC, whose resolver menu is very long and interjects links to search for related articles above its additional options (figure 6).

There are also a number of cases where identical target links are presented in the same menu. For example, a “Get it Online” link is presented for a single version of an article that is listed both in EBSCOhost Academic Search Premier and EBSCOhost EconLit with Full Text (figure 7) or in a publisher site as well as from CrossRef. At best, this adds text to the menu that is not needed when the first link works. At worst, when the first link doesn't work, the user will try the second link, thinking it is different, and that link will fail as well. This usability issue can largely be solved by adjusting the resolvers' administrative settings, although these settings may not affect CrossRef links.

Order of link presentation is a thornier issue. It would improve the user experience to be able to order links by some combination of link reliability; link depth; e.g., article-level versus journal-level; and format(s) available, listed in order of preference—HTML + PDF, PDF only, HTML only, HTML lacking figures or tables, and selected full text (i.e., some items missing).

• Link reliability is certainly the most important of these three criteria, but it is also the hardest to measure, presumably because the extent to which target links actually result in full text access is not captured by OpenURL server logs. The Pubget PDF delivery service (see chapter 4) may have unique insight into these numbers.
• Link depth should be consistent within a particular provider, so it would be particularly useful to have an administrative choice that would allow demotion of hosts based on this property. This seems particularly important for optimizing “one-click” or “direct link” functionality. When title-level links must be used, it would be extremely valuable to include a banner at the top of the journal homepage with the citation specifics (as WorldCat does, see figure 8).
• The item format(s) available differs between providers, and within providers among titles, and even within single titles. Although this information is certainly known by the provider, it is not commonly shared and was excluded from a draft list of data elements that KBART considered requiring (see section on Industry Initiatives in chapter 1). It seems reasonable to require providers to indicate whether portions of articles and even whole articles are missing for each title, but this too has not been forthcoming, except in extreme circumstances.

The resolver menus for book chapters and books at CUC need attention. They are specific to the resource type (genre) for 360 Link customers. Both menu types require a catalog search to determine whether the book is available online; it is far preferable to indicate print and online availability in the resolver menu. Furthermore, both menus are set up to search the local and union (Inn-Reach) catalog in separate steps (figure 9), even though the local catalog will send the search through to the union catalog when requested. They are also set up with separate target links by ISBN and Book Title, and the ISBN search regularly fails because the resolver adds and then searches by 13-digit ISBN, while the local catalog predominantly contains the 10-digit version. When sending book chapter searches from the CUC resolver menu to Google Scholar, a chapter title is sent, but this does not directly facilitate searching for the book title in Google Books. EKU's Google Scholar search for book content (figure 10) is preferable, although sending searches as phrases, (i.e., in quotation marks) would improve their results. Google offers results for keywords when the phrase search produces no results, so nothing is lost by sending the search in this manner.⁹

These tasks are presented roughly in order of increasing complexity. That said, they involve a wide variety of skills, so the degree of challenge of each will depend on the expertise available at each library.

• 1. Examine the “no full text link provided” report (SFX only). In addition to being a valuable collection development tool, SFX usage report Query 20, “OpenURLs that resulted in no full text services, selected by source,” provides an excellent opportunity to test for false negatives (see also chapter 2). It combines source URLs that fall into the first and last result categories (table 5), supplying a list of URLs that can be tested for access using Google Scholar links from the corresponding resolver windows. Patterns in this data may reveal whole collections that are not listed in the library knowledge base, a problem that is easily rectified. It is also easy to assess the extent of the requested content that is available on the open Web as a part of this process.
• 2. Fix dissertation target linking. EKU's usage and OpenURL failure data provide powerful justification to fix linking to this class of resource (see tables 2 and 6). Because an improved source parser provided by the link resolver vendor seems to be the ideal solution, we are requesting a global fix of this issue by Serials Solutions and Ex Libris. In the meantime, locally hosted SFX implementations can edit their source parsers to fix this problem.¹⁰ Our results showed that newspaper article linking failed almost as often as dissertation linking. Although newspapers are at least as significant a concern, their pagination and date variation, short nondistinct article titles, and frequent supplementary sections make them much more of a challenge.
• 3. Review every full text provider for item- versus title-level linking. Given the overarching goal of reducing the number of clicks from the resolver button to the full text, item-level “deep linking” is always preferable. In most cases, link level is determined by the target parser, which translates the OpenURL into a request that the full text target platform can process. Obviously, it makes sense to start with the most frequently requested providers, examining them for item- versus title-level linking and ensuring that successful item-level linking is established wherever possible. Furthermore, knowledge of this attribute is essential for establishing the order in which full text links are presented.
• 4. Reorder the full text provider links. This is an art rather than a science. It is, nonetheless, very important, because of the tendency of users to click on the first link and because one-click access is heavily dependent upon it. Key provider factors include link reliability, link depth, and format(s) available (discussed above). Once the values for each of these factors are known for each full text provider, the library can decide how to weight each factor. After the most desirable order is determined, it can be integrated into the administrative settings. By default, both systems list targets alphabetically. For 360 Link, setting the order requires entering in a rank order number for each database, not each provider. This leaves a lot to be desired because many providers have multiple databases that should receive the same rank and minor adjustments require extensive reranking. Perhaps a simple solution would be for Serials Solutions to change its system to allow priorities (i.e., 1, 2, or 3) rather than a ranking (1 to 314 for CUC), or even to offer its own order based on the factors above. SFX is significantly simpler to configure: it requires only insertion of the list of targets in the desired order in a configuration file. SFX also provides the ability to force specific targets to appear at the bottom of the list, allowing implementation of a simpler ranking (e.g., “O.K.” and “bad”).
• 5. Expand knowledge base coverage and rework resolver menus to maximize full text access. There is a delicate balance between expanding knowledge bases to cover more free and open access full text content and reducing resolver effectiveness, because these resources tend to be less well maintained.¹¹ A first step here is to maximize use of freely available collections that are covered by commercial knowledge bases (see data on error rates from Hutchens reported by Brooks-Kieffer).¹² Libraries can balance more extensive knowledge base coverage with more prominent and effective links to use Google Scholar and Google to access these resources (see section “Qualitative Observations on Resolver Effectiveness” above).
• Another key area of knowledge base expansion is the inclusion of e-books. Although there are rudimentary implementations of these in both vendors' products, there is still a great deal of room for improvement. Since libraries are investing considerable effort in representing e-books in their catalogs, the best near-term solution is probably an adaptation of David Walker's Chameleon SFX plugin to integrate e-book lookup into the full text services section. A similar JavaScript-based tool could potentially be built for 360 Link.

  Chameleon SFX Catalog Integration Plugin

  www.exlibrisgroup.org/display/SFXCC/Chameleon+SFX+Catalog+Integration+Plugin

• 6. Optimize top 100 most requested journals. According to the 80/20 rule, 80 percent of use occurs in 20 percent of the titles, so focusing on heavily used journals will address a great deal of the overall usage. Although only SFX provides a report that is specific to resolver requests, 360 Link customers can use the core Usage Statistics report “Click-through statistics by Title and ISSN” to list their 100 most popular titles. A general citation database can then be used to test resolution to articles in these journals, allowing libraries to assess the associated success rates and failure causes, as demonstrated in this chapter. When the underlying data is collected in a systematic way, spreadsheet pivot tables can be used both to examine frequencies and to show details from individual categories.¹³ This transforms the spreadsheet into a rich, easily accessible archive of examples that can be used for troubleshooting and sharing with others. Although some issues may be beyond reach, many can be addressed successfully, once they are recognized. Priorities can be established based on the frequency of the problems and the relative ease of fixing them.
• 7. Optimize top ten full text target providers. The number of click-throughs per target host (table 1, SFX Query 7) can be approximated with Serials Solutions' “Click-Through Statistics by Title and Database (Holdings)” report.
• 8. Extract and harness the resolver use data to better inform a top-down approach. The most efficient approach to improving the user experience with OpenURL linking requires identification of the fixes that will be of greatest benefit. SFX libraries can gain significant insight into usage patterns via its standard usage reports (see chapter 2 and article by Chrzastowski and others).¹⁴ However, the most powerful source of this information is the resolver server log. The structure of the OpenURL standard makes analytics on these files particularly fruitful. For example, extraction of data for “sid=” and “genre=” provides valuable information on the most used citation databases and content types. Sorting these files by Web domain separates source URLs from target URLs, and free Web analytics software (such as Funnel Web) can extract elements and reveal source platform and provider publisher frequencies. Resolver log files will be a crucial source of information for 360 Link customers, who do not have access to resolver reports like those contained in SFX. Regular collection of these files can also support database evaluation and other collection development needs.¹⁵

  Funnel Web

  www.quest.com/funnel-web-analyzer

• 9. Optimize top ten source databases by content type. Once staff at a library extract a list of the frequency of requests by content type for its most used citation databases from a log file, they can optimize resolution from these key combinations in the manner described above. For example, there may be a high volume of requests for book chapters in PsycInfo or books from MLA. Optimization of alternative content types is likely to include menu reformatting, in addition to the data- and translation-related issues common to journal article resolution. This level of analysis may also reveal peculiarities that are unique to the specific key combinations, thus revealing important issues that wouldn't be discovered in standard usage reports.
• 10. Implement, test, and optimize one-click/direct link to full text. As noted in chapter 1, discovery tools will be dependent on one-click if they are to be a viable alternative to Google Scholar. Also, it seems likely to us that in the future, link resolution will be passive and menu-free, rather than active and menu-based (e.g., see discussion of Pubget in chapter 4). The first step toward this eventuality is implementation of the one-click to full text service. We chose this as the final recommended step, not because it is the most complex, but because all of the previous improvements will make it more effective. In particular, reordering the full text provider links should be a prerequisite to this step. One link resolver feature that is needed here (not yet offered by 360 Link) is the ability to “opt out” of one-click for source databases and full text providers that are problematic. This function is available in SFX, at least for full text providers.