Luckily, at this time—the winter of 2006—a very talented graduate student in the computer science department, Rushikesh Katikar, was enrolled in a work-study program and placed in the library as a part-time software developer. Rushikesh worked side-by-side with me as we evaluated various native XML database products and analyzed their scalability and performance for querying these disparate collections. We evaluated both open source solutions and off-the-shelf commercial products. We first evaluated eXist, an open source database developed in Java. At the time, eXist was very early in its life cycle, and the current release was at a very early version number. We then evaluated a commercial solution, MarkLogic. This product is well regarded in the industry as a workhorse; however, its learning curve and the need for training forced us to push the product to the bottom of the evaluation list. The next product that was evaluated was Berkeley DB XML. Berkeley DB is also a highly regarded and heavily used database product, and its native XML counterpart felt like it would be a contender. However, after many rounds of performance testing and index tweaking, we were unsuccessful in gaining performance and scalability to meet our needs—searching a set of over one million records in under one second.

After quite a bit of frustration in tweaking configurations and the lack of scalability in the products evaluated against our collection size, we sought a different technology to support our needs. The first stop was native indexing products, whose core feature provided strengths that the native XML databases lacked. We began to play around with Apache Lucene, the industry standard for indexing. Lucence, a highly regarded and heavily used product for search, can be found in both commercial applications large and small and open source applications. When looking at how Lucene could be woven into our solution, we found Apache Solr—a Java application built on top of the Lucene product. Solr provided everything that we needed: a web-based API to develop front-end applications in any language or platform, the ability to index and store XML structured data, the ability to scale to large collection sizes, the ability to perform fast queries and facet on the search results, and a fairly low learning curve. This was the solution.

This was a very active time in the library community. Many new products were coming onto the market, and libraries were witnessing a fundamental shift from purchasing physical resources to subscribing to collections of electronic resources. We saw a growth in options around federated search products, and the next-generation-catalog product category had just been created. During my time of employment at Villanova University, the library witnessed a shift in the bulk of acquisitions spending from print materials to electronic between 2003 and 2009—a trend not uncommon at other universities in North America (see figure 1).

Wesleyan University had a similar experience between 1998 and 2009 (see figure 2).

Similar changes have happened in the publishing industry, as seen from information on the sales of O'Reilly books versus e-books (see figure 3).

Around this time, libraries began to see the value in a modern search experience that would meet user expectations, provide a highly customizable user interface, and broaden the scope of what can be discovered in the library. This is when the product line for next-generation catalogs was first established, and the race was on to implement. In a few years, thousands of libraries around the globe have adopted next-generation catalogs. This fairly rapid adoption piqued my interest, and I asked some questions:

• How will adoption of NGCs impact the services offered by libraries?
• Will adopting NGCs in an effort to provide a better solution for our patrons really make an impact in the research process and use of print materials in the long run?
• Will this highly customized and modern search tool increase the library's value to its audience?

For seven years, I have worked on technology solutions to problems faced by libraries—a custom interface to a federated search product, a digital library platform, a subject guide platform, a question-and-answer system for online reference transactions, a next-generation catalog, and now a web-scale discovery solution. Throughout this period, these questions stayed with me. After seeing a bit of maturity in both age and number of installations hit the next-generation catalog product line, it was time to set out to find some answers.