While other digitization programs on various scales also are under way (as noted above), none approaches the scale or ambition (or potential for market dominance) of Google Books. For this reason the volumes digitized by Google seemed the most appropriate objects of which to ask: Do these digitized volumes in themselves now constitute a viable general research collection? This may seem a fairly straightforward question, but it raises an antecedent question: What is a research collection?

In the abstract, a research collection is a collection of materials used primarily to support research (as opposed to one that supports teaching and learning or one that is used primarily for recreational purposes). Unfortunately, this definition does not lend itself to objective measurement, and it says nothing about the content of such a collection, since, in theory, any collection can support research of some kind.

Indeed, most research collections are developed to address the needs of a particular research community, a community that will typically reflect a variety of research interests and intensities. This variety will itself change over time. Research collections are by their nature complex. Such complexity underpins the design of the Conspectus model developed in the early 1980s by the Research Libraries Group for cooperative collection development. The Conspectus asked participants to characterize their collections according to a variety of parameters, including research area (defined by ranges within a bibliographic classification scheme or by subject descriptors), language, geographical scope, chronological periods, formats, and collection depth (this on a five-point scale, with 4 indicating “research level”).⁹ This produced a nice matrix for describing the variety of possible research collections, but it also made clear that the idea of a “generic” research collection was an oxymoron.

Ideally, in addition to being designed for a particular research community, a research collection also satisfies the needs of that community. But while research collections can come in a variety of shapes and sizes, data suggest that whatever their shape and size, local researchers will always feel that their own library's collection falls short—this despite years of earnest collection development by librarians. At member institutions of the Association of Research Libraries, for example, respondents to successive LibQual+ library service quality surveys routinely report that their libraries provide inadequate support for their research needs. On three LibQual+ items measuring collection support for research, the “adequacy gap”—the degree to which an item exceeds (or not) a user's minimum requirements—has typically been a negative number.¹⁰ To put as generous a spin as possible on the meaning of these LibQual+ responses, one could say that research collections always must be works in progress.

Although there may be no such thing as a typical research collection, at major research universities—where the research communities are larger and more multidisciplinary—a certain amount of homogeneity can be expected to develop across the associated research collections. It is not unreasonable then to treat one of these collections as approximating a “generic” large university research collection.

Having posited that the collections of large research libraries approximate to a generic research collection, a question remains about how much unique content is found in a typical research library. No one knows for sure, but overlap studies suggest it is more than one might expect.¹¹ In a 2005 study (shortly after the announcement of the Google Books Library Project) Lavoie, Connaway, and Dempsey determined that the Google 5 libraries collectively held about one-third of the resources cataloged as books in the OCLC WorldCat database, and most of these resources—61 percent—were unique to just one of the five. This percentage of uniquely held resources increased with the age of the resources involved, with 74 percent of resources published between 1801 and 1825 being held by just one Google 5 library.¹² However, there is unique and then there is unique. In subsequent research, Lavoie and Schonfeld examined a random sample of one hundred WorldCat records for such “uniquely held” resources and found that “many of the English language materials appear to be locally-produced ephemera rather than traditional published books.”¹³ This suggests that unique holdings may be less of a problem for the idea of a “generic” research collection than might otherwise appear.

Given its existing digitization agreements, it seems reasonable to expect that Google Books will eventually become a generic research collection in this sense. It will at the very least become the University of Michigan Library, since Michigan has agreed to digitize its entire library (aside from Special Collections).¹⁴

This study is limited to materials that would have been viewable in full in Google Books in 2008 under current copyright law, that is, materials then in the public domain. This was a pragmatic decision in that such materials are the only ones that can be demonstrably shown to have been digitized. But it also attempted to address the idea of the research collection as implied in Coyle's original comment—something that would be used for research.¹⁵

The author also felt that restricting the study to public domain materials would produce a more conservative estimate of overlap than would be true for more recent imprints, given the participation of publishers as well as libraries for this material in Google Books, the growth and expansion of American research libraries over the last century, and Lavoie, Connaway, and Dempsey's findings of increasing overlap within WorldCat as imprints become more recent.¹⁶ This suggested that whatever conclusions the study reached about materials in the public domain would apply with even more force to materials still covered by copyright should those materials be exposed to viewing via Google Books at some future point.

How big is the public domain? Nobody knows. The online English Short Title Catalogue, covering pre–1801 materials published chiefly in the British Empire and the United States, had accumulated nearly half a million entries by mid-2009.¹⁷ More generally, Buringh and Van Zanden estimated that more than 1,750,000 books (defined as fifty pages or more) were published in Western Europe prior to 1801.¹⁸ To bring this closer to the present, in a 2006 examination of books in OCLC WorldCat, Lavoie and Schonfeld found that roughly six million books (18 percent of the books then in WorldCat) were published prior to 1923 (a rough indicator of public domain status in 2008).¹⁹ The chart accompanying the Lavoie and Schonfeld article suggested a steady rate of increase in publishing output throughout the nineteenth century.

But as anyone who has tried to establish the copyright status of a book knows, public domain status is not simply a question of date of publication, and Google's caution increases as the date of publication approaches 1923 (with digitized volumes more likely to offer “no preview,” “snippet,” or “limited preview” access rather than “full view”). For that reason, the current study adopted a more conservative interpretation of public domain, looking for a cutoff farther back in the nineteenth century.

Given this working definition of the public domain, one question that still needs to be addressed is that of page image quality. Scholars have not been silent on this question, and anecdotal evidence suggests the frequency of poor page image quality is not insignificant, though the solution is by no means clear.²⁰ Quality control of scanned texts, unlike the initial scanning, cannot be done on an industrial scale and is labor-intensive. In theory, however, volumes to be rescanned could be prioritized on the basis of user complaints. A “flag this page” feature, presumably for this purpose, was present in earlier versions of Google Books but had been discontinued at the time of writing. Similarly, the problem of nontextual content (illustrations, maps, etc.) in the digitized volumes, while not significant for indexing, is significant for research, especially in the case of folded material, which Google scanners typically digitize in its folded form.²¹

Given both this more restricted definition of the public domain and the caveats regarding image quality and folded materials, the current study attempted to measure what proportion of the volumes in a generic research collection (as described above) would be present in digitized form in Google Books, using as a reference collection the catalog of a major research library that was actively collecting during the period, and using as control collections—against which the reference catalog also would be searched—the online catalogs of two other major research libraries, one actively collecting during the same time and one that began actively collecting at a later date.

The general research question can be stated formally as follows: Given that A and B were major American research libraries during the nineteenth century, while C became one during the same period, is there a greater probability of manifestation-level overlap between A and Google Books, between A and B, or between A and C? The question can be answered by testing two hypotheses:

1. A larger proportion of a random sample of entries drawn from A's catalog will be found in Google Books than will be found in the online catalog of B.
2. A larger proportion of a random sample of entries drawn from A's catalog will be found in Google Books than will be found in the online catalog of C.

The experiment involved six steps:

1. identifying an appropriate reference collection
2. extracting a random sample of entries from the reference collection
3. searching the metadata from those entries against Google Books for matching manifestations
4. identifying appropriate control collections
5. searching the metadata from the reference collection entries against the OPACs of the control collections
6. recording the results and performing appropriate statistical tests

Problems encountered during the current study can be subdivided into problems determining the extent of the manifestation represented by an entry in Cutter's catalog, identifying a manifestation in Cutter's catalog, locating a manifestation in Google Books, and identifying manifestations in the catalog of the control collections. These are discussed below.

The author found digitized items (matching “full view” books) representing 235 entries in Google Books out of 398 in the sample from Cutter's catalog. The match rate for Google Books was 59.05 percent ± 4.83 percent. The corresponding match rate for the older research library (268 matches) was 67.34 percent ± 4.61 percent and for the later research library (162 matches) 40.7 percent ± 4.83 percent. All rates were significant at the 95 percent confidence level. The Z-tests comparing the Google Books results with those of each of the control libraries were significant in both cases (Google Books versus the older research library: Z = 2.351; confidence level > 95 percent; versus the later research library: Z = 5.106; confidence level > 99 percent). These data are shown in figure 1.

The sample included 98 entries representing pre–1801 imprints. Google Books and the two control collections differed markedly in their match rates for these materials. Only 27 were found in Google Books, while 66 were found in the “contemporary” research library and 48 were found in the “later” research library.

For post–1800 imprints, Google Books had a slightly higher match rate than the contemporary research library and a significantly higher rate than the later library (see figure 2). Out of 300 entries in the sample representing post–1800 imprints, 208 were found in Google Books while 202 were found in the contemporary library's OPAC (Z = -0.439; confidence level > 95 percent) and 114 in the later library's OPAC (Z = 7.613; confidence level > 99 percent). (See figure 2.)

The study employed a very strict definition of manifestation, requiring matching resources to conform in all bibliographic details with the entries in Cutter's Boston Athenæum catalog. This meant that in several cases a seemingly identical text—an expression in FRBR terms—was rejected as a match. Consequently, the match rates reported in this study should be treated as floors in terms of matching expressions.

In a similar vein, the author makes no judgments as to the continued value of the expression embodied in a given manifestation vis-à-vis a later expression, one that might have superseded that earlier expression in the Athenæum catalog. A library that owned a manifestation of that later expression but not the manifestation represented by the Athenæum catalog entry was deemed not to own a match. As an example of this phenomenon, the Athenæum catalog contained a four-volume collection of the works of William H. Seward published during his lifetime, while another library had a more complete set published posthumously. Presumably the later set was both more complete and more accurate from a scholarly perspective, but with no way to determine this objectively, it was rejected as a match in this study. Again, the result is an undercount of what scholars might consider to be matches.

Beyond these effects of the experimental design on match rates, some differences were the product of the design of the digitization projects themselves. The observed differences in Google Books match rates between pre–1801 imprints and post–1800 imprints is largely explained by the collections that are being selected for mass digitization. Initially, these have tended to be low-use items in remote storage facilities. American libraries often treat 1801 as a heuristic cutoff point for rare books, with books published before that year routinely segregated to rare book libraries or similar facilities not subject to mass digitization.⁴⁰ When pre–1801 imprints have been digitized, the volumes have occurred in the general collections, especially in the general collections of European partner libraries. Of the twenty-seven pre–1801 sample imprints found in Google Books, fourteen were digitized from European collections.

While the digitization of pre–1801 imprints would be attractive to Google in terms of increasing the comprehensiveness of Google Books, research libraries have less incentive. Many of the titles involved are already accessible online to these institutions on a subscription basis from vendors that have digitized preexisting proprietary microform collections of old books (Eighteenth Century Collections Online, Early American Imprints, etc.), so bringing them into the mass digitization stream would not necessarily increase the digital content available to their researchers. Of course, this does not rule out digitization of these materials at a later time.

One side effect of indiscriminately digitizing great swathes of the print collections of large research libraries is an increasing rate of duplication as more collections are digitized, and a decreasing marginal rate of return for each newly digitized collection. For example, Lavoie estimated that while 58 percent of the holdings (approximately 18 million) of the original Google 5 were unique to a single institution, this would likely be true for only 22 percent of the holdings (approximately 8 million) of an additional Google 5.⁴¹

However, increasing duplication can introduce potential benefits in some contexts. While the study encountered many volume-level duplications, the study itself was not designed to measure this duplication and made no attempt to do so. Under ordinary conditions, one would want to keep such duplication to a minimum. But in a world where the power and storage capabilities of computers continue to increase at a staggering rate, this approach may no longer be valid, and a certain amount of redundancy may actually present advantages. In a slightly different context—that of translating a text from one language to another—Garreau observed in The Washington Post,

The explosion of the Web … has enabled a revolution. Like so many successful human approaches, it relies on brute force and ignorance. This method cares little for how any language works. It just looks—Rosetta stone fashion—at huge amounts of text translated into different languages by humans. (Dump decades of U.N. documents into the maw.) Then it lets the machine statistically express the probability that words in one language line up together in a fashion comparable to another set of words in another language.⁴²

In the context of Google Books, item-level duplication feeds into this “brute force and ignorance” method of machine-based quality control, especially as an alternative to more labor-intensive page-by-page human quality control (now mainly the responsibility of the partner libraries). One can easily imagine, for example, a two-step probabilistic method whereby Google first identifies identical images (setting a very high bar for a match in terms of visual pattern recognition), then selects from them the image from which optical character recognition (OCR) has produced the most satisfactory rendering into plain text (taking this as evidence of a relatively clear image of the original). This would simultaneously both reduce the number of duplicate volumes in Google Books and the number of poorly scanned images that would otherwise require manual intervention. A simpler process might be used to enable the supplying of missing pages when comparing two otherwise identical sequences of page images.

From the point of view of scholarly research, poor image quality and occasional missing pages may be less of a problem pragmatically than it is absolutely (as a guarantor of textual integrity). Researchers—the intended audience of a research library—are typically interested in books not as artifacts of cultural heritage or even necessarily as integral texts, but rather as containers of certain desired content. Consequently, most of them read books—at least online—only to the extent necessary to extract the desired content, and this may not be reading as we know it. This is seen, for example, in a recent large-scale survey of British academics, where most respondents reported that they “dipped in and out of several chapters” when reading e-books rather than reading continuously.⁴³ Given this user behavior, Google's post–scanning priorities might not necessarily be in image quality in general (except to the extent that this affects the ability of their software to recognize and index text) but rather in the quality of those images that people actually view (or try to view), since this can affect advertising revenue. Again, one should note that post–scanning quality control is currently the responsibility of the partner libraries.

This study has shown that—with some caveats—the pre–1872 digitized content now available in Google Books approximates that content available via the online catalog of a generic major American research library, and indeed is probably superior for post–1800 imprints. Google Books has reached this point in a remarkably short time—less than five years after the announcement of its initial Library Project—and given the large number of research library partners that have since been recruited, it seems likely that Google Books will eventually (perhaps very soon) become the single largest source for this content.

On the negative side of the ledger, two significant caveats must be recalled. The digitized images of individual pages are not always reliable—poor scanning can occasionally be so extensive as to render a digitized volume unusable—and folded maps and other illustrative matter are routinely scanned in their folded state, rendering them useless for research. One can reasonably expect that these flaws will be corrected over time, at least for high-demand texts: The users of the texts will insist on it and, at any rate, the libraries involved are committed to it.⁴⁴ Measuring the extent of this problem was not within the scope of the current study, but an extremely useful future research project would try to do so.⁴⁵

On the positive side, Google Books provides full-text indexing, something of incalculable value that would have been inconceivable had these volumes not been scanned. This indexing allows one to search both within individual volumes and across the entire collection, facilitating text-based research in general, but especially historical research and the comparison of variant texts. While this indexing is dependent in individual cases on the quality of the original page scan and the fidelity of the OCR rendering, in the aggregate the amount of hidden content that is thus exposed far exceeds the amount that remains hidden (or imperfectly rendered via OCR). Additionally, Google Books is serving as a huge laboratory for what is called “document image understanding”—the increasingly sophisticated probabilistic analysis of page images to facilitate indexing, interpretation, and other activities.⁴⁶

As noted above, in the past, large collections of works in the public domain—especially older English language works—were microfilmed by commercial firms in collaboration with various research libraries. The resulting microform collections have subsequently been digitized, either by the firms that did the original microfilming or by successor firms, and made available on a subscription basis. As Google Books and other mass digitization projects continue their progress through various research library collections, the viability of these preexisting collections may increasingly come into question as subscribing institutions weigh their annual use of these materials against the annual charges they pay for access.

Currently only a small fraction of the materials in Google Books—perhaps 15 percent—is thought to be in the public domain.⁴⁷ The great bulk is still protected by copyright, including a large but unknown number of so-called orphan works for which it is difficult or impossible to locate the current copyright holder.

These materials, many of which have been digitized in the course of the Library Project, were the object of class-action lawsuits brought against Google in 2005 by the Association of American Publishers and the Authors Guild.⁴⁸ The parties proposed a settlement of these lawsuits on October 28, 2008, but at the time of this writing the fairness of its terms is still to be determined by the U.S. District Court involved. On July 2, 2009, the U.S. Justice Department informed the court that it had opened an antitrust investigation into the settlement.⁴⁹

Among librarians and researchers, the reaction to the proposed settlement was in some ways emblematic of the ambivalence felt by many who stand to benefit from Google's mass digitization. Harvard University and others have objected to the proposed settlement on the grounds that it would grant a de facto monopoly to Google. Robert Darnton, director of the Harvard University Library, summarized his misgivings in the New York Review of Books:

Google is not a guild, and it did not set out to create a monopoly. On the contrary, it has pursued a laudable goal: promoting access to information. But the class action character of the settlement makes Google invulnerable to competition. Most book authors and publishers who own US copyrights are automatically covered by the settlement. They can opt out of it; but whatever they do, no new digitizing enterprise can get off the ground without winning their assent one by one, a practical impossibility, or without becoming mired down in another class action suit. If approved by the court—a process that could take as much as two years—the settlement will give Google control over the digitizing of virtually all books covered by copyright in the United States.⁵⁰

Given the research benefit that would accrue from providing direct integrated access to copyrighted material via Google, some mutually acceptable arrangement is likely to be reached, though its ultimate shape is hard to fathom at this point. The ramifications of any settlement are such that a lengthy court review seems likely.

Beyond the terms of the proposed settlement lies the larger question of how Google Books will ultimately affect the world of learning. By making so much of the printed record available in digital form—and so rapidly—Google Books is both transforming how scholars use the printed texts of the past and feeding a larger fundamental reshaping of the world of scholarly research. Fortunately for the author, speculation on the ramifications of these changes lies beyond the scope of the current study. But Google Books clearly is already having a dramatic effect, both on libraries and on scholarship. Indeed, a significant number of the sources cited in this study—beyond the objects of the study itself—were consulted in Google Books rather than in a physical library. As more and more scholarly research is conducted online first—and especially if the universe of digitized copyrighted works is ultimately opened up in Google Books—libraries may find that researchers are not linking out of online catalogs to versions of works available on Google Books but are rather linking to library catalogs for those cases where a version available on Google Books is not satisfactory for their purposes. We will then be entering a brave new world for both research and libraries.