In the US, digitization of valuable information resources has been supported with federal and state funding for over fifteen years. The Institute of Museum and Library Services (IMLS) has awarded National Leadership Grants and Library Services and Technology Act grants to more than five hundred digitization projects of various scales since 1998.² The National Science Foundation (NSF) has funded more than three hundred STEM digital collections.³

To offer easy access to rich pools of information objects that are available in digital format because of efforts of hundreds of digitization projects over the years, large-scale digital libraries aggregate hundreds of separate collections and function as portals to these collections and the individual items contained in them. Large-scale digital libraries provide innovative solutions in transitioning learning and teaching to the digital platform.⁴ Many of these large-scale digital libraries were created for the general public (e.g., the European Library, and IMLS Digital Collections and Content). Others serve more specific audiences of educators, students, and scholars in different disciplines or domains.

Cultural heritage materials of historical and educational value, particularly resources about local and national history, have been a priority in mass digitization initiatives, especially in the early stages in 1990s and 2000s. Therefore many large-scale digital libraries were created for users in the domain of history. In the United States, many of them function at the state level (e.g., the Portal to Texas History; http://texashistory.unt.edu), some at the regional level (e.g., Mountain West Digital Library; http://mwdl.org), and several were created at the federal level. The American Memory (AM; http://memory.loc.gov) is without doubt the most well-known large-scale digital library in the US history domain. Although comparatively small in collection size, AM aggregates the most carefully selected information resources of the highest quality. This digital library was created by the Library of Congress (LC), in cooperation with other cultural heritage institutions, in the mid-1990s with financial support from the IMLS.⁵ Similarly, IMLS funded creation of the Opening History (OH) digital library (http://imlsdcc.grainger.uiuc.edu/history). This digital library was a spin-off from the IMLS Digital Collections and Content (IMLSDCC; http://imlsdcc.grainger.uiuc.edu) portal to all digital collections supported by IMLS, with the purpose of further developing the strongest content area in the IMLSDCC (US history) and providing access to it. OH’s primary user group was broadly defined as history researchers, including both academic and nonacademic history scholars; teachers and students at undergraduate, graduate, and postgraduate levels; and genealogists and “citizen historians.”⁶ OH functioned as a separate entity from October 2008 to July 2012 and quickly became the largest aggregation of digitized content in the United States with more than 1,500 digital collections and more than a million items. In August 2012, OH was absorbed by its parent digital library, IMLSDCC.

Another important domain served by large-scale digital libraries is STEM. In the past decade, STEM digital libraries became major players in STEM education.⁷ The NSDL (http://nsdl.org) was organized by the NSF in 2000 for administrators, educators, general public, learners, parents/guardians, professionals/practitioners, and researchers. As the demand for access to high-quality resources in the area of STEM education for teachers and learners grows, NSDL serves as a starting point to locate and retrieve these discipline-specific resources in a variety of formats for different learning levels.⁸ NSDL aggregates educational resources that are available online from a wide variety of providers. The majority of these resources are free, based on Open Educational Resource (OER) access. The resources are organized by educational level (from pre–kindergarten to higher education, including informal education and professional development education). They are grouped by resource type (assessment materials, audiovisual, instructional material, reference material, and other). Subject categories are STEM disciplines, e.g., Education, History/Policy/Law, and others, including a General subject section.⁹ The NSDL offers well-defined search, browse, and help instruments, which include keyword searching, browsing collections by subject area, combining search by broad subject area and audience, limiting search to new collections, etc.¹⁰ The NSDL serves not only as a resource repository, but also provides useful services and tools for professional development of educators and for network collaboration among the members of NSDL audiences.¹¹

To effectively deliver content of large-scale digital libraries to their respective user groups, improve users’ interaction with digital libraries, and facilitate efficient information retrieval, digital library services must adjust to the evolving needs and information-seeking behavior patterns of user groups in the online environment.¹² The following will provide a historical perspective on research into information searching and review the literature that addresses how users search for information in online environment.

People engage in searches to satisfy their information needs. Searching is one of the two major types of interactions between users and discovery and access systems such as library catalogs, databases, search engines, or digital libraries.¹³ User searching is expressed through queries: sets of one or more symbols (e.g., words or phrases) combined with other syntax and used as commands for the information retrieval system to locate potentially relevant content indexed by that system. The query is a key theoretical construct in both the information-retrieval systems research and the information searching behavior research fields.¹⁴Figure 1 displays an example of a search query for “digital libraries” in one of the major search engines.

Many library catalog use studies, conducted in the 1930s to early1970s and summarized by Krikelas in 1972, compared how traditional library card catalogs were searched and how different bibliographic description elements (fields) of the catalog card were used by library patrons.¹⁵ It was found that author, title, subject headings, call number, and date of publication were heavily consulted by users while place of publication, publisher, edition, and content notes tended to be consulted less often; size, series note, and illustration statement were of less interest to library users.

The early automated catalogs of 1970s, mostly due to hardware and software limitations, had less functionality than card catalogs and were used by and designed specifically for library staff trained in technology.¹⁶ Only in the early 1980s did end users receive direct access to public access catalogs. Largely because online catalogs brought in new affordances of “search capabilities indexing,” which include keyword searching, Boolean searching, truncation, and multi-index searching, users initially expressed much greater satisfaction with online catalogs than with traditional library card catalogs.¹⁷ Such patterns of user searching as the use of Boolean operators and controlled vocabulary in online catalogs were analyzed in many studies. Boolean searching was found to be ineffective, not only because the majority of library users—even highly educated ones—experienced difficulties with Boolean logic concepts, but also because the execution order of Boolean commands was not standardized across different online catalogs.¹⁸ Moreover, some research from the early 1990s found that performance is improved in systems that do not require Boolean operators for complex queries.¹⁹ Many studies demonstrated that users of online catalogs tend to use simple keyword searches more often than any type of advanced search that allows them to search with controlled vocabulary terms.²⁰

Digital libraries have been developed since the late 1990s. User expectations of digital libraries were shaped by experiences with major easy-to-use search engines (predominantly Google), widely used transactional sites (e.g., Amazon and eBay), the popularity of computer games, and changes in the Western society in general: greater speed of developments, perceived need for immediate gratification, a more information-rich environment, and the popular heuristic of “satisficing” (i.e., an approach when the user is satisfied with “good enough” results that reach the minimum acceptability threshold by meeting some of the criteria and sacrificing other criteria).²¹ As a result, users typically expect much more from digital libraries than from conventional library services. These expectations include comprehensiveness, accessibility, immediate gratification, ease of use, and availability of data in multiple formats.²² Expectations of digital library services are often too high (although this is somewhat context-dependent) and are combined with a lack of appreciation of basic points, such as that digital library collections are created based on the knowledge of user groups’ needs.²³

A study released by British Library researchers in 2008 found that the main characteristics of user behavior include horizontal information seeking (a form of skimming activity, where searchers view just one or two pages from an academic site and then “bounce” out); extended navigation (people spend as much time finding their way around as actually viewing results); horizontal “power browsing” through titles, contents pages and abstracts; squirreling behavior, where the user saves information—particularly free content—in the form of downloads for later use but rarely revisits it; and little time spent in evaluating information.²⁴

The term domain knowledge was first coined in 1991 by Allen, who found that information-seeking behavior (i.e., selection of search strategy and tactics) and the outcomes of the search depend to a large extent on a searcher’s level of knowledge both on a specific search topic and the broader subject domain.²⁵ This observation was confirmed by numerous other studies. Researchers found that domain experts focus on the answers to search questions and have clear expectations for both the answer and the context in which it would appear.²⁶ For example, digital library user expectations, including “collection expectation”—an expectation that certain kinds of resources and information would be found in library or academic sources and not in search engines—differ by user domain and level of expertise.²⁷

Another important finding is that search tactics used by students change over time: as students acquire more domain knowledge on their research topic, they start to use wider and more specific vocabulary in their subject search.²⁸ Studies show that when the domain knowledge is low, higher numbers of searches per session occur because of an inability to initially choose appropriate terms; more domain-knowledgeable students use advanced search options more but make fewer changes to their searches.²⁹ It was also found that with an increase in the level of domain knowledge, users tend to use more terms in queries; domain experts use more effective strategy, conduct more complex searches, and incorporate more unique terms.³⁰

Many studies considered how representatives of a specific domain or discipline search for information. Studies have shown that scientists’ searching is usually aimed at specific questions or problems that they face when conducting an experiment, writing up results, or checking the accuracy of information in hand.³¹ Humanities researchers with high domain knowledge were found to use a variety of search types, with known author-title search being the least problematic. Success in more uncertain types of searches (e.g., a conceptual/discipline term search) was found to heavily depend on the level of the searcher’s domain knowledge and experience in using a particular digital library. Subject classifications were almost never used by academic searchers because the scholars’ conceptual models usually differed from that represented in the classification scheme.³²

Digital library users—humanities and social science researchers—were found to prefer searching to browsing, and some explained this by the lack of call number browsing capabilities in a digital library environment.³³ The users—both students and scholars—in these domains have been found to actively use timeline and chronological browsing, and interactive map browsing, and express the need for search limit by date.³⁴

Researchers also analyzed for what the users of information systems—in particular, scholars (i.e., domain experts)—search. Previous studies of web searching discovered, for example, that humanities scholars most often include in their search queries personal names, geographic names, chronological terms, and discipline terms. As shown by other studies, water quality researchers frequently use topical, geographical, and format or genre search terms, and occasionally, chemical formulas, dates, names, and URLs. Medical researchers’ prevailing search query types were found to include laboratory/test results, disease/syndrome, body part/organ/organ component, pharmacological substance, or diagnostic procedure.³⁵

The user base for domain-independent information systems that are aimed at a broad user audience tends to include more novice than expert users, while the audience of domain-specific information systems that are aimed at the users in certain domain (e.g., history, science) typically includes a higher proportion of domain experts. The studies discussed above analyzed information seeking behavior of the users of domain-independent and domain-specific information systems of more traditional types: bibliographic databases, including library catalogs, and some web search engines. However, information seeking by users in either domain-specific or domain-independent information systems of the new type—openly accessible large-scale digital libraries comprising of digitized and born-digital, high-quality content for education and research—has not been previously researched and compared.

Transaction logs recorded by the servers of information retrieval systems provide a wealth of data for analysis of various patterns of user information seeking expressed through queries. Transaction log analysis—“the study of electronically recorded interactions between online information retrieval systems and the persons who search for the information found in those systems”—is one of the methods actively used for unobtrusive observation of user behavior in various information retrieval systems.³⁶ For example, Markey’s summary of research results of the studies into information seeking behavior conducted over the period of 25 years, demonstrates that many of these studies used transaction log analysis method.³⁷ Transaction log data are often analyzed quantitatively. For example, Jansen, Spink, and Pedersen compared search query length and Boolean usage rates in different digital collections.³⁸ Moulaison studied transaction logs and analyzed queries by the users of a college online public access catalog and indicated that the number of terms included in a search (a query length) and the number of search limits can serve as “measures of search complexity” and a way of “documenting the sophistication of the queries.”³⁹ Transaction log analysis is frequently used in qualitative analysis. Several studies categorized web search queries into topical categories applying qualitative methods of research.⁴⁰

Despite the popularity of transaction log analysis as a research method for studying user information-seeking behavior, the potential of transaction log analysis has not been used to its full capacity to benefit large-scale digital libraries’ development. Several published earlier studies have analyzed transaction logs of domain-specific, large-scale digital libraries such as the NSDL, AM, OH, or domain-independent, large-scale digital libraries such as IMLS DCC or the European Library.⁴¹ However, only two of these studies examined the content of the user search queries.⁴² Moreover, the user search queries in the various types of digital libraries aimed at different user communities have not been previously compared.

The study reported in this paper addresses this gap through a mixed-method comparative analysis of transaction log data for patterns of user searching in two representative domain-specific, large-scale digital libraries: OH and NSDL.

Figure 2 displays search options that were enabled in the OH large-scale digital library at the time of this study. There were two types of item-level search where the user searches for individual information objects. The basic version of search for items allowed searching by keyword or phrase anywhere in item-level metadata records that describe individual items. The advanced version of search for items allowed searching by keyword or phrase anywhere in item-level metadata records, the faceted searching (i.e., a search in specific field(s) of the metadata record) by author/artist’s last name and/or by title/subject word(s), with a possibility to combine search fields and to limit results to specific digital collections selected from the dropdown menu. OH also thoughtfully provided its users with two collection-level search options to accommodate users interested in finding entire collections as opposed to individual items. The basic version of search collections allowed for a simple keyword search in all fields in collection-level metadata records that describe the collection of items as whole as opposed to individual items. The advanced version of search collections option augmented a simple keyword search in all fields in collection-level metadata records with a possibility to limit search results to collections containing one or more of eight resource types: dataset, interactive resource, physical object, text, image, moving image, and sound.

Figure 3 shows the search options enabled in the NSDL. There is one simple keyword search window, with a possibility to limit search results using one or more of the seven education levels, one or more of the twelve resource types, and one or more of the fourteen broad subject areas. The NSDL does not provide separate search options for advanced item-level search in specific fields of a metadata record (i.e., faceted search) or for collection-level search.

As shown in table 1, most of the search queries in both digital libraries were basic keyword searches, although the percentage of basic keyword searches was significantly higher in OH. The use of various advanced search options was observed in fewer than 15 percent of search queries in OH but in almost 40 percent of search queries in the NSDL. Hence the users of the NSDL engaged in advanced searches much more often than OH users (Chi-square = 1001.602, df = 1, p < .01).

Table 2 shows the use of advanced search options in two digital libraries in more detail. Both digital libraries allow the use of various search limits as part of advanced searching (e.g., limit search results to certain collections or by certain object types or audiences). These limits were used in 12.58 percent of search queries in the OH sample. More than a third (38.28 percent) of search queries in the NSDL sample included one or more of search limits. Faceted search queries are the advanced search queries in which the user is allowed to search for matches in specific fields of metadata records. OH users had an option to search by author or by title and subject words. Faceted searching was observed in a small proportion (0.61 percent) of the OH search queries in the sample but not at all in the NSDL sample, which is explained by the absence of faceted search options in this digital library. Use of another advanced search feature—quotes for bound phrases (as in the “‘climate change’ and water” NSDL query or in the “‘John Cobb’ Bonneville” OH query)—was also observed infrequently (1.32 percent of search queries in OH and 0.79 percent of search queries in NSDL).

Search query length is defined as the number of words in the query. For example, the query length of the “national parks” query is two, while the query length of the “John White’s narrative of the 1587 Virginia voyage” query is eight. Almost half of user search queries (47 percent) in the OH sample but only 29 percent of users’ search queries in the NSDL sample consisted of a single word. As shown in table 3, the search queries of the NSDL users varies more widely in length than those of OH users, which is demonstrated by higher variability—both variance (4.14 as compared to 2.58) and standard deviation (2.03 as compared to 1.61). For OH, the search query length ranged from 1 to 13 words per query, while the NSDL range was much more considerable: 1 to 53 words. Search queries in NSDL were found, with statistical significance (t = 8.09, df = 13963, p < .01), to NSDL to tend to contain more words than search queries in OH. The average NSDL search query length (2.66 words) was found to be shorter than the average OH search query length (2.36 words).

Search query frequency is the number of times certain identical queries are found in the transaction log dataset in a digital library. For example, a query “Wisconsin statehood” occurs only once over the period of twelve months in the OH sample, thus its query frequency equals 1; a query “planting” occurs six times in the same sample, thus its query frequency equals 6. Similarly, a query “Ohm’s law water” occurs four times over the period of twelve months in the NSDL sample, thus its query frequency equals 4; a query “anticoagulant properties of snake venom” occurs eleven times in the same sample, thus its query frequency equals 11. This study revealed, with statistical significance (t = 21.89, df = 13963, p < .01), that the search queries of the NSDL users occurred more frequently than the search queries of OH users. As shown in table 4, the average NSDL search query frequency (6.54) was found to be considerably higher than the average OH search query frequency (1.75).

Query frequency had a much higher variability in NSDL search queries than in OH search queries—both in variance (534 compared to 2.76) and in standard deviation (23.1 compared to 1.66). In OH, all of the search queries occurred between 1 and 25 times. Similarly, in NSDL, the vast majority of search queries (97.28 percent) occurred between 1 and 25 times. Although some NSDL queries (e.g., “chemistry,” “relativity and Einstein,” “photosynthesis” etc.) occurred as often as 100 to 885 times their proportion in the total dataset was minuscule, measured in tiny fractions of a percent. Therefore the detailed analysis of search query frequency, results of which are presented in figure 4, focused around search queries that constituted at least 0.01 percent of all search queries in either digital library, i.e., the queries that ranged in query frequency from 1 to 25 (percentage values that are below 0.01 percent are not shown on figure 4). Two-thirds (65.7 percent) of search queries in OH and almost a half (48.55 percent) of search queries in NSDL occurred only once in their respective search log samples. While the number of search queries with frequencies of 1, 11, 12, 13, 14, 16, 19, 20, and 25 were higher in the OH search log sample, search queries with frequencies of 2, 3, 4, 5, 6, 7, 8, 9, and 10 were more common in the NSDL search log sample.

As discussed above, in the categorization of user search queries, the following categories were used: work, person, family, corporate body, concept, object, event, place, ethnic group, and class of persons. As discussed above, polysemic and most phrase queries search queries were assigned to multiple categories. Qualitative results of search query categorization were quantified and are reported here as percentages of search queries in which a certain search category occurred.

The presence of particular search categories in user search queries displayed noticeable differences between the two domain-specific digital libraries. As shown in figure 5, the top two categories observed in the OH search queries were place (e.g., “Chile”) with 34 percent of searches, and object (e.g., “drinking vessel”) with 31 percent of searches. It is worth noting that both of these categories belong to FRBR Group 3 of entities, or subject entities. Another Group 3 search category—concept (e.g., “civil right”)—was the fourth most common search category with 17 percent of search queries. However, the fourth FRBR Group 3 subject entity—event (e.g., “1935 meat strike”)—was observed in the search queries much less often than the other three (10 percent). The FRBR Group 2 (agent) search categories person (e.g., “Alfred R. Glancy Jr.”) and corporate body (e.g. “Dana College,” “Kapa Alpha Psi”) were observed in 26 percent and 14 percent, respectively, of the search queries. The work search category (e.g., “Find It Illinois,” “how a colored woman aided john brown”) was observed in 9 percent of the search queries, while the class of persons (e.g., “fashion designers”) and ethnic group (e.g., “Cheyenne”) search categories were observed in 8 percent and 5 percent of user searches respectively. Finally, the family search category (e.g., “Wright brothers”) was observed in only 0.43 percent of unique search queries in OH.

In the NSDL search queries (figure 5), the distribution of search categories was different. Two search categories occurred significantly more often in the search queries of the NSDL users as compared to the queries of the OH users. This statistically significant difference was observed for the object search category (Chi-square = 292.7135, df = 1, p < .01), and the concept search category (Chi-square = 1944.959, df = 1, p < .01). Similar to the OH sample, object (e.g., “starfish student and teacher resource”) was one of the most frequently occurring search categories in the NSDL. However, this category occurred in the NSDL sample more often than in the OH sample (51 percent as compared to 31 percent of unique search queries). Unlike in OH search queries, concept (e.g., “epigenetics”) was the most frequently observed search category in the NSDL sample, with 64 percent of unique search queries containing concept. This is a considerably higher proportion of search queries than was found for OH (17 percent). Two search categories that were prevalent in OH search queries but occurred substantially less often in the NSDL search queries, with high statistical significance, included place (Chi-square = 2018.511, df = 1, p < .01) and person (Chi-square = 753.2203, df = 1, p < .01).⁵⁸ The Place category (e.g., “Chesapeake bay”) was found in only 5 percent of NSDL search queries as opposed to 34 percent of OH search queries; person search category (e.g., “Nikolai Lobachevsky”) was found in only 7 percent of NSDL search queries as opposed to 26 percent of OH search queries. It was also found, with statistical significance, that several other search categories had occurred less often in user search queries in the NSDL than in OH: corporate body (Chi-square = 567.5135, df = 1, p < .01), ethnic group (Chi-square = 277.1274, df = 1, p < .01), event (Chi-square = 276.574, df = 1, p < .01), and class of persons (Chi-square = 50.4434, df = 1, p < .01). The corporate body search category (e.g., “NASA”) was present in only 3 percent of NSDL search queries as opposed to 14 percent of OH search queries; ethnic group category (e.g., “mayans”) was present in only 0.7 percent of NSDL search queries as opposed to 5 percent of OH search queries; event category (e.g., “middle ages”) was present in only 3 percent of NSDL search queries as opposed to 10 percent of OH search queries; and class of persons category (e.g., “meteorologist”) was present in only 3 percent of NSDL search queries as opposed to 8 percent of OH search queries. The only search category where the difference in occurrence in NSDL and OH search queries was not statistically significant (Chi-square = 2.524082, df = 1, p < .2) was work search category. Finally, no family searches were observed in the user interactions with the NSDL, as compared to only 0.43 percent of OH search queries that contained the family search category.

As discussed above, in the process of analysis, polysemic search queries and most phrase search queries were assigned to multiple categories. The researchers found that although over half of all user search queries in the sample (59 percent in the NSDL and 55 percent in OH) belonged to single search category, a considerable proportion of search queries (45 percent in OH and 41 percent in the NSDL) were multi-category search queries. The number of categories in these multi-category search queries ranged from two to seven. Search queries of the OH users were found, with statistical significance (t = 8.71, df = 13479, p < .01), to contain more search categories than search queries of the NSDL users (see table 5).

As shown in figure 6, roughly a third of search queries (32 percent in the OH sample and 36 percent in the NSDL sample) combined two search categories. Representative examples include “Hisako eagle painting” (person and object), “mining Arizona Mohave county” (concept and place), “ADHD in teens” (concept and class of persons), and “Galapagos tortoise” (object and place).

The proportion of search queries with three and more search categories was low in both digital libraries. Ten percent of user search queries in the OH sample and 5 percent of queries in the NSDL sample included three search categories. Representative examples of three-category search queries include “California gold rush” (place, object, and event), “mechanics; Newton; acceleration; catapult; force; graphing; laws of motion; mass; motion” (concept, person, and object), and “Japanese culture United States” (ethnic group, concept, and place). Four-category search queries constituted 2 percent of all search queries in the OH transaction log sample, while only 0.4 percent of search queries in the NSDL sample comprised four search categories. Representative examples include “history of oil spills in the United States” (concept, event, object, and place) and “the mcgraw-hill dictionary of misspelled and easily confused words” (concept, object, corporate body, and work). A small fraction of search queries (0.45 percent in OH and 0.03 percent in the NSDL) included five search categories. For example, “fowler, h. w. 1914. fishes from the rupununi river, british guiana. proceedings of the academy of natural sciences of philadelphia v. 66:229–284” search query from NSDL included the categories: object, place, person, work, and corporate body. Finally, six category and seven category queries composed only 0.04 percent of the OH user search queries each, while no multicategory search queries with more than five search categories were observed in the NSDL search logs.