Literature Review

Several studies have addressed the uses of controlled vocabulary versus keywords in users’ catalog searches. A representative selection will be reviewed here to provide context for the current project. Carlyle conducted a study matching catalog users’ search terms with LCSH in which 47 percent of the search terms matched exactly.1 When including partial matches, word order variations, and spelling variations, the figure rose to 74 percent. Only 5 percent of users’ search terms could not be matched at all. The remaining 21 percent were matches that required two or more LCSH terms to cover the search term. In this study, users’ searches were done through subject search fields, not general keyword searches, which were not available at the time of the study. Carlyle concluded that a maximum 74 percent match rate was not an acceptable performance for LCSH and that further analysis of LCSH vis-à-vis user language was needed. The study is important because it defined levels of matching and called both for better matching against cross-references and for making LCSH semantically more flexible.

Frost investigated the utility of keywords taken from titles as “entry vocabulary” to subject searches by examining the degree of match between title keywords and controlled vocabulary.² Matches could be exact over the entire heading in direct order (11 percent of Frost’s sample), in any order (30 percent), exact main heading only (12 percent), exact in subdivision (5 percent), truncated variant in main heading (14 percent) or subdivision (1 percent), or no match at all (27 percent). Thus matches of some type occurred in 73 percent of the titles in her sample, leaving the remaining 27 percent with no matches at all. Frost concluded that keywords and subject headings are complementary.

Ansari replicated Frost’s study using medical dissertations written in Farsi.³ Her findings were very close to Frost’s; 70.3 percent of Ansari’s terms were matches of some type and 29.7 percent did not match at all, compared to Frost’s 73 percent and 27 percent, respectively. Ansari also concluded that keywords and descriptors are complementary and that keywords for which there is no matching descriptor should be considered for addition to indexing lists.

Voorbij conducted a study of title keywords and subject descriptors using somewhat different criteria for comparison.⁴ His focus was on comparing the descriptors to the keywords rather than comparing keywords to descriptors. His aim was to determine how well subject descriptors enhanced bibliographic records. The comparison defined matches in thesaural or semantic terms instead of using Frost’s more literal use of LCSH construction (i.e., main headings and subdivisions) and spelling. Voorbij categorized the results as exact match, synonym, broader term, narrower term, related term, some relation but difficult to determine, and no match. The first three categories, constituting 59.6 percent of the results (629 of 1055 descriptors), were not considered enhancements to the record. The remaining 426 descriptors (40.4 percent) were examined for the degree to which they enhanced the bibliographic record. Initially all 426 were considered as “possibly enhancing” the bibliographic record; this included 24.4 percent in the “no match” category. Further subjective examination determined that within the remaining 426 descriptors, 342 (33.0 percent of the sample) could be said to “slightly enhance” the bibliographic record, and 241 (23.2 percent) could be regarded as “considerably enhancing” it. Like Frost, Voorbij concluded that title keywords and descriptors are complementary, noting that descriptors help to reduce irrelevant hits and boost precision as well as to group synonymous terms.⁵ He further acknowledged that adding descriptors is an expensive activity that must be subjectively weighed against the value of precision and collocation.

Gross and Taylor examined transaction logs of users’ searches to see if controlled vocabulary provides additional keywords and consequently enhances both recall and precision in keyword searches of a catalog.⁶ Findings indicated an increase of up to 30 percent in the recall of relevant documents by the use of controlled vocabulary; about one-third of the keyword searches examined would have failed if the controlled terms had not been present. This percentage is similar to that of the “no match” category in Frost’s and Voorbij’s studies of title keywords and controlled vocabularies.⁷

Garrett studied the impact of adding subject headings to records in the Eighteenth Century Collections Online database of full-text documents.⁸ Preliminary results indicated that some 60 percent of searches would have failed if subject headings had not been present in the record. Terms, such as “sanitation,” that are common now were not used in the original documents and would not be retrieved without the cross-reference structure provided by controlled vocabularies.

Little has been written about author-assigned keywords. Two studies touch on them: one by Kipp and one by Gil-Leiva and Alonso-Arroyo.⁹ Kipp compared user tags with author-assigned keywords and indexer-assigned descriptors for 165 journal articles. Matching was done on a hierarchical scale (similar to Voorbij’s) of thesaural relationships, including same, synonym, broader term, narrower term, related but not in thesaurus, and not related. The focus of the study was on user tags and did not break out statistics specifically related to author-assigned keyword matches. In this study, 44.5 percent of all terms fell into the category of “related but with some ambiguity in the relationship … as well as relationships that were not formally in the thesaurus.”¹⁰ Kipp concluded that tags, as well as keywords and descriptors, can be valuable as additional access points.

Gil-Leiva and Alonso-Arroyo performed a matching study of author-assigned keywords and indexer-assigned descriptors for journal articles in four databases.¹¹ This study found an average of 24.59 percent for exact matches of keywords with descriptors and up to 45.66 percent when adding “normalized” matches (terms similar in meaning). By inference, some 54 percent of the keywords did not match, a far greater rate of nonmatch than that found in the studies related above. The authors concluded that keywords are valuable sources of information for indexers.

The debate between controlled vocabularies and keywords may be framed in terms of the issues involved with the formation (and subsequent maintenance) of new controlled terms for use by catalogers and the use of uncontrolled terms by users. As noted above, keywords may be used as guides for the creation of controlled terms, which could affect the maintenance of controlled vocabularies such as LCSH. LCSH is maintained on the principles of “literary warrant.” Historically, literary warrant for LCSH meant that terms were derived from the materials held by the Library of Congress and has since been expanded to include contributions by Subject Authority Cooperative (SACO) member libraries.¹² The standard for controlled vocabularies, ANSI/NISO Z39.19-2005, states that “the word or phrases chosen should match as closely as possible the prevailing usage in the domain’s literature.”¹³ Contrasting literary warrant is “user warrant,” which is defined by the ANSI/NISO standard as “generally reflected by the use of terms in requests for information on the concept or from searches on the term by users of an information storage and retrieval system.”¹⁴ The ANSI/NISO standard presents literary warrant and user warrant as complementary guiding principles for turning keywords into controlled terms on the basis of current literature as well as the use of terms by users who may or may not be familiar with the discipline in which they are seeking information.

Research Method

This study investigated the following questions:

• How well do author-assigned keywords match LCSH (either the established heading or a “see from” reference)?
• Conversely, how well do LCSH match keywords used by authors of electronic theses and dissertations (ETDs)?
• How many keywords are unique to their respective bibliographic records? Do these keywords add significantly more relevant terms that may increase the likelihood of their respective ETDs being found?
• Likewise, how many LCSH are unique within their respective bibliographic records; that is, how many LCSH are assigned for which there are no corresponding author-assigned keywords? Do LCSH add significantly more unique terms that may aid in the retrieval of the ETDs to which they are assigned?
• What are the implications for the way LCSH is used? What conclusions may be drawn regarding the construction or maintenance of LCSH?

Answers to these questions may corroborate the results of the studies related above and may further be used to draw conclusions regarding the use of both cataloger-assigned terms and author-assigned keywords for enhancing catalog searches.

The current project’s data set consisted of 285 eligible ETDs submitted by Ohio State University (OSU) doctoral candidates to the OhioLINK ETD Center and their associated bibliographic records in OSU’s online catalog. Eligible titles were those for which automatic e-mail notification of availability was received by catalogers in OSU Libraries’ Cataloging Department between June 1 and October 31, 2005, had author-assigned keywords, and had full text available at the time of cataloging. The cataloging of these titles was finished in 2006. Following interruptions due to a major building renovation, data collection and analysis were conducted in late 2007 through mid-2008.

The data were collected by visual inspection of the metadata page for each eligible ETD in the OhioLINK ETD Center and its bibliographic record in OSU Libraries’ online catalog, as well as the authority record for each LCSH as found through OCLC’s Connexion Client. These data included the author-assigned keywords in the ETDs, LCSH supplied in the bibliographic records, and “see from” references as indicated in the authority files. The data were recorded in Excel spreadsheets for collocation, counting, and comparisons. A total of 1,681 author-assigned keywords and 1,181 LCSH terms were collected.

To address the research questions presented earlier, the collected keywords and associated LCSH terms were assessed to answer the following working questions:

• How many keywords exactly matched LCSH, that is, could be placed in the 600, 610, 611, 650, or 651 MARC fields (fields for controlled vocabulary)?
• How many keywords were LCSH “see from” references?
• How many keywords could or could not be converted to LCSH, that is, could be placed only in a 653 field (field for uncontrolled terms)?
• How many LCSH terms had or did not have corresponding author-assigned keywords?
• How many keywords and LCSH terms could or could not be matched to corresponding words in titles and abstracts?

To categorize and codify the data, the categories of match in table 1 were used. Where more than one interpretation existed of how a keyword could be matched with a corresponding LCSH and vice versa, a rule was established to prefer the category of match in the order (top to bottom) shown in table 1.

Results and Discussion

The results of the comparisons of keywords and LCSH with each other and the matching of both in titles and abstracts yielded some patterns as well as several differences. As noted in the previous section, the total number of keywords was 1,681 and the total number of LCSH was 1,181. The average number of keywords per title was 5.9 (mode, 5), while the average number of LCSH per title was 4.1 (mode, 4). However, there was a stark contrast between the maximum number of keywords (57) that were assigned to a title and the maximum number of LCSH (13); see table 2.

Table 3 shows the raw counts of keyword and LCSH matches. The percentages of the six broad categories—exact match, all present (in a single heading), all present (needing two LCSH), partial match, variants, and no match—of keyword matches to LCSH are presented in table 4 and include the matches to cross-references, to more than one LCSH, and to abbreviations. Tables 3 and 4 summarize the data that address the issue of how well author-assigned keywords match LCSH and serve to answer the first three working questions, that is, how many keywords matched LCSH, how many keywords matched only cross-references, and how many keywords did not match LCSH. A total of 44.49 percent of the author-assigned keywords did not match cataloger-assigned LCSH (34.56 percent had no matches; 9.93 percent were variant forms); see table 4.

One explanation for the large percentage of terms not covered by cataloger-assigned LCSH is that LCSH has not kept up with current research. This issue of maintenance has been a recurring criticism of LCSH over the years.¹⁵ LCSH typically are established from evidence of a new topic found in the piece in hand, that is, from literary warrant. This is usually a monograph in hand, since articles and chapters are generally not cataloged.¹⁶ However, in some disciplines, such as the physical sciences and medicine, the journal literature is the primary publication environment for new research, and dissertations in those fields could be among the first comprehensive monographic treatments of a topic that has been otherwise extensively discussed.

Further, the distinction is becoming blurred as articles and chapters are added to bibliographic databases such as WorldCat. Although these resources are placed in research databases to aid discovery, they usually are not formally cataloged and thus are not considered as sources for new controlled terms. Yet they typically contain current terms of the disciplines in which they are written and which may or may not be familiar to users who need those resources. These terms are uncontrolled keywords that users may be likely to search on first. This use of terms for the purpose of searching is the essence of user warrant.¹⁷ As full-text access to articles and chapters becomes increasingly easier and ubiquitous, should these resources be considered as valid sources for controlled terms?

Another explanation for the unmatched keywords could be the use of different terminology for similar concepts, an issue not examined in this study. In other words, a match may not have occurred because of a lack of cross-reference in a related or semantically equivalent term, implying a different need for the maintenance of LCSH. This implication corroborates Carlyle’s conclusion about the need for the maintenance of cross-references to reflect changing user language.¹⁸

The large nonoverlap also could imply that some keywords may be spurious or not topical in nature. For example, one keyword that was used, “MD/PhD,” does not describe the topic of the document, but rather the type of degree program in which the author was enrolled.

Other keywords, such as “grounded theory,” may not have been matched because of the cataloger’s judgment of the relevance of the term to the topic of the given ETD. The cataloger may have considered such terms to be methodological and not topical. However, in some cases discussion related to such terms in the document was significant, and the terms in question could be seen to warrant inclusion in the bibliographic record.

The question of how well LCSH terms match keywords used by ETD authors was also addressed by the first three working questions as well as the specific working question of how many LCSH did or did not have corresponding author-assigned keywords. The data to address these questions are presented in table 5, which shows the broad categories of LCSH matches to keyword. The bottom half of the table shows the LCSH cross-reference matches to keyword.

As shown in table 5, 36.49 percent of the cataloger-assigned LCSH matched author-assigned keywords exactly and only 16.60 percent did not match any keywords while 31.08 percent were partial matches and 11.34 percent were variant forms of the keywords. The low total of variant matches and nonmatches could imply that keywords are used to guide the catalogers’ assignment of LCSH, consistent with the findings of Ansari, and Gil-Leiva and Alonso-Arroyo.¹⁹ Keywords, as assigned by the authors, could be seen to reflect the current use of terms in a field and can be used as points of entry for both users and catalogers. Where keywords can be translated into existing LCSH, the controlled vocabulary and cross-reference structure can then allow for meaningful sorting and organization (or “triage,” as Sclafani describes it) of search results.²⁰

In light of the professed advantages of cross-references, however, the effect of cross-references in this study was not as great as expected, although still noticeable. The total percentage of keyword matches in any form to LCSH cross-references was 9.93 percent (table 4), while the total percentage of matches of LCSH cross-references to corresponding keywords was 11.77 percent (table 5).

To answer the final working question (regarding uniqueness of terms within the bibliographic record), data were collected on the presence of the keywords and subject headings in their respective titles and abstracts. As with the keyword to LCSH matching procedure, exact and partial matches were counted as well as singular and plural differences and other variants that could affect user-search results. However, the LCSH matching procedure was varied for this portion of the study. In the previous parts of the study, base terms and subdivision strings were kept together, but for this part of the study the base terms and subdivisions were treated separately. This was done for two reasons. First, subdivided LCSH are not natural language phrases as keyword phrases were in this population of documents; exact matches over entire subdivided LCSH did not occur. Second, most of the assigned LCSH (712, or 60.29 percent) were not subdivided; that is, they were base terms only, and consequently the subdivisions were separated out to allow for the collocation of the data across all collected base terms. The percentages for the subdivisions are derived from the remaining 469 LCSH (39.71 percent) that contained them. Table 6 shows the percentages of matches that were found in titles or abstracts.

While conducting this study, the investigator learned that ETD authors were discouraged from using or relying on the titles of their works when selecting keywords. The degree to which this practice affected the results is unknown. The fact that 43.78 percent of the keywords had no match in the title and another 11.12 percent had only a variant match may reflect this instruction. Conversely, no correlation may exist. This possibility is consistent with the finding that 49.96 percent of assigned LCSH were not matched in the title and 12.36 percent were present as a variant. Further, titles are inherently limited in wording, and consequently contain a restricted number of words that could be repeated in keywords and LCSH assigned to the work.

A notable result occurred when keywords and LCSH were matched against abstracts, which are included in the bibliographic records for OSU ETDs. Author-assigned keywords exactly matched words in the abstract 54.61 percent of the time, while cataloger-assigned LCSH exactly matched only 26.84 percent of abstract words. Keyword nonmatches occurred 10.59 percent of the time, and cataloger-assigned LCSH nonmatches occurred 31.08 percent of the time. Put another way, only about one-tenth of the keywords and roughly one-third of the assigned LCSH are unique to the bibliographic records. This result corroborates Gross and Taylor’s findings in which more than one-third of the user searches that they examined would have failed if LCSH were not present in the records found.²¹ In terms of the discoverability of bibliographic records, the use of LCSH significantly complements keywords by providing further unique terms for searching and matching, even in the presence of enhancements such as abstracts.

The data gathered in this study suggest that authors performed rather effectively (when compared to assigned LCSH) in providing relevant keywords. A total of 65.44 percent of author-assigned keywords matched exactly, partially matched, or were variant forms of LCSH. Indeed, as noted above in relation to table 5, only 16.60 percent of the cataloger-assigned LCSH did not have corresponding author-assigned keywords. Authors, however, were not always concise about assigning keywords. One author assigned 57 keywords (the maximum noted in table 2), many of which are redundancies to capture variants. Table 7 shows a sample of these redundancies found in that record. One could consider this as an exemplar to demonstrate the value of controlled vocabulary.

Conclusion

In this study, LCSH demonstrated their potential to provide unique access points for approximately one-third of searches, even in the presence of bibliographic enhancements such as abstracts. Keywords provide a similar benefit, although not as strong, since they more often duplicate terms that appear in abstracts. Abstracts in the bibliographic records for ETDs are the norm for the OSU online catalog, but elsewhere this is likely not the case. Consequently, both LCSH and keywords provide significant numbers of unique terms that may increase the discoverability of ETDs in a catalog where abstracts are not present. Evidence of this can be seen by the number of nonmatches (i.e., unique terms) in the title-only comparisons of LCSH (49.96 percent) and keywords (43.78 percent). LCSH has the added benefit of collocating ETDs with like materials in other formats in the catalog.

The currency of research as found in dissertations represents a challenge to controlled vocabularies such as LCSH. Literary warrant, as it is currently practiced, makes it difficult for such systems to keep up with the pace of new research. Keywords may compensate for this lagging behind, which is inherent in the maintenance of controlled vocabularies, by serving as entry points into the catalog and as guides for the assignment of controlled terms that have already been established. This study corroborates the findings of much of the research on controlled vocabulary and uncontrolled keywords, showing that they are complementary tools for helping users find the materials that they need.