Quality of the overall catalog appears to be less frequently the subject of study, … notwithstanding the fact that both Cutter’s objects and much of FRBR’s approach are focused on the catalog as a whole rather than on individual records. Presumably, the limited extent of study at this level is due to the complexity and multi-faceted nature of the task, which now must include not only content and structure of the database, but also completeness and presentation of data on various screens, search engine execution, presence of context-sensitive help, and other elements in an environment in which users are familiar with many other Web-based information tools.1

Nevertheless, behind the variable screens of automated system interfaces, the data records that feed catalog indexes and displays are highly standardized. The widespread adoption of a core set of data standards by the U.S. academic cataloging community—the Machine-Readable Cataloging (MARC) 21 formats for mark up; the Anglo-American Cataloguing Rules, 2nd ed., rev., for description and name or title access; the Library of Congress’ (LC) Subject Cataloging Manual: Subject Headings rules for subject access; and the LC’s Name and Subject Authority Files (LCNAF and LCSAF, respectively) for authority-controlled headings—has enabled the sharing of catalog records through union databases, of which OCLC WorldCat is the prime exemplar, and the proliferation of library automated systems, all designed in their myriad ways to exploit the data contained in standard catalog records.² Studies of data quality rather than the qualities of automated system interfaces can reasonably claim to be focusing on a crucial and comparable aspect of overall catalog performance.

Past efforts to evaluate catalog data quality have generally relied on thorough review of individual catalog records. In their recent survey of the literature on quality in cataloging, Myall and Chambers found much disagreement over the definition of quality. The most common model they found for data quality analysis, reported in eight studies, calls for selecting a set of catalog records and examining the different areas of each record—fixed fields, standard numbers, title and statement of responsibility, edition and publication statements, notes, and access points—for errors, inconsistencies, and omissions.³ This kind of evaluation tends to be time and labor intensive, requiring a significant level of expertise and often the retrieval of items from the collection for comparison with the catalog records. It also raises comparability issues. Libraries’ standards for what constitutes an acceptable level of data quality and completeness vary across these several areas of description and access. For example, the extent to which libraries invest in the creation of table of contents or summary notes can vary greatly and are a matter of local policy; the trade-off between the added value of such notes and the added liability they represent as additional opportunities for error makes standard measurement difficult.

An alternative for the study of catalog data quality is to look not at a sample set of records, but at a sample set of searchable data. The “Dirty Database Test” takes this approach, offering libraries a set of typographical errors to search in the catalog.⁴ The number of errors thus found does provide a measure of data quality; however, this measure tends to lack both context and focus. The prescribed typographical error search looks for one or one set (using truncation) of erroneous variants, and ignores the number of times the term in all its variants is spelled correctly in the database. In the absence of these other counts, determining an error rate for comparison purposes with other catalogs is difficult. Typographical errors, some of which may be unlikely objects of a searcher’s query, can occur in any term in the record. Lastly, as noted earlier, libraries that include tables of contents and other notes in their records are likely to increase error counts by this measure without regard to the overall enhancement of access that such notes represent. Searching for and correcting typographical errors is an important part of maintaining access, as demonstrated by a 2007 study by Beall and Kafadar, but less effective as a comparative measure.⁵

The aim of the current study is to explore another alternative for evaluating catalog data quality. Rather than looking at whole records for a broad range of error types or searching for typographical errors, this study focuses on the consistency of selected authority controlled headings. Wolverton reports that a commitment to heading consistency as a goal is widespread among technical services departments.⁶ In describing their use of the whole-record review or “audit technique” at University of Bath, Chapman and Massey observe, “Authority control is a valuable form of quality assurance which the audit technique is weak in evaluating, compared to checking descriptive cataloguing.”⁷ The authors go on to note that their pilot study “was unable to confirm the feasibility of comparing headings to an authority file, which would inevitably increase the time required.”⁸

Departing from the whole-record audit approach to focus instead on currency and consistency of controlled headings has a number of advantages as a comparative quality measure. It highlights heading data, which is of high value for discovery. It is less prone to differences over catalogers’ judgment regarding how a particular resource should be described. It is able to sample a much wider range of records than the audit technique. Lastly, it offers a point of comparison that most if not all catalog managers would acknowledge is valid, given their widespread commitment to maintaining current and consistent headings.

Consistent headings are necessary to ensure that users can find all the items they seek under one heading. Many heading inconsistencies are obvious to the alert user in a browse search, where two similar headings for the same name may appear adjacent to each other in the index display; in other cases, where the entry term for a name or subject heading changes, the two headings may be widely separated in the browse index.

Inconsistencies can be less obvious, and therefore more of a problem, in other searching contexts. A keyword search may retrieve one of the split heading’s forms in its result set, and miss the other. A redirected search, prompted by clicking a highlighted access point in a catalog record’s display, may find only the records that match that access point exactly and exclude the variants. A “faceted browse” display of the type exemplified by North Carolina State University’s implementation of Endeca and Ex Libris’ Primo analyzes headings and other data present in a result set and presents them under such facet headings as “Authors” and “Subjects” in order of frequency in the result set. Such an analysis may find both current and obsolete heading forms in its result set but display only the more frequently used heading in its truncated list of facet terms. Because libraries’ newest items are the likeliest to carry the first instance of the revised form of a name or subject term, they are more likely to be low-posted and therefore less obvious in a faceted browse display, and possibly omitted altogether. Redirected searches are similarly prone to finding only records carrying the obsolete form of a heading and missing new resources with the current form of the heading if the heading split has not been corrected.

In addition to having a clear effect on the service the catalog provides, heading inconsistencies are relatively easy to recognize. Shared standards for heading forms are already in place. Standard forms of headings are widely distributed through the LCNAF and LCSAF, and widely relied on throughout the English-speaking library community. Determining the agreement between a library’s catalog headings and a standard heading form is also relatively straightforward and less demanding of expert judgment than the analysis of a full catalog record. Standardization makes gathering data on multiple libraries’ consistent use of a sample set of headings with a fair degree of efficiency and accuracy both possible and practicable.

One type of heading that offers an opportune focus for this kind of evaluation is a heading that has changed its authorized form. Authority files are dynamic, and the controlled terms for entities and concepts are always subject to revision for a variety of reasons. Two lists of changed headings are readily available. One is the long-standing “Library of Congress Subject Headings (LCSH) Weekly Lists,” formerly distributed in paper form, now posted online (www.loc.gov/aba/cataloging/subject/weeklylists). The weekly lists include new, deleted, and changed headings in the LCSAF. The changed headings are marked by the text “CANCEL” following the old 1XX heading form in the online list. The other list is more recent. On February 1, 2006, the LC posted a revised rule interpretation reversing its policy of discouraging changes to the authorized form of personal name headings simply to add a death date. As of that date, OCLC began compiling and posting online lists of established personal name headings to which a death data had been added in the LCNAF under the title “Closed Dates in Authority Records” (www.oclc.org/rss/feeds/authorityrecords/default.htm). Together, these two lists are a handy source for samples of authorized headings that have changed their forms.

The hypothesis for this exploratory study is that an examination of the results of searching sample sets of changed name and subject headings in a collection of catalogs will yield objective and comparable results indicative of the state of data quality control in those catalogs.

Heading consistency can be evaluated in online public catalogs in two distinct contexts. Within a given catalog, whether the library’s catalog records use the old or the revised form of a heading is arguably less important, provided the same form is used in all cases. The criteria of consistency and complete retrieval can be met in both cases. However, when a catalog’s access points are integrated with those of other libraries’ catalogs in a union catalog or federated searching environment, consistency within the local catalog may not be sufficient. The goal of consistent search results in union catalog contexts implies a shared commitment to using the latest form of an authorized heading, since that provides all contributing libraries with a common standard. For that reason, this study examines both these aspects of heading consistency: the rate at which heading “splits” (headings found in both old and new forms) occur in a single catalog; and the extent to which the new form is found to have replaced the old across a set of catalogs. By focusing on the data in the source records, interface variability can be ignored in favor of measuring adherence to commonly held goals and data standards.

Using the two identified sources, the LCSH “Weekly Lists” and the OCLC lists of “Closed Dates in Authority Records” for personal name headings, three sample sets of revised subject and name headings were compiled. The source lists are posted weekly, and the sample sets were drawn from three separate starting points within each series, both to broaden the range of heading changes gathered and to reveal any changes over time in the updating of headings in the catalogs under review. The name sample sets were collected from lists spaced approximately six months apart. The subject samples were collected from lists spaced approximately a year apart. In each case, the changed headings in the lists were reviewed from the chosen starting point forward in the list or lists until approximately fifty sample headings had been collected. For subjects, several lists were reviewed for each sample of fifty changed headings, since the number of changed subject headings in each LCSH weekly list is fairly small (approximately seven to nine). For names, each OCLC list examined exceeded the required number of fifty changed headings. With each new list sampled, the alphabetical end point of the previous sample set was used as the starting point for assembling the next set of fifty changed name headings.

Once the lists were assembled, a set of target public catalogs was selected. The home for this study is the University of Minnesota, which belongs to the Committee on Institutional Cooperation (CIC), a group of twelve large research universities.⁹ The CIC universities have a history of cooperation, including the maintenance of a federated search of CIC catalogs for a time, and are often used as peer institutions for comparison purposes. The thirteen catalogs of the CIC libraries (counting the library catalogs of the Chicago and Urbana–Champaign campuses of the University of Illinois as separate catalogs) were therefore selected to test the method being explored. Because this study is exploratory, the names of the catalogs studied have been randomly ordered and replaced by A, B, C, and so on, in the results. The LC’s public catalog was also included in the set of target catalogs, since it is the source for many of the changed headings. Because the LC’s results show a significantly greater use of the headings under review and therefore make its identity obvious, the LC’s results have been labeled.

All study data were gathered by searching the public catalogs of the target institutions. None of it depended on privileged access to information. The old and new forms of each of the sample headings were browse searched in each target public catalog, and the number of hits found for the old and new forms of the heading was recorded in a spreadsheet, the primary tool for data gathering and manipulation. Spreadsheet formulas were used to calculate the percentage of new headings found. In the spreadsheet, “0%” indicates that only instances of the old heading were found; “100%” indicates that only instances of the new heading were found; and any percentage in between indicates a split between the old and new forms. Where no use of either form was found, “NA” (not applicable) was substituted for a percentage. A sample of the spreadsheet appears in table 1. The project spreadsheets (without institution names) have been deposited in the University of Minnesota’s institutional repository, where they are available for external review.¹⁰

For the subject samples, separate counts were made of unsubdivided and subdivided forms of the old and new heading. For the name samples, separate counts were collected under the old and new form for the name as an author and as a subject. These refinements to the counting were made to assess whether heading consistency in the catalogs studied differs for authorized heading strings (the unsubdivided subject strings) versus unauthorized heading strings (most subdivided subjects) and for personal name headings in author indexes versus personal name headings in subject indexes. In expressing the results of the study for this paper, the counts of unsubdivided and subdivided subject heading forms have been merged to show a single count of old versus new main heading forms for the subject index as a whole. The results of the personal name searches are reported separately for name indexes and subject indexes.

Some deselection of the headings initially found in the sampling process proved advisable. Several types of headings were excluded:

1. Headings with old and new forms that would normalize and file identically, e.g., headings changed to remove a hyphen or to correct diacritics, capitalization, or tagging. These differences would be difficult to discern in index displays and, in any case, appear unlikely to affect access.
2. Main headings appearing multiple times with different subdivisions or with different phrase extensions (e.g., “… in art”). A few instances of this were retained in the study to explore whether the presence of an established heading-plus-subdivision string in the authority file accounted for a higher rate of correction. The minimal results from the few cases in the study sample suggest not, but are far from conclusive.
3. Headings with more than two forms, e.g., those that changed again following the “new” form’s establishment, or those that merged two earlier headings. The presence of multiple forms would require exceptional forms of counting. Given the relative rarity of instances like this, the few encountered were generally omitted. (An exception: the more heavily used older form “Hog cholera” was counted and the alternative “Swine fever” was not, though both were merged in the new heading “Classical swine fever.”)
4. Narrower subject headings merged into broader preexisting ones, e.g., the formerly established “Middle Ages—History” being merged into “Middle Ages.” Counting the number of changed headings under the preferred form would be impossible in most public catalogs.
5. Headings with an identical form in another heading system (e.g., the same term used by LCSH and MeSH) or MARC tag category (e.g., the same term used as a topical 650 and a genre/form 655 subject heading) also proved problematic. Some catalogs sort these differences into different indexes, but not all; if they appear in the same index, counting the instances of the changed bibliographic headings becomes difficult, requiring a record-by-record examination of MARC tag values.

The need for sensitivity to these kinds of problems makes the compiling of the sample sets a task that requires a cataloger’s expertise. However, once the list is compiled, the process of searching each form and counting hits can be learned quickly and requires few judgment calls. The method used here was further simplified by regarding only the established old and new heading forms on the list. Other forms were occasionally observed in the indexes (e.g., headings with typographical errors), which also caused split headings, but these were not counted. The only heading splits reported are between instances of the two forms on the sample list.

The data were gathered between January 2 and February 8, 2008, roughly one year after the date of the most recent heading change list sampled. When this task was complete, summary counts were made for each of the catalogs, showing for each set how many of the sample headings were found to be all old form, all new form, split between the old and new form, or unused in each catalog. The summary for the sample data in table 1 is illustrated in table 2.

The summary table data have been expressed as a series of doughnut graphs for quicker comprehension. Each doughnut graph shows proportionally the four states of the sample set headings found in each catalog: all instances in the old form, all instances in the new form, instances split between old and new forms, and no instances in the catalog. The outermost ring represents the sample set of the earliest changed headings, and the innermost ring represents the most recently changed sample (see figures 1–3). This arrangement of the data makes it easier to see patterns over time in the proportions of each institution’s sample sets.

Figure 1 represents the states of changed LC subject headings in the target catalogs. Some catalogs (LC, D, F, G, H, I, K) show most of their headings fully converted to the new form, while others (A, B, C, E) are largely unconverted or split for all three sample sets. Changed subjects also account for the largest proportions of heading splits overall when compared with the changed name headings. Catalogs J and L show the least use of the headings studied, while catalogs K and F and the LC’s catalog show the greatest use.

Figure 2 represents changed LCNAF personal name headings in author indexes. Changed names show a slightly higher rate of use across the CIC catalogs than did changed LCSH headings. Catalogs G, I, and K show large proportions of fully converted headings across all three samples, while catalogs C, E, and M are largely unconverted. The mixed results are seen for catalogs A, D, H, J, and L. Each shows one or two of the sample sets represented by each of the rings largely converted, but not all three. Such mixed results are more common for changed names than for subjects, suggesting that different approaches are being taken for these different kinds of maintenance.

Figure 3 represents changed LCNAF personal name headings in subject indexes. Not surprisingly, this set of results shows the lowest use rates, though even here use of less than a quarter of the sample headings is relatively uncommon. Catalogs G, I, and K again stand out as the most fully converted, while catalogs B, C, and E are largely unconverted and A, D, and H show mixed results.

This analysis does not take into account the number of hits found for each heading in each catalog. It looks only at whether the heading is present, whether all matches are on the old or the new form, and whether they are split between the two forms. The raw data, however, do include hit counts for each form of the heading and could support other kinds of analysis. For example, splits can only occur when a heading appears more than once in a catalog. Single appearances ranged from two to ten per catalog in the name samples, averaging five to six, and were always outnumbered by headings appearing multiple times. This indicates that a low count for splits in a particular catalog cannot simply be attributed to single appearance headings that could not be split.

An examination of the data gathered prompts three kinds of analysis. The first considers the extent to which the sampled headings were found to be in use in the target catalogs and whether the method chosen to assemble the sample sets of headings was effective. The second responds to the question that prompted this project—can a study of the state of changed headings in library catalogs provide a useful measure of data quality for those catalogs? The last takes up two more general questions—are the true states of headings in our catalogs sufficiently consistent with common models of data quality; and are there changes to the systems involved in revising headings that could simplify the task of heading maintenance?

The method tested in this study bears out the hypothesis that examining headings across library catalogs for currency and consistency can produce quantified, comparable results, and can serve as one useful measure of catalog data quality. A study of this kind can indicate how well a library’s catalog is performing in relation to locally established goals and to the catalogs of peer institutions, and it can indicate areas needing greater attention. The results also indicate that heading changes in the LCNAF and LCSAF are having negative as well as positive effects on catalog performance. Maintaining catalog headings is a constant challenge, and not one that is being universally or consistently met.

The proposed method for data gathering and analysis could be improved upon in several ways. Better methods might be devised for selecting sample heading sets to reduce the number of unposted or rarely posted headings. A more nuanced analytical approach might factor in the number of bibliographic records containing new versus old headings found in each catalog to produce a more balanced accounting of heading currency. Other sources of heading variations—e.g., typographical errors or unexplained variant forms—could be included to give a more complete measure of the occurrence of heading splits in the target catalogs.

More research into explanatory factors could also prove valuable. Are there common elements in the technical services operations or system implementations of those libraries that do well on this measure? Do apparent patterns in the heading currency of particular catalogs reflect changes in policies or procedures and their effect? Would closer attention to the types of changed headings that do and do not receive prompt maintenance attention suggest alternative ways of distributing this work?

Lastly, the fundamental question behind this study remains unanswered—should the library catalog’s data quality be evaluated as an outcome measure of the work of technical services? This study has attempted to demonstrate a practical method for such measurement, but it cannot answer the question of whether such measurement should be undertaken by a library or a group of libraries or included in models of library evaluation. Further discussion of that question would also be enlightening.