What is high-quality metadata? A standard definition of metadata quality is still developing. Metadata researchers do agree, however, that metadata quality involves its fitness for a specific purpose or use.¹⁴ In their article on improving metadata quality, Hillmann, Dushay, and Phipps of the National Science Digital Library, an aggregated database of science information, asserted that “the utility of metadata can best be evaluated in the context of services provided to end-users,” focusing on appropriateness.¹⁵ The authors proposed evaluation of metadata appropriateness at the element level, rather than record level, to best tailor information for users.

Further, Bruce and Hillman defined a framework in the form of a continuum of metadata quality for automated evaluation of metadata at an aggregator level.¹⁶ Their framework was developed in the context of large-scale aggregated collections, which require both domain-independent indicators, such as timeliness and provenance, and three-tiered quality levels for automated quality-assurance processes. Metadata quality indicators enumerated include completeness, accuracy, provenance, conformance to expectations, logical consistency coherence, timeliness, and accessibility. These indicators address both the correctness and appropriateness of metadata. The first-tier quality level gauges the entirety of a record, a portion of organizational expertise. This tier focuses on the presence or completeness of a record. The second tier includes additional indicators, such as use of controlled vocabularies and community-defined elements, which require organizational expertise and some subject expertise. This tier begins to address appropriateness and encompasses Hillman et al.’s recommendation to evaluate quality in each element.¹⁷ Finally, the third tier adds quality indicators such as application profiles requiring a full complement of organizational and subject-expertise skill sets.

Shreeves et al. evaluated the quality of harvested metadata at an aggregator level in order to determine how metadata quality at a local level affects searching federated collections.¹⁸ The framework used in their indicators that mapped to each indicator in Bruce and Hillman’s framework.¹⁹ These indicators study contained both intrinsic information quality (or correctness) indicators and relational information quality (or appropriateness) were used to evaluate records based on completeness, consistency, and ambiguity. They found that the quality of metadata varied between and among collections of metadata records.

With an aim to create a networked government information system, Moen, Stewart, and McClure completed a thorough analysis of metadata-quality literature and identified twenty-three distinct indicators to evaluate the implementation of a specific element set, the Government Information Locator Service (GILS).²⁰ The study focused primarily on compliance (or correctness) of metadata records created at forty-two federal agencies using GILS. The authors used the twenty-three indicators to rate compliance quality in four categories covering correctness and appropriateness: accuracy, completeness, profile, and serviceability. In both Moen, Stewart, and McClure’s and Shreeves et al.’s aggregator examples, metadata records could have been created by authors, contributors, or library and information science professionals.²¹

Greenberg et al. evaluated metadata quality at the point of metadata creation.²² Specifically, they examined the quality of eleven author-supplied metadata records using the National Institute of Environmental Health Sciences Dublin Core schema to determine baseline metadata quality. Their research argued that authors can create acceptable metadata records, as ranked by professional catalogers, using a Web template and text guidelines. Experienced professional catalogers determined the “high quality” standard for the author-generated records. They determined the acceptability of the author-supplied metadata and rated each of the records on the basis of intelligibility and correctness of the metadata.

Correctness and appropriateness are recurring themes in the metadata quality literature discussed. Specific aspects of both overarching themes appearing multiple times are completeness, accuracy, and intelligibility. These aspects apply to both the record level and element level quality. Thus metadata quality is determined by the compilation of elements used to fully describe a resource, the scheme, and the values within each element. In all but Greenberg et al., who explicitly evaluated resource creators, metadata records could have been created by authors, contributors, information professionals, or some other entity.²³ The possible effects of heterogeneity of metadata creators was not specifically identified as the cause of lower quality. Strict adherence to and availability of institutional metadata guidelines is a common recommendation to improve metadata quality.

The types of errors found in metadata quality studies are similar at both the aggregation and local levels. Within the e-print community, Barton, Currier, and Hey identified some common problems in metadata records and how they affect use and retrieval of digital resources.²⁴ The problem areas and elements include spelling and abbreviations, author and other contributors (e.g., editors and translators), title, subject, and date. These problems may also affect utility of metadata at an aggregated level; however, aggregators have another set of quality concerns. Dushay and Hillman also identified categories of errors found in metadata records at an aggregator level: missing data (e.g., empty elements), incorrect data (e.g., author data in language fields), confusing data (e.g., strings of names, or data culled from another source—a bibliography, for example—and not revised once in metadata record), and insufficient data.²⁵

Some of these errors are obviously human errors; others, such as incorrect data, could be attributed to the usage and implementation of local metadata-element sets. For example, Hanrath noted in his usability study of a Dublin Core template that metadata creators with subject expertise desired additional specific elements to accurately describe the resources.²⁶ Though he was not examining the quality of records created, the absence of a logical element for a piece of information deemed relevant resulted in metadata creators supplying incorrect data in another field, thus affecting Hanrath’s usability study. At both the aggregator and local level, the frequently suggested solution for many errors in metadata records is publicly available documentation developed and used at the local level that outlines use and content of metadata elements in a digital project.²⁷

RILM provides records for various types of scholarly works related to music. Initially, the author asked RILM for a selection of raw metadata records submitted to RILM from 1998, 2000, and 2004. The submission data supplied contained records for a majority of the record types represented in RILM, including articles in proceedings, monographs, and sound recordings. Previous metadata quality studies have evaluated between 11 and 140 records.³⁰ In order to have enough records to enable identification of patterns in the data, the target number of records for this project is 100. The author counted the occurrence of each record type (i.e., monographs, articles appearing in periodicals) to identify one type of record for evaluation. Submissions for the articles in a periodical record type appeared most frequently; therefore, record submissions for articles appearing in a periodical were evaluated in this study. RILM is an international database containing records in many languages. Because English is the only language with which the author is familiar enough to accurately assess semantic and structural errors in metadata records, the focus of the semantic abstract evaluation in this study is English-language submissions.

The raw submission data (the contributor-supplied metadata record) is the unit of evaluation in this study. High-quality metadata is appropriately fit for a purpose. A raw contributor-supplied metadata record’s purpose for RILM is to form a temporary record in the database and the basis for a final RILM record. Thus, the high-quality metadata standard in this paper is the final RILM-enhanced record from each contributor-supplied record that appears in the RILM database as displayed via OCLC’s FirstSearch interface.

Evaluating contributor-supplied metadata involved a three-step process. First, the final RILM record corresponding to each contributor-supplied record was located in the RILM database. If two records were found for the same raw submission record because the short record still remained in the database, the RILM record containing subject headings was selected for evaluation. Second, the RILM records were printed to facilitate comparative evaluation. Finally, the raw submission and final RILM-record metadata were compared element by element based on a key. Table 1 shows the metric used to evaluate each record through identification of different types of errors and value-added elements in each contributor record. For the purposes of this paper, an error occurred each time RILM made a change or addition to an element. Part A of the table contains the codes and usage notes used to mark each instance of an error type for all elements except for the “Abstract” field. Each error type in table 1 is mapped to a quality factor. Part B contains the codes and usage notes used to evaluate quality in the “Abstract” field. Analysis of the raw submission is compared to the final record based on three factors: syntax, structure, and semantic content.

In this paper, syntax refers to the overall completeness of the metadata record submitted. Each published RILM record consists of complete bibliographic information, an abstract, subject headings, and other relevant information (e.g., reference to related record in the database) if applicable. Most of these elements have corresponding fields in the submission form. Completeness is the inclusion of every required element for each type of record added to the database as defined in the RILM submitter guidelines. The appendix lists the fields required for articles appearing in a periodical. This factor is an indicator of organizational expertise.

Structure deals with the format of each element (format issues within a field). For example, values in the ISSN field should be four numbers followed by a hyphen followed by four numbers. In the submission form, text input boxes, dropdown menus, or radio buttons are provided for every field. Fields with text input boxes include “Author name,” “Periodical title,” “ISSN,” and “Page numbers.” These fields are structurally distinct from each other due to the different types of metadata that belong in each. Because the metadata may appear in different forms on the original source, variance in metadata submission may occur. Four types of structural errors were noted: non-authoritative forms, capitalization, punctuation, and other spelling and typographical errors. Structure is another indicator of organizational expertise.

Semantic content refers to the provision of content in each field. In other words, if content is supplied in a field, is that content correct, and is the content in the appropriate field? Five types of semantic errors were noted: RILM editing a portion of an element (e.g., changing “music of the church” to “church music”), RILM adding values for a non-required element (e.g., supplying ISSN), contributors leaving out required element values (e.g., contributors not supplying an author), contributors supplying an incorrect value (e.g., wrong or incomplete ISSN, wrong date), and correct data supplied in an inappropriate field (e.g., contributors putting volume and issue information in the “Volume” field). In other words, is the correct information in the appropriate field or misplaced in a different field? For example, each “Author name” field is paired with an “Author function” drop-down menu, which contains values such as preface author, editor, and translator in addition to author. Though RILM considers persons serving as translators to be “authors,” contributors may not have grasped RILM’s broad concept of authorship. On the submission form, separate fields are for information that may be concatenated on the original source (e.g., volume and issue number). Other information may simply be incorrect; that is, data for a particular element in the submission record (“Year” value is 1994) is completely different from that appearing in the final published record (“Year” value is 1999).

The primary field for elaborated content is the abstract field. RILM has outlined specific guidelines for writing abstracts.³¹ Essential content includes key concepts and names. Table 1, Part B: Abstract content analysis, is derived from the RILM abstracting guidelines document. Eight abstract elements are defined. In general, RILM requires complete names (persons or organizations, works, places) and titles. RILM prescribes the writing style, requesting concise, declarative abstracts. Because key concepts, themes, and names are appropriate and relevant to RILM’s target audience, this factor is an indicator of subject expertise.

The appearance and type of “value-added” or additional information in the submission is also noted. A “value-added” element in contributor-supplied metadata could be keywords, notice of related RILM records, or other details the contributor may deem important, such as a Web address for an online journal. Value-added elements are also those elements for which contributors supplied values, but are not present in the final RILM record. Examples include non-English language abstracts in addition to English-language abstracts, translated titles, or notice of English-language summaries within an article. The presence of value-added information is an additional indicator of subject expertise.

Syntax is the measure of completeness of the contributor-supplied metadata record. Table 5 details each element’s use frequency. Of the 104 contributor-supplied records, 100 contained values for all required elements. Required elements include Title/subtitle, Author name(s) and function(s), Periodical title, and Publication year. The 4 incomplete records were missing required additional author entries. These additional author entries were listed in RILM as translators and interviewees. Not only did a majority of records contain the basic required elements, but, taking all author name/function pairs as one, 16 of the 18 elements (88 percent) were used in more than half of the contributor metadata records evaluated (that is, occurred in 52 or more records).

The structure of each element is the format of the values supplied. Structural correctness for each record was determined based on the final RILM record. Table 4 shows the 4 types of structural errors noted in the evaluation. These 4 errors primarily assess spelling, punctuation, and other typographical errors, as well as AC formats for fields such as “Author name” and “Periodical title.” Structural errors occurred in 6 elements: “Author name,” Periodical title,” ISSN,” “Serial date,” “Issue no.,” and “Year of publication.”

Table 4 gives the number of each type of structural error. AC is the most common error in the contributor metadata with 106 instances occurring throughout the records evaluated. Of the 77 structural errors in the “Periodical title” field, 45 of them were AC. RILM added and removed subtitles and other words to and from the element for corrections. Fifty-three capitalization errors occurred in two fields, Title/subtitle (35) and Periodical title (18). Within Title/subtitle, capitalization errors occurred more frequently in the subtitle (25 of 35 times or 71 percent). Thirty-two punctuation errors were documented. Twenty-seven punctuation errors occurred in both the Title and Periodical title fields.

Semantic content of each metadata element refers to the correctness of values in a field. Table 4 also lists semantic errors. The most common semantic corrections performed by RILM were editing some portion of an element’s value (Edited) or supplying values for non-required elements (Enhanced). Edited and Enhanced errors account for 88.48 percent of the semantic errors in contributor-supplied metadata. Edited errors occurred in the widest spread of elements—9. “English abstract” field was edited in 43 of 50 supplied values. Some of these errors were semantic, such as contributors supplying translator names in the abstract. Other errors, which will be discussed later, are the result of abstract guideline infringement. The element with the next most-frequent Edited error rate is “Special features,” a controlled-value list. RILM removed terms from nine records, exchanged terms (e.g., “diagrams” to “tables,” “sound recordings” to “sound files,” “charts” to “illustrations”) in 12 records, and added new terms to 18 records. RILM supplied values that contributors left blank for 51 non-required elements across the records evaluated. Three-quarters of these values (39) were supplied in the “Translated title” field.

The remaining 3 error types were recorded 10 or fewer times. The required data missing errors (R) all surfaced in 5 author name/function pairs. RILM supplied name and function values for an interviewee, preface author, and translators. Because of the 5 missing author entries, a total of 10 R errors resulted. Incorrect errors were just that—contributor-supplied values were 100 percent wrong. In most cases the wrong year or ISSN was supplied. In 5 instances, correct values were supplied in incorrect fields (Usage––misplaced data). Four of the 5 misplaced values are related to the author elements. For example, the subject of one article was supplied as an author and translators of others were listed in abstracts. The fifth Usage––misplaced data error was unrelated to author elements—the theme for a journal issue was supplied in the “Periodical title” field.

Value-added elements in the evaluated records are those elements containing (1) correct information for a field but that may not be used in the final RILM record (Correct, unused) and (2) information in the “Additional information” element. RILM edited out Correct, unused values because they deemed the values unnecessary for an accurate description of the article. Several elements contained Correct, unused values, including “Translated title,” “Serial date,” “Special features,” “Languages of summaries published within an item,” and “Non-English abstract,” for a total of 19 instances. “English” was provided 6 times in the “Languages of summaries” element—the most Correct, unused values provided for each element. They appeared 2, 4, 4, and 3 in the remaining elements, respectively.

Contributors supplied additional information for 23 records. The type of information fell into two categories: user value (i.e., information providing additional context for the article such as keywords and other descriptive elements) and management (i.e., information helpful for the management of records in the database). Supplied content for the user-value elements include time period, theme of volume, ISBN of volume, and keywords. The management-content values included notice of related record in RILM, notification of errata in another volume, and URL of online journal. Some information appeared in some form as a part of the final records (e.g., some keyword concepts became subject headings), while others did not.