What is ONIX? According to EDItEUR, the organization responsible for coordinating the development of the ONIX standard, “ONIX stands for ONline Information eXchange; it is an XML-based family of international standards intended to support computer-to-computer communication between parties involved in creating, distributing, licensing or otherwise making available intellectual property in published form, whether physical or digital.”¹⁰ The ONIX for Books standard is used by the book industry for sending and receiving bibliographic data in support of the book supply chain from the publisher to distributors to retail stores. Figure 1 is an example of an ONIX record provided by a publisher.

ONIX 1.0 was released in 2000 in the United States and the United Kingdom as a way to provide standardized product data in a consistent format, particularly to online retailers. ONIX 2.0 followed quickly in 2001, which provided increased capabilities for transmitting richer product data. ONIX 2.1, released in 2004, became the standard currently in use by many U.S. publishers. It has remained stable since its release. The ONIX 2.x versions are both backwardly compatible with ONIX 1.0, but that eventually caused problems and confusion as increased capabilities conflicted with the original standard, leading to the development and release of ONIX 3.0 in 2009. ONIX 3.0 is not backwardly compatible with previous versions and, while the bulk of the standard is based on ONIX 2.1, some new elements in version 3.0 are not in version 2.1. While version 3.0 has been available for about three years, its adoption has been very slow with most ONIX distributors or receivers still using version 2.1. Until the book industry and other users of ONIX have a need for more of the features in ONIX 3, U.S. publishers have indicated they will stay with 2.1.¹¹

As noted, the ONIX standard was first developed in the United States and the United Kingdom as a product of the Association of American Publishers and EDItEUR. Development expanded to include the Book Industry Study Group (BISG) in the United States and Book Industry Communication (BIC) in the United Kingdom. Since then, fifteen other countries began participation in the development of the standard. Now development is overseen by the ONIX International Steering Committee with EDItEUR responsible for coordinating all of the various country user groups. BISG, through its Book Industry Standards and Communication (BISAC) arm, assigned the BISAC Metadata Committee to be the user group in the United States responsible for participating in the development of the ONIX standard. In addition to the U.S. and U.K. user groups, additional groups in are in Australia, Belgium, China, Canada, Egypt, Finland, France, Germany, Italy, Japan, South Korea, the Netherlands, Norway, Russia, Spain, and Sweden. Additionally, ONIX work is being conducted in Bulgaria, Poland, and Turkey; although no official user groups are in place yet.¹² The particular focus of this paper is on the ONIX for Books standard. Other formats use ONIX, but they will not be addressed in this paper because the LC has only implemented ONIX for Books within its workflows.

Two articles by different authors published within Publishing Research Quarterly in 2002 and 2004 begin with the statement “It is a proven fact that the more information customers have about a book, the more likely they are to buy it.”¹³ The fact that both authors from the publishing industry could begin their analyses of the utility of the ONIX standard format with the same statement is an indication of a consensus on the benefit of sharing as much information about a book as broadly as possible. In the first article, Daly, executive director of the Book Industry Study Group at that time, stated that “ONIX was developed as a solution to two modern problems: a) the need for richer book data online; and b) the widely varying format requirements of the major book wholesalers and retailers.”¹⁴ While ONIX has been used from its release to describe elements of print books, very early in its development ONIX was deemed the “ideal standard to transmit metadata about e-books. In addition many of the retailers and wholesalers in the industry sell all forms of media. ONIX could provide a platform for the transmission of metadata for all types of information and entertainment products.”¹⁵

By 2004 Beky reported that, while small publishers still did not have the resources to convert their metadata to ONIX, the standard “has been adopted by all major U.S. publishing houses, which together produce approximately 50 percent of all trade book titles.”¹⁶ That number has continued to grow during the last eight years, and services to create ONIX records have been developed by companies such as Firebrand Technologies and NetRead to assist smaller publishers who do not have the capability to work in the ONIX environment. By 2010, the primary publishers’ trade journal, Publishers Weekly, was reporting on the importance of metadata to the publishing community. A leading consultant in digital publishing services reported “accurate metadata has become a marketing tool for publishers, a shopping guide for consumers, and an absolute necessity for distributors and retailers.”¹⁷ ONIX was seen as central to these marketing efforts. The statement of 2002 “that the more information customers have about a book, the more likely they are to buy it” had morphed into “‘accurate, rich metadata sells books’” by 2010.¹⁸ To handle the creation of this metadata for publishers that do not have adequate resources, a cottage industry of metadata producers has developed, including companies such as Firebrand Technologies, NetRead, and, most important to the bibliographic record supply chain in the library community, OCLC.

An understanding of the promise and value of ONIX exists not only within the publishing industry, but within the library community as well. As noted above, On the Record fully embraced the need to use publisher created metadata—and specifically, ONIX metadata—as broadly and as effectively as possible. But even before On the Record was published, the library community recognized the value in ONIX metadata. Within the library trade magazine Library Journal, Tennant noted in 2006 that

Publishers are increasingly supplying machine-readable metadata about the publications they put out—largely to enable their books to be sold to Amazon and other online booksellers. These records could provide much enriching information to our existing MARC data if the infrastructure were in place to normalize the records. Publishers often provide cover art, pull quotes from reviews, descriptive text, author biographies, and other useful material that MARC records typically lack… . How do I know this? I walk around with over 10,000 ONIX metadata records on my laptop that I downloaded from willing publishers. If we had a service to collect these records from publishers and make them available to catalogers, we could have access to many valuable facts about library materials.¹⁹

OCLC saw the value of ONIX as well. The availability of publisher data in ONIX provided an opportunity “to break down traditional silos between library and publisher supply chain metadata.”²⁰ OCLC created a Next Generation Cataloging pilot to improve the interoperability of publisher and library data, adding value to both by leveraging the strengths of each. In 2009, the National Information Standards Organization (NISO) and OCLC solicited a white paper titled Streamlining Book Metadata Workflow from Informed Strategies to provide an industry overview of producers and stakeholders of bibliographic metadata.²¹ The first idea proposed by Luther, the author of the white paper, was to “use crosswalks between ONIX and MARC to facilitate the creation of CIP and to provide publishers with an XML feed of MARC data.”²² OCLC has moved forward on creating a crosswalk between ONIX and MARC, which is the foundation of their new OCLC Metadata Services for Publishers.²³ This fee-based service accepts titles from publishers in ONIX format and enhances them for publisher use. OCLC describes five principal benefits of use of this service by the publishing community:

1. Reduces cost and duplication of effort in bibliographic description, categorization, and name authority work.
2. More titles found = more titles sold.
3. Provides richer marketing data to support buying decisions for wholesalers, retailers, libraries, and end users
4. Adds and enhances data to support marketing, sales-analysis, and business-intelligence needs for multiple markets.
5. Opens additional channels for exposure of title metadata—for use in library workflows and to end users on the web.²⁴

In the United States, OCLC and the LC are the principal institutions working with an ONIX-to-MARC conversion program, although other organizations can map ONIX to MARC. The LC, through its management of the CIP Program and its corresponding relationship with more than 5,100 major U.S. publishers and imprints, and OCLC, with its strong penetration in the worldwide bibliographic environment (including strong connections with the publishing community), have the greatest ability to maximize use of ONIX metadata for the library community worldwide. Use of ONIX by the library community is relatively new and as such needs to be studied in more depth. Stalberg and Cronin suggest “with several concrete ONIX-MARC projects underway, analysis can now be done to determine the extent to which ONIX data are valuable for cataloging workflows.”²⁵ The information provided in this paper is a first step toward providing information on the LC’s use of ONIX and its impact on cataloging workflows. As the LC moves its ONIX-to-MARC Converter program into full production later this year, more analysis will be done and shared with the broader library community.

The LC is the world’s largest library with more than 151 million items, including more than 34.5 million cataloged books and other print materials in its collections.²⁶ In FY11, LC staff cataloged more than 363,000 new titles.²⁷ Of that total, 105,000 new titles were cataloged by the two divisions principally responsible for the U.S. national imprint collection within the Acquisitions and Bibliographic Access Directorate —the U.S. General (USGEN) and U.S. and Publisher Liaison (USPL) Divisions. Nearly all titles cataloged by the two divisions represent new monographic publications received from either the LC Copyright Office or the CIP Program. More than 51,000—almost half—of the new titles were processed through the CIP Program.²⁸ The rest were received from the Copyright Office. The bibliographic records created by the staff in the two divisions are distributed to OCLC and through other means, making them readily available to researchers and the public, thereby saving libraries of all types the expense of duplicating this effort.

The CIP Program has been in existence since 1971. Its mission has remained the same: to provide cataloging data to libraries before publication, thus saving the libraries the cost of cataloging and supporting other library functions, such as acquisitions.²⁹ However, without the continued support of the U.S. publishing community, the program would have ceased long ago. The CIP metadata created by the LC represents the “accurate, rich metadata” mentioned by Reid as being essential for enhanced sales for publishers.³⁰ Currently, more than 5,100 publishers and imprints participate in the CIP Program. Their titles represent the cream of the crop of U.S. publications. Over 95 percent of titles received through the CIP Program are retained for the LC’s permanent collections. At the end of FY11, LC staff had cataloged more than 1.5 million books received through the CIP Program since 1971.³¹

Because the librarians in USGEN and USPL are working with new publications received either from the CIP Program or the Copyright Office, close to 80 percent of the cataloging done within the divisions is original work.³² The high percentage of original work is one of the reasons why the LC has been so interested in implementing the ONIX-to-MARC Converter program. The converter will (hopefully) reduce the amount of time spent on bibliographic description of new titles so staff can focus on subject analysis or work on special collections or more unique materials.

As early as 1996, four years before the creation of ONIX, Williamson and Davis-Brown of the LC had noted “experiments involving Standard Generalized Markup Language (SGML) have demonstrated that bibliographic records can be created directly from electronic texts with little operator intervention. If a text were marked up to the MARC subfield level, a program could scan the text automatically and extract all of the data.”³³ This was an early prediction of how a standard like ONIX cross-walked with MARC could benefit the library community by reducing the amount of manual creation and input of data elements when creating bibliographic records.

The LC was an early adopter of use of ONIX data, having received ONIX data since April 2002. As the use and production of ONIX data increased, so did the number of available sources of ONIX data. Today the LC receives data directly from publishers as well as from data aggregators. Depending on the capabilities of the data supplier, ONIX data files are received containing daily, weekly, or monthly updates, with some suppliers providing occasional “full file” data files of all of their items available, as well as “delta files,” which contain only those changes made since the last file. In an average week, the LC will receive approximately 200 data files representing ONIX records from thousands of imprints and tens of thousands of individual items in the book supply chain. Many of these are update records containing changes that are of little interest to the LC, such as price changes or availability information. However, many records are of interest to various projects at the LC.

Individual ONIX records can contain a wealth of information. Elements such as author and contributor information, titles, editions, imprints, publishing dates, extent, and series may be available and can be mapped to MARC fields. Additionally, information not regularly included in a MARC record, such as summaries, tables of contents, and BISAC subject codes, may be present and can be mapped to MARC fields. Many other fields may be present that are not used in MARC records but are used in the book supply chain. These include rights information, accompanying material information, related material information, author biographies, websites, awards, affiliations, sample texts, and much more that may be displayed on the website for an online retailer.

One of the first ONIX-related projects the LC embarked on was the ONIX-TOC (tables of contents) application to extract TOCs from ONIX records and automatically link the information in the 856 (electronic location and access) MARC field. The LC enhanced hundreds of thousands of bibliographic records through this mechanism. In FY11, 17,714 records were enhanced with TOC information received via the ONIX-to-MARC Converter program. Implementation costs have been very low. In 2006, Byrum and Williamson found that the cost of manually adding a typical TOC note was about $40 per record while automatic addition of the note via the ONIX-TOC process was $0.80 or less per record.³⁴ They further found that

The ONIX costs vary depending on the size of the data file received and how many new matches can be extracted from that file. The costs to set up the processing are about eight dollars… . Once the program is running unattended, the number of successful new TOC files created, determines the cost. If ten new TOC files are created that’s about $0.80, if one hundred are created, the cost drops to $0.08, and if one thousand or more are processed, the cost is less than one cent per TOC for accomplishing extraction and linking.³⁵

The success of the TOC project was an early indication that enhanced use of ONIX at the LC could potentially provide even greater financial benefits to the library while providing enriched data to the library community.

Following the success of the TOC Program, the LC implemented a Publisher Provided Summary Program. This Program allows publishers to voluntarily add summaries to their ECIP application and extracts summaries directly from the ONIX data. The ONIX summary information is linked in the 856 MARC field, while any summary from the ECIP application is input in the 520 (summary) MARC field in the bibliographic record. In FY11, 8,303 summaries were included within ECIP records—a 44 percent increase over FY10’s 5,783 summaries. A wide range of publishers provide summaries, including children’s publishers, university presses, religious publishers, and popular presses. The LC recently expanded the program to include juvenile fiction publishers. At the close of FY11, 32,504 summaries have been provided by publishers and added to the fully cataloged CIP bibliographic records.³⁶ Enriching bibliographic records through the inclusion of tables of contents and summaries has been found to assist the user by providing more terms for retrieval or relevant titles.³⁷

The latest project to use ONIX data at the LC is also part of the Electronic Cataloging in Publication (ECIP) Program. Because ONIX data contain several of the same bibliographic elements needed in a preliminary ECIP record, a project began in June 2009 to look into using ONIX as the basis for an ECIP record, determine the quality of the ONIX data and its conversion, and see if the conversion would help decrease processing time. The LC developed an application to process the ONIX files that are received to select any prepublication records in those files and create or update them in a database of prepublication ONIX records. This database resides on a server that is accessible to all catalogers who process ECIPs.

A second application also was needed to search the database and perform the ONIX to MARC conversion. Currently in the normal ECIP workflow, when a cataloger is ready to create an ECIP record, he or she clicks on a link in the web-based ECIP application form that starts an application created in the 1990s when the LC began processing CIP applications electronically. In this project, the link starts a different application that scans the ECIP form for the ISBN of the item. The application then searches the database of prepublication ONIX records, looking for a match. If no match is found, the new application calls up the old application and the cataloger processes the ECIP as before. If a match is found, however, the associated ONIX record is retrieved from the database. A skeletal MARC record template is hard-coded in the application, which goes through the ECIP form looking for needed elements, such as additional ISBNs, contact information, and place of publication (not in ONIX records), and then adds those to the skeletal template so that a MARC record begins to take shape within the application’s memory.

The application then goes through the ONIX record looking for the various fields that can be used, starting with any authors and contributors and creating the needed 100 (personal name main entry) or 700 (personal name added entry) MARC fields for those individuals. The converter only accepts personal names because only personal names appear in the ONIX files. The ONIX record contains information identifying the order in which any authors and contributors are listed as well as the function that person had in relation to the publication; this allows the application to determine who is the person to put in the 100 field (if needed) and who goes in a 700 field and in what order. Some one hundred or so functions, such as authors, editors, compilers, arrangers, adaptors, illustrators, actors, composers, etc., are defined in the ONIX standard. The application takes each name and prepares it for inclusion in the statement of responsibility (SOR). For authors, the names are simply held in direct order for inclusion in the SOR. For other contributors, the name of the function as provided in the ONIX standard list for contributor functions is added before the name in square brackets because the application needs to inform the cataloger that the person is not an author but it does not know the exact SOR wording in the galley mock-up. For example, a name identified as an editor would be processed to show “[edited by] John Smith” in the SOR.

When the application searches the ONIX record for the title, it extracts the title, any subtitle, and, remembering the authors and contributors and their function, it creates an SOR and puts them all together to create the 245 (title statement) field. Additionally, because the catalogers discovered that the ONIX data provided did not always match what came on the galley mock-up provided by the publisher, the applications displays the title page and copyright page from the galley mock-up as well as the proposed 245 field. That way the cataloger can compare them and note if any adjustments to the title or SOR are needed. If the cataloger notices a serious problem, such as the title on the ECIP form not matching the title from the ONIX data, the cataloger can opt to use the old application to process the ECIP. If it looks like the correct title (with minor variations), the cataloger notes any differences that need to be looked at after the ONIX conversion has been completed. The cataloger then clicks a button and the application then creates the 245 field data to be added to the record being constructed and then extracts information for the 250 (edition statement) field, if available. Again, a display provides the edition information found for the cataloger to compare against the title page and copyright page from the galley mock-up, and the cataloger notes any differences.

Another click on the same button and 250 field data are created, the information for the 260 field is extracted, and another display comes up for comparison. This time, however, when the cataloger clicks the button, the application asks the cataloger to supply the place of publication for $a of the MARC 260 (publication, distribution, etc.) field and the equivalent place code in the 008 (fixed-length data elements) field because the place is not provided in the ONIX data. Finally, information for any series is extracted, if available, and the final display comes up for comparison. When the cataloger clicks the button this time, the application goes through the ONIX record and extracts the extras that will be added to the record, such as a link to a cover image, a summary, or a TOC; those are converted into their appropriate MARC fields and added to the record to be constructed.

Regarding the TOC, the information extracted from the ONIX record is not manipulated to try to make it conform to Anglo-American Cataloguing Rules, 2nd ed. (AACR2) specifications.³⁸ Instead, the data are provided more or less as given in the ONIX record. If elements are separated one per line in the ONIX record, then the application will insert two hyphens between the elements, but if the elements are strung together in one long data string, it is given as found in the 505 field. In this case, the publisher usually separates the elements with punctuation and the presence of “Chapter” or “Part” to distinguish each separate element in the TOC. The first indicator value “8” is given to indicate that no display constant will be generated by an online public access catalog or other MARC display system, and the data are then given.

For summaries, the wording is presented exactly as found in the ONIX record. The entire summary is quoted and at the end of the field “—Provided by publisher” is given to indicate the source of the quoted summary. The only manipulation the application will perform is to convert certain symbols or punctuation marks from an HTML equivalent or a Unicode value to something that can be reproduced on a keyboard, e.g., the symbol for “less than” can be represented in HTML as “<” and the application simply converts it to “<.”

The final task for the application to do is to ask the cataloger to input his or her cataloging code to put into an internal work-tracking field. The application then puts all other data elements in their correct MARC fields and writes the MARC record to a work file, calls up the cataloging client, and sends keystroke commands to the client window causing the client to import the MARC record from the workfile. The cataloger then is presented with the MARC record in the client where he or she can review it, making any of the changes noted during the conversion process.

Use of the ONIX-to-MARC Converter program to create ECIP bibliographic records has grown exponentially in the past two and a half years. In FY09, 532 ECIPs were added through use of the ONIX-to-MARC Converter program. In FY10, 2,810 records were added through the program, and in FY118, 499 records were added—an increase of 202 percent from the previous year.³⁹ The LC is on the cusp of putting its ONIX-to-MARC Converter into production later this year once a systems upgrade is completed. At that point, all of the LC catalogers responsible for processing ECIP galleys will be fully trained on use of the ONIX-to-MARC Converter.

The ONIX-to-MARC test that evaluated use of the ONIX-to-MARC Converter process described in the above section began in June 2009 and ended in August 2009. To fully understand the results of the test, one needs to know about the existing system used to create ECIP prepublication metadata.

The ONIX Pilot test was a test of both the utility of the ONIX records as a source for accurate bibliographic description and a comparison of the TCEC process to the new ONIX-to-MARC Converter process to determine which was most effective to use. The pilot began in June 2009 with two CIP program specialists who performed the descriptive cataloging and tested ONIX data from two publishers, Cambridge University Press and Wiley. The pilot evaluated several aspects:

• The availability of ONIX data for items in the CIP stream
• The usefulness of the data in cataloging
• Any problems or unexpected results from converting the data from ONIX to MARC
• Time comparison between TCEC and the ONIX-to-MARC Converter process

As the project progressed and to evaluate more data, the ONIX/MARC Conversion Committee added the imprints of Harper Collins, Palgrave Macmillan, and Oxford University Press to the pilot. In July 2009, the pilot team was joined by the National Library of Medicine (NLM). The findings of the test were shared with Acquisitions and Bibliographic Access (ABA) directorate director Beacher Wiggins in August 2009. Based on the positive results described below, Wiggins decided to continue the pilot.⁴¹ The pilot project team also was expanded to include two more LC testers in December 2009.

In October 2010 the pilot was again expanded. Each monographic section that processed ECIP applications within the ABA Directorate identified at least one staff member to work with the ONIX-to-MARC Converter. This increased the number of testers by twelve and the number of publishers was expanded to include all that might provide data in ONIX format (more than 5,100 imprints).

The LC ONIX-to-MARC Converter has been implemented successfully at the LC. The converter has allowed LC cataloging staff to directly use publisher-supplied metadata for the creation of most descriptive data elements required for a monographic record. The use of the ONIX-to-MARC Converter has benefited the LC by reducing the amount of time spent on creating records for titles received through the CIP Program while providing more enriched records for the greater library community. When the ONIX-to-MARC Converter is put into full production later in 2012, the LC anticipates that more than half of the records created for publishers through the CIP Program will have as their foundation the original ONIX record received from the publisher. At that point, even greater benefits will be realized by the LC and the library community.

However, as Stalberg and Cronin remark, more research needs to be done to determine how use of ONIX will affect the library supply chain.⁴³ The LC and OCLC, which have done the most to implement use of ONIX in their workflows, are unique because they have direct links to publishers that other libraries rarely have. The LC has a need to ingest and manipulate ONIX data as the data are directly tied to the galleys received as part of the CIP Program. OCLC has moved into the ONIX marketplace by providing a fee-based service to publishers who want their ONIX data enhanced with additional data elements, such as LC Subject Headings.

One question that will need to be answered is whether other libraries can use ONIX as effectively as the LC and OCLC. Will use of ONIX by the library community be largely confined to the LC and OCLC with work done largely behind the scenes from which the larger community benefits? Because ONIX files with information on postpublication items are available for use by the library community, can other libraries use these files as a reliable source of copy? Perhaps the postpublication data can be used to assist in removing backlogs of materials in specific areas.

How can ONIX data created by foreign publishers be used by the U.S. library community? Can the LC partner with other institutions to take advantage of ONIX metadata from foreign publishers? The Library and Archives Canada (LAC) has concurrently implemented a metadata pilot titled “Repurposing User-Generated Metadata Pathfinder” (RUGM Pathfinder) to utilize ONIX data in the cataloging of materials published by Canadian publishers and thereby reduce redundancies in cataloging.⁴⁴ EDItEUR has developed the “Linked Heritage” project, which aims “to extend and enrich the content and metadata holdings of the Europeana digital library.”⁴⁵ Some cross-sector collaboration between the LC and these institutions could be beneficial. The possibility that postpublication ONIX data for foreign publishers might be used as a source of copy for libraries exists. With all of the talk of the promise of ONIX, seeing if other libraries can implement it in their workflows will be interesting. This area is ripe for research, particularly in the large academic and research library community.

Also in question is whether greater use of ONIX by the LC, OCLC, and perhaps other libraries or bibliographic utilities in the future will, in fact, eliminate redundancies and waste as hoped for by the authors of On the Record.⁴⁶ Even with the enriched data provided by ONIX, will libraries continue to manipulate and tweak the records, thereby accruing costs to their institutions?

Many questions need to be explored in the coming years as libraries begin using and relying more on ONIX metadata supplied by publishers. This is a first foray into how ONIX can be used in a practical setting. As the LC continues to work with ONIX in the years to come, it will share its findings with the library community.