Presently, many vendors have or are developing tools to facilitate the harvest of non–MARC metadata into the online catalog. ILS vendors like Innovative Interfaces have moved to create systems to stream-line metadata harvesting directly into the online catalog. The Innovative Interfaces metadata solution known as XML Harvester, developed in cooperation with Michigan State University (MSU), is representative of most ILS vendor-supplied data harvesting tools because it provides one-way metadata conversion from a single data source into the online catalog. XML Harvester was used initially by MSU to generate MARC records in the online catalog from harvested EAD metadata, although today it can provide conversions from a number of different metadata formats.

XML Harvester’s functionality is representative of most ILS vendor-supplied metadata harvesting applications. Since this class of applications tends to run at the server level, control over how metadata crosswalking is defined will vary in granularity and generally be available only to IT staff or those at the system level. Likewise, this class of tools tends to be designed to be single project solutions, meaning that a significant amount of time is generally required for set up and testing to harvest a single collection, overhead that must be reallocated each time a new collection is set to be harvested. Because translations are tailored to specific projects or collections, work done for one project cannot be shared or used when looking to harvest other collections. This places practical limits on the types of projects that these tools can support. While the tight coupling with the ILS generally simplifies the process of loading and updating harvested metadata, it does come at a price. XML Harvester, for example, can only be used to harvest metadata into the online catalog and Encore Platform rather than as an abstract harvesting tool for providing metadata conversion services. This does tend to put very specific limits as to how useful this class of tools can be in general, particularly when considering the wide range of databases and services library technical service departments are being asked to maintain. The ability to harvest metadata and convert it into many different formats will likely become more important with time, possibly shortening the shelf life for this class of applications.

Some in the vendor community are beginning to provide better support for non–MARC metadata formats. For catalogers, the most interesting recent development for this is the inclusion of a metadata harvesting and crosswalking tool directly into OCLC’s Connexion product (www.oclc.org/connexion). Given OCLC’s influence and large number of member libraries, its software has the potential to simplify the metadata harvesting process for numerous libraries as well as lower the barriers to getting digital object metadata into the WorldCat database. As of version 2.10, the Connexion client provides OCLC members a set of basic metadata harvesting functionalities able to process records in a variety of DC flavors. The software is unique in the vendor community because it represents one of the first attempts by a vendor to shift responsibility for metadata harvesting and reuse from a library’s IT staff to its technical services staff. Nevertheless, the current implementation offers little practical functionality.

Figure 1 illustrates the current functionality provided to the user. Presently, users wanting to automatically generate MARC records from OAI DC records must download the record set locally before initiating this process. For large datasets, like UM Library’s Digital Books project, this workflow would be unfeasible because each OAI request returns only five hundred items. For example, using this method to generate the nearly one hundred thousand records made available through UM Library’s Digital Books project would require harvesting the dataset two thousand times.

One of the most unique aspects of the OCLC’s approach has been the decision, at least initially, to hide the metadata conversion process from the user. While this simplifies the overall metadata conversion process, it introduces a “fast food” approach to metadata conversion and is the process’s greatest weakness. Given the number of ways that DC elements can be interpreted and implemented between collections within a single organization, such a one-size-fits-all approach to metadata extraction and generation is not likely to be useful for meaningful record generation. Despite its limitations, the OCLC Connexion metadata extraction tool is still a significant step toward mainstreaming metadata crosswalking for technical services staff.

Overall, the vendor community has been making great strides toward simplifying the process of working with harvested metadata sets. However, at this point, their efforts still remain very project-based, making them marginally useful as general metadata conversion tools for the diverse datasets available to the library community. Each serves a need, but, as general metadata harvesting and conversion tools, their inability to allow catalogers to control metadata harvesting and customize the conversion rules is a serious impediment to adoption. For these reasons, OSU Libraries has used MarcEdit for its data harvesting and conversion needs.

MarcEdit is a freely available, metadata editing suite initially conceived in 1998 as a graphical user interface (GUI) replacement for the LC’s DOS–based MARCBreakr and MARCMakr software. Originally designed primarily as a batch MARC editing tool, the program expanded the functionality found in MARCBreakr and MARCMakr by including the MarcEditor, a notepad designed specifically for the modification of batch MARC records. Metadata needs and formats have changed significantly since 1998, and MarcEdit has changed with them. Today, the name MarcEdit is almost a misnomer because the application no longer is simply a batch MARC editing tool. Instead, MarcEdit is an application suite of metadata editing tools, including character set conversion, XML crosswalking, and metadata harvesting.

In many respects, MarcEdit has a number of things in common with OCLC’s Connexion application. They are both client-side applications, empowering users to work with data from many different sources. Likewise, the applications work with the OAI-PMH protocol and provide built-in data conversion rules for supported metadata formats. However, MarcEdit takes this one step further by providing users with the ability to customize the existing data conversion rules or create new data conversion rules. This allows users to harvest metadata from one of the supported metadata formats (DC, MODs, OAI MARC, or MARC 21 XML) as well as create conversion templates for additional metadata formats. It also allows users to customize existing conversion templates to reflect many variations in best practices used between projects. Users are given this customizability through XSLT. All of MarcEdit’s metadata conversion rules are defined as XSLT templates.

Appendix A presents the entire XSLT stylesheet used for converting OAI MARC records to MARC 21 XML. This is a good example because it underlines how readily available this type of crosswalking information already has become. This particular stylesheet was derived from an XSLT stylesheet provided by the LC and is one of many such examples currently available to the library community.¹¹ Why a conversion to MARC 21 XML? MarcEdit uses a “wheel-and-spoke” method, with MARC 21 XML sitting at the center of that wheel. This architecture allows metadata conversions to be created without the need to know directly how the individual metadata elements relate to elements within different schemas. Once a new spoke has been added to the wheel, it becomes crosswalkable to any other spoke on that wheel.

Figure 2 provides an illustration of this approach. Using this model, an EAD record could be translated to any other metadata schema on the wheel without the need to know how the elements in the EAD record relate to elements in the destination format. A user simply needs to modify or create a new XSLT template to modify the formats and behaviors of MarcEdit’s metadata conversion process. At one time, finding technical services staff with the ability to modify or create an XSLT document may have been an impediment, but the ubiquitous nature of XSLT has made this skill set much more common within the library community. At OSU Libraries, this functionality is what that makes MarcEdit’s method for metadata harvesting so valuable. Given the dearth of metadata currently available in DC, the ability to customize metadata conversion rules is essential to accommodating the variety of best practices and input standards. Staff members also have the option of either accepting the template-generated metadata or continuing to be active participants in the metadata creation process.

Like Connexion, MarcEdit simplifies the process of harvesting OAI-PMH–based metadata. Upon startup, users are greeted with the MarcEdit welcome screen (see figure 3), which includes links to commonly used functionality. Here one will find a link to MarcEdit’s OAI Data Harvester, which initiates the data harvesting service (see figure 4).

Once initialized, the user needs to provide the Metadata Harvester with only the host (URL) and set (collection name) for the set of records to be harvested. Users can optionally change the metadata type being requested from the server as well as define their own set of translation rules. Once set, the MarcEdit Metadata Harvester captures and translates the set’s metadata records from the defined metadata type to MARC. No interaction is required by the user. Users who wish to do more granular data harvests can select the advanced settings link to use some of the optional parameters supported within the OAI-PMH specification. The advanced settings function reveals a cache of additional options that can be set to define what records are to be harvested by the Metadata Harvester (see figure 5).

Using the advanced settings, users have the ability to define a subset of records (using Start and End), individual records (using GetRecord) or resume harvesting a predefined record set (using the ResumptionToken). Additionally, the harvester can translate record data from Unicode to MARC-8 as well as simply harvest and save the raw XML metadata files to a local file system. The character conversion options should be of special value for libraries that still use systems that cannot load or recognize MARC records encoded in UTF-8. Functionality has been added for users wanting to harvest XML–based metadata and create records using the legacy MARC-8 character set. Again, users need not set any of these options to harvest OAI-PMH metadata, but they are available for more granular data capture.

In January 2007, OSU joined a growing fraternity of universities whose students must submit electronic copies of their theses or dissertations in order to graduate. This policy shift by the graduate school was met with great excitement by OSU Libraries, which would take on the role of preserving and providing access for these materials through the library’s institutional repository (IR) portal, ScholarsArchive@OSU. Within the IR, these materials could find a larger audience both inside and outside the university, potentially extending the reach of the research being done by the university.

With these changes came a number of challenges for OSU Libraries’ technical services department. Like most institutions, OSU Libraries had traditionally created original MARC records for OSU theses, adding the MARC records to the local ILS as well as the WorldCat database. Cataloging for these records was done as materials were submitted to OSU Libraries by the graduate school; technical services staff usually received all of a term’s theses at one time. All record creation was performed using Connexion, meaning records were created once, dynamically becoming part of WorldCat, and then downloaded directly to the local library catalog.

The submissions of the theses and dissertations in electronic format, however, would be a much different process. First, unlike traditional print documents, electronic theses and dissertations would be submitted into the IR at any point during the term. Materials would first be vetted by the graduate school, then released to OSU Libraries, where they would be evaluated and then be made public. Technical services staff could no longer allocate fixed processing time for handling theses and dissertations because materials now would not be submitted on a fixed schedule. Secondly, metadata creation for these documents would shift from technical services staff to the document creators. When documents are submitted into the IR, submitters are required to provide metadata including abstracts and keywords. As a result, technical services staff began to use the metadata stored within the IR as the primary bibliographic data of record, meaning that metadata creation no longer took place in Connexion and was no longer being generated in MARC. This left technical services with two options: catalog materials twice (once in the IR and once in Connexion) or design a process that would allow metadata entered into the IR to be loaded directly into both WorldCat and the local catalog. Ultimately, the library choose the second option, not only to avoid the expense of rekeying records but also to support efforts to design processes that repurpose metadata rather than rekeying. As a side benefit, by using the metadata entered into the IR, the library was able to take advantage of an entirely new set of metadata elements: user contributed keywords and descriptions of the document. For materials like theses or dissertations, this information can be invaluable given the timeliness of topics, many of which are yet to be represented well within existing control vocabularies like the LC Subject Headings (LCSH).

The submission process for OSU Libraries’ ETD program mirrors those used by a number of other institutions using DSpace as their IR platform. Materials are submitted directly into the IR by their authors; in this case, it is the graduate or PhD candidate. Each submitter answers a number of questions through the submission process, entering information about their paper and topic. This metadata forms the foundation for the MARC records creation later in the process. The submitter then chooses a distribution license and uploads the document to the IR.

Once the item has been submitted to the IR, it is then vetted by the university’s graduate school. This process ensures that only approved materials are actually archived in the IR. Once approved, the item is then forwarded into the IR’s work queue, which is managed by technical services. At this point, a library staff member does original cataloging for the item in the IR. He or she then validates the user-submitted metadata and enters LCSH subject terms and any necessary descriptive notes for the document. When finished, the material is published to the IR and made available to the larger research community.

The question of whether MARC records are still needed was one with which OSU Libraries has struggled for some time. While having the records within both the local ILS as well as the WorldCat database was ideal, the reality was that staff simply did not have time to rekey data to create the MARC records. Moreover, given the increased accessibility of these documents through Web browsers like Google, questions arose regarding the need to continue producing MARC records. In the end, the library decided that having the data in WorldCat was important and set out to build a workflow that would keep staff from having to rekey the metadata from the IR into Connexion.

MarcEdit offered a solution to the rekeying issue. As a DSpace repository, the library’s IR could provide metadata for new and modified records entered into the IR through OAI-PMH. Unfortunately, the classified staff ultimately responsible for creating MARC records for IR items had no experience working with OAI-PMH or repurposing metadata from one schema to another. The ideal tool would automatically generate records from the metadata while essentially hiding this interaction from staff.

To meet this need, a special XSLT crosswalk derived from the default template was created to translate the DC metadata used for ETDs to their equivalent MARC fields.¹² This crosswalk varied from the vanilla DC-to-MARC 21 XML crosswalks provided by the LC because it used positioning within the metadata record to determine the context of some of the record’s metadata, since all metadata being harvested through OAI-PMH was unqualified DC. Using information about the generated metadata, an XSLT stylesheet was created to restore context to the harvested metadata. Once the context for these metadata elements was reestablished, the ability to create good MARC 21 records became much easier.

After being created, this custom XSLT transformation was registered with the MarcEdit application, allowing staff to use a simple, wizard-based application to harvest OAI-PMH records and convert them directly into MARC. This process has allowed staff to develop a workflow that allows them to immediately process items when submitted to the IR while doing MARC record generation for those items at the end of each week. The records are reviewed for accuracy and then uploaded directly to WorldCat and the local ILS. Appendix B provides an example of records currently generated with the OSU Libraries record-generation process. The example demonstrates how Unicode characters outside of the MARC-8 character set are embedded into the document as well as how subjects and headings are analyzed to create records that will require little or minimal manual intervention during the later steps of the process. This process has become more and more automatic as time has progressed, and the XSLT stylesheet has been refined to allow for minimal review prior to uploading metadata to the local catalog and WorldCat.

During the early testing phases of this process, several potential problems needed to be resolved. The first and most important issue was related to character encodings. All data loaded into the IR and harvested through OAI-PMH was encoded in UTF-8. While certainly desirable, OSU Libraries’ local ILS was not set up to recognize Unicode data in MARC records. Likewise, presently, the OCLC’s own importing tools will not allow for the import of Unicode data within MARC records. This meant that whatever harvesting tool OSU Libraries used to generate MARC records had to be able to facilitate some form of character set remapping between Unicode and MARC-8. Fortunately, MarcEdit’s OAI Harvester includes the ability to remap metadata from Unicode to MARC-8 on the fly, quickly solving this issue for the library.

The harvesting of granular metadata using unqualified DC remains an issue today. While much of the context lost because of the generic nature of unqualified DC can be reclaimed through careful analysis of the metadata, one ambiguity that cannot be easily resolved is the differentiation between staff-submitted LCSH subject terms and user-submitted keywords. While the XSLT stylesheet can be coded to make a very good educated guess as to the nature of these elements, the ambiguity persists enough that review is required following record generation. Fortunately, a solution to this issue has been created by the DSpace community. The recently released DSpace 1.5 simplifies the inclusion of additional supported metadata formats for harvest through OAI-PMH. This should allow OSU Libraries to modify the XSLT transformation so that it uses a more granular XML schema for harvesting, such as qualified DC or MODs and so that it retains the context associated with each harvested element producing production-ready MARC records from the harvest.