The VTDNP title selection process consisted of three main factors. The first was the formation of an Advisory Committee of scholars, historians, journalists, teachers, librarians, and archivists. NDNP grant application guidelines stipulate the articulation of a plan to involve an advisory board in the selection process.³⁸ The second was to refine the long and unwieldy alphabetical lists of approximately 500 titles published between 1836 and 1922 that the project planning group had identified and to confirm the availability of corresponding master microfilm negatives. Finally, once the titles on master negatives were ascertained and compiled into coherent title families arranged by county, the Advisory Committee would be engaged to review the list of titles and to rank them based on NDNP selection criteria.³⁹ Titles for digitization would be selected based on compiled ranking results.

A provisional Advisory Committee was formed in the planning stages of the grant proposal. Once the VTDNP was funded, Advisory Committee members⁴⁰ confirmed their commitment to the project and their engagement in the title selection process. Lorraine Lanius, on behalf of the Project Planning Group, led efforts to compile an Advisory Committee Briefing Book to provide general background information about the NDNP and the VTDNP to guide Advisory Committee members in the selection of newspapers to be digitized and to suggest other areas of engagement for committee members.⁴¹ Reference to a timeline of historical events in Vermont was also provided.⁴² An Advisory Committee “project” space was established on Basecamp in the fall of 2010 where the Project Management Group (previously the Planning Group) and Advisory Committee members could exchange correspondence and post documents.

During the planning phase of the VTDNP, an alphabetical list of more than 500 titles published between 1836 and 1922 available in microfilm format was generated from the VTNP database. The title list was subsequently sorted geographically by each of Vermont’s fourteen counties and further culled to identify significant titles with long runs in each of the counties. However, newspapers are serials and in the nineteenth and early twentieth century they often changed name, absorbed competing titles, and were themselves absorbed, or split into daily and weekly editions, often without warning or apparent reason. It soon became apparent that the alphabetical lists would be better classified as title families—composed of directly successive titles making up a newspaper “family.” By consulting geographical title lists developed by the Project Planning Group and bibliographic records in the VTNP catalog, with their corresponding preceding and succeeding titles (as identified in MARC record fields 780 and 785), the project librarian grouped logical title families together and, with further research, developed a unique list of fifty-seven title families, representing newspapers from Vermont’s fourteen counties.

The VTDNP endeavored to create a system whereby the advisors could rank titles in a manner consistent with the selection criteria outlined by the NDNP.⁴³ With an estimate of approximately 850,000 pages of newspapers on master microfilm negatives identified within the scope of the grant, and a high level of public interest in project outcomes, it was important to have a clear, transparent process in place for selection of the 100,000 pages to be digitized. A clearly defined, quantifiable selection process was essential to anticipating public concerns. To accomplish this, the VTDNP developed a title ranking form modeled on a form produced by the Pennsylvania NDNP project in their selection process and distributed the form to the advisory committee with their briefing handbook.⁴⁴

The ranking form provides information in two sections: the “Title” area provides title and place of publication information, dates of publication and dates of available master negatives, and an estimated page count for those negatives. The “Ranking Criteria” section enables advisers to rank each title based on a scale of 1 through 5 (5 being the highest), as applied to three NDNP criteria: Research Value, Geographic Coverage, and Temporal Coverage. This ranking system yields a score between 3 and 15. Advisers can augment this ranking by adding a point if the title is a complete run, an orphan title, or expands the diversity of selections. “Complete run,” means that the negatives available are a full collection of the title and do not have significant gaps. “Orphan title” means that the title has no direct descendent still in publication. “Diversity” means titles that may be of particular value to researchers because they are from either a significant minority community, or contain diverse or distinctive social or political perspectives. In Vermont, this includes abolitionist or antislavery newspapers, ethnic newspapers in Italian and French, and Democratic Party papers, in contrast to the dominant Whig and Republican titles in the mid- to late nineteeth century.

In April 2011, the University of Vermont hosted a day-long meeting where project planners and advisers met in person. Skype’s online conferencing capability was used to establish interactive connections with Advisory Committee members who could not be physically present. During this meeting, advisers familiarized themselves with project background and goals, as well as the selection process and ranking terminology, and came to a procedural understanding for ranking titles using the same rubric. The meeting also offered the occasion for lively group discussion of individual titles. The strength of the VTDNP Advisory Committee is that its members bring to the table a blend of diverse newspaper experiences and perspectives with strong regional and statewide knowledge of Vermont history. Meeting attendees came away with valuable insights into the newspaper publishing history in Vermont and appreciation for titles under consideration for digitization.

Due to the strict production timetable for deliverables defined by the grant cycle, the Project Management Group proposed three titles to the Advisory Committee to launch digitization: the Vermont Farmer, Rutland Herald, and Burlington Free Press. The Vermont Farmer was chosen as a start-up sample reel for its size and format on a compact reel and as a title that offers perspectives into agricultural practices and methods in northeast Vermont during the 1870s.⁴⁵ The publisher was well known as an innovative plant breeder and orchardist. The title is not solely an agricultural journal, as it also contains local news, obituaries, and advertisements, and is rightfully called a “newspaper.” The Burlington Free Press and Rutland Herald are well-known as the two longest-running and historically significant titles in the state. These titles were universally approved by the Advisory Committee.

With production underway for the three titles, Advisory Committee members selected and ranked additional titles over the course of the next month. Advisers with stronger regional knowledge tended to rank newspapers with which they were most familiar. Those with broad knowledge of historical Vermont newspapers tempered this regionalism by ranking titles that encompass a variety of geographical areas and times within the grant period. Overall, the Advisory Committee provided a well-balanced selection of newspapers. The project librarian subsequently compiled rankings. In June 2011, the Advisory Committee agreed on a list of twelve title families comprising thirty-seven unique bibliographic titles from ten of Vermont’s fourteen counties (see appendix).

The NDNP requires that state projects work with existing master negative microfilm collections as source material because they yield the highest quality digital images of newspapers. It is important to start with the best available microfilm negative to achieve NDNP standards, which are intended as “best practices” for digital preservation.⁴⁶ This requirement puts the charge on NDNP state projects to locate preservation microfilm negatives. Unlike public-use films, or service copies, of which there may be many, there is generally only one master negative, that is, the film that went through the camera at the time of filming. Subsequent generations of film of a given reel are derived from the negative. Availability and condition of master negatives for newspapers is often contingent on preservation archiving policies in a given state, and may vary.

Newspaper pages are to be scanned in 8-bit grayscale, as opposed to bi-tonal pure black or white, at 300–400 dpi.⁴⁷ This 300–400 dpi is relative to the original page, meaning that the scan, enlarged to the size of the original newspaper would contain 300–400 dots per inch. For each newspaper page, the NDNP requires delivery of an uncompressed TIFF 6.0, and a JPEG2000 and PDF derived from the original TIFF. Also required is “structural metadata to relate pages to title, date, and edition, to sequence pages within issue or section and to identify associated image and OCR files.”⁴⁸

Submitted XML files at the reel, issue, and page level fit a delineated hierarchical structure that enables users to access newspapers by title, date, issue, and page. Attention to detail in the gathering of this data leads to accurate results for those searching newspapers.

OCR files with word-bounding boxes and columns at the page level are required for NDNP work.⁴⁹ OCR files are XML-encoded representations of the text as interpreted by OCR software, such as ABBYY FineReader.⁵⁰ This software interprets the scanned TIFF images to generate files of text. OCR text is the basis of keyword searching. When a user enters a keyword into a Chronicling America search, the results are derived from the OCR files. OCR text is located, word by word, to areas on the image, so that a keyword can be highlighted on the page. The quality of image scans directly impacts the accuracy of OCR interpretation, and therefore has a direct effect on the usability of the resource. A common problem with OCR interpretation of poorly scanned pages is letter “blocking.” A newspaper page that has shortcomings in the scan may cause certain letters to “block.” This is similar to when too much ink is applied on a hand pressed page, and all of the white space inside the formation of letters is filled in. In poor scans, the characters on a page may be eye-readable, but the OCR interpreter may not be able to discern the difference between an “a,” “c,” “e,” or “o,” for example. A human reading a page that has a relatively poor scan of the words “Ethan Allen” may be able to easily tell the letters, but a computer may very easily read the words as “Ethon Allon,” or a similar variant. A researcher looking for Ethan Allen would not receive a “hit” for this occurrence of the search terms. This is a primary factor in why it is important to use master negatives when scanning newspapers: the closer one can get to the original, the higher fidelity one will achieve in scanned images.

LC recognizes that many non-NDNP digitization projects will not fully implement the NDNP specifications,⁵¹ but it is fundamental to image quality to use master negatives for scanning. Some newspaper digitization projects, in both commercial and public sectors, do not always use master negatives, instead using at-hand positive service copies as the source material. Scanning from public-use positive reels yields mixed results. Such film often has scratch marks, tape repairs, and other kinds of damage from public use that decreases the quality of scans made from those reels. Any damage to the film will naturally be visible in the scans.

While most libraries do an admirable job of maintaining their microfilm collections for public use, service microfilm is naturally subjected to conditions that are unfavorable for preservation. A large percentage of film still in use is acetate.⁵² Acetate is an early generation plastic used into the 1980s, when polyester-based film stock came into widespread use. Acetate microfilm is best stored in constant temperatures, ideally around 35 degrees Fahrenheit, with relative humidity below 30 percent.⁵³ Acetate film stored in ideal conditions from the onset can go decades with little or no degradation.⁵⁴ Polyester film is more resistant to unfavorable storage conditions, and its life is extended from proper storage as well. Properly stored polyester film has an estimated lifespan of more than 500 years.⁵⁵

Ideal storage conditions are plainly impossible to achieve for film in public use areas. Increases in either temperature or relative humidity can dramatically shorten the life of acetate microfilm by bringing on vinegar syndrome. Fluctuations in these storage factors can also accelerate vinegar syndrome.⁵⁶ Vinegar syndrome is so named for the strong vinegar odor that the chemical breakdown of acetate film base produces as it ages. This smell intensifies as vinegar syndrome progresses. Beyond the smell, the separation of the film layer base with the gelatin is where the real problem lies. As vinegar syndrome advances and this separation intensifies, it can make film reels unreadable.

For a relatively small number of reels, it would make sense to simply duplicate these reels on modern polyester stock to create a duplicate master negative, but the scope of microfilming preservation efforts in the mid- to late twentieth century is staggering. Millions of pages of newspapers were microfilmed from the 1950s to the 1990s in the US. The vast majority of these film masters are acetate, but duplicating such large collections on new polyester negatives is cost prohibitive. Today, there is little room in most library budgets for such an undertaking. Indeed, in the last ten to fifteen years many public microfilming operations have closed, and the master negatives dispersed, often ending up in the hands of commercial companies.⁵⁷

Most of this twentieth century film is commonly referred to as “prestandards” film. That is, the film was produced according to local practices, and not with an overarching set of standards in mind.⁵⁸ Without such standards, the organization and quality of this film can vary widely. Nevertheless, the vast majority of such films can be used to successfully create images that meet NDNP specifications. Much of this prestandards film has not been stored under ideal conditions and has varying degrees of vinegar syndrome. This aging resource is ripe for digital preservation.

The Vermont Digital Newspaper Project Planning Group initially projected the use of master negatives produced under the USNP Preservation Microfilming Guidelines for the Vermont Newspaper Project.⁵⁹ These master negative microfilms are on polyester stock and stored in archival conditions, so there was an expectation that high-quality, easy to use film would be readily available.

As the VTDNP prepared to move forward with title selection, it became clear that the longest runs for the most historically significant titles were filmed in Vermont before the USNP effort on “prestandards” microfilm. Through the VTDNP partnership with the Vermont Department of Libraries (VTLIB), notably through Paul Donovan, Law and Documents Librarian at VTLIB, an agreement was arranged between the Vermont State Archives and Records Administration (VSARA) and the VTDNP, to permit borrowing master negatives on an as-needed basis for the duration of the project.

VTLIB stores the master negatives generated by the Vermont Public Records Division microfilm program in the vault of the VSARA. Through this cooperative arrangement, Archivist Scott Reilly provided the VTDNP with a list, a 400-plus page document, which provides a reel-by-reel inventory of the master negatives held in the vault. Comparing this list of available masters with title lists generated during the VTDNP planning phase showed all of the titles that fit in the grant period of 1836–1922. Synthesizing the available reels with the title lists yielded an inventory of approximately 850,000 estimated pages of newspapers in significant runs. This abundance of film ensured that the VTDNP would have an ample supply of titles from which to select for the grant period.

In January 2011, the project director issued an RFP to solicit bids for NDNP compliant digitization services, including OCR creation, and provision of related metadata. The RFP stipulated that vendors convert a sample test reel according to NDNP specifications. A sample reel of the Vermont Farmer was duplicated and distributed to vendors who expressed interest in offering proposals. The VTDNP received four competitive proposals, with complete digital conversions. The digital conversions enabled us to evaluate the quality of each vendor’s work by comparing sample output for page images, XML and OCR files side by side. Vendor selection criteria outlined in the RFP was used to evaluate proposals. Evaluative areas included: experience with similar digitization projects; quality and completeness of the proposal and sample conversion images and metadata; organizational capacity and commitment to serve a long-term, high-volume, deadline-driven digitization project; availability of skilled and reliable project personnel to carry out work and communicate progress; and, of course, overall cost. After close evaluation of each bid, iArchives of Lindon, Utah was selected as VTDNP’s digitization vendor.

iArchives is an experienced news-paper digitization partner, evidenced by their work with several other NDNP state projects. A list of iArchives NDNP partners is included on their website (www.iarchives.com/demo-loc.shtml). Shortly after the digitization contract was signed, two representatives from iArchives came to Vermont for a series of meetings that established procedural workflows, production timetables, and scheduling for ongoing communication. Their familiarity with NDNP specifications enabled production to begin immediately. However, VTDNP personnel had to provide them with content to get underway.

NDNP metadata, that is, data about the newspaper pages, issues, and reels of film, must be generated from a thorough review of each reel of film. Since archival masters and printing masters should not be fed through a microfilm viewer, it is necessary to match a positive, public-use set to the masters for the purpose of review. “Thorough review” for the NDNP involves inspecting every frame on every reel to create an exact metadata record of what is on each reel. Due to the high volume of pages, or frames, to be scanned, thousands of pages per monthly batch of deliverables, this process is very labor-intensive and requires a great deal of lead time and planning.

The initial selection of three titles enabled the VTDNP to launch digitization workflows with iArchives with little delay. Master negatives were sent to a film duplication vendor, Archival Microfilming Services of Connecticut, to create duplicate negatives for digitization. Upon completion of the duplication, production began. The frame by frame review for the first 10,000 page batch was completed single-handedly by the project librarian and shipped to iArchives on May 17, 2011.

It became quickly apparent that frame by frame inventory of film for a production schedule of 8,000 to 10,000 pages per month conducted by one person, as it was for the first batch, would not be sustainable. The grant proposal’s staffing plan provided for a half-time, NEH-funded digital support specialist. While a search effort to fill that position was in progress, it had not been completed as production was starting. To bolster production start-up, the project director allocated three cataloging/metadata specialists (combined 0.3 FTE) from UVM’s Resource Description and Analysis Services unit to contribute to frame by frame inspection and metadata collation. This support was vital to launching production. The hiring of the digital support specialist (0.5 FTE) in July 2011 brought the project to a full staffing complement, well-positioned to meet production goals of approximately 10,000 pages per month over the remaining year of the grant.

Nineteenth century newspapers, with all of their oddities, mixed with the sometimes idiosyncratic nature of prestandards microfilm, make for many diverse and unusual reels of microfilm. The frame by frame inventory of these reels demands attention to detail and requires a learning curve to understand what information is important to record, and what is of lesser importance. The decisions made during the inventory can profoundly affect the end user experience. In this process, information about each reel is recorded, including reel dates, titles on a reel and their corresponding LCCN (Library of Congress Control Number, or field 010 in catalog records), issue dates, supplements, pages in each issue, duplicate images or issues, the number of targets, incorrect dates, other printing errors, and volume and issue numbers for each issue. Many irregularities exist, such as undated supplements filmed in sequence with dated issues, temporary title changes, issues of completely unrelated titles filmed with another title, partially filmed issues, or damaged issues. Each person completing the frame-by-frame film inventory must make case by case decisions about these frames on the film.

These inventory files are the basis of the XML files that are associated with the scanned newspaper images. To understand their function in file structure and information retrieval, it is helpful to think of XML files as pointing toward a given title, issue, or page in an issue. If mistakes are made in the frame by frame inventory, the XML files generated from that data will not point to the correct titles, issues, or pages. This frame by frame inventory has a critical role in the usefulness of the images created.

The VTDNP delivers raw metadata to iArchives along with a duplicate negative for each reel. iArchives scans the duplicate negative reels to generate TIFFs. From those TIFFs, they create a PDF and a JPEG2000 file. These PDF and JPEG2000 derivative files are compressed versions of the TIFF. Since they are compressed, they are much easier to deliver to online users. TIFFs are simply too large and resource-intensive to be delivered in a timely manner. The functionality of JPEG2000 is excellent. While they are compressed files, they are “lossless,” and the clarity they deliver to end users is adequate.⁶⁰ The PDF files give end users a choice of formats in which to view or save an image.

iArchives also generates the XML data associated with the image files using the raw metadata generated by the VTDNP production team. The Metadata Object Description schema, or MODS, is used as the NDNP standard for descriptive metadata. iArchives converts the raw VTDNP metadata into MODS, inside the Metadata Encoding and Transmission Standard, or METS, wrapper. The newspaper metadata files and the images are hierarchically indexed in this highest level file. Within each batch file, there are reel files. Within each reel file, there are issue files, and in each issue file, there are page files. For each of these levels, there are corresponding XML “pointer” files to the appropriate image or group of images.

iArchives also generates OCR files for each page. These OCR files correspond to zones on a page image, with a computer-generated set of words that match the page. The OCR files are the basis for keyword searches of the newspapers on Chronicling America. A keyword search of Chronicling America yields pages with the search term(s) highlighted. Terms are derived from the OCR files which match the terms to the page images and the location of the words on the pages. OCR output is continually improving with successive generations of software, but still has its limits. Nineteenth century, hand-pressed, small-type newspapers, sometimes with damage from a hundred-plus years of storage, filmed without clear standards on degrading acetate film, and then finally digitized, are often a challenge for human eyes, as well as for OCR. Still, OCR provides a powerful window into the vast newspaper collections on Chronicling America.

The project librarian ships reels, raw metadata, and a 1 TB drive in a protective case to iArchives. iArchives loads the images and data files on the drive in a single batch, and returns the drive to Vermont. The project librarian employs a quality-review process to ensure that NDNP specifications are met by checking for image quality, irregularities, and correct bibliographic data. Software tools used for quality assurance include the Oxygen XML editor, ThumbsPlus, and the NDNP’s Digital Validation View (DVV). Each of these tools is described below.

The Oxygen XML editor (www.oxygenxml.com/) is used to check the XML files. Oxygen displays the XML files in a color coded, line by line format. The color-coding serves as a visual marker for different elements within the XML files. When changes to the XML data are needed, this color-coding makes it significantly easier than with a black and white XML viewer.

ThumbsPlus imaging software (www.cerious.com/thumbnails.shtml) is used to examine the images. This software can process an entire batch and load all of the images on the batch into the viewer. Every TIFF and JPEG2000 image in the batch is examined as part of the VTDNP quality assurance. In the thumbs preview, one can easily scroll through issues and see each image. Typically, images in the preview are large enough to spot problem pages, which can then be enlarged in the software for full sized viewing and closer inspection.

LC provides the most essential tool, the NDNP or DVV,⁶¹ which is used to examine the OCR files and to check that the image headers are correct. The DVV orders the information in a manner that allows one to check OCR zoning and for correct bibliographic information. The critical function of the DVV is to validate and verify the digital signatures of the files in a batch. Validation checks to ensure that the batch data meets the structure of NDNP technical specifications, though not the specific information within that structure. Digital signatures are then written into the batch. The presence of these digital signature files provides a ready way to quickly see that a batch has been validated. Verification is a similar process that checks the data structures, but it does not write files to the batch. Whenever data are moved, either virtually or physically, verification is used to ensure that no damage or errors occurred in transfer.

Following validation, the batch is shipped to LC on the 1 TB hard drive. LC verifies the batch and performs their quality control processes before ingesting the batch into Chronicling America. The process is not always straightforward. There are sometimes problems with batches that require communication and coordination between the project librarian, the vendor, and LC so that adjustments can be made. Once the batch is ingested, LC notifies the VTDNP that its newspaper contents are available in Chronicling America.