Sellitto provided an extensive review of the literature on Web site persistence of cited Web resources.³ He selected 123 papers from the 1995 to 2003 AusWeb conference series, examined the 2,168 references cited, and found Web resources in almost 50 percent of them. Of the Web resources cited, 45.8 percent were not found at the URL cited. Sellitto determined the average half-life for these Web citations to be 4.8 years. In a study of papers in dermatological journals published between 1999 and 2004, Wren and colleagues found that 81.7 percent of URLs cited in papers published in 2004 were available.⁴ The percentage decreased with time to an availability rate of 65.4 percent for URLs cited in papers published in 1999.

Dellavalle and colleagues examined Internet references from the papers of three high-impact scientific journals (NEJM, JAMA, and Science) published over a six-week period in three different years (2001–2003).⁵ They found that the number of inactive Internet references increased over the twenty-seven-month period, reaching a high of 21 percent for JAMA, to a low of 11 percent for Science. Spinellis examined the Web citations in two computer science publications from 1995 to 1999.⁶ Almost 50 percent of the 4,224 references he checked were not available after four years. Dimitrova and Bugeja examined Web references from a sample of papers published in communication journals between 2000 and 2003, and found 37 percent of the URLs no longer led to the content cited.⁷ In April 2004, Crichlow, Aguillo, and Prieto examined the Internet references in the articles of five major medical journals that were published in January 2004, and, by cutting and pasting the URLs into a Web browser, found that five were inaccurate (7.4 percent) and three had inaccessible sites (4.4 percent).⁸

Markwell and Brooks studied biochemistry and molecular biology Web-based educational material and found that, in a twenty-four-month period starting in August 2000, more than 20 percent of the 515 sites examined had changed content, moved with no automatic forwarding, or were broken links.⁹ Ortega and colleagues compared Web content in 738 Web sites in 1997 and again in 2004. As part of this study, they examined the persistence and stability of the links within these pages; they found that 74.28 percent of the 145,092 links were “broken (linkrot) or not operative.”¹⁰ Bugeja and Dimitrova looked at Internet references cited in Association for Education in Journalism and Mass Communication online conference papers.¹¹ They found that after less than a year, 55 (51 percent) of the 108 citations led to the Web content cited when they clicked on the link. However, when they cut and pasted the URLs into a Web browser, the number of found citations increased to 65 (60 percent).

Bar-Ilan and Peritz conducted a broad-ranging study of Web pages concerning the subject “informetrics.”¹² They looked at the number of Web sites retrieved on this subject in 1998, 1999, 2002, and 2003 and identified 866 URLs in 1998 that met their search criteria. Of those, 299 (34.5 percent) were not available in 1999, with 496 (57.3 percent) and 552 (63.7 percent) not available in 2002 and 2003, respectively. Of the 1,297 URLs identified in 1999, 643 (49.6 percent) were not available in 2002, and 769 (59.3 percent) not available in 2003. Of the 3,746 URLs identified in 2002, 682 (18.2 percent) were not available in 2003. Koehler continued his study of Web page persistence begun with 361 URLs identified in 1996 and found that by 2003, two-thirds of the original sample of URLs were gone.¹³

Kushkowski analyzed the citations in economic theses and dissertations in print and electronic format to see if different patterns emerged in the citations to Web resources.¹⁴ He selected master’s theses and doctoral dissertations from Virginia Tech, where electronic theses and dissertations were required, and Iowa State University, where electronic copies were not accepted, and found that a small percentage of citations in these works were to Web resources (Virginia Tech had 5.4 percent, and Iowa State had 2.2 percent), with Web citations increasing between 1997 and 2002. Kushkowski also looked at the persistence of Web citations and found that approximately 55 percent of Web citations from both universities’ theses and dissertations led directly to the documents cited.

Wu briefly discussed the importance of persistence in Web documents in legal research, noting, particularly, the loss of interim documents, which are of great importance to both legal research and analysis.¹⁵ In 1995, librarians at the National Library of Australia identified fifty publications on the Web that were considered sufficiently important for preservation.¹⁶ They found that 22 percent of these publications were still available in 2004 at their original URLs, and an additional 64 percent were still accessible on the Internet, although the content or style of some had changed. They predicted that in about five years, users will be able to find only 50 percent of the original publications.

URL persistence decreased by 17.4 percent during the period between the original and follow-up studies. These data are presented in table 1. In the original study, 282, or 56.4 percent, of the sample URLs, were persistent; that is, they pointed to the cited content or to Web pages that referred or redirected the researcher to the cited content. Of the 500 citations studied, the content cited by 213, or 42.6 percent, could not be found at the URLs included in the citations and, therefore, were considered to be impermanent. In the follow-up study, 195, or 39 percent, of the sample URLs were found to be persistent, and 305, or 61 percent, were impermanent.

Cited content was considered to be accessible if, after failing to find it at the URL included in the citation or at a referred page, the researchers were able to locate it elsewhere on the Web. In the original study, the researchers had to search for the content of 213 impermanent URLs, while in the follow-up study they had to search for 300. The results of these searches are presented in table 2.

The researchers found content cited in 3.8 percent of the original study impermanent URLs by truncating them, and in 4.2 percent by identifying errors that, when corrected, led to the cited information. In the follow-up study, the percentage of content found by truncation dropped slightly to 3.0 percent, with a 2.2 percent decrease in content found by correcting URL errors. In the follow-up study, the researchers had relatively less success finding content at the URL cited by browsing and searching the Web site, and relatively more success using Google to locate content elsewhere on the Web. In the original study, they located content cited in 25.4 percent of the impermanent citations by browsing or searching the site to which the URL led them, while in the follow-up study this percentage was 21.3 percent. In the follow-up study, the researchers found 30.7 percent of the impermanent URLs by using Google, an increase of 5.3 percent from the original study.

The number of citations for which the cited content could not be found rose from 83, or 16.6 percent of the 500 citations in the sample, in the original study, to 124, or 24.8 percent of the sample, in the follow-up study. This represents an 8.2 percent increase in unavailable content.

The researchers searched the Internet Archive using the Wayback Machine (www.archive.org/index.php) to determine if the URLs included in the sample citations had been archived. The results are presented in table 3. The percentages of all citations and of those that were considered persistent that the researchers were able to find in the Internet Archive changed very little from the original to the follow-up study. The largest difference was in the accessible content category. In the follow-up study, the researchers found 64.3 percent of these URLs in the Internet Archive, almost 14 percent more than in the original study.

The researchers were able to access 39, or 47.0 percent, of the 83 citations that they could not find at the URL cited or elsewhere on the Web in the original study by using the Wayback Machine. This raised the overall availability rate of the cited content from 81.4 percent to 89.2 percent, or 7.8 percent. In the follow-up study, they found 63, or 50.8 percent, of the 124 citations in the “Content Not Found” category, raising the overall availability of cited content from 68.0 percent to 80.6 percent, or 12.6 percent.

The researchers ran cross-tabulations to determine the extent to which the citations in the sample changed categories between the studies. Table 4 presents the status in the follow-up study of the URLs that were persistent, accessible, not found, or could not be determined in the original study. Of the 282 citations that were persistent in the original study, 188 were found to be persistent in the follow-up study, and all except 37 were found elsewhere on the Web by either truncating the URL (1 citation), browsing and searching the site (19 citations), or using Google (37 citations). Four of the 8 citations that were found by truncating the URL in the original study were found using this method in the follow-up study, one was found by browsing and searching the Web site, and another was found by using Google. Thirty-two of the 54 citations that were found by browsing and searching the Web site in the original study also were found using that method in the follow-up study, while 15 were found using Google, and 4 were not found. The researchers found 29 of the 54 citations for which they had to use Google in the original study by using Google in the follow-up study. They could not find the content cited by 14 of those 54 citations.

The researchers expected that many of the citations in the sample would, over time, move from persistent to accessible to not found. However, 20 progressed in the opposite direction. These citations are identified with an asterisk in table 4. In 5 cases, URLs that were only accessible in the original study were found to be persistent in the follow-up study. This group includes 2 citations for which content was found in the original study by truncating the URL and using Google, and one whose content was found by browsing and searching the Web site. In addition, 2 of the 83 citations in the not found category in the original study were found to be persistent in the follow-up study, and 13 others were found to be accessible by truncating the URL (1 citation), browsing and searching the Web site (4 citations), or using Google (8 citations).

The researchers examined the 20 citations that moved from accessible to persistent and from not found to persistent or accessible to try to identify patterns that would explain this improvement in availability. They found that when they searched for 12 of these citations in the original study, they initially received a “URL not available” message. This required that they wait a week and then search the URL again. The researchers found the content cited by 3 of these on the Web but not at the URL in the citation. In the follow-up study, their status changed from accessible to persistent because when the researchers looked for the content at the URL included in the citation, they were referred to a page that contained that content. The researchers also initially received a “URL not available” message in the original study for the remaining 9 of these 12 citations, but when they waited a week and searched again they were not able to find the cited content anywhere on the Web. In the follow-up study the researchers did find the content, but not at the URL included in the citation and not at a referred page. Therefore, the status of these cases changed from not found to accessible. Based on these findings, the improvement in availability appears to be the result of referrals and automatic redirects not in place during the original study. Improvements in Web site indexing capabilities with more sites offering tools to search their sites and enhancements to Google’s Web search features also may have affected availability. Finally, in some cases, the improvement may be simply because the Web sites containing the cited content were not working at the time the original study was conducted, but were functioning during the follow-up study.

The cross-tabulation of cited content availability in the Internet Archive in the original and follow-up study is presented in table 5. Fifty of the 344 URLs found in the Internet Archive in the original study no longer led to the cited content when the follow-up study was conducted. The content for 46 of the 146 citations that were not found in the Archive in 2002 was found in the Archive during the follow-up study.

The researchers ran a series of cross-tabulations to identify the characteristics of the cited URLs that could be associated with URL persistence and content availability on the Web and in the Internet Archive. Chi-Square Tests of Independence were performed to identify the statistically significant relationships. In order to run these tests, the researchers had to filter out some of the cases in the could not determine category and reclassify some of the variable values into broader categories. The results of the Chi-Square tests for the original and the follow-up studies are presented in table 6, and those that are significant at the p < 0.05 level are identified with an asterisk. Cross-tabulations were not run on citation content and availability in the Internet Archive variables, as the Wayback Machine only accepts URLs and, therefore, the presence or absence of additional bibliographic information in the citation could not affect, or be associated with, availability in the archive.

In the original study, the Chi-Square tests indicated that citation content, domain, and URL directory depth were associated with content availability. Specifically, the amount of information in the citation, implied domain, and directory depth were associated with content found at the URL cited or at a referred page; that is, persistence. Original and implied domain, as well as directory depth, were associated with content that was either found at the URL cited or elsewhere on the Web; that is they were “persistent or accessible.” Finally, original domain and directory depth were found to be associated with content availability in the Internet Archive. Although domain, directory depth, and citation content characteristics were found to be associated with content availability in the follow-up study, they are not associated with the same types of availability as in the original study. In addition, the source journal (print or e-journal) and page type (navigation or content) were found to be associated with availability in the follow-up study, although not in the original study.

The cross-tabulations for the characteristics with significant Chi-Square values are presented in table 7. The cross-tabulation between source journal and persistence indicates that the URLs in the sample citations taken from print journals were found to be persistent more often than the URLs in citations taken from journals that were published only in electronic format. Specifically, 41.5 percent of the citations in the sample that came from print journals were found to be persistent, while the persistence rate for URLs from electronic-only journals was only 20.5 percent.

Page type and directory depth also were found to be associated with persistence. More than 52 percent of the content cited by navigation pages was found at the URL included in the citation, in comparison to 35 percent of the content cited by content pages. In general, persistence rates decreased as the directory depth increased. More than 62 percent of the content was found at the URL cited when the URL was at the server level, and only 13.3 percent of the content was found at the URL cited when the URL was at level five or lower. However, for URLs at level four, the persistence rate (43.8 percent) was higher than that for citations containing URLs at level one (40.3 percent). This suggests that some other variable may be affecting the relationship between directory depth and persistence. This pattern also was observed in the original study, where directory level was found to be associated with availability on the Web and in the Internet Archive, but where page type was not found to be associated with availability in either place.

Citations containing complete bibliographic information along with the URL were more likely to lead to available content than those with only partial bibliographic information or those that contained only URLs. Almost 80 percent of the citations with complete bibliographic information led to content that was either at the URL cited or accessible elsewhere on the Web. When the citation included only partial bibliographic information, this percentage dropped to 71.8 percent; for citations that included only URLs, the availability rate was only 62.1 percent.

The cross-tabulations of domains with content availability indicated that content cited by URLs with original domains of .gov, .edu, and .org were more likely to be available and found in the Internet Archive than content cited by URLs with .com or other types of domains. Between 81.7 and 82.2 percent of the content cited by URLs residing on education, government, and organization servers was found either at the URL included in the citation or elsewhere on the Web. Less than 60 percent of the content cited by URLs residing on commercial servers was found to be available. For content found in the Internet Archive, the pattern was similar. The researchers found 76.6 percent of the content cited by URLs residing on education servers and only 55.8 percent of the content cited by URLs on commercial servers. URLs in the combined category include those on military and network servers and those with geographic designations. Content availability in the Internet Archive for citations with URLs in this combined category was 70.6 percent, which is higher than the percentages of available content for citations containing government and organization URLs.

The cross-tabulations indicated that content availability on the Web and in the Internet Archive also was associated with implied domain. The original domains in the sample citations’ URLs were translated into implied domains by folding the country code top-level domains that included information about the organization hosting their content into the appropriate generic top-level domains. This reduced the number of sample citations in the “Geographic Designation and Other” category and increased the number in the other four domain categories. The citation URLs with implied education domains were found to have the most available content (82.2 percent), and those with commercial implied domains had the least available content (58.7 percent). Three-quarters of the content cited by URLs with government implied domains and 80.8 percent of content cited by those with organization implied domains were found to be available. This pattern of content availability also was found in the Internet Archive. The content cited by URLs in the education implied domain was found to be most accessible (76.9 percent), followed by URL citations with organization (70.6 percent) and government (67.7 percent) implied domains.

The Chi-Square test also indicated that page type was found to be associated with availability in the Internet Archive. More than three-quarters of the navigation pages (76.5 percent) were found in the Internet Archive, while 65.6 percent of the content pages were found there.

In the original study, the researchers reviewed the instructions for authors published by the journals from which the sample was drawn for the period of the study (1999–2000), and again as that manuscript was being prepared, in order to determine if these journals had established policies or instructions on citing content on the Web. In the original study, only 6 of the 34 journals included examples of citations to electronic resources for authors to follow. Three of these also provided further instructions on citing Web resources. One additional title referred authors citing content on the Web to the American Psychological Association’s Web site (APAStyle.org). Fifteen of the journals referred authors to the fourteenth edition of The Chicago Manual of Style, which, having been published in 1993, does not address references to digital resources. None of the instructions for authors addressed Web site persistence.

By the time the follow-up study was conducted, 2 of the 34 journals had ceased publication, and 2 had merged. Of the 31 remaining journals, 12 included examples of citations to electronic resources in their instructions to the authors. Only 5 of the journals instructed authors to use the fourteenth edition of The Chicago Manual of Style, whereas 6 referred authors to the fifteenth edition, which includes instructions for citing content on the Web. Three of the 31 journals included some mention of Web site instability, with 1 journal requiring authors to include digital object identifiers in citations to journal articles on the Web.