In her conclusion to a 2003 overview of journals in communication disorders, Shpilko recommended that future researchers use citation analysis to follow trends in the literature because core journals are likely to change over time.² Various criteria (including lists compiled by subject experts, extent of coverage by indexes, library holdings, surveys of faculty, and citation data) for selecting the best journals to support research within a discipline have been used by collection development librarians.³ Citation analysis is the study of relationships between authors, articles, journals, concepts, institutions, nations, and other elements by compiling and counting references in documents.⁴ Broadly stated, citation analysis is the study of relationships among published works based on the principle that a reference in a published paper implies a relationship between the cited and the citing documents.⁵ One type of citation analysis is to count citations to identify leading journals in a discipline. The present study employed the method of counting citations to identify the most frequently cited journals in a discipline, in this case communication disorders.

Half of the data for this study come from the author's 2001 LRTS article on a core collection in communication disorders. To determine the degree to which the core list changed over a decade, works cited were compiled for the appropriate volumes of the two journals used in the initial study: Journal of Speech, Language, and Hearing Research (JSLHR) and Journal of Communication Disorders (JCD). The broader sample of four journals used by Slater may better represent the literature, but the sample corresponding to this author's first study was used to minimize extraneous variables.³⁹ The volumes, number of articles published, number of works cited, and average number of works cited per article are shown in table 1.

For the 2001 study the author entered the data by hand from printed issues. Thankfully, new tools are now available to speed the process of compiling works cited. The journals used for this study are covered by the PsycINFO bibliographic database. PsycINFO has the very useful feature of including the works cited of every article whether the full text is available or not. The author exported the works cited from every issue of the sampled journals from PsycINFO (via EBSCOhost) into the Refworks bibliographic management program. A drawback of downloading works cited instead of entering them by hand is that PsycINFO tags works from any type of series as “journal article.” The author was thus unable to re-create the proportion of citations from journals, conferences, tests, books, and others that appeared in figure 1 of the 2001 study.⁴⁰ Databases hosted by EBSCO can export directly into several bibliographic management programs, any of which could be used for compiling citation data. RefWorks was chosen as the most cost-effective choice. Exported citations were organized into one folder for each volume of each journal, and each folder was downloaded into a Microsoft Excel worksheet. Keeping data from each volume discrete allowed the analysis that appears in table 2.

The author used Excel to compile and analyze the works cited data from the six journals. A time-consuming but important task was fixing discrepancies in titles. For example, title variants Audiol Neurootol, Audiol Neurotol, Audiology & Neuro-Otology, Audiology & Neurotology, and Audiology and Neuro-Otology were edited to match the title as it appears in OCLC WorldCat, Audiology & Neuro-otology. Even with careful editing, the author deemed alphabetizing lists and counting citations to be better rather than relying on Excel's COUNTIF function, because any overlooked discrepancy in punctuation or spelling could cause a citation to go uncounted. Citations to books, standards, tests, chapters in series, and anything other than citations to journals were ignored. Title changes posed something of a dilemma. Does the new title count as a new work to be treated separately or as a continuation of the same work? The author chose to aggregate counts for journals that changed title, but the author treated splits and mergers as separate works.

Counts for journals cited at least 5 times were copied to a separate worksheet for each citing journal. Worksheets were created to aggregate results by decade, one for 1997–99 and one for 2007–9. All lists were sorted by citation count to create ranked lists of cited journals. The core list published in 2001 included 104 journals that represented 80 percent of citations to journals. This base was used to establish 104 as the maximum number of journals in the ranked lists used here.

The Spearman's rho rank coefficient statistical test was used to measure whether rankings vary more over time than they do between the titles. The data for each pair of journals shown in table 2 were assigned ranks. For example, articles in the JCD 1997–99 had 89 journals cited at least 5 times. Many of these received an equal number of citations, but to calculate r_s, one must have 89 ranks for 89 journals. Ties were summed and averaged, e.g., 2 titles receiving 19 citations each were both ranked 16.5, and 6 titles receiving 11 citations were each ranked 34.5. Each pair of ranked lists included titles with no match. The nonmatched titles were given a rank one below the bottom. For instance, when correlating with the ranked list of 89 titles from JCD 1997–99, journals from the other lists that did not match were ranked 90. The correlations of ranked lists are shown in table 2.

Calculating rank correlations is a valid way to measure reliability; analyzing data from more than one perspective can be helpful. Another way to examine whether rankings vary more over time than they do between the titles is to calculate coefficients of variation. Coefficient of variation is the standard deviation divided by the mean. Standard deviation is only meaningful in the context of sample size as a measure of the dispersal of scores. Dividing standard deviation by the mean produces a relative dispersal that allows comparison of the spread of scores among different sample sizes.⁴¹ Coefficients of variation were calculated for the cited journals appearing in all four ranked lists from the volumes of JSLHR. The average variation and examples representing the range of variation are shown in table 3.

The formula for Spearman's rho takes into account journals that change dramatically in rank. But because it treats subjects as a group, the statistical measure can obscure significant changes at the level of individual journals. To account for this, journals that moved most in rank between 1997–99 and 2007–9 were identified and scrutinized. Myriad characteristics might affect the impact of a journal. The author chose to investigate dates of publication, publisher, number of articles published in 1997, number of articles published in 2009, price, and availability online. The author retrieved dates of publication from WorldCat and determined publishers from the journals’ websites. Article counts came from databases including Medline, PsycINFO, and Communication and Mass Media Complete, with the exception of Journal of Medical Speech-Language Pathology, which was determined by examining the journal. The author used EBSCOnet to look up prices. For purposes of comparison the author used prices for print subscriptions when available. For titles with tiered or otherwise differentiated pricing, subscription prices applicable to a masters-level college were recorded. Online availability was found in EBSCOnet and confirmed from Serials Solutions’ data, searching in the identified databases, and checking publishers’ websites.

The comparison of works cited between 1997–99 and 2007–9 shows growth in both the number of articles published per volume and the number of works cited per article. The most obvious case of growth in the literature is JCD’s substantial increase in both the number of articles published and the average number of works cited per article. The JSLHR also experienced growth in articles and works cited. The sample as a group experienced a 32 percent increase in works cited over the decade.

The list of most frequently cited journals for 2007–9 from the JSLHR and JCD is shown the appendix. The sample contained 15,631 works cited, a total that includes citations to books, conferences, tests, and other resources. Ranks are based on the percentage of citations received, and the rightmost column indicates the change in rank relative to the 2001 list of 104 journals representing 80 percent of citations to journals.⁴² The method of downloading works cited from PsycINFO obscured the type of resource being cited, so the percentages of works cited shown in the appendix total less than 80 percent. Titles that were not on the list of 104 titles published in 2001 were assigned a rank of 104, thus counting them as ties for last place. The Spearman rank correlation coefficient of the 1997–99 ranks against the 2007–09 ranks is r_s = 0.73. In essence, the Spearman correlation formula squares the differences in rankings and divides that by how many items are being ranked. By design, dramatic changes in rank by many items in the measured group will cause r_s to approach zero, i.e., no relationship between the two ranked lists. No changes in rank between lists would result in r_s = 1, a perfect correlation and total reliability between the two measurements. Any coefficient above 0.70 is generally considered a very strong positive correlation that indicates a high degree of reliability. However, the r_s = 0.73 must be considered within the context of two key factors: random variability in the journals authors cite and possible variation due to the composition of the sample.

Some of the change in which journals are cited is due to random variability. This makes determining the degree to which changes in rank are due to real changes in the literature or simply due to natural variability in which journals authors choose to cite difficult. Table 2 shows reliability of ranked lists among the volumes comprising the sample. As one might intuitively expect, the strongest correlations are between volumes of the same journal. Reliability as measured by Spearman's rho rank correlations for volumes of JSLHR range from r_s = 0.60 for the 1997 and 2007 volumes to r_s = 0.71 for both 1997–99 and 1999–2007. The weakest correlations are for the 1997–99 volumes of the JCD. This may be due to differences in topic coverage, but is likely because JCD 1997–99 contained the fewest number of works cited. The smaller sample size may have caused the lower reliability. Striking is that the strong positive r_s = 0.73 between the entire 1997–99 and 2007–9 groups indicates greater reliability than exists from any of the volumes shown in table 2. In that context, the simplest answer to the question “how much does the core change over a decade?” is “not much.” In fact, the between-group correlation of r_s = 0.73 compared to the within-group correlations shown in table 2 allows one to reasonably infer that most of the measured change in core journals over one decade was due to random variability.

The effect of composition of the sample can be inferred from the correlations shown in table 2. Each volume represents its own sample, so one can see that the reliability of ranked lists from these six samples ranges from r_s = 0.52 to r_s = 0.71, all indicators of strong, but not perfect, correlations. Another way to analyze the effect of sample selection is to compare these results to Slater's list of frequently cited journals in communication disorders.⁴³ Slater used a method very similar to this authors’ except that her sample was based on four journals. In addition to JSLHR and JCD, she also counted works cited in the American Journal of Speech-Language Pathology and the British Journal of Disorders of Communication/European Journal of Disorders of Communication. The period for Slater's study was 1991 through 1993. The correlation coefficients for Slater's results versus the present results are r_s = 0.54 for the 1997–99 list and r_s = 0.38 for the 2007–9 list. The difference in samples makes it difficult to determine how much of the drop in reliability is because of true changes in the literature. However, the fact that reliability of ranked lists drops from r_s = 0.54 after six years to r_s = 0.38 after sixteen years suggests a continual, albeit rather gradual, change in core journals over time.

Spearman's rho measures reliability on the basis of correlations of ranked lists. Another way to look at the degree of variability is to calculate coefficients of variation using the method explained above. Because the scope of JCD is somewhat different from that of JSLHR, only ranked lists from JSLHR were used for this analysis. Fifty-seven cited journals appeared on all four rankings from the volumes of JSLHR. As shown in table 3, the average coefficient of variation was 38 percent. One can infer from this that roughly one-third of the variation in times cited is because of random variation. But this too varies, from the highly reliable 4 percent rate for the top journal to the very unpredictable 79 percent for Dysphagia. One sees a general trend for the most-cited journals to have the lowest coefficients of variation. That would come as no surprise to Line, who emphasized the difficulty of predicting use of journals on the fringe of a core.⁴⁴

A reliability coefficient of r_s = 0.73 indicates that the core list of communication disorders journals changed relatively little over a decade, but this does not mean ranks of individual titles all stayed about the same. Table 4 shows the 10 journals that moved up the most in rank and selected characteristics that might suggest reasons for the changes. Trends in Cognitive Sciences and Developmental Science changed for the obvious reason that they were new launches. Length of time in publication also could be a major reason for increased citations to Journal of Medical Speech-Language Pathology, NeuroImage, and Psychological Science.

The case of the Journal of Medical Speech-Language Pathology is very interesting. It defies conventional wisdom to become so well established in the discipline despite being published only in print by a smaller publisher. Moreover, it is only indexed in Cumulative Index to Nursing and Allied Health Literature and Linguistics Language and Behavior Abstracts, not in the commonly used databases for communication disorders such as Medline, or PsycINFO, or Communication and Mass Media Complete. This shows that researchers manage to find their way to highly relevant literature. That also could be an explanation for the success of Cleft Palate-Craniofacial Journal. A possible factor in the success of those two journals is their relatively low subscription prices.

The remaining titles on the list of journals with greatest increases in rank are Language and Cognitive Processes, Journal of Neuroscience, PNAS, and Journal of Applied Physiology. All of these are either Open Access (OA) or in full-text in Communication and Mass Media Complete after an embargo period. Knowing the exact effect of OA on use by researchers is impossible, but of interest is that OA is more prevalent among journals that moved up in table 4 than it is among journals that moved down in rank. However, the four journals all experienced substantial increases in the number of articles published per year, which may be a more significant factor than online availability.

Several titles decreased in rank; see table 5. The journal experiencing the greatest drop was for the blindingly obvious reason that it ceased publication. It merged into ASHA Leader, the newspaper of the American Speech-Language-Hearing Association. Three of the journals that dropped in rank published fewer articles in 2009 than in 1997: Seminars in Hearing, Language, and Annals of Neurology. Two others had no significant growth in articles published: Otology and Neurotology and Topics in Language Disorders. Decreases in rank for Archives of Neurology, Neurology, and Annals of Neurology hint at less use of general neurology journals by communication disorders researchers, but natural variability in what is cited prevents drawing any conclusions about that.

One may conclude on the basis of the method used here that the core journals in communication disorders have changed little over a decade. But even though the list as a whole was stable, some journals did change significantly in rank. This study has several implications for collection development. One is that a core of journals carefully selected to support researchers can be expected to need only minor adjustments over time. If the case of communication disorders can be generalized to other fields, librarians can rest assured that core journals change gradually. Normal ongoing collection development activities (regrettably including cancellation projects) should be sufficient to keep the core aligned with researchers’ needs.

A second implication is that collection development is somewhat easier for libraries serving a range of disciplines than it is for libraries serving a special population. A discipline like communication disorders draws from the literature of many fields. A research library will probably have many heavily cited journals not specific to the discipline, such as Nature, Science, Cognition, and Brain. Smaller, specialized libraries are less likely to have such a broad-based collection to build upon.

A third implication is that variability in what is cited as shown in tables 2 and 3 warns librarians and scholars to treat with skepticism all ranked lists of journals. A ranked list can be a good starting point for collection development, selecting a journal to submit an article to, or suggesting appropriate places to browse. The principle of using ranks as only one source of information among many important considerations has been established in court. Judge Leonard B. Sand heard the case of publisher Gordon and Breach against Henry Barschall, who had published a ranked lists of journals based on cost relative to impact.⁴⁵ Sand wrote that “[Gordon and Breach] have proved only the unremarkable proposition that a librarian would be ill-advised to rely on Barschall's study to the exclusion of all other considerations in making purchasing decisions.”⁴⁶ One should use rankings with the understanding that they have limited reliability. A different selection of sampled journal titles or years of coverage will result in a different ranked list. Rankings drawn from small samples should be treated with increased caution. Lower reliability due to a limited sample is an inherent problem with ranked lists in a subspecialty not categorized in JCR or Scopus.

How reliable is a ranked list over time? The primary purpose of this study was to determine how much a ranked list of core journals changed during a decade. This was accomplished by replicating a 2001 study and calculating the Spearman's rho rank correlation coefficient of the two ranked lists. A second research question was whether variability in ranked lists was greater over time than it was among citations in source journals from the same timeframe. To answer that question, correlations were calculated to compare rankings derived from citations in two leading journals. In the process of answering those questions, discoveries were made regarding growth in literature and journals that changed most in rank were identified. This study supports these conclusions for the case of core journals in communication disorders:

• Journal literature grew substantially during the decade both in articles published and works cited per article.
• As a group, core journals remained quite stable over time (r_s = 0.73).
• Despite high reliability of core journals over time, some titles changed dramatically in rank.
• Approximately one-third of change in ranked lists derived from works cited in single journal volumes is due to random variability.

The stability of a core list over time means that normal collection development activities should be sufficient to keep a collection aligned with a discipline's core of most frequently cited journals. While ranked lists are useful collection development tools, random variability in rankings cautions librarians to treat any ranked listing with some skepticism.