ILS Satisfaction

As mentioned in chapter 2, a few products—notably Apollo and OPALS—received exceptionally high satisfaction ratings. These products typically serve small libraries; therefore both the products and the libraries may not be reflective of trends for other types of libraries. Through the linkages to lib-web-cats, it's possible to associate survey responses with additional data elements to bring in factors such as library type or size of collection. We were also able, by using data from lib-web-cats, to correlate library collection size for most, although not all, of the libraries in the survey. This let us examine ILS and customer support satisfaction for three sizes of library—smaller, larger, and very large—and two types—public and academic. There were not enough respondents from other library types in the survey to permit meaningful comparisons.

One caveat: collection size is self-reported, and libraries may use different metrics; for instance, they may make different choices about how to count electronic content. Therefore the breakdown by library size should be taken as a broad approximation.

When segregating the survey data by the size of the library, in this case determined by those with collections over 50,000 volumes, much more interesting results are revealed than from the total aggregate data. This view of the data filters out the ultrapositive responses submitted primarily by small libraries, providing a more fair comparison of those companies and products that serve all but the smallest libraries. Removing the smallest libraries, and the 347 for which collection size could not be determined, leaves 1,043 out of the 2,102 total survey responses (figure 1). Note also that only products with at least 20 active sites generate discrete entries in the summary tables.

In this view of the data filtering out smaller libraries, Polaris stands out as the ILS with the most positive ratings, with an average of 7.84. Millennium ranks second with 7.19. A middle group of products (Library•Solution, Aleph, and Evergreen) all rank between 6.4 and 6.75. Voyager, Symphony, and Horizon received similar ratings around 5.9.

By looking at the comments, we can see some of the reasons behind these ratings. While top-ranking Polaris has usability concerns for some customers, many agree that “customer support is beyond excellent,” including increased responsiveness over the past year. Libraries feel that “they really listen to our issues, thoughts, and suggestions”; for example, “via the Polaris Users Group enhancement process.” One library feels that “The customer service is unparalleled. I would not say that I would stand on a street corner in a clown suit to sell it, but it's close.” Notably, few of these comments are about the software itself; rather, they note the libraries’ exceedingly positive relationship with their vendor, which seems to translate into satisfaction with the product.

As for Millennium, a common theme seems to be “We are happy with III/Millennium except for the cost.” Indeed, while satisfaction ratings are generally high—and there is a great deal of interest in Innovative's discovery interface, Encore—many comments are negative. Libraries express concern about maintenance costs and about having to pay individually for new features, including “basic functionality (faceted browsing, spell check etc) that is part of other companies’ standard web catalogs, but must be purchased from III.”

Reasons for dissatisfaction with the lower-ranked products vary. Some Horizon customers expressed concern that closures and consolidations of SirsiDynix offices resulted in important institutional knowledge being lost, with a corresponding decline in customer service. They also wonder what their upgrade path will be as SirsiDynix shifts its focus to Symphony. Symphony libraries speak somewhat more positively of customer support, although they note that quality varies. Many expressed concern regarding high maintenance costs.

Comments on Voyager are more likely to center on technical issues. While some customers are concerned that the product may be outdated and underdeveloped, others speak well of the technical support resources available. Voyager is also praised for participating in the open source community and exposing APIs.

It seems that, to paraphrase Tolstoy, happy libraries are all alike, in their satisfaction with customer service; every unhappy library is unhappy in its own way. Although cost is a major concern for users of many different ILS products (including those who are otherwise satisfied), unhappy libraries also wonder about companies’ strategies for their product lines; usability, functionality, and feature development; and customer service, especially responsiveness.

Indeed, one key theme to emerge from the comments was listening. (This will be treated more fully in chapter 4.) Libraries that are satisfied with their ILS frequently commended their vendors for listening to them in a caring, responsive way. Libraries that are dissatisfied mentioned unresponsive customer service personnel and companies that did not act on bug reports or feature requests, which one profoundly dissatisfied library summarized with, “It would be nice to have vendors listen to our needs for a change.”

Finally, recall that many libraries, while they did submit ratings, noted that they could not fully evaluate their satisfaction with the company because their consortium (or, in some cases, their IT department) is the point of contact for service issues. These libraries do not have a direct way to communicate their feature requests or other needs to their vendors and do not have direct experience of their customer service. It is not clear what their ratings indicate, nor can they be filtered out of the data.

We can also disaggregate the data for larger libraries by library type to see if trends differ. Among these libraries, only academic and public library types had enough responses to examine (see figures 2 and 3). Both academic and larger public libraries show generally high satisfaction with Millennium, driving its second place overall satisfaction ranking among larger public libraries and top place with academics. And both types of library are less happy with the two SirsiDynix products, Horizon and Symphony.

However, public and academic libraries diverge in which ILSes they use. While some systems such as Millennium, Horizon, and Symphony find use across library types, ALEPH 500 and Voyager target academic and research libraries, while Polaris and Library‧Solution appeal more to public libraries. While some implement these systems outside their target markets, the numbers don't reach the thresholds to be included in the tables. For example, ALEPH 500 and Voyager, both strongly represented in the academic sample, are scarcely used in the public library world (neither meets the threshold of 20 sites required for inclusion here). Similarly, Polaris and Library‧Solution have only a handful of academic sites. This sheds light on two aspects of the overall ranking in figure 1.

First, the tiers of rankings correspond roughly to library type. Number 1 Polaris and number 3 Library‧Solution are both public library ILSes; number 4 ALEPH 500 and number 6 Voyager are both academic library products.

Evergreen, an open source ILS used primarily by public library consortia, earned moderate ratings (6.40) in the view of the data showing only larger libraries. However, Evergreen does not have enough installed sites in these libraries to be included in figure 2.

Second, academic libraries are simply less satisfied than public libraries overall. Public libraries are more satisfied than academics with all three of the ILSes (Millennium, Symphony, and Horizon) in common use in both library types. They are more likely to be highly satisfied regardless of product; note the mode of 8 for public libraries versus 7 for academic libraries.

The survey does not specifically address the reasons why public libraries may have generally higher levels of satisfaction than their academic counterparts. We speculate that the traditional automation systems in use today covered by the survey don't fare as well with the more complex collections of academic libraries and their increased orientation toward electronic resources. Public libraries continue to rely on their automation systems handling their physical collections, which these systems continue to do quite well.

When we turn our attention again to the 719 smaller libraries (collection size under 50,000) in the survey, we find they differ from larger libraries in several ways (figure 4). First, they are more likely to be extremely satisfied with their products. As noted above, high marks for Apollo and OPALS are part of this picture. However, even when comparing satisfaction levels with the same product, smaller libraries tend to be happier than larger ones. Note the ratings for Library‧Solution (mean of 7.24 for smaller libraries versus 6.75 for larger ones), Horizon (6.85 versus 5.89), and Symphony (6.18 versus 5.92). Millennium (6.79 versus 7.19) is an exception to this trend, but its ratings among smaller libraries still indicate moderately high satisfaction.

One aspect of the very high satisfaction seen in small libraries lies in automation products that offer the right level of functionality for their needs. These libraries may feel overwhelmed by the complexity of some of the higher-end products, especially Millennium. Fully managed and hosted products, such as Apollo and OPALS, remove much of the burden of automation from these libraries, offering an ample number of features through a Web-based system with no need to manage software on local servers or workstations. Many of the libraries implementing Apollo have migrated from long-outdated systems such as Winnebago Spectrum, Athena, InfoCentre, or Circulation Plus or may have automated for the first time. These right-sized solutions in tandem with high quality and personalized support may underlie the superlative perceptions of these libraries.

Other smaller libraries also appreciated other fully hosted systems, though at slightly lower levels, such as Koha supported by ByWater Solutions (7.81) and AGent VERSO from Auto-Graphics (7.28). Library‧Solution from the Library Corporation also fared well with smaller libraries (7.24). It's no surprise that Winnebago Spectrum, a system that has not been actively developed since about 2005, received low marks (5.29). Destiny, the current product from Follett Software Company, geared toward school libraries, did not rank that much higher (6.45) from this group of mostly public library responders. Full-featured systems such as Horizon, Millennium, and Symphony fell in the lower tier of satisfaction scores, presumably due to their complexity relative to the modest needs of these small libraries.

Second, smaller libraries use a different and more diverse set of ILSes: twelve products are used by at least 20 smaller libraries, versus the eight products commonly used by larger libraries. Eight of these twelve are products not used by larger libraries at all.

Many small libraries are public libraries, so we can separate out their responses. Comparing the number of responses for each of the ILS products in figure 5, smaller public libraries, to those in figure 4, smaller libraries overall, we see that public libraries are the majority of those responding from this tier of libraries with smaller collections. Therefore public library satisfaction drives most of the rankings.

On first glance it appears that, as with larger libraries, the overall rankings reflect the public rankings but incorporate additional ILSes used by different library types. However, digging further into the data, nearly all of the ILSes in figure 4 are chiefly used by public libraries; they just do not meet the threshold for inclusion in figure 5. The only exception is OPALS, which is used almost exclusively by school libraries. Therefore, in this survey, the outcome for smaller libraries’ ILS satisfaction is chiefly based on public library responses.

Finally, we also examined very large libraries, with collection sizes above one million (figure 6). As with the comparison of larger and smaller libraries, we do see some confirmation that the larger the library, the lower the satisfaction scores given. We might infer that these very large libraries press these systems to the limits, causing problems and gaps in functionality to surface, and impose more difficult support scenarios.

Company Satisfaction

We focus on the question dealing with ILS satisfaction as the core of the survey. As we turn to responses to the other questions, such as that of company satisfaction (figures 7 and 8), we note a very strong correlation (0.81) with ILS satisfaction scores. The survey results do not, for example, reveal any specific cases where libraries hold the product in high regard but do not like the company or its support quality.

As a result of industry consolidation, some companies take responsibility for multiple ILS products: Symphony and Horizon both reside within SirsiDynix; Ex Libris owns Aleph and Voyager; Follett Software Company offers Destiny as its current flagship product, superseding its legacy products including Winnebago Spectrum, Athena, InfoCentre, and Circulation Plus. In most cases, we did not see large differences among the products supported by the same company, except for Follett Software Company, where Winnebago Spectrum received drastically lower company satisfaction scores than Destiny or Circulation Plus.

Interest in Open Source

As noted in chapter 2, libraries have mixed feelings about open source ILSes. When we break this down by library type and current ILS, we see a more nuanced but still very mixed picture.

If we compare academic libraries’ interest in open source (figure 9) to the data on academic libraries’ satisfaction with their current ILS and company (figures 3 and 8), we see a roughly inverse relationship. (Figure 3 excludes smaller libraries, but the picture is quite similar when all are included.) Unsurprisingly, dissatisfaction with the status quo, largely based on proprietary products, correlates with increased interest in open source. Within the Ex Libris fold, libraries using Voyager expressed higher interest (4.30) than those using ALEPH 500 (3.32). For those operating Millennium, academic libraries showed less interest (3.22) than public libraries (3.76).

This increase is slight; average interest in open source is low across the board, with 0 as the most common response. However, the distribution is nothing like a bell curve, or even a straight line. Even though the plurality response is 0, there are as many libraries that answered 8 or 9.

Public libraries (figure 10) have one important difference from academics (figure 9), which is that public libraries are much more likely to already be using open source ILSes. For instance, there are several public library consortia that have adopted open source products. (Survey respondents are individual libraries, not consortia, so a single consortial adoption can lead to numerous survey responses.) Of the 47 libraries using Evergreen, all are public; of the 131 using Koha, 56 are public and 30 are academic (the rest represent a wide cross-section of library types). Thirty-nine of 40 Koha libraries who purchase support from ByWater, and 11 of 31 LibLime customers (the plurality), are public libraries.

The higher adoption in open source ILS products by public libraries may have to do with the fact that the functionality in Koha and Evergreen currently fits public libraries better than academics. Both were originally developed for public libraries, though their functionality, and to some degree adoption, has expanded to academic libraries as well.

Those libraries that are already using open source ILSes naturally tend to rate their interest as 9. Aside from that, the overall picture is similar for public and academic libraries. Most public libraries are uninterested, or not very interested, in open source ILSes, with the mode score again 0. However, even when factoring out public libraries already using open source, there is a second peak at 9.

For larger (figure 11) and very large (figure 12) libraries, the picture is similar: low overall interest, with a mode at 0, but a second, smaller peak at 9. Large libraries are much more likely to need functions such as acquisitions, serials control, or reserve book room modules that are not well-developed yet in these open source products—indeed, several comments specifically mentioned the lack of an integrated acquisitions module as a barrier to adoption of open source. However, they may also have more in-house technical knowledge to draw on.

Thus the overall picture in 2010 is of low, even zero, interest in open source, but with a notable and diverse population of highly interested libraries. How has this picture changed over time? Figure 13 shows the percent of respondents expressing any given level of interest in open source products from 2007 through 2010.

Examined this way, we see a polarizing of interest: slightly more libraries have 0 interest, substantially more libraries have maximum interest, and there are fewer libraries at almost every interest level in between. This picture, however, may be slightly misleading, as it includes libraries that have adopted open source ILSes, nearly all of which have very high interest. Figure 14 shows what happens when those libraries are removed from the sample.

This view amplifies the concerns of the nonadopters; the 0 peak is growing much faster in this view, and the 9 peak more slowly. There is clearly a modest, and modestly growing, interest in open source among people who have not yet adopted it; the increased interest does not solely represent the passion of early adopters. However, once we factor those early adopters out of the picture, the growth in the population strongly uninterested in open source is much more pronounced than the growth in those strongly interested.

Of course, there is a difference between interest and adoption. Do these strongly interested libraries represent future open source users? And are strongly uninterested libraries guaranteed to stick with a proprietary ILS? To investigate this question, we compared libraries’ level of open source interest in 2007 against the same libraries’ ILS products in 2010. Figure 15 shows what percent of libraries above each interest level in 2007 were using Koha, Evergreen, or OPALS in 2010.

In short, open source interest seems to be predictive of open source adoption. Under 10% of the libraries in this sample (that is, libraries that answered the survey in both 2007 and 2010) are now using open source products—but a third of the most interested libraries are, and likelihood of adoption rises steadily with interest.

Interest is, however, not a guarantee. Even some of the profoundly uninterested libraries have gone open source, and two-thirds of the most interested libraries have not. Investigating these libraries’ comments suggests their reasoning. Uninterested libraries may have been pushed to migrate as part of a consortium, jumped ship from a discontinued product, or faced severe cost constraints; some of these libraries became steadily more interested in open source from 2007 to 2010. Interested libraries that did not migrate cite the cost or difficulty of migration, concerns about viability, lack of interest among consortial partners, or lack of in-house technical knowledge. (Similarly, high-interest libraries that did make the switch sometimes refer to high levels of in-house technical knowledge.)

To summarize, the overall picture of open source interest is conflicted, and data can be found to support (or undermine) many hypotheses. One might look at the low average interest in open source, and the high number of respondents expressing zero interest, and conclude that these products will not be broadly adopted. One could look at the increasing levels of high interest in open source products, coupled with the elevated likelihood that highly interested libraries will switch to open source, and conclude that we are at the threshold of an explosion—or, considering the comments and the impact of libraries that have already gone open source, conclude that interested libraries with high technical knowledge have already switched, and the rest may be frustrated. Indeed, the same question might appear to have different answers depending on whether one looks at means, modes, medians, or distributions; 2010 alone, or trends over time. Readers are cautioned against drawing conclusions beyond the scope of the data.

Company Loyalty and Migration

The survey asked libraries both their degree of company loyalty and whether they were shopping for a new ILS. Figure 16 shows the percent of libraries expressing each level of company loyalty that are also shopping for a new ILS. We see, unsurprisingly, that there is a strong negative correlation between loyalty and shopping; libraries that feel strongly disloyal to their current ILS vendor are highly likely to shop around, and libraries with high loyalty are seldom inclined to shop. However, scores run the gamut, and there are 37 libraries that expressed maximum loyalty to their current vendor but are still shopping. How can this be?

If we look more deeply into the data, we discover differences between the low-loyalty and high-loyalty libraries that are shopping. Among the 94 libraries with loyalty 0, 61 are considering open source, either as one of several options or as their only contender; given the overall levels of open source interest discussed above, this is staggering. By contrast, of the 37 libraries expressing loyalty 9, 21 are considering migration to a newer product line from the same company (often as the sole product under consideration), and only 6 are considering open source (one of which has elected to remain with its current, proprietary, vendor). Many of these libraries are using products no longer under development, which seems to be a key factor motivating them to shop despite high loyalty. A few belong to consortia that are considering migrations; thus their high loyalty may not be a factor as decisions are made for the consortium as a whole.

We also see systematic differences in company, support, and ILS satisfaction among libraries that are shopping for a new ILS, depending on their level of company loyalty. These are summarized in figures 17, 18, and 19.

We also wondered whether respondents that indicated that they were planning to migrate to a new system actually followed through on such plans. As the Perceptions survey has now been conducted for four years, it possible to investigate this question by making comparisons between the survey data and migrations documented in lib-web-cats.

Of the libraries that responded to the 2007 Perceptions survey, 269 have since migrated to a new ILS. Of these libraries, 176 indicated on their 2007 survey response that they were shopping for a new ILS; 88 that migrated since 2007 did not, at that time, indicate that they were planning for a new ILS. For context, 425 total libraries indicated in 2007 that they were considering a new ILS, and 1,334 said they were not; that is, 41% of libraries that indicated in 2007 that they were shopping have migrated, compared to 7% of libraries that said they were not. These percentages must be taken with a grain of salt. Libraries that were shopping in 2007 and have not migrated may still be considering their options with the intent to migrate in the future. Those which were not shopping in 2007 but have since migrated may have had a very quick migration process and may have indicated in later surveys that they were shopping. Nonetheless, it seems clear that libraries that say they are shopping for a new ILS have a substantially increased likelihood of acquiring one.

Trends over Time

Average satisfaction level for each of the ten most popular ILSes (figure 20) has remained roughly constant over time. The graphs for average company satisfaction and customer support satisfaction (figures 21 and 22) are similar. For some ILSes there may be slight upward trend, although its magnitude is dwarfed by the variability of the data; it may instead be merely statistical noise.

In comparison with these relatively flat lines, it is interesting to look at company loyalty over time. In Figure 23 we see the percent of libraries expressing each level of company loyalty, 0–9.

There is a dramatic increase in the percent of libraries expressing maximum loyalty to their vendors, along with a modest decrease in very dissatisfied companies. How can it be that libraries are so much more likely to be loyal when their average satisfaction with products and support has not markedly changed?

Two explanations spring to mind. One is money. Many libraries expressed concerns about cost in the comments, in some cases specifically noting that concerns about cost prevented migration; for instance, “most customers are loath to move to better products due to hassle and cost. The last thing I ever want to do is another system migration!” This concern was especially pronounced for libraries interested in open source (e.g., “Although it is appealing, I doubt that we can afford the programmers to do open source.”) And, of course, the economic downturn since 2008 has wreaked widespread havoc on library budgets. However, macroeconomics cannot be the whole explanation since the trend was already evident in 2007–2008. (The Perceptions survey went live in August 2008, when the economy was still strong.)

The second is that libraries have been switching from companies they don't feel loyal to; then, upon adopting a new ILS, they have about the same level of satisfaction as all the other users. In this way, average satisfaction with each product would not change. However, libraries might have much higher loyalty to their new companies, whether because they are optimistic about the reasons they selected it, or because the nightmare of migration is fresh enough in their minds they don't want to think about doing another one.

Indeed, if we examine the libraries that had low loyalty in 2007 (< 3) and high loyalty in 2010 (> 6), we find that the overwhelming majority of them—36 out of 49, 73%—have switched ILSes. (For comparison, among all libraries that responded to both the 2007 and the 2010 survey, only 35% have switched ILSes.)

Unfortunately, of the 13 whose loyalty increased sharply while they remained with the same company, most did not comment, so it is not clear why their feelings changed. (Interestingly, both of the libraries that did comment expressed an interest in open source in 2007, but were unable to make this switch.) Among the libraries that did not comment, the same respondent filled out the survey in nearly all cases, so the difference in loyalty does not reflect a personnel change.

Among those which did switch ILSes, about a quarter were either automating for the first time or abandoning a discontinued product line. Some of the remainder comment on cost concerns or strong dislike of their current ILS. Many, however, do not comment, so (except for libraries abandoning discontinued products) there is no clear reasoning separating libraries whose loyalty increased following a switch, and those whose loyalty increased without one.