Bibliographic Management Tool Adoption and Use

A Qualitative Research Study Using the UTAUT Model

Hannah Gascho Rempel and Margaret Mellinger

Hannah Gascho Rempel (hannah.rempel@oregonstate.edu) is science librarian and Graduate Student Services coordinator, and Margaret Mellinger (margaret.mellinger@oregonstate.edu) is engineering librarian at Oregon State University Libraries, Corvallis, Oregon.

This study explores how researchers choose a bibliographic management tool and what makes them continue using this tool. This exploratory, observational study combined a naturalistic work-practice method, interviews, and journal reflections to collect qualitative research data from researchers actively using a bibliographic management tool. The unified theory of acceptance and use of technology (UTAUT) model was used as a guiding framework to help provide a better understanding of these researchers’ bibliographic management adoption and use behaviors. Findings indicate participants adopt tools because of an expectation of enhanced research productivity, but participants persist in using the tools because of ease-of-use experiences. Librarians were found to have opportunities to influence tool adoption decisions but may have somewhat less influence over researchers’ decisions to continue using bibliographic management tools.

Workshops on bibliographic management software tools (e.g., EndNote, Zotero, and Mendeley) are common in academic libraries. This is the case at Oregon State University (OSU) Libraries, and although a wide range of research-skills workshops are offered at OSU Libraries, almost half of the total workshop attendees participate in the bibliographic manager tool workshops. Bibliographic management tools promise to make researchers’ workflows more simple, sources more easily re-findable, and bibliographies simpler to create. During the sessions, workshop instructors witness participants’ positive responses to the tools but must speculate about how participants use what they have learned after they leave the classroom and whether attendees feel the tools live up to their claims.

Some bibliographic management workshop attendees schedule follow-up research consultations with librarians, but many do not. Consequently, librarians leading these workshops typically resort to instruction informed by what makes the tools useful and usable for their own research process or based simply on the help guides provided by the bibliographic management tools themselves. This lack of systematic, in-depth examination of how other researchers choose and use bibliographic management tools results in an incomplete picture of researchers’ potential bibliographic management tool uses and can hamper librarians’ ability to provide well-rounded approaches to using the tools. Gaining insights from a wide range of users could help librarians leading bibliographic management tool workshops to more specifically guide researchers as they choose a tool that works best for them, to provide more meaningful examples, and to give more compelling reasons for why these tools may be useful.

The purpose of this study was to observe and explore how researchers choose bibliographic management tools, how they use the tools after they leave the workshop setting, and how they deal with problems that arise so that workshop instructors can provide improved instructional experiences. To examine how researchers adopt and actually use bibliographic management tools after attending a workshop, we devised an exploratory, qualitative study consisting of nonintrusive observations, participant responses to journal prompts, and interviews. Because bibliographic manager use has not been documented in this way before, this combination of observational methods was intended to uncover patterns in researcher adoption and use behaviors and to explore participants’ attitudes toward these tools in a descriptive way so that OSU Libraries workshop instructors could make informed adjustments to workshop promotion and delivery methods. While the intention of this exploratory study was not to systematically compare bibliographic manager use by specific user populations or disciplines, the results provide a framework for considering new tool use that can assist other librarians who want to explore those questions in more depth.

Literature Review

Libraries’ Approach to Bibliographic Managers

While bibliographic managers have been used for decades, the literature has primarily focused on discussions and comparisons of the features of these tools, especially as new offerings have emerged. For example, Jon Ritterbush provides an early description of Zotero’s capabilities and specifically advertises this tool to a librarian audience as a way to promote Zotero to users; similarly, Holt Zaugg et al. provide a thorough description of Mendeley to a higher education audience.1 Several librarians have undertaken in-depth comparisons of a range of bibliographic management tools’ features to help users determine the best tools to match their workflow requirements.2 Another common approach for discussing bibliographic management tools includes suggestions and best practices for instructing learners in their use.3 Librarians have widely embraced the responsibility for instructing users in how to get the most out of these often complicated tools. In a 2009 review of Association of Research Libraries websites, H. Stephen McMinn found that library support for bibliographic management tools is widespread.4 Seventy-two percent of the libraries examined provided instruction on at least one bibliographic management tool.

Despite the proliferation of library instruction on the use of bibliographic management tools, little research has been published on the question of whether and how researchers actually continue to use the tools after an instructional session. However, two recent studies have examined aspects of these questions while exploring researchers’ perceptions of bibliographic management tools to help determine what tools libraries and librarians should support in terms of licensing and instruction. One study focused on undergraduates’ potential use of bibliographic management tools. Librarians from Kent State University held focus groups to see if undergraduates might use the institution’s RefWorks license or the institutional version of EasyBib. They found the participating undergraduates’ preferred bibliographic management processes primarily consisted of tracking their sources in a Microsoft Word document. The researchers determined that undergraduate students were not willing or ready to adopt a more sophisticated bibliographic management tool because their citation management tasks are not complex enough for them to see a return on investment for familiarizing themselves with one of these tools.5

In contrast, Jenny Emanuel conducted a survey targeting advanced researchers, including graduate students and faculty at the University of Illinois Urbana Champaign (UIUC), who already used a bibliographic management tool, to explore: how researchers chose a particular tool, what features they looked for in a tool, how much support they needed to use a particular bibliographic management tool, and if they saw the library as having a role in that support.6 On the basis of participants’ feedback, Emanuel concluded hands-on library workshops on bibliographic management tools were valuable, but the libraries’ web presence should focus on advising users in the tool selection phase. In addition, the UIUC libraries chose to continue to support a variety of bibliographic management tools, reasoning that different users have different needs that cannot be supported with a single tool.

Measurements of Bibliographic Management Tool Use

A few attempts have been made to quantitatively measure bibliographic management use across broad disciplinary groups or groups based on academic status. Statistics on bibliographic management use are scattered and rapidly go out of date, but a survey conducted in 2005 at the University of Minnesota found that 36 percent of social scientists and 8 percent of humanists responding to this institution-specific survey used a bibliographic management tool.7 These researchers also found 27 percent of responding graduate students at their institution used a bibliographic manager as compared to 17 percent of responding faculty. Marni Harrington found the master’s students responding to her survey were more likely than the PhD participants to use bibliographic management tools, and less than half of the survey respondents used a bibliographic management tool at all.8 Interviews with fifteen Australian researchers determined that just over half used EndNote or a similar tool.9 While some groups may use these tools more than others, this snapshot of bibliographic management tool use demonstrates that there is a significant opportunity for growth in the adoption and use of these tools regardless of disciplinary area or academic status.

Adoption and Use Models

The variability in use of bibliographic management tools across disciplinary groupings and academic status raises the following questions: what leads to the adoption of various technology tools, and why do some people feel the effort to learn a new tool outweighs the costs?10 The field of software adoption and use studies these questions across a broad range of tools and technologies. Software developers use models to help determine how new products will be created and if they will be profitable. For example, the technology acceptance model (TAM) created by Fred Davis focuses on users’ willingness to accept a new technology based on the factors of perceived usefulness of a tool and the ease of using that tool.11 Davis found both factors were significantly correlated with actual use, but that usefulness is somewhat more predictive of user behavior than ease of use, stating, “No amount of ease of use can compensate for a system that does not perform a useful function.”12 TAM has been widely tested and validated across a variety of fields and on a wide variety of technologies. A variant of TAM, called the unified theory of acceptance and use of technology (UTAUT) was created by Viswanath Venkatesh et al. to include the influence of other people on the technology adoption process.13

The UTAUT model was created to help managers make decisions about how employees would use new technologies. UTAUT is especially relevant for this study because it not only reflects the potential influence of other people, such as graduate advisors or research collaborators, on technology adoption, but it was developed alongside training programs designed to introduce the new technology. Technology adoption and acceptance was then measured at various intervals after the training intervention. UTAUT contains four direct determinants of user behavior: performance expectancy, effort expectancy, social influence, and facilitating conditions (see table 1 for determinant definitions). In addition, UTAUT includes four variables that moderate the degree of technology acceptance and behavior: previous technology experience, voluntariness, gender, and age.

Since its development, UTAUT has been used in a variety of contexts, including library and information science applications. Anders Avdic and Anders Eklund used UTAUT to help study why Swedish students do or do not use library databases.14 Sarah-Jane Saravani and Gaby Haddow looked at technology training needs of Australian and New Zealand library staffers using a qualitative adaptation of UTAUT.15 And Leila Khalili and Diljit Singh used UTAUT to study the acceptance of open access journals by researchers in Iran.16

These studies demonstrate that UTAUT has the potential to help explain technology software adoption and use questions within a library context. At this time, no one has explored bibliographic manager adoption and use through the lens of UTAUT. In addition, the data for the existing studies examining researchers’ use of and attitudes toward bibliographic managers has been gathered through surveys and interviews rather than through naturalistic observations of the actual research process. This exploratory observational study combined a naturalistic work-practice methodology, interviews and journal reflections within the guiding framework of UTAUT to help provide a better understanding of researchers’ bibliographic management adoption and use behaviors.

Based on the exploration of the literature and personal experiences conducting bibliographic management tool workshops, the objective for this study was to observe and explore how attendees of bibliographic management tool workshops choose and use bibliographic managers. More specifically, the study sought to gain insight into four research questions:

1. Do researchers adopt bibliographic managers based primarily on influences from their advisors or peers?
2. Do workshop participants continue to use the bibliographic manager after the workshop?
3. What makes workshop participants more or less likely to continue using the tool?
4. What resources do bibliographic manager workshop attendees use to troubleshoot their bibliographic manager questions?

The findings resulting from these guiding questions will be explored throughout the following sections.

Methods

Data Gathering

Study participants were recruited from the rosters of bibliographic management software workshops taught between December 2011 and July 2012 at OSU Libraries. Individual workshop sessions were offered for the bibliographic managers EndNote, EndNote Web, Mendeley, and Zotero and were open to anyone in the OSU community, including undergraduates, graduate students, staff, and faculty. Of the 128 past attendees emailed, 15 potential participants responded to the study recruitment; these respondents were screened to select those who were both using bibliographic management software and actively working on the literature review portion of their research. Screening took place in an initial interview where researchers gathered basic demographic information about the participant, explained the study’s purposes and activities and then, for consenting participants, described the processes of setting up screen-capture software and depositing study documents into a shared Dropbox account. After this initial screening process, one participant dropped out of the study without participating in any further study activities, leaving fourteen participants.

Because this study required a time commitment of four to five hours over two months, incentives for completion were offered. The ten participants who completed all of the study activities received $50 in gift certificates from a local department store or the Fandango movie ticket website. The four participants who completed a portion of the study activities received partial compensation corresponding to their level of activity.

Participants completed three types of tasks in the two-month study period (August through September 2012): screen captures, journaling, and an interview. Using one of two free screen capture tools (BB Flashback Express for PC users or QuickTime for Mac users) participants self-recorded their screens while carrying out research and writing activities that included at least some use of a bibliographic manager on their own computers and in a naturalistic research environment of their choosing. They collected recordings three times over the course of the study in one-hour sessions at least a week apart. Participants were directed to stop the screen recording during those one-hour sessions if they chose to begin any screen activities of a personal nature that they did not wish the study researchers to view. After each session, participants deposited the screen capture recordings in an invitation-only Dropbox folder shared only with the study researchers.

After each screen capture session, participants wrote journal responses to several prompting questions. To better understand the factors that would influence participants’ to continue using the tool, the journal prompts focused on participants’ experiences with the tools’ performance by asking them to describe how the tool either enhanced or detracted from their work during the recording sessions. In addition, to better understand what troubleshooting resources participants were likely to draw from, participants were asked how they dealt with any problems they may have encountered during each recording session. Participants were asked to spend approximately fifteen minutes writing the journal entry and were told they did not have to answer all the questions if it would take them more than fifteen minutes. Journal entry files were also deposited in the study’s invitation-only shared Dropbox folder.

At the end of two months, researchers conducted semi­structured interviews with each of the ten participants who had completed all of the preceding study activities. The interviews were thirty to forty-five minutes long and were audiorecorded. The interview questions were designed to address the research questions of this study. To learn about factors influencing tool adoption, each participant was asked why he or she adopted a particular tool and whether advisors or peers affected that choice. Participants were asked whether they would continue to use the tool and follow-up questions delved into what conditions might encourage or discourage continued use. Further questions explored what resources participants used to troubleshoot their bibliographic manager problems.

Of the fourteen participants in this study, ten participants completed the full study, which included an initial interview, three one-hour screen-recording sessions, followed by three journaling sessions, and a final interview. Four participants partially completed a range of the study activities but did not complete the final interview. Participant demographics are further described in table 2.

Data Analysis

Responses to the journal prompts and interview responses were analyzed by the study researchers using NVivo, specialized software for the analysis of qualitative data. The two study researchers used the UTAUT model as a guiding framework for analyzing the data.17 As noted in the literature review, UTAUT is used to predict adoption and use of new technologies and was chosen as a way to make sense of users’ behaviors in this study because it includes determinants based not only on performance and ease of use but also on the effect of social influences and facilitating conditions, which can include training opportunities.

For the purposes of this study, the study researchers modified the determinants slightly to accommodate the fact that the model was being used in an academic rather than a business setting. Venkatesh et al. define the Performance Expectancy determinant as “the degree to which an individual believes that using the system will help him/her to attain gains in job performance.”18 For this study, the definition was adjusted to read “research performance,” rather than “job performance.” The researchers also modified the example statements for the Facilitating Conditions determinant to include workshops and training. In addition, the determinants were modified to reflect the fact that participants were not predicting their tool use behaviors prior to actual use but that their active use of the tools was being observed and discussed. As a result, the determinants Performance Expectations and Effort Expectations were changed to Performance Expectations and Experiences and Effort Expectations and Experiences, respectively (see table 1).

Limitations

Because this was an exploratory, qualitative study based on a sample of participants who attended in-person bibliographic manager workshops, the responses may not apply to all user groups, and is not intended as a comparison between researchers who attend workshops and those who do not. Participants do not evenly represent all bibliographic management tools or all disciplinary, gender, or status groups. This type of study is not designed to be generalizable, and the findings are not intended to demonstrate differences in adoption and use between specific tools or to compare different groups to each other. Rather, because so little is currently known about how researchers adopt and use bibliographic management tools, the results are intended to help build a baseline understanding to which further studies can contribute and expand.

An additional limitation of the study is that the authors are also involved in teaching bibliographic management tool use at OSU Libraries. The authors were involved in the interview process and consequently, participants may have tempered some of their responses because of their previous encounters with the authors in the classroom or because of their expectations of future encounters.

Findings and Analysis

Influences on Adoption

The first research question asked if researchers adopt bibliographic managers based primarily on influences from their advisors or peers. Within the UTAUT model the role this type of influence plays falls in the Social Influence determinant. One of the primary reasons the UTAUT model was chosen for this study was because unlike several other models, it includes the impact of social influence, such as the influence of a mentor, advisor, or peers, on tool adoption and use. At the outset of the study, the initial hypothesis was that social influence would positively predict tool selection and use. However, only forty percent of the ten participants who fully completed our study indicated their advisors or supervisors used any type of bibliographic manager at all; only twenty percent of this participant pool indicated their advisors or supervisors used the same bibliographic manager they did, and as a result had some influence over the participants’ use of the tool (see table 3). Peer usage of bibliographic managers was somewhat higher at sixty percent.

While not all of the participants were graduate students, we assumed that advisors or mentors would influence bibliographic management tool adoption and use for those participants who were graduate students. Instead we observed that graduate students were given a large amount of free choice in their bibliographic manager decisions. In addition, graduate student participants in this study were infrequently mentored as to how to choose a bibliographic management tool or any organizational system at all. Participant 8, a graduate student, explained, “I think the expectation is just that you’ll know how to organize any online sources that you use; like by yourself you’ll just figure it out.”

In contrast, junior faculty or faculty researchers in settings where a lead supervisor expressed a bibliographic manager preference used the supervisor’s tool, regardless of their own preferences or previous experiences with other bibliographic management tools. For example, both participants 2 and 4 used EndNote at some point in their career to align their bibliographic management practices with the specific tool used by their supervisors, and because they found it much simpler to use the same bibliographic manager to share resources with a research team that was using the same tool.

These findings mesh with those of Venkatesh et al. who found that social influence was only significant in “mandatory settings” where rewards or punishments could be given and the moderating variable of “voluntariness” was limited.19 In this study when supervisors dictated or strongly suggested the use of a particular tool, participants adopted that tool. However, when clear direction was absent, as was the case for most of the graduate students, participants chose a tool based on other factors.

The Social Influence determinant as defined in the UTAUT model goes beyond the influence of a single mentor or peer interaction and also includes the social impact of the larger institution. At the time of this study, OSU provided 500 MB of free storage space to Zotero users (this has now increased to 1 GB), and some labs purchase copies of EndNote for their students, thereby facilitating a cost-free experience while students or faculty are on campus. Institutional support in the form of covering the cost of the tools or free options was important to some participants. As Participant 5 noted, “I found that Mendeley is my favorite one. I think the…reason is that it is free.” Some student participants were worried about how they would access bibliographic management software once they graduated. Another institutional-level barrier occurred for Participant 4 when she transitioned to a state government job where access to open source tools and the ability to load programs onto her own computer was denied. The result was that she needed to switch tools to a workplace-sanctioned choice. Institution-level barriers may impede adoption or continued use for some researchers.

Based on participants’ responses, the UTAUT determinant Facilitating Conditions had a relatively small impact on participants’ adoption and continued use of the tools. However, Facilitating Conditions is the UTAUT determinant that falls most directly within the library’s realm of influence as it contains the ideas of training support and troubleshooting. In our context, training opportunities in the form of workshops served not only as an opportunity to learn how to use the tools, but also as a prompt for participants to learn that such tools existed via workshop promotional venues like the library’s website. When participants are initially considering adopting tools, having appropriate Facilitating Conditions like knowing that workshops are available can be an important factor. For several participants this training impacted their tool adoption behaviors and made them willing to use the tools more. For example, Participant 5 noted that he did not initially know anything about such tools, “but when I finished the workshops, I found them pretty useful.” Similarly, Participant 8 stated, “Well, I think the first thing that drew me to it [using the tool] was taking your workshop in Zotero.” Moreover, Participant 2 noted, “When I first starting using EndNote, the intro class that they had here at the library; that was very helpful.”

But the long-term impact of Facilitating Conditions, especially in-person training opportunities, diminishes for participants who have already adopted a particular tool the longer they use that tool. Venkatesh et al. found that the influence of the Facilitating Conditions determinant, particularly in terms of support available, declines as time after training increases, as long as the tool continues to be perceived positively in terms of the determinants Effort Expectations and Performance Expectations. As a result, librarians will likely see higher rates of participation in introductory-level workshops than in advanced-level workshops, once participants have already adopted a tool.

Impacts on Continued Use

The second and third research questions asked if bibliographic manager workshop participants continue to use the bibliographic management tool after the study, and if so what impacts their willingness to continue using the tool. Eight out of the ten final participants indicated they would continue using the tool (see table 3). In addition, participants demonstrated their transition from intending to use the tools to actual use of the tools over the course of the study as six out of the ten final participants ranked themselves as more proficient users of the tool than when they began the study. However, it should be noted that three participants were already fairly comfortable using the tools, and as a result their understanding of how to use the tools remained stable during the study period (see table 3).

In Venkatesh et al.’s model of user acceptance and usage behavior, the determinant Performance Expectancy was found to be the strongest predictor of usage intention.20 In this study, analysis showed that the Performance Expectations and Experiences determinant also had a major impact on participants’ usage behavior. Participants incorporated the bibliographic management tools into existing workflows and were able to devise new ways of working with the tools. Participant 1 commented, “I feel like it’s already a part of everyday use. I mean I haven’t really been writing today, and I have bookmarked a couple things and brought them into Zotero.” In particular, participants discussed enhanced methods of working electronically as compared to working with paper, such as annotating PDFs, searching their libraries and creating tags. Participant 9 was particularly enterprising in creating a variety of supporting resources around his primary source. He took screenshots of figures of the articles he was reading to help him better emphasize significant elements of the paper and to view multiple figures at one time, and he then saved all of these elements together in his Zotero library.

Another positive productivity element is the ability to work more quickly or efficiently. Saving time is a key feature of the Performance Expectancy determinant, and participants observed that the bibliographic management tools could help them avoid duplicate effort and manage their work in fewer steps. Participant 3 stated, “All in all, Zotero saved me a lot of time in terms of saving all these images where I could find them again with the bibliographic information I needed.” Saving time on tasks like bibliographic management has the potential to allow researchers to be more efficient in their work, and as a result, spend more time on other aspects of the research process.

On the other hand, some participants did not experience the gains in productivity they had expected from using a bibliographic manager. Participant 15 felt distracted by the availability of the various sorting, organizing and tagging features that for her were simply a new variant of procrastination techniques that prevented her from prioritizing more important tasks. Participant 5 acknowledged this tension between wanting an easier workflow and continuing to put forth the necessary effort himself, reflecting that at some point he realized “I cannot totally depend on the software.”

For participants trying to transition to a new workflow, frustrations often came from a lack of understanding of their particular bibliographic management tool. For example, Participant 3 had not learned how to create a stand-alone bibliography using Zotero, and Participant 2 had not learned how to limit her EndNote search in the database PubMed. Such issues prevented these participants from using the bibliographic managers as successfully as they might have. Neither participant sought assistance for their questions, either online or in person, instead assuming a resigned attitude toward the tools. For example, Participant 3 decided “for that particular thing, Zotero wouldn’t work.” However, some participants found legitimate drawbacks to the tools and described difficulty with scanned papers that do not yield citation data, or the inability to easily incorporate websites into an EndNote library.

Technological glitches plagued several participants at various points in their interaction with the bibliographic managers and detracted from their potential performance experiences with their chosen tools. Two participants noted that their bibliographic managers, Zotero and EndNote, did not automatically save as they had expected. Three participants complained of inconsistent and mysterious access to PDFs. One participant accidentally deleted her library; and one participant did not get a needed update. In addition, one Zotero user noted the de-duping option did not always work as expected. Some problems were related to user error and some were related to problems with the software itself.

Participants’ larger technological problems arose from the interaction of the bibliographic manager with another system, such as off-campus access to library databases, PDFs received from interlibrary loan without OCRed text (optical character recognition), or the use of unfamiliar computers or operating systems. Because bibliographic managers are not used in a vacuum as standalone entities, they require a certain level of knowledge about the larger online information environment. Without baseline knowledge of how to interact with databases, interlibrary loan, and proxy servers, researchers may not be able to make it over the hurdle to use a bibliographic manager to enhance their scholarly productivity.

One final performance barrier arose as participants experienced a range of frustrations related to citations. The output of the bibliographic managers was sometimes inconsistent resulting in “garbage citation information,” incorrect metadata, words in all capital letters, difficulty working with foreign languages, or struggling to find the required citation style. The quality of citations and bibliographies generated by bibliographic managers has been documented as researchers have conducted comparisons to determine which tools create superior bibliographies. However, each of the tools examined still produced some errors.21 Unfortunately, while a certain level of errors may be acceptable for software developers and many users, some users will likely identify with Participant 4, who noted, “At times it may have been faster to write in the citations myself.”

The Performance Expectancy and Experiences determinant was an important consideration for most participants, who kept using the tools even when they encountered some technical glitches. However, Performance Expectancy and Experience issues were referred to with much less enthusiasm than issues related to Effort Expectancy and Experiences. Responses in the interviews and journal prompts indicated that Effort Expectations and Experiences played a larger role than expected in influencing participants’ continued use of bibliographic managers. As Participant 1 noted, “I think for some people, and maybe I’m one of those people, it almost has to be that easy. In order to really stick with doing it.” Likewise, Participant 8 commented, “Oh, this is a lot easier than I thought, and I just kept using Zotero.” This willingness to keep using a tool because of the ease of use is notable because in the TAM model, Davis found that Performance Expectancy played the larger role for predicting continued use.22

The Effort Expectations and Experiences determinant emphasizes users’ ability to easily learn, understand, and use a tool or technology; and in this study participants noted these characteristics especially in terms of transparency of use and portability. For example, Participant 3 pointed out how easy it was for her to understand how to add images to her Zotero library. Because web-based bibliographic managers can be used from any computer, or even from mobile devices, they become easily portable tools. Participant 5 noted that he could use the Mendeley iPhone app to “read and edit even when I’m in bed.”

For users still in the adoption phase, ease of use was an important factor as they evaluated features of the various tools and then discarded tools that did not match their expectations. For example, Participant 1 was still exploring a variety of tools and felt that “Mendeley’s interface was not as intuitive as Zotero’s.” And Participant 2 commented that “EndNote is bulky and complex; it did not work the way I wanted it to work.” This prioritization of Effort over Performance may be because of the relatively large range of bibliographic management tools available, which make Performance features like Cite While You Write feel commonplace rather than extraordinary.

In contrast to the undergraduate students in Salem and Fehrmann’s study, participants in this study, who were primarily graduate students, faculty, or staff who had already obtained an advanced degree, more readily determined that the return on investment for learning how to use a bibliographic manager was high enough to make the effort worthwhile.23 Once they began using a bibliographic manager, these participants were likely to keep using the tool. It should be noted that a key to this transition to actual tool use is regular practice and sustained use of the tool. The study design, which required participants to spend three hours working with a bibliographic manager, could certainly have impacted participants’ usage behaviors. However, being immersed in the regular activities of carrying out a literature review, as these participants were, should be another contributing factor to sustained use.

Troubleshooting Preferences

The fourth research question asked what resources do bibliographic manager workshop attendees use to troubleshoot their bibliographic manager questions. External inputs, such as workshop training, are one facet of the UTAUT determinant Facilitating Conditions. Another important component of Facilitating Conditions in this study was participants’ ability to use a range of personal resources such as the ability to troubleshoot problems or find support. Unfortunately, participants in this study did not demonstrate a particularly wide range of troubleshooting strategies. Only one participant indicated the option of consulting with librarians on bibliographic management tool questions, and instead participants were much more likely to rely on a Google search to solve specific problems. Participant 4 commented, “Just Googling it is the easiest thing. You don’t want to have to call somebody with ‘this is wrong’ unless it’s a big problem.”

The ability to explore and troubleshoot was a strong indicator of continued tool use. Those participants who were able to navigate the trial-and-error environment of learning new tools and possessed a level of self-efficacy, or trust in their abilities to solve problems, were more resilient and confident in their ability to keep learning about their particular bibliographic manager. Participant 1 reflects this attitude well as she described herself as “very much a hands-on figure-it-out yourself kind of learner.” But for some participants a lack of confidence in their problem solving abilities or the inability to allocate enough personal resources to solving problems held them back. For example, time allocation was a concern for Participant 3 who noted, “I tried it repeatedly, got frustrated, and then quit and did something else, thinking ‘well, when I have more time, I’ll come back and learn how to do this’.” Lack of technology facility was another personal resource constraint. Participant 13 described her lack of technology skills and her resulting reliance on outside expertise this way, “Sometimes I will go actually through the tabs—go through you know the edit, the references and try to problem solve it myself. But I almost never solve it myself, because there’s so many choices, and I say ‘Crap, that’s why I paid all this money for tech support, I’m going to wait and call them, I’ve already spent x amount of time’.”

Unfortunately, finding ways to troubleshoot specific problems was not always as straightforward as it could have been. Two EndNote users found the available documentation particularly frustrating to use: Participant 2 felt so much EndNote documentation was available that it was overwhelming, and Participant 13 found the in-person technology support to be unhelpful.

While librarians have created a variety of in-depth tutorials to support learners in their adoption and use of bibliographic management tools, participants did not view step-by-step tutorials as a desirable solution for finding help because of their perceived inability to directly answer troubleshooting questions. As Participant 15 commented, “Unless I’m like just learning how to use a product, and then tutorials, that’s usually where those come in a little more.” Participants were much more willing to draw on the impersonal support provided by the bibliographic managers’ online forums, as six of the participants mentioned having used forums as a way to find solutions to their problems.

Roles for Librarians

While the library’s role in providing Facilitating Conditions in the form of workshops and research consultations remains the most clear, the library can also influence users’ Performance Expectations by promoting knowledge of the performance and productivity enhancing features of bibliographic management tools. Venkatesh et al. found that the influence of the Facilitating Conditions determinant, particularly in terms of support available, declines as time after training increases, as long as the tool continues to be perceived positively in terms of Effort Expectations and Performance Expectations. Promoting the tools’ ease of use and productivity enhancing features can happen not just in bibliographic manager-specific workshops, but also in research consultations, reference desk interactions, and course-based instruction.

Similar to Emanuel’s recommendations, this study also demonstrated the value in offering workshops in a variety of bibliographic management tools to introduce users to and support users in their various tool needs.24 Not all researchers voluntarily select their bibliographic management tools, and having support available for those researchers who need more assistance adapting to a range of new tools can serve a valuable instructional function.

While Emanuel also recommended using the library’s web guides to support researchers during the tool selection phase, the participants in this study were not looking for online support during the adoption phase. Instead, these participants were primarily looking for online support in how to troubleshoot specific problems that arose with the tools. Participants clearly preferred on-demand responses to specific questions in an anonymous setting, such as an online forum. Online tutorials that walked through the process of learning a tool were not highly valued by participants, as they had already received this type of training and were at the point of needing more targeted assistance for specific problems. For tools like EndNote, Mendeley, and Zotero that have tutorials available on their websites, perhaps a better use of librarians’ time in supporting researchers would be to help answer questions on the online forums rather than to create their own versions of bibliographic manager tutorials. Alternatively, librarians might consider building easily searchable and findable FAQs for the tools that do not have these resources to support users with specific bibliographic manager questions.

Despite the lack of use during the study period of librarian consultation services (Participants 3 and 13 did consult with librarians on bibliographic manager questions outside of the study period), the option of consulting with a librarian in a one-on-one reference appointment continues to be a valuable, high-touch service for some users. Use of consultation services may be driven by a variety of moderating variables, including previous tool experience, voluntariness of tool use, and age. Librarians should continue to support researchers via research consultations, but may consider evaluating the factors that drive users to make use of these high-touch services as opposed to workshops or online support options.

Finally, one of the underlying goals of this research study was to help inform the promotion and delivery of bibliographic management tool workshops at OSU Libraries. While the study was not designed to compare researchers from different disciplines to one another, the data gathered did reveal some preliminary data about needs and approaches to using these tools that were different between disciplines. More importantly, the data revealed some small differences between the participating researchers’ approaches and the library instructors’ approaches to using the tools. Observing these differences has resulted in some changes to how bibliographic manager workshops are delivered. For example, instruction on how to create standalone bibliographies in addition to bibliographies generated from in-text citations is now provided. Also, a range of approaches to note taking and attaching files, such as images or figures, are discussed to reflect differences in how disciplines rely on various information gathering processes and source materials. Librarians at other institutions might be well served to push beyond their traditional disciplinary framework to observe other techniques and research approaches.

Conclusion

This exploratory, qualitative research study of bibliographic manager workshop attendees’ adoption and use behaviors provided a greater understanding of participants’ own views of these tools by observing how participants used the tools and through interviews that delved into the factors that influenced their adoption and usage behaviors. The lens of the UTAUT model provided a nuanced view of what influences researchers’ willingness to adopt and continue using the tools. This study demonstrated that bibliographic manager adoption is not as strongly influenced by mentors or peers as expected. Participants were more strongly influenced to adopt a tool by their own perceptions that it would be easy to use and by their expectations that using such a tool would increase their productivity. The promise of the tools’ ease of use, in combination with specific performance gains, made the return on investment of learning how to use a bibliographic manager worth the effort for participants and led to adoption of a bibliographic manager into the research workflow along with continued use of the tool.

Providing training opportunities enabled participants to start using the tools and served to promote many of the productivity options of the bibliographic management tools that were previously unknown to the participants. While attending bibliographic management workshops did not increase study participants’ preference for in-person assistance at the library, the use of online forums was a valued method of receiving assistance. Librarians should be encouraged to participate in these online forums as a way to reach out to users with troubleshooting needs. Workshop trainings serve an important function in the process of bibliographic manager adoption but play a diminishing role in influencing users’ continued use of the tools.

Future studies could explore whether disciplinary affiliation or academic standing has an effect on tool adoption and use. In addition, this study could be expanded to more users to compare the rates of adoption and use of specific bibliographic management tools. Finally, future research could explore whether the simple act of requiring or suggesting regular, sustained use of the tool, such as using it three times a month after attending a workshop, leads to increased long-term tool use.

Acknowledgement

The authors would like to thank the Robert Lundeen Faculty Development Award for supporting this research study.

Referencess

1. Jon Ritterbush, “Supporting Library Research with LibX and Zotero: Two Open Source Firefox Extensions,” Journal of Web Librarianship 1, no. 3 (July 2007): 111–22, http://dx.doi.org/10.1300/J502v1n03-08; Holt Zaugg et al., “Mendeley: Creating Communities of Scholarly Inquiry through Research Collaboration,” TechTrends 55, no. 1 (January 2011): 32–36, http://dx.doi.org/10.1007/s11528-011-0467-y.
2. Ron Gilmour and Laura Cobus-Kuo, “Reference Management Software: A Comparative Analysis of Four Products,” Issues in Science and Technology Librarianship 66, no. Summer (2011), http://dx.doi.org/10.5062/F4Z60KZF; Yingting Zhang, “Comparison of Select Reference Management Tools,” Medical Reference Services Quarterly 31, no. 1 (March 2012): 45–60, http://dx.doi.org/10.1080/02763869.2012.641841; Michael Steeleworthy and Pauline Theresa Dewan, “Web-Based Citation Management Systems: Which One Is Best?,” Partnership: The Canadian Journal of Library and Information Practice and Research 8, no. 1 (2013), http://gir.uoguelph.ca/index.php/perj/article/view/2220; Laura Pope Robbins, “Mendeley or Zotero: Which Should the Mobile Researcher Use?,” The Charleston Advisor 14, no. 2 (October 2012): 5–11, http://dx.doi.org/10.5260/chara.14.2.5.
3. M. Kathleen Kern and Merinda Kaye Hensley, “Citation Management Software: Features and Futures,” Reference & User Services Quarterly 50, no. 3 (Spring 2011): 204–8; Khue Duong, “Rolling Out Zotero Across Campus as a Part of a Science Librarian’s Outreach Efforts,” Science & Technology Libraries 29, no. 4 (October 2010): 315–24, http://dx.doi.org/10.1080/0194262X.2010.523309; Megan Fitzgibbons and Deborah Meert, “Are Bibliographic Management Software Search Interfaces Reliable? A Comparison between Search Results Obtained Using Database Interfaces and the End Note Online Search Function,” Journal of Academic Librarianship 36, no. 2 (March 2010): 144–50; John Chenault, “From Citation Management to Knowledge Management: Developing and Implementing Innovative Endnote Training and Support Services on the Health Sciences Campus,” Kentucky Libraries 75, no. 1 (Winter 2011): 6–11; Dawn Childress, “Citation Tools in Academic Libraries,” Reference & User Services Quarterly 51, no. 2 (December 2011): 143–152, http://dx.doi.org/10.5860/rusq.51n2.143.
4. H. Stephen McMinn, “Library Support of Bibliographic Management Tools: A Review,” Reference Services Review 39, no. 2 (2011): 278–302.
5. Jamie Salem and Paul Fehrmann, “Bibliographic Management Software: A Focus Group Study of the Preferences and Practices of Undergraduate Students,” Public Services Quarterly 9, no. 2 (2013): 110–20, http://dx.doi.org/10.1080/15228959.2013.785878.
6. Jenny Emanuel, “Users and Citation Management Tools: Use and Support,” Reference Services Review 41, no. 4 (November 2013): 639–59.
7. University of Minnesota Libraries, University of Minnesota Digital Conservancy: A Multi-Dimensional Framework for Academic Support: Final Report (Minneapolis: University of Minnesota Libraries, 2006), http://conservancy.umn.edu/handle/5540.
8. Marni R. Harrington, “Information Literacy and Research-Intensive Graduate Students: Enhancing the Role of Research Librarians,” Behavioral & Social Sciences Librarian 28, no. 4 (December 2009): 179–201, http://dx.doi.org/10.1080/01639260903272778.
9. Kirsty Williamson et al., “Research Students in the Electronic Age: Impacts of Changing Information Behavior on Information Literacy Needs,” Communications in Information Literacy 1, no. 2 (August 2008): 47–63, http://dx.doi.org/10.7548/cil.v1i2.21.
10. Emanuel, “Users and Citation Management Tools.”
11. Fred D. Davis, “Perceived Usefulness, Perceived Ease of Use, and User Acceptance of Information Technology,” MIS Quarterly 13, no. 3 (September 1989): 319–340, http://dx.doi.org/10.2307/249008.
12. Ibid., 333–34.
13. Viswanath Venkatesh et al., “User Acceptance of Information Technology: Toward a Unified View,” MIS Quarterly 27, no. 3 (September 2003): 425–78.
14. Anders Avdic and Anders Eklund, “Searching Reference Databases: What Students Experience and What Teachers Believe That Students Experience,” Journal of Librarianship and Information Science 42, no. 4 (December 2010): 224–35, http://dx.doi.org/10.1177/0961000610380119.
15. Sarah-Jane Saravani and Gaby Haddow, “The Mobile Library and Staff Preparedness: Exploring Staff Competencies Using the Unified Theory of Acceptance and Use of Technology Model,” Australian Academic & Research Libraries 42, no. 3 (September 2011): 179–90.
16. Leila Khalili and Diljit Singh, “Factors Influencing Acceptance of Open Access Publishing among Medical Researchers in Iran,” Libri 62, no. 4 (December 2012): 336–54, http://dx.doi.org/10.1515/libri-2012-0026.
17. Venkatesh et al., “User Acceptance of Information Technology.”
18. Viswanath Venkatesh, “Technology Acceptance Models,” accessed January 17, 2014, www.vvenkatesh.com/it/organizations/Theoretical_Models.asp#Con=structdefs.
19. Venkatesh et al., “User Acceptance of Information Technology.”
20. Ibid.
21. Gilmour and Cobus-Kuo, “Reference Management Software.”
22. Davis, “Perceived Usefulness, Perceived Ease of Use, and User Acceptance of Information Technology.”
23. Salem and Fehrmann, “Bibliographic Management Software.”
24. Emanuel, “Users and Citation Management Tools.”