A Model for Developing and Implementing a Systematic Review Service for Disciplines outside of the Health Sciences

Amy Riegelman and Megan Kocher

Amy Riegelman (aspringe@umn.edu) is Social Sciences Librarian for Government Information, Psychology, and Institute of Child Development at the University of Minnesota. Megan Kocher (mkocher@umn.edu) is Science Librarian for Food Science and Nutrition, Animal Science, and Soil Water and Climate at the University of Minnesota.

Correspondence concerning this column should be addressed to M. Kathleen Kern, RUSQ Editor, email: rusqeditor@gmail.com.

Support for systematic reviews and meta-analyses in the social sciences is an innovative service that makes advanced use of the expert skills of reference librarians and subject specialists. This column provides a deep look into the launch of one systematic review service to provide a model that is adaptable for other academic and special libraries.—Editor

There is a growing need for academic libraries to support systematic review research. Currently the library literature does not represent needs outside of the health sciences. This article is a descriptive narrative of a systematic review service development and launch. The described service model supports the needs of several disciplines including social sciences, agriculture, physical sciences, and other disciplines. Primary foci of the article include direction from library administration, service development and launch, and plans for assessment and evaluation.

A systematic review includes conducting a systematic and exhaustive search, assessing quality, and synthesizing evidence.1 Librarian involvement in systematic reviews has been documented in the library literature and referenced by authoritative organizations like Cochrane Review Group and the Campbell Collaboration.2

In some disciplines there is an increased interest in systematic reviews and meta-analyses as research methods, and this is evident in literature output discoverable in subject databases. Since both systematic reviews and meta-analyses are typically designated in article titles, it is fairly easy to track their prominence and growing interest as a research method over the last five years. According to searches for “systematic review*” OR “meta-analys*” conducted in CAB Abstracts, PsycINFO, Education Source, and Sociological Abstracts, each disciplinary database saw a jump in published systematic reviews and meta-analyses in the last five years (see figure 1).

At the University of Minnesota, subject liaison librarians experienced an uptick in the number of requests from faculty and students for assistance with the search methodology required for systematic reviews and meta-analyses. Some of the librarians had previously received training on this level of research synthesis while others felt ill equipped. It became clear that serving research synthesis needs should be formalized as a service and that library staff needed training specific to supporting systematic reviews.

Charge from Library Administration

A team of liaison librarians serving the agricultural and social sciences approached library administration about the need to develop services for researchers outside the health sciences working on systematic reviews, meta-analyses, and other forms of evidence synthesis. The administration charged a group to:

• collect and evaluate evidence from existing systematic review services (e.g., our bio-medical library’s existing service and external services) and other resources to inform service model development;
• determine and document competencies for service team members, and propose training for existing and/or potential new team members;
• document service level(s) and librarian staff/support needs and propose a model to manage and support the service;
• develop a communications plan that includes web-based documentation for target audiences as well as marketing and advertising strategies; and
• develop and track indicators of service success.

The group of five subject liaison librarians (two in agricultural disciplines and three in social sciences) and two support staff was given eight months to develop a model and propose a service launch to administration for approval.

Service Development

Developing our systematic review service depended on gaining knowledge generally about systematic reviews and learning from systematic review services that had been developed by medical libraries. This work included reading the library science literature, formal training, and informal discussions with librarians at Cornell University and other institutions.

The group scanned the library science literature to make informed decisions about service development. A goal of the literature scan was to gather literature related to libraries adopting a systematic review service. This search and discovery revealed that services in existence prioritized supporting medical disciplines, and support of non-medical systematic reviews occurred on an ad hoc basis.

Themes emerged regarding a higher rate of reproducibility when librarians served on systematic review teams.3 Other articles and conference proceedings included information on various library roles within conducting a systematic review and how roles may vary depending on patron type (e.g., faculty vs. student).4 Some institutions had well-established services that were integrated into the curriculum with the health sciences.5 Ludeman et al. wrote a case study about launching systematic review services in academic medical libraries.6 This article shared useful information regarding staff training, guidelines, web page development, and service development.

The group concluded that select staff needed formal on-site training, and different training options were explored. We decided that the best option given our needs was to bring in an in-person trainer. Margaret Foster from Texas A&M University was invited to lead training based on her experience on systematic review teams, publications, and experience leading training programs.7 In April 2017, Foster led a two-day training for the systematic review team and other library staff. The training was required for group members, but it was open to other library staff as well. There were eighteen total participants in the training. In addition to the in-person training, Foster provided us with her training materials and resources to refer back to as needed.

Beyond the more technical aspects of conducting systematic reviews, group members also identified a need for expert search training in various databases. The definition of expert searching from the Medical Library Association is multifaceted, and we chose to focus on the aspects identified in table 1.

To meet the need for expert searching skills, we developed a training model that benefited our group as well as others in our library system. This training was branded as Expert Search Camp, and five sessions were developed, with each session being led by subject liaison librarians or other expert searchers in their respective fields. Attendees included the systematic review group and other library staff. The sessions focused on the following: (1) education databases, specifically ERIC and Education Source; (2) psychology databases, specifically PsycINFO via Ovid; (3) agriculture databases, specifically CAB Abstracts and Agricola; (4) grey literature discovery; and (5) medical databases including Ovid Medline, Embase, and the Cochrane Library.

Service Model

Most subject liaison librarians outside the health sciences do not receive formal training in working on systematic reviews. The members of our group had varying levels of hands-on experience with systematic reviews outside of the training and Expert Search Camps we had arranged, with the majority of members never having been provided any formal support for this type of research. Because of this, we determined that the best model for our service would include ongoing training and peer support, using the expertise of some members of the group while building the skills of those with less or no experience.

We devised a unique approach for our systematic review service model wherein the initial planning group now constituted the Systematic Review Service; in our model, each systematic review request that comes to the service is assigned by the Systematic Review Service cochairs to a team. This team is comprised of the following:

• Lead: This is an experienced member of the systematic review group. This person leads communications for the team, directs the intake process, and provides expertise in systematic review methods.
• Subject Liaison: Liaison librarians are included in systematic reviews for their departments regardless of whether they are on the systematic review group. They provide subject knowledge and disciplinary database expertise and foster relationships with researchers.
• Reviewer: The Reviewer is consulted to provide peer review for a search strategy once the Lead and Subject Liaison have drafted it. While a search strategy reviewer is not required for a systematic review, we chose to build this into our service as a way to both ensure quality and learn from each other.
• Apprentice: As a way of training less experienced group members, an apprentice assists the Lead throughout the entire process and helps develop the search strategy. Once all members of the systematic review team have experience working on a full systematic review and feel comfortable in a Lead role, there are no apprentices.

Depending on whether the subject liaison is also on the systematic review group, each team has two to three members. The varying levels of commitment required by team members enable us to share and build expertise within the group without overextending our capacity. We also find this to be a very sustainable model as new members can rotate on to the systematic review group and have a built-in device for skill development through the apprentice role. It also means that not every subject liaison librarian serving departments outside of the health sciences needs to have formal systematic review training (a considerable undertaking for a large institution) to help support systematic reviews in their disciplines.

A flowchart (figure 2) was then created to be a visual map of the process and help acclimate team members to the new service model. It was helpful in reporting on the progress of our service development at the director level, and it was useful for team members becoming acclimated to the newly launched service model.

To accompany the service, a guidelines document (appendix A) was created to help group members and service users understand expectations including co-authorship. This document was intended to be featured on our service’s web page and to be used in conversations with service applicants. The document explains the potential roles of librarians in serving on a systematic review team. It cites evidence of working with a librarian enhancing quality and reproducibility.8 Librarian co-authorship is described in terms of “substantial contributions,” which is language used by the New England Journal of Medicine.9

Service Launch and Promotion

Once the service was approved and ready to launch, we identified two key audiences for communications and developed a strategy to reach both of them. The first audience was internal—our library’s staff. We decided that it was crucial that this audience be our first priority so that everyone in our library would be aware of the service and able to answer basic questions about it (and systematic reviews generally) if questions arose at a service point or within departments. We also needed all liaison librarians to understand their role in our service model and answer any questions they might have. With this in mind, our internal communication plan consisted of a newsletter announcement and emails to all librarians and library staff to establish basic awareness. More in-depth awareness came from “Roadshows” where co-chairs of the group visited staff meetings in each department and presented data about the numbers of systematic reviews being performed in that department’s subject areas, described systematic review methodology and the need for librarian involvement in the process, explained the team model, and answered questions.

The second prong of our strategy was the external communications plan to reach faculty and researchers. We were cautious in developing this plan to take a gradual approach so that the service did not get an overwhelming response at the outset and we were able to test out our service model and its capacity over a period of months. Systematic reviews are time- and labor-intensive projects. Depending on the scope of the research and needs of the researchers, librarian involvement can range from a single consultation to in-depth collaboration over a period of months or even years. With that in mind, over the course of one semester we launched our webpage, provided liaisons with editable descriptive language announcing the launch of our service, and advertised in a campus-wide newsletter to all faculty, staff, and students. Finally, we took a more targeted approach. We used our institution’s subscriptions to Elsevier’s Pure database and Clarivate’s Web of Science to compile a list of faculty at our institution who had published a systematic review or meta-analysis in the last five years. Using this list, we sent individual emails to each of these researchers announcing the service and explaining the roles librarians could play on a systematic review team.

Each wave of communications in this section yielded new applicants to our service, for a total of nine by the end of our first semester of service in December 2017. Our ongoing strategy for communicating about our service to faculty and researchers includes promotion in emails sent at the beginning of each semester, a case study presented on our library’s news webpage, and featuring the service occasionally on our library home page.

Statistics, Assessment, and Evaluation

As a new service, it was important to build in mechanisms for evaluation and assessment. Our service uses Google Forms to collect service requests and track their progress, which allows us to easily keep metrics on numbers of requests over time, subject/department affiliations, requestor status (e.g., faculty vs. graduate student), level of service needed, team member participation, and time to complete. In addition to this, we incorporated a time tracking protocol in which each request is assigned a unique hashtag that team members use when logging time spent in Desk Tracker software. A feedback mechanism is also built into our service model so that one of our support staff follows up with researchers to administer a feedback survey after the completion of a project. Team members are also asked to provide feedback separately. Both sets of feedback are used for ongoing evaluation and improvement of our service.

Astrid Schmeid, a PhD student in the Educational Psychology department at the University of Minnesota, submitted these comments about the service: “I requested the Systematic Review Service at the Library to refine a project on the non-medical use of brain stimulants in educational systems. While I had some background in PRISMA protocols, they provided deeper guidance about standards for good research practices in systemic review processes. I strongly recommend using this Service; it provides support and adds professionalism to your research products.”

Conclusion

The future development of the service will depend on statistics gathered and support from library administration as well as outcomes from our service assessments. Because few models exist for supporting evidence synthesis across a wide range of disciplines, we believe that the model we have developed and outlined here can be useful to other libraries as they work to meet their researchers’ needs in this area. Although we developed this model to serve a large research university, it could be scaled to meet the needs of other institutions regardless of size or type. At a smaller institution, for example, all liaisons may comprise such a group and support each other in distributing work using the team approach. Degree offerings of various institutions also may determine which systematic review skill trainings are necessary and which databases require expert search training.

References

1. Maria J. Grant and Andrew Booth, “A Typology of Reviews: An Analysis of 14 Review Types and Associated Methodologies,” Health Information and Libraries Journal 26, no. 2 (2009): 91–108, https://doi.org/10.1111/j.1471-1842.2009.00848.x.
2. Jonathan B. Koffel, “Use of Recommended Search Strategies in Systematic Reviews and the Impact of Librarian Involvement: A Cross-Sectional Survey of Recent Authors,” PLOS ONE 10, no. 5 (2015), https://doi.org/10.1371/journal.pone.0125931; Genevieve C. Gore and Julie Jones, “Systematic Reviews and Librarians: A Primer for Managers,” Partnership: The Canadian Journal of Library and Information Practice and Research 10, no.1 (2015), https://doi.org/10.21083/partnership.v10i1.3343; Carol Lefebvre, Eric Manheimer, and Julie Glanville, “Searching for Studies,” in Cochrane Handbook for Systematic Reviews of Interventions, edited by Julian P. Higgins and Sally Green (Chichester, UK: Wiley, 2008): 95–150, https://doi.org/10.1002/9780470712184.ch6; “Producing a Review,” The Campbell Collaboration, accessed February 28, 2018, http://archive.campbellcollaboration.org/systematic_reviews/categoryPrinterPage.shtml.
3. Jonathan B. Koffel and Melissa L. Rethlefsen, “Reproducibility of Search Strategies Is Poor in Systematic Reviews Published in High-Impact Pediatrics, Cardiology and Surgery Journals: A Cross-Sectional Study,” PLOS ONE 11, no. 9 (2016), https://doi.org/10.1371/journal.pone.0163309.
4. Robin MN Parker and Maggie J. Neilson, “Lost in Translation: Supporting Learners to Search Comprehensively across Databases,” Journal of the Canadian Health Libraries Association/Journal de l’Association des bibliothèques de la santé du Canada 36, no. 2 (2015): 54–58.
5. Misa Mi, “Leveraging Research Synthesis for Promoting and Expanding Library Services and Educational Programs,” The Journal of Academic Librarianship 42, no. 2 (2016): 151–53.
6. Emilie Ludeman et al., “Developing a Library Systematic Review Service: A Case Study,” Medical Reference Services Quarterly 34, no. 2 (2015): 173–80.
7. Margaret J. Foster and Sarah T. Jewell, eds., Assembling the Pieces of a Systematic Review: A Guide for Librarians, Medical Library Association Books (Lanham, MD: Rowman & Littlefield, 2017).
8. Koffel, “Use of Recommended Search Strategies in Systematic Reviews”; Koffel and Rethlefsen, “Reproducibility of Search Strategies Is Poor in Systematic Reviews.”
9. “Editorial Policies,” The New England Journal of Medicine, http://www.nejm.org/about-nejm/editorial-policies.

Appendix A. Guidelines for the Systematic Review Service

Guidelines for the Systematic Review Service

We are looking forward to working with you on this project.

International associations that support the creation of systematic reviews advise the inclusion of a librarian or information specialist on a systematic review team. Studies have shown that working with a librarian has enhanced both quality and reproducibility of the searches (Rethlefsen et al., 2015).

• Potential roles of librarians
  • Guide researchers on search strategies and database choices
  • Develop and conduct literature searches
  • Document search strategies
  • Write the relevant portions of the methods section
  • Advise on software for managing and reviewing
  • Substantial contributions by the librarian usually result in co-authorship
• Researcher responsibilities and expectations

  In order to ensure your work goes more smoothly, we have found it useful to have the following pieces in place as you start the systematic review process:

• Establish Systematic Review team members and their roles
• Determine your research question
• Finalize your inclusion/exclusion criteria beforehand to make screening less biased
• Work with librarians to develop search protocols and documentation
• Decide which software your team will use
  1. Citation management
  2. Systematic review management
  3. Collaborative manuscript editing
• Keep librarians abreast of your progress/questions
  • Responsibilities of both parties
  • Communicate regularly about the progress
  • Negotiate timelines as systematic reviews are often time consuming
  • Organize and manage sources within reference management software; team will agree on which software will be selected
  • Review manuscript as appropriate