rusq: Vol. 53 Issue 4: p. 300
So Text Me—Maybe: A Rubric Assessment of Librarian Behavior in SMS Reference Services
Erin Dorris Cassidy, Angela Colmenares, Michelle Martinez

Erin Dorris Cassidy (ecassidy@shsu.edu), is Associate Professor, Web Services Librarian, Sam Houston State University, Huntsville, Texas
Angela Colmenares is Reference Librarian/Associate Professor, Lone Star College–Montgomery, Conroe, TX
Michelle Martinez is Assistant Professor, Reference Librarian, Sam Houston State University, Huntsville, Texas

Abstract

Assessment has become a persistently hot topic in the library world, particularly when it comes to establishing value for academic library services. In an effort to assess performance and develop training tools to improve text/SMS reference services for an academic library, we used the Reference and User Services Association’s Guidelines for Behavioral Performance of Reference and Information Service Providers as the framework for an analytic rubric. We then used the rubric to assess academic librarian responses collected over a three-year period as part of text/SMS reference service. Results include implications for librarian friendliness, response time, attentiveness, and follow-up, as well as patron return rates. Comparative trends in text reference and the physical reference desk response times are also examined.


According to the Pew Internet and American Life Project’s “Smartphone Research,” approximately half of all American adults (45%) and two-thirds of young adults own a smartphone, and as of May 2011, texting is one of the top uses of this technology.1 Part of the recent change in librarians’ best practices is to go where the students are, to reach out to them, and this is achieved in part by offering text reference services. With the introduction of any new service, the need arises to assess performance in providing that service. It is of particular importance for libraries to evaluate and report their effectiveness in order to demonstrate their value and receive continued funding and support from US policymakers and academic governing boards.2


BACKGROUND

Sam Houston State University (SHSU) is a Carnegie Doctoral Research University located in East Texas. The Newton Gresham Library (NGL) serves the SHSU campus where 18,478 students were enrolled in fall 2012. Currently NGL employs nineteen faculty librarians, twenty-eight staff members, and two administrative faculty librarians.

NGL has offered virtual reference services via live chat since 2004, and through email for even longer. In the fall of 2009, the reference department decided to expand reference services to provide help via SMS/text messages, and the library began subscribing to the Text a Librarian service from Mosio. Staff training was conducted in late 2009, and the new service was launched in January 2010.

Training focused predominantly on the technical workings of the system itself rather than on the ideal characteristics of reference transactions in the text medium. For instance, staff were taught to recognize alert sounds when a new message arrived, enter responses and watch how many characters remained, send messages to patrons, recognize follow-up comments and questions, and close a thread when a conversation ended.

During training, personnel were also given some reference material about understanding text-speak—such as the use of u to mean you or 2nite to mean tonight—and were reminded to practice concision in their text-based responses. In the preexisting live chat service, it was common for personnel to send a pre-scripted welcome message such as “Hello, and thank you for contacting SHSU Ask a Librarian,” but such practices were discouraged in text messaging due to the constraints that the medium placed on message length. Beyond the one recommendation of concision, guidelines were not provided concerning how best to provide quality reference service in the unique context of this new medium.

The service was implemented so that text messages from patrons would be answered as they were received at the reference desk by the individual on rotation. One generic login account was used by all personnel, rather than individual usernames, so patrons did not know which individual responded to their question, and the library could not definitively attribute responses to a specific librarian, paraprofessional staff member, or student assistant.

At the end of the 2012 fall semester, with three full years of data collected, we evaluated the quality of the service being provided. Text messaging is a unique reference tool, due to characteristics such as the restrictions in message length—which renders traditional reference interviews difficult or impossible—and the dichotomy between the medium’s inherent asynchronicity and the student expectation of near-immediate response. The researchers sought to determine how well a librarian’s reference skills translate into this tool and how well librarians are serving customers in the manner which they expect.

We hope that a close evaluation of current performance will serve as a training tool, help to clarify and communicate expectations for text-based reference service, and lead to the development of guidelines and best practices which will improve overall staff performance in this service.


LITERATURE REVIEW

RUSA developed “Guidelines for Implementing and Maintaining Virtual Reference Services” in 2004, based on proposed guidelines from Bernie Sloan’s 1998 article in Reference and User Services Quarterly, which suggested that for almost half a decade the practice of virtual reference itself was very ad hoc.1 With most libraries being on the same virtual page when it comes to reference services, due to guides like RUSA’s, it becomes important to evaluate how well librarians are using the SMS tool to provide service.

Evaluating newly implemented reference services and assessing librarians’ effectiveness in these services has long been a priority to libraries.2 Pomerantz recommends content analysis as permitted by virtual reference services (transcripts are available and the interaction captured between librarian and patron) “because it opens reference interactions up for a range of evaluation metrics” such as those identified by RUSA’s “Guidelines for Implementing and Maintaining Virtual Reference Services”; however, the author discusses chat sessions rather than texting.3 Standards and tools for evaluation have varied for examining virtual reference services prior to the introduction of texting, including survey questions,4 a proxy method in which fake librarian patrons asked chat questions during certain times and filled out an evaluation on the service,5 and the TAM or technology acceptance model created in 1989 by F. D. Davis,6 and later adapted to evaluate the adoption of instant messaging.7 The International Federation of Library Associations and Institutions (IFLA) provides digital reference guidelines that contain elements included in rubrics for scoring library services, such as “respond to 100% of questions that are assigned,” “acknowledge receipt of patron question . . . [respond] as quickly as possible,” and “show professional courtesy and respect”; however these are all “guidelines for chat sessions,” which are not necessarily applicable to text messaging, and they are not broken up as in RUSA’s guidelines for specific evaluation of Interest, Follow-up, etc.8

According to the Pew Internet and American Life Project, 80% of all cell phone users send or receive text messages.9 Libraries use SMS as a system to send patron notices, such as overdue book alerts, library closings, and other information, as well as in reference services. Sims Memorial Library at Southeastern Louisiana claims to be “the first library in the United States to develop a text messaging reference service” in 2005.10 Over the past decade, both public and academic libraries across the United States have been catching up with the trend of providing reference via text messaging.

Since 1996, RUSA has published “Guidelines of Behavioral Performance of Reference and Information Service Providers,” which provide guidance for information professionals performing reference interviews in order to improve service to the end-user. The original guidelines dealt with face-to-face transactions but grew to include remote transactions, such as telephone, email, and instant messaging. However, text messaging generates unique challenges such as the cost of transaction and word limits that may require the use of text-speak. Luo addressed how RUSA guidelines could be applied to text reference, examining each section (approachability, interest, listening/inquiring, searching, and follow-up) individually, explaining the importance of each portion in text reference, and suggesting how each section might be adhered to while using text reference; however, Luo did not provide any case studies or surveys that used RUSA’s rubric to evaluate their services.11

Using RUSA’s guidelines to create a scoring rubric for chat reference, Maness, Naper, and Chaudhuri scored librarians’ performance “in which a patron was determined to be acting inappropriately” rather than scoring librarians’ performance as a whole with any and all chat interactions.12 Other libraries have adopted the RUSA guidelines to create a rubric on which to score individual librarians’ chat sessions; however it is unclear who is scoring whom—whether it is the patron receiving the checklist to score the librarian after the transaction or a supervisor scoring librarians on each chat or the librarians’ self-scoring.13 Using rubrics based on non-RUSA guidelines and rubrics, libraries such as the University of California Libraries and Maricopa Community Colleges Libraries have assessed digital reference transcripts for chat reference; however, their guidelines recommend only selecting a small sample and using them to explore and discuss problems or good quality service, yielding only a tiny snapshot which does not show the whole picture of reference service.”14

Publications about text messaging in libraries follow a pattern similar to that seen in articles about instant messaging, one of the latest technologies adapted prior to texting. Devine, Paladino, and Davis clearly group these articles in their paper on chat reference, and these categories can be applied to texting/SMS reference and mobile library services in general: anecdotal papers, “which summarize implementation and training”;15 “books and articles that present training practices for starting or modifying existing . . . services”;16 studies that “describe and analyze skills and competencies that are necessary for the effective provision” of the service;17 and academic library surveys, 18 “which, at various times, have asked one or more questions regarding the training practices”.19 To this list, we would add the evaluation of particular software/services—whether in articles jointly written by a library and its service provider or by a group using particular software, or even as a comparison with other software—as well as discussion of how patrons use or perceive the service provided or usability study results.20 Similar groupings are visible in discussing SMS/text reference.

This is the first paper, as far as the researchers can determine, that systematically evaluates and scores librarians’ interactions in SMS text reference, though there have been suggestions that best practices should be created in order to provide better service.21 Although a few papers have touched on the concept of librarians’ skills in using the service22 or offering statistics of librarians’ responses such as length of time taken, how many words used per message, and friendliness,23 this information is typically generalized and not addressed by systematic assessment as in the current study.


METHOD

We downloaded the complete listing of questions and answers from the Mosio platform. This data included Patron ID codes that uniquely distinguished between patron phone numbers without revealing the actual phone numbers or other personally identifying patron information.

The precise date and time of each question and answer were also included in the data. The researchers encountered some initial difficulty in calculating response time from the data provided, but Mosio’s technical support staff was highly responsive and promptly upgraded the data export feature so that we obtained data with a clearly calculated response time. The researchers are grateful for the company’s responsiveness in providing information and assistance that made this study possible.

All text messages sent and received were grouped by the researchers into transactions; a transaction might consist of several individual messages from patron and librarian, including automatic system responses, representing one logical session of conversation. The final dataset consisted of 385 transactions, or conversational threads. This does not equate to the total number of messages ever submitted to the service, because messages where a question was never raised, such as “test,” “hi,” or similar were not included; this study is instead based solely on actual substantive questions to which a librarian did respond or could/should have responded.

The data was first evaluated statistically with regards to characteristics such as response times and how often patrons revisited the service. A portion of this statistical data was also compared against a seven-month sample of similar data from face-to-face and telephone transactions at the reference desk.

After these quantitative characteristics were measured, the researchers developed a rubric in order to assess how well librarians performed in answering each question (appendix A; also online at http://library.shsu.edu/libfac/TextRubric.pdf). One member of the research team holds a Masters of Education degree, and she led the development and testing of the rubric. A three-scale, analytic rubric was created using the “Guidelines for Behavioral Performance of Reference and Information Service Providers,” as published by the Reference and User Services Association (RUSA), a division of the American Library Association (ALA).24 RUSA’s guidelines helped to ensure content and construct validity by providing a criterion-referenced rubric framework. The guidelines include general, in-person, and remote reference librarian behaviors across the five categories of Approachability, Interest, Listening/Inquiring, Searching, and Follow-up. While a list of remote behaviors is available, behaviors in each of the three formats (general, in-person, and remote) were considered in constructing the rubric.

The framework was then modified to accommodate a focus on text reference services, which necessitated the elimination of the Approachability category from the project. The Approachability category as identified by RUSA for the remote format emphasizes elements of web design, and while an evaluation of a library’s website is certainly an important and worthwhile pursuit, the purpose of this project was to provide training for librarian behavior. The rating scales included Beginning (1 point), Developing (2 points), and Accomplished (3 points).

In addition to eliminating the Approachability category, minor customizations were also needed to accommodate the limitations inherent in a text-messaging service, and the researchers used RUSA’s overall category descriptions to stay on target during the modification stage. For example, RUSA’s description of the Listening/Inquiring category advises that the “librarian must be effective in identifying the patron’s information needs and must do so in a manner that keeps patrons at ease;”25 hence, the descriptor “Uses a tone of voice and/or written language appropriate to the nature of the transaction”26 became the descriptor “Uses concise language, abbreviations if appropriate, limiting responses to one text message (or 160 characters) per patron query” (see appendix A).

The limitations of text messaging as a reference platform also impacted the creation of the Searching criterion. For example, RUSA suggests that the librarian “Finds out what patrons have already tried, and encourages patrons to contribute ideas;”27 since many patrons may be billed per text message, this criterion was omitted from the rubric. Similar minor customizations occurred in each of the four major criteria, with the Searching criterion being the most extensively customized, including the addition of two local rules: The first rule was that librarian answers that were clearly inaccurate to the researchers received the Beginning (1) score, and the second rule was that closed-ended questions that required no actual searching on the part of the librarian (for example, library hours of operation) received the Accomplished (3) score (see appendix A).

The creation of the analytic rubric provided an educational opportunity for us, and following rubric development, we evaluated our draft rubric using A Rubric for Judging Your Rubric, a tool developed by The Center for Faculty Development at the University of Colorado Denver.28 Following minor modifications for measurable and observable descriptors, we normed the rubric for inter-rater and intra-rater reliability using two sets of five transactions across three norming sessions spaced several weeks apart. Next, the evaluation of all 385 transactions was conducted in equal portions by the three researchers. Since staff names were not part of the data collected, the results give a general picture of the Reference Department’s performance without critiquing individuals.


RESULTS
Rubric Evaluations

All text messages were grouped into transactions, and each transaction was evaluated according to the project rubric to assess the librarian’s performance in four criteria: Listening and Inquiring, Interest, Searching, and Follow-Up. In each criterion, a transaction could earn a score of 1 point (Beginning), 2 points (Developing), or 3 points (Accomplished). Average scores per criterion ranged from 1.96 to 2.49 (see figure 1).

When the scores from each criterion were added together, a transaction could earn a cumulative score between 4 and 12 points, with 12 representing the most Accomplished performance and 4 representing the most Beginning performance. Overall, the average cumulative score was a 9.1. Although only 1.3% of transactions met the Accomplished standard with the highest possible score of 12, 73% of transactions earned a score between 9 and 12, indicating that the majority of transactions did exceed the Developing standard, represented by an overall score of 8 (see figure 2).

Response Rate

For this study, response rate was defined as how soon after an initial question’s arrival a human response was sent to the patron. In some transactions, the patron may have received an immediate, automatic system response indicating that the question was sent outside of the library’s hours of operation, but this was not counted as a “response” for the purpose of calculating response rate. Overall, 51% of questions received human responses within 10 minutes and 88% within twenty-foure hours or less; 4% of questions never received a human response (see figure 3).

Response rates were compared based on the month, weekday, and time of day when a question was sent to identify any trends. For each of these comparisons, three response times were omitted because these extreme outliers of seven months or longer skewed the averages.

In a comparison of months (see figure 4), May and June showed a longer average response time (about 1 day 7 hours), while October saw the fastest average response time (about 2.6 hours). In a comparison of weekdays (see figure 5), Fridays and Saturdays showed a longer average response time (16–20 hours), while the fastest average response time was seen on Wednesdays (about 2 hours).

Finally, times of day were compared, according to seven time ranges (see figure 6). The poorest response time was seen for questions received 5–7:30 a.m. (about 1 day 5 hours). In contrast, the fastest average response time (about 3.5 hours) was seen 3–6 p.m.

Virtual Reference versus Physical Reference

Data concerning more traditional “physical” (face-to-face and telephone) transactions at the reference desk (not including text, chat, or email transactions) were analyzed for the sample seven-month period of June 1 through December 31, 2012, to determine the busiest hours of the day for non-virtual reference at the reference desk (see figure 7). The 10 a.m., 11 a.m., 12 p.m., and 2 p.m. hours stood out as the four busiest hours on average.

Next, text messages from this same June through December 2012 period were split into two groups, based on whether the questions were sent during busy or non-busy reference desk hours. Response times were averaged in each group, minus one abnormally long response time. The average response time during busy hours was 2295.11 seconds (about 38 minutes), while the average response time during non-busy hours was 9143.06 seconds (about 2.5 hours), suggesting that text messages were actually answered more quickly during hours when the reference desk was busiest with traditional face-to-face and telephone reference.

Repeat Patrons

From 2010 to 2012, 81% of patrons who used the service used it only once (see figure 8). No patron used the service more than seven times—at least, not from the same cell phone number.

Response Length

Within 385 transactions, there were a total of 447 individual replies sent by library personnel. Of these individual replies, 93% comprised one text message of no more than 160 characters, 6.7% used two or three text messages, and only one reply required four text messages to be sent to the patron (see figure 9).


DISCUSSION

Useful implications are suggested by the results of a rubric analysis of text-reference transactions, particularly the transaction scores, response rates, and patron-return rates. Overall, the average score across the entire transaction pool is promising. Since a score of Developing (2) in all four criteria would earn a total score of 8, the cumulative average score of 9.1 indicates that librarian performance, on average, meets or exceeds the Developing (2) level, even though it does not quite meet the Accomplished (3) level. In fact, nearly three quarters of the transactions scored between 9 and 12, indicating that the majority of transactions exceeded the Developing standard (see figure 2). However, while the average score of all transactions is helpful to appreciate the big picture of text reference performance, a more detailed analysis of each criterion is needed to generate quality feedback.

The rubric used to score transactions clustered criteria into four categories: Listening and Inquiring, Interest, Searching, and Follow-up. The category with the highest average score was the Searching category (see figure 1). As this is one of the most traditional and visible skills in the librarian profession, we find the evidence of a high performance level personally and professionally rewarding. Of greater use in an assessment capacity, however, are the low performance scores. The lowest-scoring criterion was the Follow-up category, with the average situated between the Beginning (1) and Developing (2) rubric scores. The Listening and Inquiring category had the second-lowest score, with the average falling between the Developing (2) and Accomplished (3) rubric scores. Both the Listening and Inquiring and Follow-up categories contain criteria that focus on librarian communication skills: Listening and Inquiring focuses on cordial yet concise communication and Follow-up emphasizes the invitation for library patrons to return if they have additional questions or need more assistance. In the Follow-up category, any transaction that did not invite the patron’s future use of the service did not receive the Accomplished rating, which significantly impacted the overall average for this category. Perhaps the expectation of an invitation to return in each transaction is a bit idealistic, but this criterion is included by the Reference and User Services Association (RUSA), a division of the American Library Association (ALA), and the rating group of three professional librarians expected that the highest-performing library professionals would strive to achieve it. The resulting low scores indicate a clear need for heavier emphasis in initial and refresher training on this aspect of performance.

Another implication of such a low score in the Follow-up category is that librarians may be struggling to balance the goals of providing detailed, quality information with an invitation to return (follow-up), all while being expressly cordial within a single 160-character message. One solution may be to generate a canned response or a signature line for staff use, automatically providing a succinct welcome to return, while emphasizing the most clear text-shortening methods that would still resonate well with the library’s target population. Such signature lines could be as basic as “Use us again!” or as abbreviated as “Thx&T2UL8R” (Thanks and talk to you later). Although the canned response or signature line would take up part of the 160-character limit, the easy availability of a concise, cordial message would contribute to higher-quality text reference by providing assistance that the staff needs, as evidenced by the low Follow-up transaction scores.

While average category scores are useful for looking at the big picture, analysis of smaller data clusters yields more specific trends. For instance, only 18 transactions (4.7%) earned a Beginner (1) rating across all categories, 16 of them because they were never answered (see figure 2). On the surface this may indicate that additional technical training may be needed to ensure that staff are able to recognize incoming questions and then successfully respond to them. An alternate implication is that a higher level of attentiveness from library staff may be needed. The remaining two transactions that did include a librarian response were answered inexpertly enough to receive a total score of 4, representing a Beginner (1) rating in each category. While these were clearly problem transactions, it is encouraging that the area of needed development is in refresher training on identifying “waiting” questions and the technical response procedure, which is less disturbing than a need to retrain groups of librarians on the fundamentals of reference techniques. Only five transactions were rated Accomplished (3) across all four criteria, earning a total score of 12 (see figure 2). Although this is a disturbingly low number, it clearly validates this project’s aim: to evaluate staff performance with an eye to documenting expected standards and best practices, then providing training for library staff.

Unique to the medium of text reference, and therefore one of the main concerns of this rubric project, is the limitation of response length (usually 160 characters). While concision is a goal for many reference services, instant message, email, phone, and even face-to-face reference services have a more reasonable expectation of back-and-forth communication than text reference does. Many users, however, are subject to cell phone plans that may limit the quantity of text messages sent or even charge fees for each separate text message, a scenario that libraries promoting text services should seek to accommodate. With this limitation in mind, responding with a single text message whenever possible is imperative for librarians, even to the point of forgoing an immediate acknowledgement text which would assure patrons that their queries were received and that an answer was forthcoming. Analysis of the collected data reveals that 93% of librarian responses were constrained to a single message (see figure 9). This high result suggests that most library staff are clearly aware of the need to limit response length.

In addition to transaction scores and response length, librarian response times for transactions are also instructive (see figure 3).29 The majority of transactions (59%) received human responses within 30 minutes of the initial question, which was the timeframe established for the Accomplished (3) rating. This thirty-minute window may seem lengthy, given most students’ perception of text messaging as an instantaneous communication medium, but it allows the single librarian on duty at our library’s reference desk to balance the complete range of reference services for which she is responsible, including in-person and telephone reference questions as well as virtual queries by email, instant message, and text message. This time allotment also permits the librarian to locate the requested information and then edit the delivery of that information down to a single text message when possible. The fact that most text messages are personally handled by library staff quickly is encouraging, particularly since 51% of that 59% were answered within ten minutes. Approximately 5.7% of transactions received a human response within thirty minutes to one hour, the timeframe established for the Developing (2) rating, reinforcing the focus on response time as an important aspect of virtual reference.

The remaining portion of transactions (34%) includes the 23% that received human responses in more than 1 hour but less than one day and an approximate 5% that were handled within one to four days, as well as 2% of transactions took more than four days to receive a reply and 4% that never received a human response. This last cluster of response times (34%) points to a clear need for consistent training and reinforcement of expected standards for library staff. Transactions that never received a response (4%) may indicate the need for additional or reinforced technical training with the text reference system: either librarians were not aware of the texted query or were unable to answer it, which could indicate either inattentiveness or an inability to use the system correctly.

Clearly the majority of text messages are being addressed quickly, according to the standards crafted in the rubric for this project, despite a time limit not having been identified in previous virtual-reference training for library staff. Perhaps the initial thirty-minute expected response rate is too idealistic, and a longer timeframe should be the expectation for providing a substantive answer (not merely acknowledging the question). Expected response times differ between a text-reference service and an instant chat service, at least in the context of this study. However, students today rely heavily on texting, which may serve to blur the lines between text reference and instant chat services. Further research into the expectations of both patrons and librarians, with regards to perceived differences in communication methods and expected response times across multiple virtual reference platforms, would inform such discussion.

Closer analysis of the librarian response-time data reveals that Fridays and Saturdays had the longest time between initial question and librarian response, an average of approximately 16 hours on Friday and 20.6 hours on Saturday, compared to the lowest average on Wednesday of around 2 hours (see figure 5). One explanation of this gap may relate to the library’s operational weekend hours. The library closes early on Friday (6 p.m.), opens late and closes early on Saturday (10 a.m. to 7 p.m.), and opens even later on Sunday (2 p.m.). Since averaged response times did not take into account the library’s nonoperational hours, texted queries that were received after closing on Friday were not answered until mid-morning on Saturday, resulting in a 16-hour gap. The same is true of queries sent after closing on Saturday and not answered until Sunday afternoon—a 19-hour gap. While most weekdays averaged only two queries submitted after library closing, the majority of queries submitted outside of operational hours occurred on the day with the latest opening time: Sunday. This may indicate a need to reconsider Sunday’s operational hours or at least the need to explore options to provide virtual reference services during reduced weekend operational hours.

Analyzing response time by month also yields trends to consider (see figure 4). May and June had the poorest response times by a large margin, suggesting that the beginning of the summer semester saw librarians busy with other activities that may have prevented them from noticing text-reference questions. This pattern seems to repeat for other major semesters as well: the opening six weeks or so of each semester saw the poorest response times for that semester, suggesting that librarians, like teaching faculty, may begin the semester with an extraordinary amount of work already on their plate. October and April, on the other hand, are the months with the fastest response times, further suggesting that librarians may focus more effectively on fielding text-reference questions after the semester is well underway. Another possible explanation for poor response time in May and June is that librarian response time may be impacted by the library’s limited operational hours: May includes the spring interim, which sees a reduction in library operating hours, and June also sees reduced hours.

Further analysis of response times by hours indicates that the poorest response times—averaging almost 1 day 5 hours—are associated with questions received between 5 a.m. and 7:30 a.m., the opening time for the library and the few hours before. This may indicate problems with the library staff’s procedure for opening the reference desk or a need for technical training to recognize the presence of a waiting text-reference question. The fastest average response times are seen with questions received between 3 p.m. and 6 p.m., averaging around 3.5 hours. To get a wider perspective on possible response-time issues by hour, data concerning physical reference desk transactions (including face-to-face and telephone, but not chat, email, or text) were also analyzed for the period of June 2012 to December 2012, revealing that the busiest hours of the day at the physical reference desk included 10 a.m., 11 a.m., 12 p.m., and 2 p.m. (see figure 7). In comparison, the busiest text-reference hours (by number of text referenced questions received) were the four hours from 12 p.m. to 4 p.m.

Admittedly, this seven-month sample of physical reference desk statistics, correlated with text-reference activities, provides only a limited quantity of data to compare, so these results are more anecdotal than statistically significant. Nevertheless, the results of the comparison are surprising. The researchers suspected that text-reference response times might suffer during peak periods when reference desk personnel were busy assisting face-to-face patrons. Surprisingly, the opposite was true: the busiest hours at the physical reference desk correlated with the fastest text responses of the day. Similarly, the busiest text-reference hours of the day also correlated with the fastest text response times of the day. The researchers hypothesize that reference desk personnel may become more alert when traffic picks up at the physical desk, thus resulting in better virtual response times as well, due simply to increased attentiveness. This supposition also seems to hold true for the analysis of librarian response time by month: slower months at reference, such as May and June, see a weaker response time, while October—the second-busiest month at the reference desk—sees the strongest response time.

In addition to transaction scores and response times, the researchers also examined the number of repeat patrons (see figure 8). Approximately 11% of patrons returned to use the text-reference service for a second time, while another 8% used it between three and seven times. Admittedly this is a small number of repeat patrons, with only 19% of all users returning to the service again after their first use. However, statistics on repeat patrons were determined based on cell phone numbers. It is possible that at least a few users of the text reference services may have switched phone numbers, and current statistics cannot account for this possibility.

Cell phone technologies aside, these numbers still raise questions about why most patrons do not tend to revisit this service. Many patrons may have been deterred simply by the economics of pay-per-message cell phone plans. This study does not support conclusions about return use, but it is possible that the various factors evaluated in the rubric—such as response time, cordiality, and the amount of detail or direction provided for an information source—played a role in whether a patron chose to use the service again. One goal of the researchers is to collect follow-up data on repeat patrons in the future, after best practices and staff training have been implemented for a reasonable period, to see whether repeat patronage might increase in correlation with improved service. Further research may also be warranted concerning student attitudes toward the service, whether library marketing for this service is reaching students, whether marketing and student expectations match the actual student experience, and how students’ cell phone plans address text messages, including whether pay-per-text plans deter use of the library service.

A final issue highlighted by this study relates to maintenance procedures, especially concerning hours of operation. The researchers encountered multiple transactions which lacked the appropriate automatic system response that should have been sent during nonoperational hours. This was not a technical error in the service platform, but rather a failure of the library’s Web Services personnel to consistently program auto-responses for holiday closures, limited interim hours, and so forth. This emphasizes the need for a clear and consistent maintenance routine so that patrons always receive prompt indications of the library’s status during nonoperational hours, rather than mistakenly awaiting a response while feeling ignored and poorly served.


CONCLUSIONS

This study demonstrates that significant issues exist with staff attentiveness to monitoring text reference. Some improvement is also needed in areas of friendliness and follow-up. These findings confirm a need for the reference department to offer more in-depth staff training than has previously been provided. Over the next academic year, the researchers plan to create and implement various support tools to educate staff about performance targets. These tools may include sample message templates or signature lines and guidelines for best practices in providing text-reference service. The researchers hope that the assessment rubric resulting from this study will provide a template for other libraries to adapt and implement in their own self-assessment.


STUDY LIMITATIONS

Although the study covers three years of SMS reference data, the total number of transactions is still relatively small. Examination of a more substantial data pool may allow additional insights. The researchers are interested in partnering with other libraries to provide assessment of de-identified text reference data provided by those partners. This would not only increase the quantity of available data but also allow for a broader view of the issue, rather than focusing on only one library.

Additionally, although most questions are answered by professional librarians or long-time paraprofessional library staff, questions may occasionally have been answered by a student assistant, and the use of a single login account makes it impossible to identify the exact staff member who answered a specific question. For the purposes of our study, this does create a minimal amount of uncertainty about whether any lower-quality answers could be attributed to less proficient staff.

Finally, the researchers recognize that it would be useful to examine response time only in regards to the library’s operational hours, rather than including nonoperational hours in the calculated time for response. If a question is asked after closing and a response is sent eight hours later, but within two minutes of the library reopening, is it more accurate and helpful to report the response time as eight hours or two minutes? However, in the context of this study, the researchers determined that it was not feasible to adjust the response time calculations to include only operational hours, largely because the plan for this study did not predate and thus could not influence the manner of data collection. The data examined spanned three years, during which time the library followed at least eight different schedules of hours, and conclusively determining the operational hours of each transaction proved difficult. Furthermore, the response times as reported have been electronically calculated from the mathematical difference between the question’s arrival date/time and the date/time that the first human response was sent; adjusting this calculation to account for operational hours would require manual recalculation for all 385 transactions by subtracting nonoperational hours from the electronically calculated time. This level of manual recalculation would present a significant opportunity for human error.


APPENDIX A. REFERENCE TEXT PROJECT RUBRIC—EXPLODED VIEW
Project Purpose

The purpose of this project is to give librarians feedback on virtual reference skills such as Listening/Inquiring, Searching, Interest, and Follow-up. RUSA has identified each of these skills as important guidelines for behavioral performance of reference and information service providers.

Rubric Purpose

The purpose of this rubric is to provide measurable criteria to assess the text message reference skills of professional academic librarians for the last 3 calendar years. Results of this rubric are intended to be used as a teaching/training tool to communicate expectations and give informative feedback. The assessment goal is to improve the performance of professional librarians in the area of text reference message service at Newton Gresham Library.


References and Notes
1. “Guidelines for Implementing and Maintaining Virtual Reference Services,” Reference and User Services Association (RUSA), American Library Association (ALA), accessed August 5, 2013, www.ala.org/rusa/resources/guidelines/virtrefguidelines; Bernie Sloan, “Electronic Reference Services: Some Suggested Guidelines,” Reference & User Services Quarterly 38, no. 1 (Fall 1998), accessed June 21, 2013, www.jstor.org/stable/20863431
2. See, for example: Firouzeh F. Logan, “A Brief History of Reference Assessment: No Easy Solutions,” Reference Librarian 50 (2009), http://dx.doi.org/10.1080/02763870902947133; Anne Pemberton, Jerome Hoskins, and Caitlin Boninti, “Minding the Gap: Identifying Performance Issues Using the Human Performance Technology Model,” Reference Services Review 39(2011), http://dx.doi.org/10.1108/00907321111135457; Firouzeh F. Logan and Krystal Lewis, “Quality Control: A Necessary Good for Improving Service,” Reference Librarian 52 (2011), http://dx.doi.org/10.1080/02763877.2011.557314; Buff Hirko and Mary Bucher Ross, Virtual Reference Training: The Complete Guide to Providing Anytime, Anywhere Answers (Chicago: American Library Association, 2004)
3. Jeffery Pomerantz,  “Virtual Reference Services: Evaluation of Online Reference Services,” Bulletin of the American Society for Information Science and Technology 34, no. 2 (September 2008): 16, http://dx.doi.org/10.1002/bult.2008.1720340206
4. María Pinto and Ramón A. Manso,  "“Virtual References Services: Defining the Criteria and Indicators to Evaluate Them,” ,"  Electronic Library  (2012)   30, no. 1:  51–69.
5. David Ward,  "“Measuring the Completeness of Reference Transactions in Online Chats: Results of an Unobtrusive Study,”" in Reference & User Services Quarterly,   accessed December 3, 2013, http://www.jstor.org/stable/20864287Fall 2004
6. Davis F. D.,  "“Perceived Usefulness, Perceived Ease of Use, and User Acceptance of Information Technology,”,"  MIS Quarterly  (1989)   13:  319–40.
7. Wang C.WangC. ,  Hsu Y.WangC. ,  Fang W.,  “Acceptance of Technology with Network Externalities: An Empirical Study of Internet Instant Messaging Services,” Journal of Information Technology Theory & Application 6, no. 4, (2008): 15–28; Jung-Jung Chang and Chyan Yang, “Viable or Vital? Evaluation of IM Services from Patrons’ Perspectives,” Electronic Library 30, no. 1 (2012), http://dx.doi.org/10.1108/02640471211204079
8. International Federation of Library Associations and Institutions, “IFLA Digital Reference Guidelines,” IFLANET, accessed December 3, 2013, http://archive.ifla.org/VII/s36/pubs/drg03.htm
9. Pew Research Center, “Cell Phone Activities 2012,” accessed June 22, 2013, http://pewinternet.org/∼/media/Files/Reports/2012/PIP_CellActivities_11.25.pdf
10. Hill J. B.HillJ. B. ,  Madarash Hill CherieHillJ. B. ,  Sherman Dayne,  "“Text Messaging in an Academic Library: Integrating SMS into Digital Reference,”,"  Reference Librarian  (2007)   47, no. 1:  17–29.
11. Luo, “Text Reference Service.”
12. Jack M.. Maness, Sarah Naper,  and Jayati Chaudhuri,  “The Good, the Bad, but Mostly the Ugly: Adherence to RUSA Guidelines during Encounters with Inappropriate Behavior Online,” Reference & User Services Quarterly 49, no. 2 (Winter 2009), http://dx.doi.org/10.5860/rusq.49n2.151
13. Examples of these rubrics can be seen on askusnow.info under “stellar transcript team rubrics from other services” at http://askusnow.info/staff/node/379
14. California Academic & Research Libraries Association, “Rubric Used by University of California Libraries to Assess Digital Reference Transcripts 2009,” CARL-ACRL.org, accessed December 2, 2013, www.carl-acrl.org/ig/carldigs/Files/UCRubric%C2%A0for%C2%A0Ask%C2%A0a%C2%A0Librarian%20Chat_basedonQPandMaricopa.pdf
15. See, for example: Virginia Cole and Baseema B. Krkoska, “Launching a Text a Librarian Service: Cornell’s Preliminary Experience,” Reference Librarian 52 (2011), http://dx.doi.org/10.1080/02763877.2011.521881; Char Mairn, “Three Things You Can Do Today to Get Your Library Ready for the Mobile Experience,” Reference Librarian 53(2012), http://dx.doi.org/10.1080/02763877.2012.678245
16. See, for example: Amada Bielskas and Kathleen M. Dreyer, IM and SMS Reference Services for Libraries (London: Facet Publishing, 2012); Charles Harmon and Michael Mesina, Mobile Library Services: Best Practices (Lanham, MD: Scarecrow Press, 2013); Lili Luo, “Text Reference Service: Delivery, Characteristics, and Best Practices,” Reference Services Review 39 (2011), http://dx.doi.org/10.1108/00907321111161449; Lili Luo, “Professional Preparation for ‘Text a Librarian,’” Reference & User Services Quarterly 52(2012), http://dx.doi.org/10.5860/rusq.52n1.44; Diane Zabel, ed., Reference Reborn: Breathing New Life in to Public Services Librarianship (Santa Barbara, CA: Libraries Unlimited, 2011); Alexa Pearce, Scott Collard, and Kara Whatley, AmpldquosemicolonSMS Reference: Myths, Markers, and Modalities,” Reference Services Review 38 (2010), http://dx.doi.org/10.1108/00907321011045016
17. John Maxymuk,  "“Library Texts,” The Bottom Line: Managing Library Finances 22"http://dx.doi.org/10.1108/088804509109554132009
18. See, for example: Marvie Brooks and Mark Zubarev, “Another Lane on the Information Highway? A Case Study of Experimenting with Text Message Reference,” Reference Librarian 53 (2012), http://dx.doi.org/10.1080/02763877.2011.611095
19. Christopher Devine, Emily Bounds Paladino,  and John A. Davis,  "“Chat Reference Training after One Decade: The Results of a National Survey of Academic Libraries,”"; in Journal of Academic Librarianship,   http://dx.doi.org/10.1016/j.acalib. 2011.02.0112011 197-198
20. See, for example: Sian Brannon, “SMS Reference,” Reference Librarian 52 (2010), http://dx.doi.org/10.1080/02763877.2011.523817; Beth Stahr, “SMS Library Reference Service Options,” Library Hi Tech News 26(1999), http://dx.doi.org/10.1108/07419050910979955; Lili Luo and Emily Weak, “Texting 4 Answers,” Reference & User Services Quarterly 51 (2011), http://dx.doi.org/10.5860/rusq.51n2.133; Margie Ruppel and Amy Vecchione, “‘It’s Research Made Easier!’ SMS and Chat Reference Perceptions,” Reference Services Review 40 (2012), http://dx.doi.org/10.1108/00907321211254689; Andrew Walsh, “Mobile Phone Services and UK Higher Education Students, What Do They Want from the Library?” Library and Information Research (2010), accessed June 22, 2013, www.lirg.org.uk/lir/ojs/index.php/lir/article/view/218; Lorraine Paterson and Boon Low, “Student Attitudes Towards Mobile Library Services for Smartphones,” Library Hi Tech 29 (2011), http://dx.doi.org/10.1108/07378831111174387; Gillian Nowlan, “Going Mobile: Creating a Mobile Presence for Your Library,” New Library World 114 (2013), http://dx.doi.org/10.1108/03074801311304050
21. Thomas A. Peters and Lori Bell, eds., The Handheld Library: Mobile Technology and the Librarian (Santa Barbara, CA: ABC-CLIO, 2013).
22. See, for example, Luo, “Professional Preparation”; Cole and Krkoska, “Launching a Text a Librarian Service”; Alexa Pearce, “Text Message Reference at NYU Libraries,” Reference Librarian 51 (2010), http://dx.doi.org/10.1080/02763877.2010.503314
23. See, for example, Pearce, Collard, and Whatley, “SMS Reference.”
24. “Guidelines for Behavioral Performance of Reference and Information Service Providers,” Reference and User Services Association (RUSA), American Library Association (ALA), accessed August 5, 2013, www.ala.org/rusa/resources/guidelines/guidelinesbehavioral
25. “Guidelines for Behavioral Performance,” section 3.0
26. “Guidelines for Behavioral Performance,” section 3.2
27. “Guidelines for Behavioral Performance,” section 4.1
28. “A Rubric for Judging Your Rubric,” The Center for Faculty Development at the University of Colorado Denver, accessed March 5, 2013, www.ucdenver.edu/faculty_staff/faculty/center-for-faculty-development/Documents/Tutorials/Rubrics/4_quality/5_rubric_rubric.htm
29. Please note: in order to improve the readability of the discussion, some response-time percentages have been rounded and thus may appear slightly differently than in the figures

Figures

Figure 1

Average scores by criterion.



Figure 2

Total scores of transactions.



Figure 3

Library response rates.



Figure 4

Average response times by month.



Figure 5

Average response times by day of the week.



Figure 6

Average response times by hours of the day.



Figure 7

Face-to-face transactions by hour, June through December 2012.



Figure 8

Number of questions asked per patron.



Figure 9

Number of text messages per library reply.



Tables
Accomplished—3 Developing—2 Beginning—1
Listening/
Inquiring
The reference interview is the heart of the reference transaction and is crucial to the success of the process. The librarian must be effective in identifying the patron’s information needs and must do so in a manner that keeps patrons at ease. Strong listening and questioning skills are necessary for a positive interaction.
1. Communicates in a clearly receptive/cordial/ encouraging manner.
2. Uses concise language, abbreviations, if appropriate, limiting responses to one text message (or 160 characters) per patron query.
3. Uses open-ended questioning techniques, if appropriate, to encourage the patron to expand on the request or present additional information. Some examples of such questions include:
• Please tell me more about your topic.
• What additional information can you give me?
• How much information do you need?
4. Uses closed questions if appropriate to refine the search query. Some examples of clarifying questions are:
• What types of information do you need (books, articles, etc.)?
• Do you need current or historical information?
1. Communicates in a receptive/cordial/ encouraging manner.
2. Uses concise language, abbreviations, if appropriate, limiting responses to no more than two text messages (or 320 characters) per patron query.
3. Does not use open-ended questioning techniques even when appropriate to encourage the patron to expand on the request or present additional information.
4. Uses closed questions to refine the search query. Some examples of clarifying questions are:
• What types of information do you need (books, articles, etc.)?
• Do you need current or historical information?
1. Communicates in an abrupt manner.
2. Does not use concise language, sending responses that exceed two text messages (over 320 characters) per patron query.
3. Does not use open-ended questioning techniques even when appropriate to encourage the patron to expand on the request or present additional information.
4. Does not use closed questions to refine the search query.
Interest
A successful librarian must demonstrate a high degree of interest in the reference transaction. While not every query will contain stimulating intellectual challenges, the librarian should be interested in each patron’s information need and should be committed to providing the most effective assistance. Librarians who demonstrate a high level of interest in the inquiries of their patrons will generate a higher level of satisfaction among users.
1. An automatic response acknowledges user questions submitted outside of library operation hours (hours during which the library is open).
2. Provides an initial response to the patron in 30 minutes or less during library operation hours (hours during which the library is open).
3. Responds to follow-up questions, if appropriate, in 30 minutes or less during library operation hours (hours during which the library is open).
1. No automatic response acknowledges user questions submitted outside of library operation hours (hours during which the library is open).
2. Provides an initial response to the patron in a timely manner, between 30 and 60 minutes during library operation hours (hours during which the library is open).
3. Responds to follow-up questions, if appropriate, between 30 and 60 minutes during library operation hours (hours during which the library is open).
1. No automatic response acknowledges user questions submitted outside of library operation hours (hours during which the library is open).
2. Provides an initial response to the patron after more than 60 minutes during library operation hours (hours during which the library is open).
3. Responds to follow-up questions, if appropriate, after more than 60 minutes during library operation hours (hours during which the library is open).
Searching
The search process is the portion of the transaction in which behavior and accuracy intersect. Without an effective search, not only is the desired information unlikely to be found, but patrons may become discouraged as well. Yet many of the aspects of searching that lead to accurate results are still dependent on the behavior of the librarian.
1. Names the sources to be used, when appropriate.
2. Works with the patron to narrow or broaden the topic when too little or too much information is identified.
3. Recognizes when to refer the patron to a more appropriate guide, database, library, librarian, or other resource.
4. Offers detailed search paths or links/URLs to needed electronic resources. Excessively long links have been converted to a shorter link (for example, using Tiny.URL).
5. If appropriate, detailed directions to physical resources are given, for example:
- Call #s and Floor #s
- Room #s
1. Names the sources to be used, when appropriate.
2. Indicates that the patron needs to narrow or broaden the topic when too little or too much information is identified.
3. Recognizes when to refer the patron to a more appropriate guide, database, library, librarian, or other resource when appropriate.
4. Offers detailed search paths or links/URLs to needed electronic resources.
5. If appropriate, general directions to physical resources are given, for example—either call #s or floor #s, but not both.
1. Does not name the sources to be used when appropriate.
2. Does not work with the patron to narrow or broaden the topic when too little or too much information is identified.
3. Does not refer the patron to a more appropriate guide, database, library, librarian, or other resource when appropriate.
4. Does not offer detailed search paths or links/URLs to needed electronic resources.
5. Even if appropriate, directions to physical resources are not given.
Rubric Notes for Searching:
1. Librarian answers that were clearly inaccurate to the scoring group received the “Beginning” (1) score.
2. Close-ended questions that required little or no searching on the part of the librarian received the “Accomplished” (3) rating.
Follow-up
The reference transaction does not end when the librarian leaves the patrons. The librarian is responsible for determining if the patrons are satisfied with the results of the search, and is also responsible for referring the patrons to other sources, even when those sources are not available in the local library.
1. Encourages the patron to return if they have further questions by making a statement such as— “if you don’t find what you are looking for, please come back and we’ll try something else” or similar.
2. Makes the patron aware of other reference services, if appropriate (email, instant chat, phone, etc.)
3. Makes arrangements, when appropriate, with the patron to research a question even after the reference transaction has been completed.
4. Refers the patron to other sources or institutions when the query cannot be answered to the satisfaction of the patron.
5. Takes care not to end the reference interview prematurely.
1. Does not encourage the patron to return if they have further questions.
2. Makes the patron aware of other reference services, if appropriate (email, instant chat, phone, etc.).
3. Does not make arrangements, when appropriate, with the patron to research a question even after the reference transaction has been completed.
4. Does not refer the patron to other sources or institutions when the query cannot be answered to the satisfaction of the patron.
5. Takes care not to end the reference interview prematurely.
1. Does not encourage the patron to return if they have further questions.
2. Does not make the patron aware of other reference services even when appropriate (email, instant chat, phone, etc.).
3. Does not make arrangements, when appropriate, with the patron to research a question even after the reference transaction has been completed.
4. Does not refer the patron to other sources or institutions when the query cannot be answered to the satisfaction of the patron.
5. Ends the reference interview prematurely, before answering or addressing all parts of a question.


Article Categories:
  • Library Reference and User Services
    • Features

Refbacks

  • There are currently no refbacks.


ALA Privacy Policy

© 2019 RUSA