Sam Houston State University (SHSU) is a Carnegie Doctoral Research University located in East Texas. The Newton Gresham Library (NGL) serves the SHSU campus where 18,478 students were enrolled in fall 2012. Currently NGL employs nineteen faculty librarians, twenty-eight staff members, and two administrative faculty librarians.

NGL has offered virtual reference services via live chat since 2004, and through email for even longer. In the fall of 2009, the reference department decided to expand reference services to provide help via SMS/text messages, and the library began subscribing to the Text a Librarian service from Mosio. Staff training was conducted in late 2009, and the new service was launched in January 2010.

Training focused predominantly on the technical workings of the system itself rather than on the ideal characteristics of reference transactions in the text medium. For instance, staff were taught to recognize alert sounds when a new message arrived, enter responses and watch how many characters remained, send messages to patrons, recognize follow-up comments and questions, and close a thread when a conversation ended.

During training, personnel were also given some reference material about understanding text-speak—such as the use of u to mean you or 2nite to mean tonight—and were reminded to practice concision in their text-based responses. In the preexisting live chat service, it was common for personnel to send a pre-scripted welcome message such as “Hello, and thank you for contacting SHSU Ask a Librarian,” but such practices were discouraged in text messaging due to the constraints that the medium placed on message length. Beyond the one recommendation of concision, guidelines were not provided concerning how best to provide quality reference service in the unique context of this new medium.

The service was implemented so that text messages from patrons would be answered as they were received at the reference desk by the individual on rotation. One generic login account was used by all personnel, rather than individual usernames, so patrons did not know which individual responded to their question, and the library could not definitively attribute responses to a specific librarian, paraprofessional staff member, or student assistant.

At the end of the 2012 fall semester, with three full years of data collected, we evaluated the quality of the service being provided. Text messaging is a unique reference tool, due to characteristics such as the restrictions in message length—which renders traditional reference interviews difficult or impossible—and the dichotomy between the medium’s inherent asynchronicity and the student expectation of near-immediate response. The researchers sought to determine how well a librarian’s reference skills translate into this tool and how well librarians are serving customers in the manner which they expect.

We hope that a close evaluation of current performance will serve as a training tool, help to clarify and communicate expectations for text-based reference service, and lead to the development of guidelines and best practices which will improve overall staff performance in this service.

RUSA developed “Guidelines for Implementing and Maintaining Virtual Reference Services” in 2004, based on proposed guidelines from Bernie Sloan’s 1998 article in Reference and User Services Quarterly, which suggested that for almost half a decade the practice of virtual reference itself was very ad hoc.¹ With most libraries being on the same virtual page when it comes to reference services, due to guides like RUSA’s, it becomes important to evaluate how well librarians are using the SMS tool to provide service.

Evaluating newly implemented reference services and assessing librarians’ effectiveness in these services has long been a priority to libraries.² Pomerantz recommends content analysis as permitted by virtual reference services (transcripts are available and the interaction captured between librarian and patron) “because it opens reference interactions up for a range of evaluation metrics” such as those identified by RUSA’s “Guidelines for Implementing and Maintaining Virtual Reference Services”; however, the author discusses chat sessions rather than texting.³ Standards and tools for evaluation have varied for examining virtual reference services prior to the introduction of texting, including survey questions,⁴ a proxy method in which fake librarian patrons asked chat questions during certain times and filled out an evaluation on the service,⁵ and the TAM or technology acceptance model created in 1989 by F. D. Davis,⁶ and later adapted to evaluate the adoption of instant messaging.⁷ The International Federation of Library Associations and Institutions (IFLA) provides digital reference guidelines that contain elements included in rubrics for scoring library services, such as “respond to 100% of questions that are assigned,” “acknowledge receipt of patron question . . . [respond] as quickly as possible,” and “show professional courtesy and respect”; however these are all “guidelines for chat sessions,” which are not necessarily applicable to text messaging, and they are not broken up as in RUSA’s guidelines for specific evaluation of Interest, Follow-up, etc.⁸

According to the Pew Internet and American Life Project, 80% of all cell phone users send or receive text messages.⁹ Libraries use SMS as a system to send patron notices, such as overdue book alerts, library closings, and other information, as well as in reference services. Sims Memorial Library at Southeastern Louisiana claims to be “the first library in the United States to develop a text messaging reference service” in 2005.¹⁰ Over the past decade, both public and academic libraries across the United States have been catching up with the trend of providing reference via text messaging.

Since 1996, RUSA has published “Guidelines of Behavioral Performance of Reference and Information Service Providers,” which provide guidance for information professionals performing reference interviews in order to improve service to the end-user. The original guidelines dealt with face-to-face transactions but grew to include remote transactions, such as telephone, email, and instant messaging. However, text messaging generates unique challenges such as the cost of transaction and word limits that may require the use of text-speak. Luo addressed how RUSA guidelines could be applied to text reference, examining each section (approachability, interest, listening/inquiring, searching, and follow-up) individually, explaining the importance of each portion in text reference, and suggesting how each section might be adhered to while using text reference; however, Luo did not provide any case studies or surveys that used RUSA’s rubric to evaluate their services.¹¹

Using RUSA’s guidelines to create a scoring rubric for chat reference, Maness, Naper, and Chaudhuri scored librarians’ performance “in which a patron was determined to be acting inappropriately” rather than scoring librarians’ performance as a whole with any and all chat interactions.¹² Other libraries have adopted the RUSA guidelines to create a rubric on which to score individual librarians’ chat sessions; however it is unclear who is scoring whom—whether it is the patron receiving the checklist to score the librarian after the transaction or a supervisor scoring librarians on each chat or the librarians’ self-scoring.¹³ Using rubrics based on non-RUSA guidelines and rubrics, libraries such as the University of California Libraries and Maricopa Community Colleges Libraries have assessed digital reference transcripts for chat reference; however, their guidelines recommend only selecting a small sample and using them to explore and discuss problems or good quality service, yielding only a tiny snapshot which does not show the whole picture of reference service.”¹⁴

Publications about text messaging in libraries follow a pattern similar to that seen in articles about instant messaging, one of the latest technologies adapted prior to texting. Devine, Paladino, and Davis clearly group these articles in their paper on chat reference, and these categories can be applied to texting/SMS reference and mobile library services in general: anecdotal papers, “which summarize implementation and training”;¹⁵ “books and articles that present training practices for starting or modifying existing . . . services”;¹⁶ studies that “describe and analyze skills and competencies that are necessary for the effective provision” of the service;¹⁷ and academic library surveys, ¹⁸ “which, at various times, have asked one or more questions regarding the training practices”.¹⁹ To this list, we would add the evaluation of particular software/services—whether in articles jointly written by a library and its service provider or by a group using particular software, or even as a comparison with other software—as well as discussion of how patrons use or perceive the service provided or usability study results.²⁰ Similar groupings are visible in discussing SMS/text reference.

This is the first paper, as far as the researchers can determine, that systematically evaluates and scores librarians’ interactions in SMS text reference, though there have been suggestions that best practices should be created in order to provide better service.²¹ Although a few papers have touched on the concept of librarians’ skills in using the service²² or offering statistics of librarians’ responses such as length of time taken, how many words used per message, and friendliness,²³ this information is typically generalized and not addressed by systematic assessment as in the current study.

We downloaded the complete listing of questions and answers from the Mosio platform. This data included Patron ID codes that uniquely distinguished between patron phone numbers without revealing the actual phone numbers or other personally identifying patron information.

The precise date and time of each question and answer were also included in the data. The researchers encountered some initial difficulty in calculating response time from the data provided, but Mosio’s technical support staff was highly responsive and promptly upgraded the data export feature so that we obtained data with a clearly calculated response time. The researchers are grateful for the company’s responsiveness in providing information and assistance that made this study possible.

All text messages sent and received were grouped by the researchers into transactions; a transaction might consist of several individual messages from patron and librarian, including automatic system responses, representing one logical session of conversation. The final dataset consisted of 385 transactions, or conversational threads. This does not equate to the total number of messages ever submitted to the service, because messages where a question was never raised, such as “test,” “hi,” or similar were not included; this study is instead based solely on actual substantive questions to which a librarian did respond or could/should have responded.

The data was first evaluated statistically with regards to characteristics such as response times and how often patrons revisited the service. A portion of this statistical data was also compared against a seven-month sample of similar data from face-to-face and telephone transactions at the reference desk.

After these quantitative characteristics were measured, the researchers developed a rubric in order to assess how well librarians performed in answering each question (appendix A; also online at http://library.shsu.edu/libfac/TextRubric.pdf). One member of the research team holds a Masters of Education degree, and she led the development and testing of the rubric. A three-scale, analytic rubric was created using the “Guidelines for Behavioral Performance of Reference and Information Service Providers,” as published by the Reference and User Services Association (RUSA), a division of the American Library Association (ALA).²⁴ RUSA’s guidelines helped to ensure content and construct validity by providing a criterion-referenced rubric framework. The guidelines include general, in-person, and remote reference librarian behaviors across the five categories of Approachability, Interest, Listening/Inquiring, Searching, and Follow-up. While a list of remote behaviors is available, behaviors in each of the three formats (general, in-person, and remote) were considered in constructing the rubric.

The framework was then modified to accommodate a focus on text reference services, which necessitated the elimination of the Approachability category from the project. The Approachability category as identified by RUSA for the remote format emphasizes elements of web design, and while an evaluation of a library’s website is certainly an important and worthwhile pursuit, the purpose of this project was to provide training for librarian behavior. The rating scales included Beginning (1 point), Developing (2 points), and Accomplished (3 points).

In addition to eliminating the Approachability category, minor customizations were also needed to accommodate the limitations inherent in a text-messaging service, and the researchers used RUSA’s overall category descriptions to stay on target during the modification stage. For example, RUSA’s description of the Listening/Inquiring category advises that the “librarian must be effective in identifying the patron’s information needs and must do so in a manner that keeps patrons at ease;”²⁵ hence, the descriptor “Uses a tone of voice and/or written language appropriate to the nature of the transaction”²⁶ became the descriptor “Uses concise language, abbreviations if appropriate, limiting responses to one text message (or 160 characters) per patron query” (see appendix A).

The limitations of text messaging as a reference platform also impacted the creation of the Searching criterion. For example, RUSA suggests that the librarian “Finds out what patrons have already tried, and encourages patrons to contribute ideas;”²⁷ since many patrons may be billed per text message, this criterion was omitted from the rubric. Similar minor customizations occurred in each of the four major criteria, with the Searching criterion being the most extensively customized, including the addition of two local rules: The first rule was that librarian answers that were clearly inaccurate to the researchers received the Beginning (1) score, and the second rule was that closed-ended questions that required no actual searching on the part of the librarian (for example, library hours of operation) received the Accomplished (3) score (see appendix A).

The creation of the analytic rubric provided an educational opportunity for us, and following rubric development, we evaluated our draft rubric using A Rubric for Judging Your Rubric, a tool developed by The Center for Faculty Development at the University of Colorado Denver.²⁸ Following minor modifications for measurable and observable descriptors, we normed the rubric for inter-rater and intra-rater reliability using two sets of five transactions across three norming sessions spaced several weeks apart. Next, the evaluation of all 385 transactions was conducted in equal portions by the three researchers. Since staff names were not part of the data collected, the results give a general picture of the Reference Department’s performance without critiquing individuals.

Useful implications are suggested by the results of a rubric analysis of text-reference transactions, particularly the transaction scores, response rates, and patron-return rates. Overall, the average score across the entire transaction pool is promising. Since a score of Developing (2) in all four criteria would earn a total score of 8, the cumulative average score of 9.1 indicates that librarian performance, on average, meets or exceeds the Developing (2) level, even though it does not quite meet the Accomplished (3) level. In fact, nearly three quarters of the transactions scored between 9 and 12, indicating that the majority of transactions exceeded the Developing standard (see figure 2). However, while the average score of all transactions is helpful to appreciate the big picture of text reference performance, a more detailed analysis of each criterion is needed to generate quality feedback.

The rubric used to score transactions clustered criteria into four categories: Listening and Inquiring, Interest, Searching, and Follow-up. The category with the highest average score was the Searching category (see figure 1). As this is one of the most traditional and visible skills in the librarian profession, we find the evidence of a high performance level personally and professionally rewarding. Of greater use in an assessment capacity, however, are the low performance scores. The lowest-scoring criterion was the Follow-up category, with the average situated between the Beginning (1) and Developing (2) rubric scores. The Listening and Inquiring category had the second-lowest score, with the average falling between the Developing (2) and Accomplished (3) rubric scores. Both the Listening and Inquiring and Follow-up categories contain criteria that focus on librarian communication skills: Listening and Inquiring focuses on cordial yet concise communication and Follow-up emphasizes the invitation for library patrons to return if they have additional questions or need more assistance. In the Follow-up category, any transaction that did not invite the patron’s future use of the service did not receive the Accomplished rating, which significantly impacted the overall average for this category. Perhaps the expectation of an invitation to return in each transaction is a bit idealistic, but this criterion is included by the Reference and User Services Association (RUSA), a division of the American Library Association (ALA), and the rating group of three professional librarians expected that the highest-performing library professionals would strive to achieve it. The resulting low scores indicate a clear need for heavier emphasis in initial and refresher training on this aspect of performance.

Another implication of such a low score in the Follow-up category is that librarians may be struggling to balance the goals of providing detailed, quality information with an invitation to return (follow-up), all while being expressly cordial within a single 160-character message. One solution may be to generate a canned response or a signature line for staff use, automatically providing a succinct welcome to return, while emphasizing the most clear text-shortening methods that would still resonate well with the library’s target population. Such signature lines could be as basic as “Use us again!” or as abbreviated as “Thx&T2UL8R” (Thanks and talk to you later). Although the canned response or signature line would take up part of the 160-character limit, the easy availability of a concise, cordial message would contribute to higher-quality text reference by providing assistance that the staff needs, as evidenced by the low Follow-up transaction scores.

While average category scores are useful for looking at the big picture, analysis of smaller data clusters yields more specific trends. For instance, only 18 transactions (4.7%) earned a Beginner (1) rating across all categories, 16 of them because they were never answered (see figure 2). On the surface this may indicate that additional technical training may be needed to ensure that staff are able to recognize incoming questions and then successfully respond to them. An alternate implication is that a higher level of attentiveness from library staff may be needed. The remaining two transactions that did include a librarian response were answered inexpertly enough to receive a total score of 4, representing a Beginner (1) rating in each category. While these were clearly problem transactions, it is encouraging that the area of needed development is in refresher training on identifying “waiting” questions and the technical response procedure, which is less disturbing than a need to retrain groups of librarians on the fundamentals of reference techniques. Only five transactions were rated Accomplished (3) across all four criteria, earning a total score of 12 (see figure 2). Although this is a disturbingly low number, it clearly validates this project’s aim: to evaluate staff performance with an eye to documenting expected standards and best practices, then providing training for library staff.

Unique to the medium of text reference, and therefore one of the main concerns of this rubric project, is the limitation of response length (usually 160 characters). While concision is a goal for many reference services, instant message, email, phone, and even face-to-face reference services have a more reasonable expectation of back-and-forth communication than text reference does. Many users, however, are subject to cell phone plans that may limit the quantity of text messages sent or even charge fees for each separate text message, a scenario that libraries promoting text services should seek to accommodate. With this limitation in mind, responding with a single text message whenever possible is imperative for librarians, even to the point of forgoing an immediate acknowledgement text which would assure patrons that their queries were received and that an answer was forthcoming. Analysis of the collected data reveals that 93% of librarian responses were constrained to a single message (see figure 9). This high result suggests that most library staff are clearly aware of the need to limit response length.

In addition to transaction scores and response length, librarian response times for transactions are also instructive (see figure 3).²⁹ The majority of transactions (59%) received human responses within 30 minutes of the initial question, which was the timeframe established for the Accomplished (3) rating. This thirty-minute window may seem lengthy, given most students’ perception of text messaging as an instantaneous communication medium, but it allows the single librarian on duty at our library’s reference desk to balance the complete range of reference services for which she is responsible, including in-person and telephone reference questions as well as virtual queries by email, instant message, and text message. This time allotment also permits the librarian to locate the requested information and then edit the delivery of that information down to a single text message when possible. The fact that most text messages are personally handled by library staff quickly is encouraging, particularly since 51% of that 59% were answered within ten minutes. Approximately 5.7% of transactions received a human response within thirty minutes to one hour, the timeframe established for the Developing (2) rating, reinforcing the focus on response time as an important aspect of virtual reference.

The remaining portion of transactions (34%) includes the 23% that received human responses in more than 1 hour but less than one day and an approximate 5% that were handled within one to four days, as well as 2% of transactions took more than four days to receive a reply and 4% that never received a human response. This last cluster of response times (34%) points to a clear need for consistent training and reinforcement of expected standards for library staff. Transactions that never received a response (4%) may indicate the need for additional or reinforced technical training with the text reference system: either librarians were not aware of the texted query or were unable to answer it, which could indicate either inattentiveness or an inability to use the system correctly.

Clearly the majority of text messages are being addressed quickly, according to the standards crafted in the rubric for this project, despite a time limit not having been identified in previous virtual-reference training for library staff. Perhaps the initial thirty-minute expected response rate is too idealistic, and a longer timeframe should be the expectation for providing a substantive answer (not merely acknowledging the question). Expected response times differ between a text-reference service and an instant chat service, at least in the context of this study. However, students today rely heavily on texting, which may serve to blur the lines between text reference and instant chat services. Further research into the expectations of both patrons and librarians, with regards to perceived differences in communication methods and expected response times across multiple virtual reference platforms, would inform such discussion.

Closer analysis of the librarian response-time data reveals that Fridays and Saturdays had the longest time between initial question and librarian response, an average of approximately 16 hours on Friday and 20.6 hours on Saturday, compared to the lowest average on Wednesday of around 2 hours (see figure 5). One explanation of this gap may relate to the library’s operational weekend hours. The library closes early on Friday (6 p.m.), opens late and closes early on Saturday (10 a.m. to 7 p.m.), and opens even later on Sunday (2 p.m.). Since averaged response times did not take into account the library’s nonoperational hours, texted queries that were received after closing on Friday were not answered until mid-morning on Saturday, resulting in a 16-hour gap. The same is true of queries sent after closing on Saturday and not answered until Sunday afternoon—a 19-hour gap. While most weekdays averaged only two queries submitted after library closing, the majority of queries submitted outside of operational hours occurred on the day with the latest opening time: Sunday. This may indicate a need to reconsider Sunday’s operational hours or at least the need to explore options to provide virtual reference services during reduced weekend operational hours.

Analyzing response time by month also yields trends to consider (see figure 4). May and June had the poorest response times by a large margin, suggesting that the beginning of the summer semester saw librarians busy with other activities that may have prevented them from noticing text-reference questions. This pattern seems to repeat for other major semesters as well: the opening six weeks or so of each semester saw the poorest response times for that semester, suggesting that librarians, like teaching faculty, may begin the semester with an extraordinary amount of work already on their plate. October and April, on the other hand, are the months with the fastest response times, further suggesting that librarians may focus more effectively on fielding text-reference questions after the semester is well underway. Another possible explanation for poor response time in May and June is that librarian response time may be impacted by the library’s limited operational hours: May includes the spring interim, which sees a reduction in library operating hours, and June also sees reduced hours.

Further analysis of response times by hours indicates that the poorest response times—averaging almost 1 day 5 hours—are associated with questions received between 5 a.m. and 7:30 a.m., the opening time for the library and the few hours before. This may indicate problems with the library staff’s procedure for opening the reference desk or a need for technical training to recognize the presence of a waiting text-reference question. The fastest average response times are seen with questions received between 3 p.m. and 6 p.m., averaging around 3.5 hours. To get a wider perspective on possible response-time issues by hour, data concerning physical reference desk transactions (including face-to-face and telephone, but not chat, email, or text) were also analyzed for the period of June 2012 to December 2012, revealing that the busiest hours of the day at the physical reference desk included 10 a.m., 11 a.m., 12 p.m., and 2 p.m. (see figure 7). In comparison, the busiest text-reference hours (by number of text referenced questions received) were the four hours from 12 p.m. to 4 p.m.

Admittedly, this seven-month sample of physical reference desk statistics, correlated with text-reference activities, provides only a limited quantity of data to compare, so these results are more anecdotal than statistically significant. Nevertheless, the results of the comparison are surprising. The researchers suspected that text-reference response times might suffer during peak periods when reference desk personnel were busy assisting face-to-face patrons. Surprisingly, the opposite was true: the busiest hours at the physical reference desk correlated with the fastest text responses of the day. Similarly, the busiest text-reference hours of the day also correlated with the fastest text response times of the day. The researchers hypothesize that reference desk personnel may become more alert when traffic picks up at the physical desk, thus resulting in better virtual response times as well, due simply to increased attentiveness. This supposition also seems to hold true for the analysis of librarian response time by month: slower months at reference, such as May and June, see a weaker response time, while October—the second-busiest month at the reference desk—sees the strongest response time.

In addition to transaction scores and response times, the researchers also examined the number of repeat patrons (see figure 8). Approximately 11% of patrons returned to use the text-reference service for a second time, while another 8% used it between three and seven times. Admittedly this is a small number of repeat patrons, with only 19% of all users returning to the service again after their first use. However, statistics on repeat patrons were determined based on cell phone numbers. It is possible that at least a few users of the text reference services may have switched phone numbers, and current statistics cannot account for this possibility.

Cell phone technologies aside, these numbers still raise questions about why most patrons do not tend to revisit this service. Many patrons may have been deterred simply by the economics of pay-per-message cell phone plans. This study does not support conclusions about return use, but it is possible that the various factors evaluated in the rubric—such as response time, cordiality, and the amount of detail or direction provided for an information source—played a role in whether a patron chose to use the service again. One goal of the researchers is to collect follow-up data on repeat patrons in the future, after best practices and staff training have been implemented for a reasonable period, to see whether repeat patronage might increase in correlation with improved service. Further research may also be warranted concerning student attitudes toward the service, whether library marketing for this service is reaching students, whether marketing and student expectations match the actual student experience, and how students’ cell phone plans address text messages, including whether pay-per-text plans deter use of the library service.

A final issue highlighted by this study relates to maintenance procedures, especially concerning hours of operation. The researchers encountered multiple transactions which lacked the appropriate automatic system response that should have been sent during nonoperational hours. This was not a technical error in the service platform, but rather a failure of the library’s Web Services personnel to consistently program auto-responses for holiday closures, limited interim hours, and so forth. This emphasizes the need for a clear and consistent maintenance routine so that patrons always receive prompt indications of the library’s status during nonoperational hours, rather than mistakenly awaiting a response while feeling ignored and poorly served.

This study demonstrates that significant issues exist with staff attentiveness to monitoring text reference. Some improvement is also needed in areas of friendliness and follow-up. These findings confirm a need for the reference department to offer more in-depth staff training than has previously been provided. Over the next academic year, the researchers plan to create and implement various support tools to educate staff about performance targets. These tools may include sample message templates or signature lines and guidelines for best practices in providing text-reference service. The researchers hope that the assessment rubric resulting from this study will provide a template for other libraries to adapt and implement in their own self-assessment.

Although the study covers three years of SMS reference data, the total number of transactions is still relatively small. Examination of a more substantial data pool may allow additional insights. The researchers are interested in partnering with other libraries to provide assessment of de-identified text reference data provided by those partners. This would not only increase the quantity of available data but also allow for a broader view of the issue, rather than focusing on only one library.

Additionally, although most questions are answered by professional librarians or long-time paraprofessional library staff, questions may occasionally have been answered by a student assistant, and the use of a single login account makes it impossible to identify the exact staff member who answered a specific question. For the purposes of our study, this does create a minimal amount of uncertainty about whether any lower-quality answers could be attributed to less proficient staff.

Finally, the researchers recognize that it would be useful to examine response time only in regards to the library’s operational hours, rather than including nonoperational hours in the calculated time for response. If a question is asked after closing and a response is sent eight hours later, but within two minutes of the library reopening, is it more accurate and helpful to report the response time as eight hours or two minutes? However, in the context of this study, the researchers determined that it was not feasible to adjust the response time calculations to include only operational hours, largely because the plan for this study did not predate and thus could not influence the manner of data collection. The data examined spanned three years, during which time the library followed at least eight different schedules of hours, and conclusively determining the operational hours of each transaction proved difficult. Furthermore, the response times as reported have been electronically calculated from the mathematical difference between the question’s arrival date/time and the date/time that the first human response was sent; adjusting this calculation to account for operational hours would require manual recalculation for all 385 transactions by subtracting nonoperational hours from the electronically calculated time. This level of manual recalculation would present a significant opportunity for human error.