Two studies in particular explored what motivates a user to choose chat service.²⁰ Though the response categories for this question were not identical across the two studies, some integration can be made.

Table 5 shows that the convenience of chat service is the main motivation for choosing it in each study. A somewhat miscellaneous category of reasons was the second highest between the two studies including familiarity, a rejection of other services, as well as curiosity and serendipity.

Some studies investigated users’ preferred mode of reference services.²¹ Cummings et al. reported that 72 percent (n = 197) of users surveyed would be willing to use chat reference service.²² In this study, users also were asked to rank their preferred mode of reference service: both users inside and outside of the library placed “chat” at the top of the list, above asking a friend, e-mail, reference librarian, telephone, and librarian website. When asking distance students if telephone reference service was preferred over chat reference service, Lee found that only 8 of 34 students either agreed or strongly agreed with this statement while 22 students either disagreed or expressed no opinion.²³ Connaway et al. compared nonusers and users of chat reference and “Net Gen” (born between 1979 and 1994) with adult information seekers to look at reference service preferences. Almost all users of chat services surveyed (100 percent of Net Gen users and 95 percent of adult users, n = 137) indicated that convenience was a significant factor in deciding to use chat reference service.²⁴ Seventy-six percent (n = 184) of the Net Gens stated that chat was the least intimidating form of reference. But 71 percent of nonchat reference users indicated that they prefer face-to-face reference service and 49 percent enjoy face-to-face interactions.²⁵

An important finding throughout all of these studies is that many users were not aware of chat reference service. Connaway et al. reported that the main reason Net Gen students are not using chat reference service is because they did not know that it existed.²⁶ As recently as 2008, focus group interview research showed that all 45 subjects had never heard of chat reference.²⁷ Several studies also reported that participants expressed the belief that a range of reference service types should be available, and some should be available 24/7.²⁸

Ten research questions, in eight studies in the data set, explored variations on the question of users’ satisfaction with chat service. In addition to the direct question “are users satisfied with chat reference?” other aspects of user satisfaction explored included differences in satisfaction between complete answers and referrals;²⁹ the factors that contribute to (dis)satisfaction with chat;³⁰ user and librarian correlations with satisfaction assessments;³¹ satisfaction with outsourced chat service;³² user satisfaction relative to librarians’ performing Reference and User Services Association (RUSA) behaviors during sessions;³³ and whether the degree of formality in the librarian's language correlates with user satisfaction with the librarian.³⁴

For those studies that looked directly at users’ satisfaction, the data show that users were quite satisfied with chat service. Pomerantz and Luo reported 92 percent (n = 292) of the respondents were satisfied or very satisfied with chat service.³⁵ Fifty percent (n = 46) of the students in Lee's study were very satisfied.³⁶ Using “willingness to return” as an indirect satisfaction measure, Nilsen and Ross found that 59 percent (n = 85) of respondents indicated a willingness to return.³⁷

Other findings related to satisfaction are summarized here. User satisfaction was highest with completely answered questions, and users whose questions were referred, or partially or not answered, were far less satisfied that those with a complete answer.³⁸

Comparing user reported satisfaction evaluations with librarians’ evaluations of satisfaction with the same transactions, the average patron-reported satisfaction score was 86.7 percent whereas the librarian's average assessment of the transactions was 74.2 percent. Further, patron data revealed 15 “perfect” satisfaction scores, while librarian data showed no perfect scores, with the highest scores ranging from 86 and 97 percent.³⁹

User provided satisfaction scores based on interactions with local librarians compared with interactions with outsourcing service providers were quite similar ranging from 85 to 94 percent for local librarians and 80 to 93 percent for outsource service providers, although users expressed greater dissatisfaction with outsource chat providers on the dimension of answering the question.⁴⁰

User satisfaction was statistically significantly higher when these six RUSA behaviors were performed by library staff: used patron's name, communicated receptively and listened carefully, searched with or for the patron, provided pointers, asked if question was completely answered, and invited patron to come back. Further, the following five behaviors were statistically significant predictors of user satisfaction: receptive listening, searching with or for users; offering pointers; verifying question was completely answered, and encouraging patrons to return.⁴¹

User satisfaction, (measured by librarian helpfulness) was greater in transactions where the librarians used fewer scripted words, and when librarians used with frequency compensator devices, specifically ellipses.⁴²

Integrating this collection of satisfaction-related findings, the data suggest that users of chat service tend to rate their experiences fairly highly in terms of satisfaction, both in general satisfaction and satisfaction toward particular dimensions of the transaction. Librarians’ assessments of chat transactions, however, were not quite so positive. Users tend to be more satisfied when they receive complete answers, and rate outsourcing chat providers lower when they are unable to fully answer a question, which may be an inherent limitation with outsourcing chat service. User satisfaction was higher in transactions where librarians performed critical behaviors that connect them to the user, including listening and communicating, demonstrating expertise in searching and offering pointers, and providing full closure to the transaction by checking that the answer was complete and inviting the user back to the service. Along these same lines, librarians who were considered to be very helpful used fewer scripted words and used ellipses during the chat transaction.

Eight studies from the data set investigated this theme as a research question. Five additional studies reported findings on the type of question asked, although it was not stated research question for their studies. For this analysis data from both cases were included when possible.

Taking into account the various schemas in use for categorizing the questions, no one question type was predominant. Looking at the two highest percentage question categories reported by each study, four basic question types were observed with nearly equal frequency: reference (6);⁴³ specific search/known item (7); policy/procedure (5); and information/directional (6). The one outlier is the category for circulation-related questions, which was only observed in the single study reporting data from a public library chat service. Table 6 reports the findings on question types.

A few studies examined the questions asked of the chat service from the standpoint of how the question was subsequently handled by the operator. The two most commonly used outcome measures were: (1) measures of completeness of the answer, and (2) question referral. Table 7 summarizes the findings from three studies that measured the extent to which the answer provided was a complete answer.

These numbers show that in the studies included in the data set, relatively few questions received a partially or fully incomplete answer. Across the data set, the highest percentage of questions reported not answered in real time (a measure of incompleteness) came from questions that were handled by chat operators within a consortium, but not from the local library staff, thus it might be expected that being less familiar with the specific patrons would result in a greater degree of questions not fully answered.⁴⁴

The other outcome path studied in the data set was the percentage of questions that were referred by the chat operator, either to another librarian on staff, or to a users’ home library institution. Table 8 summarizes the percentage of questions that were recorded as referred to another librarian or another service.

The number of questions that were referred varied among the studies included, but followed predictable patterns. The Kwon and Wikoff studies that reported the largest percent of questions referred were both analyzing chat transcripts from consortial-based chat services where not all the operators were from the same home institution as the user, and thus were more likely to be unable to answer specific account questions or other local kinds of questions. Kwon reported on a public library chat consortia where circulation-related and local library information questions were the most referred (44.8 percent and 39.3 percent, respectively).⁴⁵ Wikoff found that 15 percent (n = 32) of the percentage total questions referred (33 percent, n = 69) were because the questions were deemed more appropriate for home libraries.⁴⁶

In making sense of what the research shows about the questions asked of chat service, in nearly all of the studies that examined the type of questions asked of the chat reference service, there were not large differences among the percentages of categories of question types, indicating a fairly balanced distribution of question types. Even accounting for different manners of categorizing the questions, these data show that users are asking a range of questions through chat reference. This suggests that users of chat service do not associate this medium of communication with one or only a few types of information needs. Rather, users of the service are willing to express a variety of information needs via chat, which requires that libraries equip their operators to handle that same variety of questions.

In the case of consortial chat services, this can become problematic. The benefit of consortial chat service is obvious; greater access for many users with modest investment by any one library. But the trade-off comes with a lower degree of familiarity with local library information needs, and a lesser degree of customization of service. Libraries must consider those outcomes bearing in mind the impact on the user. If a user with a question about an overdue book reaches out to her library via chat, and through a consortial arrangement ends up interacting with an operator not from her local institution who is unable to answer the question, is she better off for having had easy access to chat service? Or is she is more likely to be frustrated for not having her information need met? Making referrals in chat reference also illustrates this point. Referring a user to another person or service is in many cases the best answer a librarian can give, and in one respect is a complete answer. But from the user's perspective, does the instant access nature of chat bring the expectation of a full and final answer in real time, so that a referral might be considered a less than satisfactory answer? Kwon found that users were most satisfied when receiving completely answered questions.⁴⁷ Their satisfaction with referrals was statistically significantly lower than with complete answers, and was at about the same level as those who received an incomplete answer or no answer.

Researchers explored an array of stylistic components revealed in chat sessions transcripts, including the use of abbreviations, contractions, acronyms, emoticons, sentence fragments, dropped words, punctuation, capitalization, font, slang, colloquialisms, and scripted messages. These features of chat reference were interpreted through different lenses: as characteristics of chat;⁴⁸ as a basis with which to compare chat with F2F reference;⁴⁹ as indicators of relational elements of the interaction;⁵⁰ and as indicators of (in)formality in the interaction.⁵¹

A general conclusion from these analyses is that such stylistic elements affect the degree of formality of the interaction between the librarian and the user. A number of studies reported that in general chat users invoke stylistically informal features more frequently than librarians do, although, as the interaction progresses, the librarian may mirror the level of formality employed by the user.⁵² Further, the use of these features to increase the informality of the exchange can have the effect of maintaining positive face between the librarian and the client and may for younger users “transform the reference transaction into a more familiar form of discourse.”⁵³ Maness explored the co-occurrence of stylistic features of chat and user satisfaction and found evidence to suggest that use of scripted words negatively relates to user satisfaction while use of ellipsis (as a nonverbal compensator) positively relates to user satisfaction.⁵⁴

A wide range of contextual elements were observed in chat transcripts including:

• Showing vulnerability, excessive gratitude, self-deprecation, apologies, mild humor, group identity, and invoking library policy⁵⁵
• Greeting and closing rituals, rapport building, deference, self-disclosure⁵⁶
• Reaching shared meaning through dynamic, constructed, situation-specific processes, assumptions of trust and mutual reciprocity, indexicality (meanings conditional on the situation) and the use of classification, instruction and instructed action, and sequential organization of the interaction⁵⁷

These contextual elements were studied as indicators of the degree of formality in the chat interaction;⁵⁸ as indicators of facilitators and barriers in relational communication;⁵⁹ and as indicators of the techniques and communication features used in the meaning making process that occurs through a chat interaction.⁶⁰

A general conclusion to be made from the studies that focused on contextual elements is that the chat medium is rich in context, even without the nonverbal elements present in face-to-face communication. These studies demonstrate that just as in face-to-face interactions, communication through synchronous, virtual communication is also subject to the fundamental complexities of human interaction: creating socially constructed understandings through linguistic, semantic, syntactic, and conversational devices, under the imposition of cultural, social, institutional and individual norms.

Two studies explored the overall service quality of chat service, and both studies supplied a list of dimensions of quality through which chat service could be evaluated. White et al. determined that an efficiency dimension could be assessed by considering the extent to which there was a focus on the main objective of the question combined with the length of the session, minus any down time during the transaction.⁶¹ Their effectiveness measure came from considering the accuracy and completeness of the answer provided. Their research also revealed a need for a measure of the quality of the experience of the chat interaction, which they propose should include the accuracy of the answer, the extent to which there was a focus on the main objective of the question, librarian traits such as patience and helpfulness, the interaction, down time, questioner frustration, time spent in the queue, lag time, and technical problems.

Pomerantz et al. evaluated service quality of chat sessions with thirteen items that can be grouped into these categories: characteristics of the answer, librarian behaviors; user satisfaction, and librarian traits.⁶²

Finally, Radford set out to determine what relationship may exist between the content and relational dimensions in determining the quality of chat service and concludes that service quality assessment can only truly come from asking the users of the chat service.⁶³

This whole category of research is a significant departure from some of the earlier research on chat reference, by moving beyond counting of types of questions, categories of users, times of day questions come in, or length of chat session. The stylistic and contextual findings integrated here suggest a much richer, and arguably more valuable line of inquiry for research that will have a much greater impact on service.

Two studies report that in both IM and commercial chat service, instruction is being provided, 83 percent (n = 146) and 82 percent (n = 118) of the time (when possible) respectively.⁶⁴ Ward reported that two instruction elements were present 78 percent (n = 72) of the time in the transcripts reviewed, and one of the two elements was found in 12 percent of the transcripts reviewed.⁶⁵ In Ford's study, 53.92 percent (n = 102) of chat reference transactions included instruction.⁶⁶

The following instructional techniques were found to be used through chat service, in varying degrees of frequency: resource suggestion;⁶⁷ modeling (explaining how to find information);⁶⁸ explaining how to use information sources;⁶⁹ term suggestion;⁷⁰ leading (explaining in step-by-step fashion while the user follows along);⁷¹ providing search tips and tricks.⁷²

Data show that users do want instruction through a chat interaction. In a pair of studies, Desai and Graves report that 43 percent (IM, n = 146) and 52 percent (chat, n = 136) of the transcripts reviewed were occasions where users asked for and received instruction.⁷³ Further, 50 percent (IM) and 30 percent (chat) of the transcripts showed that users did not ask for instruction, but still received it.⁷⁴ In only 2 percent and 3 percent respectively did the transcripts show that a user asked for, but did not receive instruction.⁷⁵ When users in the IM study were surveyed, 62 percent (n = 50) reported either a desire or willingness for instruction, while 30 percent were indifferent toward it.⁷⁶ From the chat study user survey, 82 percent (n = 62) either wanted or were willing to have instruction, while only 15 percent were indifferent.⁷⁷ Finally, 98 percent (IM) users and 92 percent (chat) users reported that chat was a good way to learn.⁷⁸

The findings in these cases are positive toward the idea of offering library instruction via chat. They demonstrate that instruction can be provided (and evaluated) in chat service, that many chat interactions include instructional elements, and that users do ask for instruction. It is important here to note that the samples for these studies all come from academic libraries which may place more emphasis on providing instruction in any reference transaction, compared with public library chat services.

From the studies in the data set, only two sought exclusively to evaluate the staff's views of the chat service. In the Casebier study, a questionnaire was issued in 2003 and 2004 and the results show only lukewarm attitudes toward chat service.⁷⁹ Results from the 2003 survey period showed that 50 percent (n = 10) believed chat was not the best tool for instructing users on database searching; 80 percent of those who regularly staffed the service did not enjoy it; and 60 percent felt it did not enhance reference services. In the next year, the outlook improved somewhat. 66 percent (n = 19) believed chat service did enhance services at the library; only 33 percent did not enjoy it, and 66 percent felt is should be used mostly for ready reference.

Huston also used a survey instrument to gather the attitudes and opinions of reference librarians regarding the uses, impact and feasibility of chat reference.⁸⁰ 71.8 percent (n = 103) either agreed or strongly agreed that they had the technology skills necessary to perform chat reference, and over half (63.2 percent) believed that their coworkers also had the necessary technological skills. On the other hand, 64.1 percent either agreed or strongly agreed that it was difficult to keep up with emerging technology. In general, the librarians surveyed expressed a concern for insufficient staffing and training, with only 16.5 percent expressing no opinion or not agreeing that chat reference implementation had increased or would increase their workload. An interesting finding in this study is that over half (61.1 percent) of librarians did not agree that there would be an overwhelming demand for chat reference.

Comparing commercial chat service software with instant messaging (IM) software as alternative means for offering chat, Steiner and Long found that more librarians preferred IM over chat, younger librarians tended to favor IM, and 40 percent (n = 302) believed IM was insufficient for handling in-depth reference questions.⁸¹ Ward and Kern examined similar issues and reported that staff were hesitant about learning new software, but over time, many came to express a preference for IM.⁸² However, the increased volume of chat transactions that came as a result of adding IM as a service point caused stress among librarians.

To understand what librarians perceive as successful chat reference transactions, Ozkaramanli interviewed 40 librarians employing critical incident technique.⁸³ The study found that successful and unsuccessful chat reference transactions were largely determined by the attitudes and behaviors of both librarians and patrons. Eleven librarians indicated that communication skills added to the success of a transaction; ten stated that knowledge of reference work and sources was also important. 25 percent noted that the ability to adjust to a new format (i.e. chat) and deal with the unexpected would also contribute to a successful chat reference transaction. The reference interview and question negotiation were also important, with seven librarians indicating that overall success of the interview was helpful and 12 librarians stating that it was important to find out exactly what the patron wants.

In this study, librarians were also asked to consider how users impacted the success of a chat reference transaction.⁸⁴ Ten librarians thought it was helpful when patrons were familiar and/or comfortable with chat and 8 indicated that patrons should also know what they want. Over a third of librarians interviewed (35 percent) stated that appreciative patrons also lead to a successful chat reference transaction. 25 percent stated that less successful transactions were a result of communication difficulties, such as patrons being impatient (6)⁸⁵, demanding (5), frustrated (4) or unengaged (2). When asked how chat service could be improved, librarians gave the following suggestions: have more librarians involved, improve hours of operation, provide formal training, and provide more practice and hands-on experience.

Luo explored specific competencies for providers who offer chat reference services.⁸⁶ From reviews of literature and interviews with librarians, she arrived at a list of 30 competencies in 8 areas and then validated those competencies through a survey of other librarians. Results showed of the 30 competencies, 21 were considered essential chat reference competencies, which were grouped into categories of general reference competencies (e.g., reference interview skills), reference competencies highlighted in chat (e.g., ability to work under pressure), and reference competencies specific to chat (online communication skills).

RUSA guidelines also served as an indicator against which to assess chat provider competencies.⁸⁷ Walter and Mediavilla found that in 114 transcripts which ultimately ended in a referral to a homework help service, very few of the RUSA guidelines were observed.⁸⁸ Most of the transactions analyzed showed evidence of a friendly greeting and clear communication, but analysis found that the providers’ did not typically probe for further information nor did they check that the information was clearly understood. Similarly, van Duinkerken, Stephens, and MacDonald, examined 1,435 chat transactions for the presence of RUSA-recommended behavior.⁸⁹ Their data showed strong evidence of the behaviors associated with the RUSA categories of approachability and interest, but found mixed results in the remaining categories of listening/inquiring (i.e., rephrasing the question, using open and closed end questions appropriate in the reference interview), searching (i.e., asking the user what searches they may have already tried, helping the user broaden or narrow their topics), and following up (asking if the question was fully answered, encouraging the user to return to the service, inviting the user to call or visit the library).

Harmeyer approached RUSA guidelines from a different perspective, asking how their presence might contribute to an accurate answer.⁹⁰ 333 transcripts were analyzed in this study, with particular attention paid to the RUSA competencies of librarian interest, librarian approachability, and question negotiation. In the category of librarian interest, 5 variables were measured: hold time, gaps between the librarian's typed responses, service time, the librarian's response to the patron's “Are you there?” statements and the number of URLs co-browsed by the librarian with the patron. Significant correlation was shown between answer accuracy and only 2 of those variables: gaps between the librarian's typed responses and total service time. On average, the gap between responses was 3.8 minutes. Librarians with response gaps of 1.85 minutes or less reached statistically higher levels of accuracy. Harmeyer indicated that, though overall a short gap time is related to a higher accuracy score, it is a weak relationship. In terms of service time, the mean value was 16 minutes, and transactions lasting 8.3 minutes or less reached higher levels of accuracy than those over 8.3 minutes. Again, though shorter service time is related to a higher accuracy score, the author notes that this is also a weak relationship. Librarian approachability was measured by a sense of friendliness and a lack of using jargon. The study showed that neither variable had significance with answer accuracy, and, in most transactions, the librarian showed friendliness (89.2 percent) and did not use jargon (95.5 percent).

In the category of question negotiation, Harmeyer considered 4 variables: the use of open-ended questions, the use of close-ended or clarifying questions, asking the user if the question has been answered completely (follow-up question) and whether the librarian maintained objectivity.⁹¹ Significance was found with all variables, except for the last. Librarians who did not ask open-ended questions and should have were less likely to give an accurate answer than those who did not ask this type of question and did not need to. Similarly, librarians who did not ask close-ended or clarifying questions and did not need to were more likely to provide accurate answers than those who should have asked these types of questions and did not. For 45 percent of the transcripts analyzed, the follow-up question was either not present or not appropriate. The study found that, overall, librarians who asked a follow-up question, whether or not the transaction called for it, achieved higher levels of answer accuracy than those who did not.

The strength of the systematic review as a method for integrating knowledge is illustrated when pairing some of the findings from the staffing category with the user category. Data on user satisfaction indicates that users are more satisfied when librarians communicated receptively and listened carefully, asked if the question was completely answered, and, invited the patron to come back.⁹² Yet, van Duinkerken et al., reported inconsistent use of librarians’ listening/inquiring behaviors, and with following up behaviors such as asking if the question was fully answered, encouraging the user to return to the service, and inviting the user to call or visit the library.⁹³ This highlights a potential service gap area that chat service providers should examine in their local practice.