Librarians Applying Information Literacy Standards as Evaluators of Peer-to-Peer Course Content in a First-Year College Success Course

Peter Klubek

Peter Klubek (klubekp@mybrcc.edu) is Reference Librarian/Assistant Professor at the Magnolia Library, Baton Rouge (LA) Community College.

Information literacy is a crucial skill in library science. As experts in information literacy, librarians are able to both use and promote techniques relevant to finding, evaluating, and presenting information. They are relied on by end users to provide the most pertinent resources, and are expected to do so as part of their jobs. This study describes how librarians applied their professional background in the evaluation of student peer-to-peer material in order to provide the most appropriate content for an introductory college success course.

Peer-to-peer learning is a fairly new instruction model that encourages active learning, engagement, and student-to-student communication. O’Brien et al. describe peer-to-peer learning as a process where students learn from one another, and student teachers can better reach student learners because of shared perspectives. In simpler terms, it acts as a tool that creates a bridge between teachers and pupils.1 That said, it is important that the information students are sharing with one another in peer-to-peer learning sessions is accurate and authoritative. The ability to determine accuracy and authority is one of the key tenets of information literacy, and librarians use this skill when they evaluate peer-to-peer content.

Information literacy is a multifaceted topic that academic librarians address on a daily basis. Understanding how information is presented and how it may be interpreted is a basic principle of information literacy, and this skill is commonly taught to first-year students in contemporary higher education. Demonstrating how to effectively evaluate and interpret information is also one aspect of what the college librarian does when assisting freshmen during their transition from high school to college.

Oakleaf and Owen (2010) examined collaborative efforts between high school and college librarians. The librarians’ goal was to identify areas of overlap in students’ information acquisition skills between senior year of high school and the first year of college in order to enhance these skills and ultimately increase college retention rates. Specifically, the college librarians could use this information in the development of information literacy instruction, course session development, and in the improvement of librarian instruction skills. High school librarians could also use these results to pinpoint the skills needed for college-bound high school students.2 In this example there were benefits for both library populations: those in high school and those in college.

Smalley (2004) also examined the impact of a school librarian presence on information literacy skills, by studying freshmen at a small open-admission college. The library courses taught at this college drew students from three local school districts, two of which lacked the support for full-time librarians. Students that graduated from these two districts and enrolled in the library research course were compared with those from the district that did have librarians. The students’ research skills were assessed and the comparisons were drawn from midterm grades and final grades.3

At midterm, 57 percent of the top third of students enrolled in a library research course came from the district with librarians. When final grades were turned in, 66 percent of those from the district with librarians earned As, compared to 43 percent and 37 percent, respectively, for the two districts that did not have librarians.4

A similar paper (“What Works” 1997) compiled comments from dissertations on the topic of high school students in the transition from high school to college. The authors of these dissertations found that those students with previous access to an academic library were better prepared to conduct research in college courses, evaluated more resources, and referenced more sources of information in their papers than those without this access and background. High school teachers were found to focus on their individual disciplines and rarely gave a lesson on library use or material evaluation. The task of teaching information use and evaluation was generally left to the school librarians.5

These studies all provide strong evidence that librarians serve as instructors of information literacy skills for first-year students At Baton Rouge Community College (BRCC), a two-year college in the Louisiana state capital with an approximate enrollment of nine thousand students, librarians regularly serve as facilitators of information literacy to all students, including incoming freshmen. Traditional methods such as one-shot bibliographic instruction sessions are held, workshops for faculty and students are offered, and a course in basic information literacy and library science is taught every semester. These efforts are very similar to those described in the studies mentioned above, and they also help bridge the information skills gap between high school and college. But beyond these roles, the actual use of librarians as content evaluators of material presented by students in class is an innovative application that was not examined in these studies and has not been attempted.

At BRCC, a three-credit-hour course titled College Success Skills is taught each semester. This course introduces students to college and college life, and builds the skills needed to navigate and succeed in college. Some of the topics covered include how to select a major, how to manage time, how to budget money for college, and how to conduct research effectively and use library resources. Instructors of this course are called college success faculty, just as English instructors are referred to as English faculty, history instructors are called history faculty, etc. The college success faculty have diverse backgrounds of study, with many coming from the liberal arts. The faculty that participated in this study consisted of two members with degrees in psychology, one in English, and one in higher education. The college success faculty wanted to create an online peer-to-peer handbook to be put on the college website and used as a guide for the college success skills class. These instructors also wanted this information to be authoritative, reliable, and not just the voice of student opinion. This project required that students apply information literacy skills learned from their college success class and construct online compositions in the form of digital handbooks. This created an opportunity for a collaborative effort between the college success faculty and the library faculty. Three peer-to-peer student handbooks were reviewed for potential inclusion in the college success course. Because of the unique information literacy skills they possess, the librarians were called upon to evaluate the virtual handbooks, examine the content, and identify which handbook presented the most useful information. The handbook, or combination of handbooks, identified as having the most accurate information could then be placed on the website for in-class use and general consumption by any user visiting the website.

The purpose of this research is to use information literacy standards and an established evaluation tool in order to evaluate and disseminate student handbooks created for peer-to-peer learning. The handbooks were written by students and for students, and their content and nature were considered for inclusion in classroom use and website distribution. The use of peer feedback in the classroom as a learning tool is not without precedent; however, a review of the literature reveals that a professional application of information literacy standards to this feedback mechanism is an educational tool that has yet to be utilized.

Literature Review

The studies described above establish that information literacy skills offer users of those skills an authoritative voice when presenting information. But in order to begin an evaluation of the handbooks, learning processes must be examined, including peer-to-peer learning.

Peer feedback is an active learning strategy that has gained prominence and is being practiced in more classrooms of higher education. Asghar (2010) outlines how Reciprocal Peer Coaching (RPC) was successfully implemented in the physiotherapy courses at Leeds Metropolitan University in the United Kingdom. She begins with a description of the program and illustrates how self, peer, and tutor assessment were essential elements of the program. Through the utilization of RPC, Asghar states that student learning developed in an environment that supported cooperative learning but also had the advantage of tutor support. She adds that students who used RPC techniques developed mutual goals while they simultaneously developed individual accountability for their learning outcomes.6

The practice of RPC, as described by Asghar, involved the creation of student groups of no more than four individuals who participated in individual assessment sessions. Prior to the start of each session, the instructor explained what was being assessed and what was expected of each learner in that particular session. At the conclusion of this first part of the session, participants were asked to complete a reflective skills log on what they just learned. Following this RPC exercise, the instructor would then select one individual from each group to demonstrate the skill explained in the assessment. Credit was given to the group as a whole when all aspects of the assessment were fully explained by that one individual.7

Asghar reports three outcomes were achieved when these methods were tried at the university level, including motivated learning, learning as a group, and contextualized learning. The author states that each of these outcomes was a driver of learning and the use of RPC methods developed a sense of self-regulation among learners. She concludes that these techniques ultimately helped students develop the skills necessary to be autonomous learners.8 In this study, peer-to-peer learning was applied through group efforts using traditional classroom models. However, with the rise of blended and on-line learning environments, new opportunities for student-to-student learning have also been created.

Ertmer et al. (2010) report on student reactions to peer learning in a large undergraduate course. The course used in this examination incorporated elements from online learning and face-to-face learning. These authors state that using this blended approach offered them the opportunity to examine the use of educational technology while still incorporating the human interaction element. The technique most readily available to the students that combined these elements was feedback to student discussion posts. The authors add that in an online course environment, little time is available for the instructor to provide the feedback required, and that peer feedback may function as an alternative.9

In order to gauge the effectiveness of peer feedback, the authors evaluated student perceptions using a rating system and scale. The students were provided instructions on the rating system and scale, and points for participating in this research. One group was permitted to respond to fellow postings with peer feedback, and the other group was not. The second group only received feedback from the instructor. A series of discussions were then created using the discussion board format on the learning management system Blackboard. Upon completion of the discussion board task, students completed a Likert scale–based survey on their experiences of peer feedback in the discussion forums. The authors report that the peer feedback group rated their experience more positively than their counterparts did theirs.10 In this example, student perceptions of peer feedback were tracked and were found to be beneficial by participants.

Electronic methods of integrating peer feedback were also used by Willey and Gardner (2010), who incorporated Spark Plus to encourage student peer feedback and assessment. These authors report that these processes were very successful for individual students in achieving the learning outcomes in specific classes. They note that participants found the peer learning exercises increased course engagement and helped them to learn more efficiently.11

Ertmer et al. and Willey and Gardner relied on new technologies to incorporate peer-to-peer feedback. Asghar used more traditional classroom methods to incorporate this learning method. All three articles reported excellent participation and greater learning through the use of peer feedback. These are the outcomes that the college success faculty at BRCC were trying to duplicate by assigning their students the creation of a peer-to-peer handbook. The concept was taken one step further by adding the element of evaluations by librarians. Parallel to the Ertmer et al. and Willey and Gardener reports, the processes in this experiment were focused on the use of technology. As more electronic tools for education and peer-to-peer content creation become available, new ways of evaluating these results must also be developed, including the involvement of information professionals. The Asghar report used the traditional learning method of having students present their work to their fellow students. The project in Louisiana was similarly focused, but reached a broader audience through Internet use and involved librarians before distribution.

Methodology

Copies of three handbooks, identified by the student editors’ last names and labeled Handbook One, Handbook Two, and Handbook Three, were distributed to all fifteen college success skills instructors and all five full-time librarians. None of the participants were familiar with the handbook authors, and by using only the student editors’ last names, anonymity could be maintained. A modified version of the evaluation rubric created by Peeters and Sahloff was also distributed to each evaluator.

Peeters and Sahloff (2010) designed a rating rubric for evaluating student presentations for a capstone pharmacy course at the University of Toledo (OH) College of Pharmacy and Pharmaceutical Sciences. The design for this rubric included twenty-five categories with a four-point rating scale for each category. In their paper highlighting this rubric, Peeters and Sahloff emphasize the importance of student evaluations through presentations in education. They state that evaluation of student presentations represents a higher order of learning because students must demonstrate and talk about what they have learned, incorporating aspects of both knowing and doing.12 This pedagogical concept is in keeping with the goals of peer-to-peer learning, which made it ideal for this study. Additionally, the rubric design was tested to determine reliability. The reliability rating was found to be at 98 percent.

The Peeters and Sahloff rubric also aligns to the first four Information Literacy Competency Standards of the Association of College and Research Libraries (ACRL).13 The first ACRL standard (the student “defines and articulates the need for information”) is covered by Peeters and Sahloff’s category for evaluating the stated objective of the information, and the category that identifies the opening statement and relevance of the information to the audience. The second standard (the student “accesses needed information effectively and efficiently”) is addressed in the category of references and the category of appropriateness of selected literature. Standard three (the student “evaluates information… and incorporates [it] into his or her knowledge base”) is included in the rubric’s category of balanced representation of material, its category of application of material, and its category of generating discussion, responses, and questions. The fourth standard (the student “uses information effectively to accomplish a specific purpose”) is addressed in the “slide effectiveness” category and the “organization of handbook/planned coherently” category.

This rubric was also selected in part because it evaluates Microsoft PowerPoint presentations. The handbooks were created using PowerPoint, so the evaluation rubric complemented the finished handbooks despite their not being actual presentations. Modifications to the Peeters and Sahloff rubric for the handbook evaluation included removal of sixteen category items specific to presentations (for example, good eye contact, spoke with a clear voice, timing). Items related to information imparted (slide effectiveness, content, organization, cited sources, etc.) remained intact and were used by the college success faculty and library faculty in evaluating the student handbooks.

Nine categories from the Peeters and Sahloff rubric were used by both the librarians and the college success faculty in evaluating the handbooks created using PowerPoint. These categories included: PowerPoint slide effectiveness, references, stated objective, opening statement/relevance to audience, balanced representation of material, appropriateness of selected literature, organization of handbook/planned coherently, application of material, handbook generates discussion/responses/questions. A copy of the rubric used in this study can be seen in appendix A.

Participants were asked to rate each handbook using the criteria set by Peeters and Sahloff. The evaluators also had the opportunity to write any comments based on these handbooks. The handbooks were evaluated by a total of eight individuals. Four out of five librarians responded for a response rate of 80 percent, while four out of 15 college success skills instructors responded for a response rate of 27 percent. The total response rate was 47 percent.

In order to maintain the anonymity of the evaluators, evaluations were returned from each respondent to the principal researcher and assigned a number before the responses were examined. Response rates and comments were then examined by this study’s author and the three handbooks were ranked first, second, or third. The evaluation data were analyzed by the average mean of the nine criteria ratings (evaluators’ scores combined) of Peeters and Sahloff for each handbook, and where appropriate, a paired mean comparison, Students T test (Snedecor and Cochran, 1980), was made between the selected average means.14 Statistical significance was set with an acceptable error of 5 percent.

Results and Findings

A copy of the modified Peeters and Sahloff rubric is available in appendix A. Each category is examined in the following results and findings using the heading created by the authors for evaluation. It should be noted that reviewer one gave the same scores in each category to handbooks one and two, and gave zeros for handbook three. Reviewer four rated each handbook the same across every rubric category. For the three reviewers that provided comments only (reviewers two, seven, and eight), scores of zero were assigned to their ratings and used in the calculations.

Under “PowerPoint Slide Effectiveness” (figure 1), Handbook One was rated the highest, with only reviewer four giving a non-zero rating lower than 3 points. Only Handbooks One and Three received 4 points from more than one reviewer, and none of the handbooks rated below 2.0 points, excluding ratings of zero. In the comments section, reviewers described Handbook Three as having the most visually appealing presentation of information.

For “References” (figure 2), Handbook One again received the highest rating. Handbooks One and Three each received a rating of 4 points from three reviewers. Only two reviewers gave Handbook Two 4 points. In addition, reviewer four commented that Handbook Two needed to present facts in a more concise format, and that Handbook Three effectively used sources to support individual topics. Reviewer six commented that Handbook Two presented an inaccurate description of the library, which caused the reviewer to question the validity of other statements. Reviewer seven stated that the data provided in Handbook Two regarding transfer credits were confusing, and that the references in Handbook Three were poor. Reviewer eight wrote that slides for Handbook One were crowded with information, which made it difficult to find the cited sources. (See appendix B for written comments.)

Under “Stated Objective” (figure 3), Handbook One scored the highest, and Handbook Two scored above Handbook Three. Handbooks One and Three each received ratings of 4 points from three different reviewers. Handbook Two received 4 points from reviewers five and six only. Reviewer six also commented that Handbook Three provided an objective and then gave detailed information in support of each thesis. Reviewer eight wrote that the objectives were listed in a logical beginning in Handbook Two. (See appendix B for written comments.)

For “Opening Statement/Relevance to Audience” (figure 4), Handbook One received the highest ratings, with a score lower than 3 points from only reviewer four. Handbook Two was next, with reviewers one and three rating it 3 points or above, and five and six each giving a rating of 4 points. Handbook Three was last with only reviewers four, five, and six providing a score of 4 each. There were no written comments for this section.

“Balanced Representation of Material” (figure 5), had scores with Handbook Two leading and Handbooks One and Three tying. Handbook Three received marks of 4 points from reviewers four, five, and six. Handbooks One and Two each received 4 points from two reviewers (reviewers one and five for Handbook One, and reviewers five and six for Handbook Two). Reviewers one and three both rated Handbook Two at 3 points. Reviewer three rated Handbook Three above 3 points with a 3.1 score, and Handbook One below 3 points with 2.8 score. The remaining scores for each handbook were below 2.5 points. Reviewer six commented that Handbook One lacked a statement about textbooks on reserve in the library. Reviewer eight stated that Handbook Two did not provide a fair and balanced representation of the material, and used outdated information in one chart. Reviewer eight also stated that Handbook Three unfairly described the advising services. (See appendix B for written comments.)

Under “Appropriateness of Selected Literature” (figure 6), Handbook One scored the highest with reviewers one, five, and six giving a rating of 4 points. Reviewers five and six provided a rating of 4 points for every handbook. Reviewer four rated both Handbook Two and Handbook One below 3 points. Comments in this section were mixed, with some reviewers offering contradictory opinions. Reviewer four stated that all the sources referenced in Handbook Three supported the topic for each section discussed. Reviewer six commented that Handbook Two was thorough and that the handbook could be presented as a credible source. However, reviewer seven stated that Handbook Two’s information about advising was prejudiced, and was also confused about its information regarding transfer credits. (See appendix B for written comments.)

For “Organization of Handbook/Planned Coherently” (figure 7), Handbooks One and Two tied. Reviewers one and five each rated Handbook One at 4 points. Reviewer three gave this handbook 2.32 points, and reviewer six gave it 3 points. Handbook Two received 4 points from reviewers five and six, while also receiving 2.32 points from reviewer three. Reviewer one rated Handbook Two at 3 points. Handbook Three was rated with 4 points by reviewers four, five, and six, while reviewer three gave it 3.8 points. Reviewer four said that Handbook One needed better organization to help with the flow of contents, but noted that Handbook Three was well organized. Comments from reviewer six for Handbook One included a recommendation to break the afterword slide into three or four individual slides. Reviewer eight commented that there was too much text provided on each slide for Handbook One and that Handbook Two was comprehensive and offered a logical beginning. (See appendix B for written comments.)

Under “Application of Material,” (figure 8), Handbook One scored highest, followed by Handbook Two, with Handbook Three last. Handbook One received 4 points from reviewers one, five, and six. Handbook One also received 3.5 points from reviewer three, who gave this score to all handbooks. Handbooks Two and Three each received 4 points from reviewers five and six. Handbook Two edged out Handbook Three where reviewer one gave this handbook 3 points, and zero for Handbook Three. Under this category, reviewer six commented that the information in Handbook Three could be applied to students at any university or college. Reviewer seven stated that the information in Handbook One could be applied as long as it was prefaced by a statement acknowledging that this was the work of students and not the view of the college. Reviewer eight made similar comments on Handbook One. (See appendix B for written comments.)

For the final category of “Handbook Generates Discussion/responses/questions,” (figure 9), Handbooks One and Two tied with Handbook Three trailing. Handbook One received 4 points from reviewers one and five, 3.8 points from reviewer three, 3 points from reviewer six, and 2.32 points from reviewer four. Handbook Two received the same total number of points but from different reviewers. Reviewers five and six rated Handbook Two at 4 points, reviewer three gave 3.8 points, reviewer one gave 3 points and reviewer four gave 2.32 points. Handbook Three was rated by reviewers four, five, and six at 4 points, and 3.8 points from reviewer three. There were no written comments for this final section.

Comments from the evaluators for all categories can be found in the appendix. Based on the average score across reviewers and categories, Handbooks One and Two were rated significantly higher than Handbook Three. Handbook One had an average score of 2.12, Handbook Two had an average score of 2.07, while Handbook Three had an average score of 1.89. Nevertheless, despite the higher ranking for Handbooks One and Two, it was generally decided by the librarians and the instructors that rather than publishing an entire handbook, sections could be posted in Blackboard within the appropriate module for that topic. For example: The portion describing time management and organizing your social life while in college could be added to the module on “Time Management.” It was also found that Handbook One had highly skewed information related to advising. The advising portion would have to be reworked before posting on Blackboard in order to provide the most accurate information.

The general consensus of the librarians and the instructors was that when posting a handbook section or module on Blackboard, a disclaimer should follow indicating that these are the views of students and that the opinions expressed are not necessarily those of the instructor or the college. Such a policy would help keep the tone of the handbooks neutral.

Conclusion

Instructors and librarians alike were in agreement that the overall idea of peer instruction in the form of handbooks for incoming students was excellent. The information supplied by students was found to be useful, and through this opportunity students also gained a sense of ownership in their collegiate educational process. One commenter even noted that the handbooks could be generalized for use at other colleges and universities. The interdepartmental and collaborative nature of this project also enabled the library and the librarians to engage directly in active learning processes occurring on campus. Future studies may examine how institutional cultures change and strengthen as a result of more involvement with the library and multiple departments across campus.

The takeaway from this project was to develop and reinforce the importance of information literacy skills in first-year college students. By creating a platform for students to communicate directly with other students in a dialog that is authoritative, learners gain a hands-on understanding of the information process. The integration of librarians in this environment, with their professional training in how authoritative information is transmitted and received, strengthened the standards by which the students created their handbooks.

Although peer-to-peer learning is primarily done in the classroom, this project offered a way for librarians to participate in this learning strategy. The dynamic formed in this project between instructors, students, and librarians ties three groups together, from which all participants have a vested learning opportunity. Academic librarians are also always looking for new opportunities to interact with students, and this project offers another platform for that interaction. Integrating librarians in peer-to-peer instruction also helps to strengthen and foster learning communities.

Drawbacks to bringing librarians into the peer-to-peer learning model might relate to how students choose to interact with one another. One idea behind peer-to-peer learning is for students to communicate openly with one another in a way they might not feel comfortable using with the instructor or the class a whole. In reference services, librarians may act as gatekeepers of information and can therefore be perceived as intimidating. Bringing this intimidation factor into the peer-to-peer format may cause some students to hesitate to participate within an active learning community. How students respond to librarians in a peer-to-peer setting might be worth investigating. It is important to remember that librarianship is changing, and as libraries move from a more traditional model of service with books in a physical space, to one that handles multiple information types in both physical and digital forms, the role librarians play will also change. Despite potential intimidation factors, librarians working one-on-one with students, embedded librarians in the classroom, and other newly charted roles are bound to expand. The delineation between instructor and librarian is likely to blur.

In this study, despite both the librarians’ and instructors’ participation, the disparate response rates between instructors and librarians should be addressed. As evaluators, the librarian reviewers returned a higher response than did their instructor counterparts. Of the five librarians asked to review the handbooks, four (80 percent) responded, while only four out of fifteen instructors (27 percent) responded. Ratings from the librarian reviewers also included more comments on the content and how it was structurally conveyed. Fifty percent of the librarian reviewers (two out of four) provided comments in addition to ratings, while only twenty-five percent of the instructor reviewers (one out of four) offered both comments and ratings.

Although there were proportionately fewer responses from the total number of instructors versus the total number of librarians, there was an even distribution among reviewers (that is, four librarians and four instructors). In addition, the librarian participants have professional training on how information is transmitted, received, and interpreted. Given this background, it is logical that this group would offer more information using both of the communication tools provided by this study. The librarians were more vocal in expressing their views, but this should not diminish the value of analysis from the instructors. Further research might explore librarian responses to surveys in general, and how they differ from the responses of other faculty on campus.

The literature review suggests that peer feedback opens the classroom up to active learning, and this project demonstrated how students can actively articulate what they have learned about college life. This study also helped illuminate the changing nature of librarianship and offered an additional role for academic librarians to move the profession forward in the arena of instruction.

References

1. Heather L. O’Brien et al., “Investigating a Peer-to-Peer Community Service Learning Model for LIS Education,” Journal of Education for Library & Information Science 55, no. 4 (October 2014): 322–35.
2. Meagan Oakleaf and Patricia Owen, “Closing the 12–13 Gap Together: School and College Librarians Supporting 21st Century Learners,” Teacher Librarian 37, no. 4 (April 2010): 52–58, accessed July 14, 2016, http://meganoakleaf.info/oakleafowensyllabi.pdf.
3. Topsy N. Smalley, “College Success: High School Librarians Make the Difference,” Journal of Academic Librarianship 30, no. 3 (May 2004): 193–98, accessed July 14, 2016, www.researchgate.net/publication/222519116_College_success_High_school_librarians_make_the_difference.
4. Ibid.
5. “What Works—The Teacher-Librarian as Link between the High School and College Library,” Emergency Librarian 24, no. 5 (May 1997): 33.
6. Amanda Asghar, “Reciprocal Peer Coaching and Its Use as a Formative Assessment Strategy for First-Year Students,” Assessment & Evaluation in Higher Education 35, no. 4 (July 2010): 406.
7. Ibid., 407.
8. Ibid., 415.
9. Peggy A. Ertmer et al., “Peer Feedback in a Large Undergraduate Blended Course: Perceptions of Value and Learning,” Journal of Educational Computing Research 43, no. 1 (September 2010): 67–88.
10. Ibid., 75.
11. Keith Willey and Anne Gardner, “Investigating the Capacity of Self and Peer Assessment Activities to Engage Students and Promote Learning,” European Journal of Engineering Education 35, no. 4 (August 2010): 436, accessed July 14, 2016, www.sefi.be/wp-content/abstracts2009/Willey.pdf.
12. Michael J. Peeters, Eric G. Sahloff, and Gregory E. Stone, “A standardized rubric to evaluate student presentations,” American Journal of Pharmaceutical Education 74, no. 9 (November 10, 2010): 171, accessed July 14, 2016, http://www.ncbi.nlm.nih.gov/pmc/articles/PMC2996761/pdf/ajpe171.pdf.
13. Association of College & Research Libraries, “Information Literacy Competency Standards for Higher Education,” American Library Association, www.ala.org/acrl/standards/informationliteracycompetency.
14. George W. Snedecor and William G. Cochran, Statistical Methods, 7th ed. (Ames: Iowa State University Press, 1980).

Appendix A. Handbook Evaluation Rubric

	1 point	2 points	3 points	4 points
PowerPoint Slides Effectiveness	Slides so poorly constructed they detract from presentation.	Many slides ineffective-too wordy.	Too many/few slides. Poor color, font selection, graphs/tables not described.	Effective slides which enrich the presentation and are easily read.
References	No references listed on slides.	References listed inappropriately (e.g. References listed as slide titles).	Occasional reference missing/inappropriate format.	References formatted appropriately throughout.
Stated Objective	Handbook was not related to assigned purpose.	Some objectives addressed.	Most objectives addressed.	Handbook matched announced purpose and met all objectives.
Opening Statement/Relevance to Audience	No useful introduction to handbook. Readers have no idea what the handbook is addressing.	Minimal opening statement with little mention of relevance of topic to audience.	Introduction present, may state how topic impacts audience.	Effective opening which states what the presentation is covering and how the topic impacted the author and the reader.
Balanced Representation of Material	Handbook heavy in introduction/background material with little emphasis on application.	Presentation is one sided or biased. Too much emphasis on background.	Balanced inclusion of introduction and background.	Balanced inclusion of introduction and background, presentation of literature, and application.
Appropriateness of Selected Literature	Selected literature does not support theme of handbook.	Significant gaps in literature presented, or selected literature appeared to be biased.	Missing some important portion of the literature without sating the limited scope.	Selected literature supported theme of the handbook and was well balanced.
Organization of Handbook/Planned Coherently	Many points left out, handbook disorganized.	Majority of points glossed over, insufficient depth of topic.	Majority of points covered in depth, some important points may be unclear, minor organization issues.	Thoroughly explains all points. Makes essential points obvious. Well organized.
Application of Material	No application or conclusion provided.	Opinions on application and conclusion presented, but are not supported by data.	Superficial conclusions or opinions presented with limited reference to data.	Valid conclusion presented which were supported by data.
Handbook Generates Discussion/Responses/Questions	Avoided discussion, or did not stimulate thoughtful questions.	Answers to questions superficial.	Questions were answered somewhat vaguely.	Answered questions appropriately.

Appendix B. Written comments

Reviewer two wrote:

While they provide good information, I’m unsure if all of them should be added. Maybe if we condense it to one presentation, it would be more receptive to our students.

Reviewer four wrote:

(Handbook 1) Over all, this handbook looks good, but it should have connection and need more organization of contents in order to keep audiences following each topic smoothly. More contents, links, or strategies/practical information should be added (e.g. enrollment services, financial aids, bursar, student ID, IT, disability services, testing center, shuttle bus, add/drop/withdraw classes, etc.).

(Handbook 2) Overall, this handbook looks good and clear in the way of presentation. However, it seems that this handbook provides information in general (For example, “Facts” should be concise and focus on “useful information /issues specific to new students.). It would be better if it provides more contents or information the incoming students really need to know for their success (e.g. enrollment services, financial aids, bursar, student ID, IT, disability services, testing center, shuttle bus, add/drop/withdraw classes, etc.)

(Handbook 3) Overall, it looks good and clear throughout the presentation. This presentation seems to be well organized and designed when compared to another two. All sources used in this presentation support the topics. However, it would be very good source if some information about tutoring services, student ID (how and why), information on add/drop/withdraw classes, etc.

Reviewer five wrote:

I like all of the presentations except for the first because it has too much verbage [sic].

Reviewer six wrote:

(Handbook 1) This is an excellent presentation, and I think that students will find the information and recommendations found in it very relatable to their own circumstances. I especially liked the section on friends’ and peers’ influences. [This handbook should] add info about textbooks on Reserve. The afterword slide is too dense; break into 2 or 4 slides.

(Handbook 2) A “4” indicates content was thorough and could be presented as credible source. But on slide 7, the number of library staff is shown as 15, which is not correct. If the reference for this is BRCC.edu, that page/Web Site needs to be updated. This misstatement of facts makes me question the accuracy of other statements. I tried locating that info on the BRCC Web Site and could not find where that statement is made. Changed this score to 3. I like the images that were used on the slides, but think that the works citied slide is very hard to read. This is not enough to drop the 4 grade I have given to the PowerPoint slides.

(Handbook 3) Very well organized presentation. Quality of information is excellent. It’s presented simply, with the “big ideas” very clearly stated and details that support them. It is visually pleasing-Good design choices! Consistency of slide background was a very good choice. Love the use of quotations and choice of images to match text! Very practical advice is given, such as “get and use a calendar,” “write down everything,” “prioritize your tasks.” This manual is geared to BRCC, but it would be useful for a freshman at any college or university. That it is created by college students with some experience behind it makes it very relatable.

Reviewer seven wrote:

(Handbook 1) Presentation is good. I think it can be used as is, as long as it is presented as a resource prepared by students.

(Handbook 2) A bit prejudicial about advising and should be used with some critical thinking pointers. Also, I don’t understand the data that is given on the pages about transferring credits.

(Handbook 3) This presentation fits between the above two, I think. Their references are not very good. I also think that their section on advising needs to be discussed.

Reviewer eight wrote:

(Handbook 1) There is a LOT of text on most of the slides; many have backgrounds so busy that visually impaired students or older adult students may have difficulty reading. In places, I think text may not have been appropriately attributed; does not sound like student voice. Some references difficult to attach to resource listing at the end, i.e. slides 17, 19, and 27. Maybe disclaimer should be added: Views expressed in this handbook are those of the students who are sharing their experiences.

(Handbook 2) “Like the energy and vibe—sounds like student voices. Very comprehensive and logical beginning—includes campus map and pictures of major buildings new students need to know. Most students complain about parking, slide 16. After griping, at least they do recommend taking early classes or coming to campus earlier to find a space. I am very sensitive about advising complaints. This group’s treatment was very one sided. It sounded like the negative experience of one student colored slide 17: “Most advisor do not know enough . . . ” The slam was even more glaring because of slide 18, lifted verbatim from the BRCC website about Disability Services without attribution. DS is “da bomb” and Advising is the pits in the opinion of this group. Not a fair and balanced treatment. New Student Checklist on slide 19 is no longer accurate. Divider slides are great, as are most graphics, except one on slide 28; very graphic depiction of suicide. May be offensive or upsetting for some. Liked reasons for going to class. Not so much the suggestion to cut class if really burnt out—after the first cut, it is much easier to skip others. Transfer info for other universities also out of date. The LA Transfer Degree tracks and specific 2+2s created by BRCC and others around specific degrees covers. Great summation, slide 37. Liked the student bios and the group shot at the end with the BRCC Bear.

(Handbook 3) Slide 5 zaps advising—all three quotes about advising are negative; pre-nursing students are very dissatisfied with advising, because the department does not see them individually until they are admitted to the program. Professional advisors try to meet the need, but there are more than 1,000 students in the pipeline, many of whom are very weak students who will need several semesters of prerequisites to be eligible to apply for admission—very easy to shoot the messenger, in this case, us. Perhaps a visit from CADST is in order to present a balanced view and to answer questions. Slides 6 and 8 cover advising in a more fair manner and could be used exclusively to cover the topic. Slide 10 seems to be an attempt to put a positive spin on a slanted intro. As with any multi-step process it tends to be much easier if one begins much earlier—waiting until the last minute is asking for Murphy’s Law to apply. Good graphics in the Financial Aid section. Good job in handling Time Management, Study Skills, Library, balancing social and school life and alcohol abuse. Good Summary—wordy, but good. Like the bios—especially the personal advise for new students.