Chapter 4. Six Roadblocks to Designing Digital Badges

Chris Gamrat and Brett Bixler*

In previous work, Gamrat, Zimmerman, Dudek, and Peck defined badges as “online representations of learning experiences and activities that tell a story about the learner’s education and skills.”1 While at first glance this definition seems straightforward and badges appear easy to adopt, we offer that such may not be the case. In this chapter, we identify three internal challenges faced by badge creators and issuers and three external challenges faced by the larger education community. We argue that these six challenges present some of the largest barriers to the adoption of digital badges in education.

Three Internal Reasons

Digital badges require a significant degree of internal reflection about what they represent for an organization. Digital badges offer additional value and complexity because of the associated metadata and the transparent representation of the skills, abilities, and experiences valued by the badge issuer. For these reasons, the badge creators and issuers may invest significant time before they can reach a consensus on the badges that positively represent them and their institutions. Variations in badge design, assessment practices, and attention to the many details surrounding badge creation all contribute to this time factor.

Variation in Badge Design

Badges can range in size and scope. Badge design affords a great deal of flexibility regarding the scope of the task required to earn the badge. An organization could create badges that are awarded for attending a one-hour workshop or that completely align to an undergraduate or graduate degree. Digital badges’ advantage is the capture of detailed information explaining the learning experience, what is required of the learner, and documented evidence of the completion of these requirements. Conventional wisdom might suggest that mapping a badge or suite of badges to something as large as a degree might be too complex, and also unnecessary because a series of data explaining the student’s learning experience already exists—otherwise known as a transcript.

Since badges have value in different forms and represent different types of experiences, badge issuers may want to consider why they are issuing the badge, and badge earners may want to consider the number and types of badges they might share with a potential employer. Badge issuers should evaluate their goals for issuing badges, as this will help to determine the size and scope of the badges. For example, the badges may act as an additional layer of information that might supplement recognition that is already awarded, such as those associated with the completion of a class, degree, or certification program. Other badges could document skills and accomplishments beyond what an organization traditionally recognizes—for example, soft skills such as teamwork or problem solving.

Badges offer richer detail about achievements and experiences, which makes a badge a potentially valuable supplement to a résumé. However, in research conducted by Raish and Rimland, employers reported interest in reviewing badges within a limited scope, suggesting that the rules of résumés (brevity and conciseness) still apply.2 Badge scope might require the consideration for how many criterion points to include in a badge—that is, how many steps or submitted artifacts are required to complete the badge. Digital badges may help learners to better elaborate on their learning experiences, but a badge author may want to consider this function of a badge and how it can help to translate the badge beyond its original context.

Badges Require Excellent Assessment Practices

As with any credential, various forms of badge assessments exist. Quizzes and written papers are often used for competency-based badges. Project-based badges may require more complex evidence of success, including portfolios or other tangible artifacts. Participation-based badges usually require the completion of a workshop, project, or course, where the assessment is not necessarily about the quality of a produced artifact but rather on social interaction and completion.3 When used in concert, these assessment techniques may provide a rich perspective on what students know and can do.

Educators, administrators, and employers all want to ensure the claims students make about their earned credentials, certificates, or degrees are authentic. Rigorous assessments help to validate the claims about student achievement made by certificates and degrees in most educational environments, but often the assessment is hidden from the public. If badges are to be generally accepted as an alternative credential, then the assessment provided by the badge issuer must be visible to all and valid. As a badge ideally makes public the criteria and assessments for earning the badge and thus can expose weak assessment practices, badge creators and organizations are opened up to scrutiny and criticism.

The badge issuer ideally uses reliable and valid assessment instruments built into the badge criteria. Several statistical methods exist to ascertain the reliability and validity of multiple-choice quizzes. Badges that require portfolios or written materials may rely on a grading rubric that should be examined for intra-rater and inter-rater reliability.4 If possible, badge issuers should include evidence of the reliability and validity of assessment instruments used in the badge to increase the badge’s veracity. The quality of the assessment instruments used in badges is critical.

Complexities in Badge Design

The authors recently participated in an advisory committee helping interested faculty and staff through think-aloud exercises focused on the creation and implementation of digital badges. From this advisory role and our own experiences authoring badges, we can confidently say that the details of badge design can quickly become complicated. The concept of badging is easy to grasp. The processes needed to implement a badging system are complex. For example, just the instructional design considerations for badges are multifaceted, including aspects of content, assessment, and program scalability.5 At first, implementing digital badges can seem easy, but addressing these complexities can be overwhelming. This may be especially true when the badge issuer is an organization with multiple stakeholders, inputs, and concerns.

As described above, the scope of a badge can be difficult to determine, but goal articulation will help to narrow what the badge means and what experience it is intended to recognize. To avoid becoming overwhelmed by the detailed questions presented by Gamrat, Bixler, and Raish, we recommend badge issuers start by considering primary reasoning for using digital badges.6 Ahn, Pellicone, and Butler found that most researchers have approached digital badges with the goal of exploring student motivation, pedagogical considerations, or impacts to credentialing.7 After determining the primary goal for creating and issuing badges, badge creators may be better able to consider content, assessment, relationships between digital badges, and program scalability.

Probably the most challenging detail for implementing a digital badging system is to determine how to scale the initiative. Badge ecosystems—the mechanisms needed to create, store, and issue badges—require a great deal of thought to conceptualize and significant resources (personnel and computer systems) to implement. Among the many questions to address are, Who handles the computing infrastructure? Who handles the uploading and storing of digital badges? Who updates the information?8 Infrastructure questions add to the details of pedagogical and administrative questions, resulting in a balance among the three. Implementation and pedagogical strategies that work at the scale of a few dozen learners might be impossible with hundreds or thousands. Issuing digital badges in large numbers requires careful reflection, balancing high-quality assessment and grading timeliness. Decisions regarding implementing badges in large numbers likely cannot be made unilaterally, and achieving consensus can be a time-intensive process.

Three External Reasons

Badges present a value proposition at different levels of education: elementary and secondary education, higher education, and continuing education or professional development. Within informal and formal education, badges might be helpful in providing structure or motivation to students. However, when learners want to make use of their earned badges within a different context, such as a job application, potential barriers external to where the badges were earned arise.

Digital Badge Examination

Open badges were initially created by Mozilla as a way of establishing a set of standard metadata associated with digital badges. IMS Global picked up the open badges efforts from Mozilla and established the Open Badges 2.0 (OBv2) specifications in June 2018.9 The specifications support the ability to transport digital badging information from system to system. The specifications on stored data are broad, so actual data from badge to badge can vary greatly while still following the Open Badges standard. This variability presents a challenge similar to that of comparing résumés or portfolios of work. While some large organizations have automated methods for streamlining résumé processing, manual review is still required to make decisions about what is valuable and how to compare across multiple and differing claims. To use an example, consider two applicants for a job, each claiming to have experience with multimedia production and both pointing to educational experiences to support the claim. Using digital badges or a course transcript presents similar comparison challenges. However, the digital badge for the experience has an advantage in that it fully describes the experience and provides evidence of the student’s claim, whereas the transcript provides only the name of the course, the weight in course credits, and a letter grade received. The badge evidence is superior to the transcript but requires more effort to interpret. In an era where expert systems are used for initial comparison of job applicant credentials, digital badges will continue to require human appraisal.

Badges’ Value Proposition

Badges form representations or claims of educational experiences that may or may not be valued by others. Similar to other educational currencies such as degrees or professional certifications, badges can offer a way of representing academic achievement through digital metadata. That is, for learners and employers, digital badges function as a mechanism to represent learning. We argue that digital badges, like other educational recognition, act as a currency to varying degrees in the three largest areas for learning: elementary and secondary learners preparing for postsecondary school, postsecondary learners preparing for jobs, and on-the-job learning for professional growth and advancement. In this chapter, we offer these three generalized scenarios for the adoption of badges as educational currency.

• Elementary and secondary education. Badges earned by learners in elementary and secondary schools document progress over time and motivate children to continue to learn and explore. In 2012, the Digital Media and Learning competition funded projects to design and implement badging systems. Many of these were for elementary and secondary student populations through 4-H and NASA, among many other projects.10 More recently, Davis and Klein researched an afterschool science program for high school students.11 This initiative offered the opportunity to use digital badges and explore student perception with this new credential. Some of the earliest educational research with digital badges in elementary and secondary contexts examined student engagement and motivation.12 In these examples, the badges had value either in motivating students to engage with the content or in representing educational achievement for college applications.
• Job preparation. Digital badges are beginning to emerge as a tool for students to stand out in a crowd of applicants for a job. Purdue University and Brigham Young University have adopted a series of badges for their preservice teacher education programs.13 Universities are also exploring the use of digital badges in a variety of areas outside the credit-bearing course. Penn State University Libraries offer a series of badges to recognize student learning in the area of information literacy.14 In September 2018, Northeastern University and IBM announced a partnership in which Northeastern will accept badges offered by IBM to count toward college credit.15
• Lifelong learning on the job. Digital badges for tracking professional growth may help to encourage, track, and organize a lifetime of on-the-job learning. In 2012, Gamrat and colleagues developed and researched digital badges for teacher professional development.16 This work helped to inform future efforts for the use of digital badges in independent professional development settings.17 Since then, other organizations have also adopted an approach to open-ended professional development in which the learner can choose what to learn and when. For example, IBM has provided a series of badges through a web resource, IBM Skills Gateway. The IBM Skills Gateway offers access to a library of technical and managerial modules and offers assessments upon completion of the content.

As indicated in the three generalized scenarios above, badge authors might view badges as metaphorical currency because the badges represent something they value. The currency metaphor must also pass the value test for both the primary and secondary audiences. That is, does the learner (primary audience) value the badge enough to put in the work to earn it, and do college admission offices and employers (secondary audience) see value in the badges that are earned? This is a significant external barrier to adoption as it requires significant numbers of people to agree on the value of a badge.

Buy-in for Badges

If badges are not deemed acceptable currency by the majority who interact with them, they will never be widely accepted. Badge earners need at least one badge in a given area for badges to be effective, but is there a true minimal number needed for badges to be accepted by earners, teachers, administrators, and employers? Although all these groups must be considered, this chapter examines buy-in at the individual level.

How are individual goals related to the acceptance of badges? At the individual level, goal setting is critical, and it is generally believed that grades and personal satisfaction are generally tied to academic goals. Although research is sparse here, the findings of one relevant study by Fanfarelli and McDaniel indicated the number of badges earned seems to correlate with a higher final grade for undergraduate males but not females. However, the authors infer that women may derive greater satisfaction from a badging system and earning badges than men.18

Research by Denny discovered a correlation between the number of times individuals viewed information about badges (possible goal setting and confirmation) and the number of badges collected.19 The mechanisms that prompted individuals to view badge information seemed to play a role in the number of views. It must be noted that this was a study limited to badges collected on a single platform, not about badges collected over multiple platforms or offerings, making generalizations difficult. While these findings are preliminary and more research is needed in this area, it may be that the importance of the number of badges earned varies from individual to individual, and also from group to group. Mechanisms prompting students to view badges may contribute to the perception of the importance of badges at the individual level.

The structure of the goals within a suite of badges may also contribute to the acceptance and adoption of badges by the individual. Conceptualizing badge creation with proximal and distal goals may assist in the development of a suite of badges. The learner sees an immediate need to complete proximal goals—also called subgoals—and their value is clear to the learner. Distal goals are less urgent, and their value may be less clear.20 A suite of badges, each with a small focus that contributes to a larger goal, is used in some programs, such as the Penn State Information Literacy Badge, and may be one method of creating the critical mass needed for a suite of badges.21 The Penn State Information Literacy Badge consists of ten sub-badges that can be used to earn three “meta badges.” If students earn the three meta badges, they can earn the overall “über badge,” indicating they have successfully earned all the badges in the suite. The sub-badges here relate to proximal goals, such as “Refining Your Search,” and the “Savvy Searcher” meta badge provides the distal goal.

Conclusion

Many educational institutions have not broadly implemented digital badges. Variations in badge design, evidence provided, the potential exposure of substandard assessments, complex support structures for badges, the variable value of badges, and buy-in by potential stakeholders all contribute to a slow pace of adoption. While it is possible from a technical standpoint to author and award digital badges, it is likely that the reasons outlined above are some of the major hurdles delaying implementing large-scale efforts with digital badges. We hope that the sections above offer the reader perspective on the challenges for adoption and some ideas on how they may be mitigated.

Notes

1. Christopher Gamrat, Heather T. Zimmerman, Jaclyn Dudek, and Kyle Peck, “Personalized Workplace Learning: An Exploratory Study on Digital Badging within a Teacher Professional Development Program,” British Journal of Educational Technology 45, no. 6 (November 2014): 1136, https://doi.org/10.1111/bjet.12200.
2. Victoria Raish and Emily Rimland, “Employer Perceptions of Critical Information Literacy Skills and Digital Badges,” College and Research Libraries 77, no. 1 (2016): 87–113, https://doi.org/10.5860/crl.77.1.87.
3. Daniel T. Hickey and James E. Willis III, “Where Open Badges Appear to Work Better: Findings from the Design Principles Documentation Project,” Center for Research on Learning and Technology, Indiana University, May 1, 2017.
4. Anders Jonsson and Gunilla Svingby, “The Use of Scoring Rubrics: Reliability, Validity and Educational Consequences,” Educational Research Review 2, no. 2 (2007): 130–44, https://doi.org/10.1016/j.edurev.2007.05.002.
5. Christopher Gamrat, Brett Bixler, and Victoria Raish, “Instructional Design Considerations for Digital Badges,” in Digital Badges in Education: Trends, Issues, and Cases, ed. Lin Y. Muilenburg and Zane L. Berge (New York: Routledge, 2016), 71–81, https://doi.org/10.4324/9781315718569.
6. Gamrat, Bixler, and Raish, “Instructional Design Considerations for Digital Badges.”
7. June Ahn, Anthony Pellicone, and Brian S. Butler, “Open Badges for Education: What Are the Implications at the Intersection of Open Systems and Badging?” Research in Learning Technology 22 (2014), https://doi.org/10.3402/rlt.v22.23563.
8. John C. Foster, “The Promise of Digital Badges,” Techniques: Connecting Education and Careers 88, no. 8 (November/December 2013): 31–34.
9. “Open Badges 2.0 (OBv2),” IMS Global Learning Consortium, accessed January 28, 2019, https://www.imsglobal.org/activity/digital-badges.
10. Daniel T. Hickey, Nate Otto, Rebecca Itow, Katerina Schenke, Cathy Tran, and Christine Chow, “Badges Design Principles Documentation Project: Interim Report January 2014 Update,” Center for Research on Learning and Technology, Indiana University, January 2014, http://dpdproject.info/files/2014/05/DPD-interim-report-v4-january.pdf.
11. Katie Davis and Eve Klein, “Investigating High School Students’ Perceptions of Digital Badges in Afterschool Learning,” CHI ’15: Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems (New York: ACM, 2015), 4043–46, https://doi.org/10.1145/2702123.2702413.
12. Samuel Abramovich, Christian Schunn, and Ross Mitsuo Higashi, “Are Badges Useful in Education? It Depends upon the Type of Badge and Expertise of Learner,” Educational Technology Research and Development 61, no. 2 (April 2013): 217–32, https://doi.org/10.1007/s11423-013-9289-2.
13. Timothy Newby, Casey Wright, Erin Besser, and Elizabeth Beese, “Passport to Creating and Issuing Digital Instructional Badges,” in Foundations of Digital Badges and Micro-credentials: Demonstrating and Recognizing Knowledge and Competencies, ed. Dirk Ifenthaler, Nicole Bellin-Mularski, and Dana-Kristin Mah (Cham, Switzerland: Springer International, 2016), 179–201.
14. Emily Rimland and Victoria Raish, “Design Principles for Digital Badges Used in Libraries,” Journal of Electronic Resources Librarianship 29, no. 4 (2017): 211–20, https://doi.org/10.1080/1941126X.2017.1378540; Daniel L. Randall, J. Buckley Harrison, and Richard E. West, “Giving Credit Where Credit Is Due: Designing Open Badges for a Technology Integration Course,” TechTrends 57, no. 6 (November 2013): 88–95, https://doi.org/10.1007/s11528-013-0706-5.
15. News@Northeastern, “Northeastern University and IBM Partnership First to Turn Digital Badges into Academic Credentials for Learners Worldwide,” news release, September 25, 2017, https://news.northeastern.edu/2017/09/25/northeastern-university-and-ibm-partnership-first-to-turn-digital-badges-into-academic-credentials-for-learners-worldwide.
16. Gamrat et al., “Personalized Workplace Learning.”
17. Christopher Gamrat and Heather T. Zimmerman, “Teacher Learning Journeys: A Design Case Study of a Learner-Centered Stem Digital Badging System,” in Digital Badges in Education: Trends, Issues, and Cases, ed. Lin Y. Muilenburg and Zane L. Berge (New York: Routledge, 2016), 215–25, https://doi.org/10.4324/9781315718569.
18. Joseph R. Fanfarelli and Rudy McDaniel, “Exploring Digital Badges in University Courses: Relationships between Quantity, Engagement, and Performance,” Online Learning 21, no. 2 (2017), https://doi.org/10.24059/olj.v21i2.1007.
19. Paul Denny, “The Effect of Virtual Achievements on Student Engagement,” in Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (New York: ACM, 2013), 763–72.
20. Zui Cheng, Sunnie Lee Watson, and Timothy James Newby, “Goal Setting and Open Digital Badges in Higher Education,” TechTrends 62, no. 2 (March 2018): 190, https://doi.org/10.1007/s11528-018-0249-x.
21. Rimland and Raish, “Design Principles for Digital Badges.”

---