Skip to main content

Gamification of graduate medical education in an emergency medicine residency program

Abstract

Objectives

Our program implemented East EMWars, a year-long, longitudinal game that added competition to our existing curricular content. We surveyed residents to investigate the impact of gamification in emergency medicine residency training. We hypothesized that residents would report higher levels of motivation, engagement, and challenge with gamification compared to traditional didactics. Furthermore, we hypothesized that residents would exhibit generally positive perceptions about gamification as a learning tool and that it would translate to improved performance on the annual in-training examination.

Methods

This was a single-center, prospective pre- and post-intervention survey study at a community-based emergency medicine residency program. Given the multiplicity of research questions and inherent nature of educational research, a mixed methods approach was utilized. We utilized nonparametric testing for quantitative data with paired responses pre- and post-intervention. We solicited comments on the post-intervention that were categorized under thematic approach and presented in complete and unedited form in the results.

Results

Eighteen (100%) of eligible residents in our program participated in both surveys. There were statistically significant increases in reported levels of motivation, engagement, and challenge with gamification compared to traditional didactic methods. Residents also reported overwhelmingly positive general perceptions about gamification and its broader generalizability and applicability. We did not reach statistical significance in determining if in-training exam scores were associated with our gamification initiative.

Conclusions

This study was a first-of-its-kind look into a longitudinal game in an emergency medicine residency program. Although our results are encouraging, medical educators need further research to determine if this increase in motivation, engagement, and challenge will be associated with an increase in examination scores or, more importantly, healthcare outcomes. Theory-based, broader-scale, prospective studies are needed to further explore and help establish these associations and outcomes.

Introduction

What really is gamification? Ten years ago, it was unheard of. Now, it is in the armory of many educators across the country. Gamification has been defined as “the use of game design elements in non-game contexts” [1] or “the craft of deriving all the fun and addicting elements found in games and applying them to real-world or productive activities.” [2] But how do we create it, apply it, and explore the potential benefits to resident education?

Long before we had the terminology, gamification existed in popular culture. The American Airlines’ frequent flyer program began in 1981, rewarding loyal customers with incentives as they ascended the ranks of the program. More recently, gamification has made its way into most other areas of daily life; Starbucks is one example which gives users a platform to earn rewards and a mobile payment method all in one [3]. Despite a more recent uptick in research on gamification in the educational realm, there remains very little specific to graduate medical education [4,5,6].

Although emergency medicine may be leading the way, existing gamification projects are typically single sessions, narrowly focused on a particular content area, and heavily reliant on quiz-style gaming [7,8,9,10,11,12,13,14,15]. Instead, we sought to investigate the impact of a long-running game held longitudinally throughout an entire academic year. This innovation allowed for team building and competition over the course of an entire academic year, rather than a single session, or shorter term. While there were no changes made to the methods for curricular content delivery, we sought to investigate the impact of adding gamification elements to existing curricula to determine levels of motivation, engagement, and challenge compared to non-gamified traditional didactic sessions. We hypothesized that residents would report increased levels in all categories participating in the gamified model compared to the traditional didactic curriculum. We hope to encourage future educational researchers to design studies that will help to determine if this increased motivation, engagement, and challenge may also lead to improved knowledge translation, better performance on certifying examinations, and improved outcomes in patient care.

Methods

Study design and population

This was a prospective, observational research study designed to explore the perspective of emergency medicine residents in a newly implemented gamified graduate medical education curriculum. Given the multiplicity of research questions and inherent nature of educational research, a mixed methods approach was utilized. Surveys were administered to participants before and after their participation in a longitudinal game, held over a 10-month period between August 2020 and June 2021. The study was approved under exempt review by the AdventHealth Orlando Institutional Review Board.

Our study took place in a single-center, community-based, 3-year emergency medicine program in the southeast. Participants were voluntarily enrolled from the pool of 18 emergency medicine residents currently in the program. Participation was 100%. Several measures were taken to avoid any type of coercion to participate in the study.

Of note, the implementation of the game was an innovation to our existing curriculum, and as such, participation was required as a normal part of conference attendance and progression through our training program. Participation in this research study, by completing pre- and post-intervention surveys, was completely voluntary, and residents received no incentive to participate or consequence to not participate. For more information about game design, play mechanics, and scoring, see attached (Additional files 1 and 2: Appendices A and B).

Survey content and administration

Participants were recruited from the available pool of residents in our program. A flyer advertising the study was placed in the resident sign-out room within the emergency department. Additionally, after an overview presentation about the game, which was given during regularly scheduled didactic conferences, the investigators provided detailed verbal information about the game and study (completing the surveys), held a Q and A session, and presented the letter of invitation to those residents that expressed interest in participating.

Two surveys were distributed to all participants. The pre-intervention survey was distributed prior to the initiation of the game (August 2020) and included questions regarding perceived level of motivation, engagement, and challenge with the current (non-gamified) didactic curriculum as well as their perceptions about the implementation of gamification. The post-intervention survey was distributed after the completion of the game (June 2021) and included questions regarding perceived level of motivation, engagement, and challenge with the elements of gamification that had been added to the curriculum in comparison with the traditional didactic curriculum. This survey also solicited their perceptions about feasibility and generalization of gamification in graduate medical education.

Paper surveys were distributed in sealed envelopes with each participant’s name printed on the front. Each survey was uniquely coded so that responses remained anonymous but were matched pre-intervention and post-intervention to the same participant for data analysis purposes. All 36 surveys were printed and sealed in envelopes prior to starting the study. This ensured we were able to pair responses between survey periods but maintain anonymity of each individual resident’s responses. Surveys are attached (Additional files 3 and 4: Appendices C and D).

Data analysis

The primary objective of this study was to evaluate resident motivation, engagement, and challenge with a longitudinal, gamified medical education curriculum. The secondary objective was to evaluate whether increased participation with this gamified curriculum translated to improved scores on the annual in-training examination. Levels of motivation, engagement, and challenge were analyzed with the Wilcoxon signed-rank test, a nonparametric test comparing pre- (traditional didactic curriculum) and post-intervention (gamified curriculum) survey responses on a 5-point Likert scale. In-training examination percentile scores were self-reported via survey, where second- and third-year residents reported their 2020 ITE percentile score on the pre-intervention survey, while all respondents were asked to report their 2021 ITE percentile score on the post-intervention survey. Open-ended survey questions were summarized using qualitative methods, specifically thematic approach. Given this triangulation model of mixed methods data collection, there existed a possibility for contradiction in quantitative versus qualitative data [16]. Educational researchers have suggested that if resource limitations prevent further exploration of these contradictions, the results should be presented together, and authors should define directions for future studies [17].

Results

Of the 18 eligible participants, all 18 (100%) completed all required items on both pre-intervention and post-intervention surveys. Among the participants were 6 (33.3%) first-year residents, 6 (33.3%) second-year residents, and 6 (33.3%) third-year residents. The primary objective of the study was to evaluate resident motivation, engagement, and challenge with a longitudinal, gamified medical education curriculum. The secondary objective was to evaluate whether increased participation with this gamified curriculum translated to improved scores on the annual in-training examination.

Motivation, engagement, and challenge

For motivation, 13 (72.2%) respondents indicated increased motivation with a gamified curriculum compared to traditional didactics as evidenced by higher scores on a 5-point Likert scale post-intervention paired to pre-intervention responses; 3 (16.7%) reported no difference, and 2 (11.1%) had decreased motivation scores. The Wilcoxon signed-rank test supported rejection of the null hypothesis (p-value = 0.0028), so we concluded there is a significant increase in resident motivation.

For engagement, 13 (72.2%) respondents indicated increased engagement with a gamified curriculum compared to traditional didactics as evidenced by higher scores on a 5-point Likert scale post-intervention paired to pre-intervention responses; 5 (27.8%) reported no difference, and none had decreased engagement scores. The Wilcoxon signed-rank test supported rejection of the null hypothesis (p-value = 0.0002), so we concluded there is a significant increase in resident engagement.

For challenge, 10 (55.6%) respondents indicated increased challenge with a gamified curriculum compared to traditional didactics as evidenced by higher scores on a 5-point Likert scale post-intervention paired to pre-intervention responses, 7 (38.9%) reported no difference, and 1 (5.6%) had decreased challenge scores. The Wilcoxon signed-rank test supported rejection of the null hypothesis (p-value = 0.0068), so we again concluded there is a significant increase in resident challenge. Descriptive statistics are displayed in Table 1.

Table 1 Descriptive statistics for motivation, engagement, and challenge

In-training exam scores

In-training exam percentile scores were self-reported by second- and third-year residents on both pre- and post-intervention surveys. Scores were reported in ranges of 10 percentile points and coded to a 10-point scale (1st–9th percentile = 1, 10th–19th percentile = 2, 90th–99th percentile =10, etc.). There were a total of 12 respondents, 6 (50%) second-year residents and 6 (50%) third-year residents. Five (41.2%) respondents indicated an increase in ITE percentile scores after intervention, 2 (16.7%) respondents had no change in ITE percentile scores, and 5 (41.2%) respondents indicated an decrease in ITE percentile scores. A paired t-test was conducted (p-value = 0.198), failing to reject the null hypothesis. Subsequent power analysis (0.64) determined that the study was insufficiently powered to detect a difference if one exists. Unfortunately, we were limited by the number of eligible participants, not to mention a myriad of confounding factors that limit the usability of the obtained data.

General perceptions of gamification

Lastly, we asked participants several questions regarding their perceptions of gamification on the post-intervention survey. Respondents were asked to rate their level of agreement with several statements on a 1 to 5 Likert scale, from “strongly disagree” to “strongly agree.” Results are summarized in Table 2. Overwhelmingly, residents indicated agreement with positively worded statements about the impact of gamification on their educational experience.

Table 2 Descriptive statistics for perceptions on gamification

Residents also provided summative qualitative feedback through open-ended comments on the post-intervention survey. Their complete responses are included in Table 3 below. All comments were rated positively and associated with positive perceptions of gamification. Comments were categorized into “general” and categories specific to certain aspects of our game.

Table 3 Qualitative feedback through anonymous comments

Discussion

In this prospective, pre- and post-intervention survey, we determined that residents in an emergency medicine program perceive increased engagement, motivation, and challenge in a gamified curricular model compared to traditional, non-gamified, lecture-based didactics. Residents reported overwhelming agreement with positive statements regarding the implementation of gamification in their educational experiences, but our study was not powered to detect any significant difference in in-training examination scores before and after the intervention. Furthermore, there are a multitude of confounding variables affecting examination scores that were unable to be controlled for in this specific study.

For over half a century, educational researchers have recognized the need for change in our approach to adult learning [18]. Traditional pedagogical methods have long been shown to fall short. Couple this with an upcoming generation of adult learners who grew up in the technological age, not knowing life (or education) without computers, cellphones, and the internet. Emergency medicine educators have been leading the way in innovative strategies to improve adult learning for this new generation. Gamification is just one approach.

Over the past decade, the implementation of gamification models into educational curriculum has been met with tremendous success in a variety of settings [4, 5]. Systematic reviews of gamification in health professions education confirm that gamification has attracted the attention of educational researchers during the last few years, but there remains a scarcity of evidence and need for further theory-driven research [19,20,21]. In emergency medicine, we have seen success in gamification in various forms: game show-style quizzing [7,8,9], simulation competitions [10,11,12], escape rooms [13,14,15], and more. However, to our knowledge, this is the first study to consider these impacts of gamification in a longitudinal fashion.

The results supported our primary hypothesis that the game would increase learner motivation, engagement, and challenge. Selecting motivation as a measurable variable was important in that we believe it is a necessary precursor to self-directed learning. And the motivation we incur from gamification is the likely reason for its perceived success.

Perhaps the most broadly researched macro-theory of motivation is self-determination theory. SDT describes the core incentives of human behavior to reside with the needs for competence, relatedness, and autonomy [22]. Making learning into a game can allow residents the autonomy to set goals and obtain achievements, relate and socialize with their colleagues, and show their competence with progression in the game. In addition to these intrinsic needs, our game added extrinsic motivators in the form of gift cards and other prizes as awards for achievement. This was a strategic addition to solicit buy-in and further increase attention, engagement, and motivation. Although extrinsic motivators, like rewards and prizes, must be utilized cautiously, they have the ability to increase engagement, which can assist learners in identifying the value of an educational activity as it may apply to their circumstances, leading to autonomous motivation or “motivation arising out of genuine interest or personal endorsement or valuing of an activity.” [23]

Although our study was not powered to detect a significant difference in in-training examination scores, SDT supports that increasing levels of motivation, engagement, and challenge will lead to improved learning and outcomes. Further prospective, theory-based research is needed to determine whether gamification in emergency medicine residency programs can lead to improvements in both board examination scores and pass rates as well as patient outcomes.

Limitations

Medical education research, in general, suffers many methodological limitations [24]. Our study was no exception. This study was conducted at a single, community-based residency program limited to a very small sample size of only 18 residents, even with 100% participation. Furthermore, survey data, especially data obtained from residents, is prone to observer bias and social desirability bias as residents know they are being studied and may select the responses they believe to be desired by the investigators [25, 26].

Our main limitation, however, was the lack of power to detect a significant difference in performance on the in-training exam, likely due to small sample size. Even still, there are multiple confounding variables that affect examination scores and would prove difficult to control across a 1-year period separating subsequent in-training exams: other curricular components, quality of conferences, clinical experiences, self-directed learning, and more. Furthermore, we only obtained data on percentile rankings for each respondent (not raw score) which vary from year to year over administration periods and are compared to scores of peers in the same level of training across the country. A future consideration would be to utilize raw scores of in-training exams or, more broadly, pass rates on the ABEM qualifying exam, as the latter is already used in program evaluation. This would likely require a much larger sample size to yield statistically significant differences across exam administration periods.

Another potential limiting factor to the generalizability of our results is the intensive time requirement on the part of faculty to set up and conduct the game throughout the year. We estimate that a minimum of 30–50 faculty hours were volunteered throughout the game-play period to tabulate team scores, update leaderboards, and award prizes. This was in addition to time spent designing the game and performing the research component. Implementation of a large-scale game, like ours, would likely necessitate program support for protected faculty time and other resources. Despite these limitations, our study can be used as a basis to design future gamified curricular content and set up future prospective studies on its effectiveness.

Conclusions

We performed a prospective, pre- and post-intervention survey study to explore the perspective of emergency medicine residents in a newly implemented gamified graduate medical education curriculum. Our results supported the hypothesis that residents reported increased levels of motivation, engagement, and challenge with gamification compared to traditional didactics. Furthermore, residents overwhelmingly indicated agreement with positively worded statements about the impact of gamification on their educational experience. The study produced insufficient evidence to determine whether in-training exam scores are influenced by exposure to gamification compared to a traditional didactic curriculum.

Our study was a first-of-its-kind look into a longitudinal game in an emergency medicine residency program. Although our results are encouraging, medical educators need further research to determine if this increase in motivation, engagement, and challenge will be associated with an increase in examination scores or, more importantly, healthcare outcomes. Theory-based, broader-scale, and prospective studies are needed to further explore and help establish these associations and outcomes.

Availability of data and materials

Data sharing not applicable to this article as no datasets were generated or analyzed during the current study.

References

  1. Deterding S, Dixon D, Khaled R, Nacke L. From game design elements to gamefulness. In: Proceedings of the 15th international academic MindTrek conference on envisioning future media environments. MindTrek’11. New York: ACM Press; 2011. Accessed on 20 May 2020 at https://www.slideshare.net/dings/meaningful-play-getting-gamification-right/12-Makebelieve_Rules_challenges_Goals_Feedback/.

    Google Scholar 

  2. Chou Y. What is gamification? Yukai Chou: Gamification and behavioral design. (Accessed on 10 Oct 2019 at https://yukaichou.com/gamification-examples/what-is-gamification/)

  3. McEachern A. A history of loyalty programs, and how they have changed. Smile.io. (Accessed on 20 May 2020 at https://blog.smile.io/a-history-of-loyalty-programs/)

  4. Johnson SL. Gamification of MIS3538: social media innovation. All Social. 2012; Accessed on 20 May 2020 at http://community.mis.temple.edu/stevenljohnson/2012/05/19/gamificationof-mis3538-social-media-innovation/.

  5. Barata G, Gama S, Jorge J, Goncalves D. Improving participation and learning with gamification. In: Proceedings of the first international conference on gameful design, research, and applications; 2013. p. 10–7.

    Chapter  Google Scholar 

  6. Ahmed M, Sherwani Y, Al-Jibury O, Najim M, Rabee R, Ashraf M. Gamification in medical education. Med Educ Online. 2015;20:29536.

    Article  Google Scholar 

  7. Smart J, Olson AS, Muck A. Asynchronous curriculum “socially synchronized”: learning via competition. West J Emerg Med. 2019;20(1):6–8.

    Article  Google Scholar 

  8. Wolfe J, Safdar B, Sethuraman KN, Greenberg MR, Madsen TE, Jarman AF, et al. Engaging emergency medicine influencers in sex- and gender-based medicine: lessons learned from the sex and gender interest group in emergency medicine and the SAEM jeopardy game. AEM Educ Train. 2019;4(2):161–5.

    Article  Google Scholar 

  9. O'Connell A, Tomaselli PJ, Stobart-Gallagher M. Effective use of virtual gamification during COVID-19 to deliver the OB-GYN core curriculum in an emergency medicine resident conference. Cureus. 2020;12(6):e8397.

    PubMed  PubMed Central  Google Scholar 

  10. Salerno N, Papanagnou D, Mahesh P, Bowers KM, Pasichow SH, Paradise S, et al. Challenging hazards amidst observational simulation in the emergency department: advancing gamification in simulation education through a novel resident-led skills competition. Cureus. 2018;10(11):e3563.

    PubMed  PubMed Central  Google Scholar 

  11. Kobner S, Grassini M, Le NN, Riddell J. The challenging case conference: a gamified approach to clinical reasoning in the video conference era. West J Emerg Med. 2020;22(1):136–8.

    PubMed  PubMed Central  Google Scholar 

  12. Lai AKH, Noor Azhar AMB, Bustam AB, Tiong XT, Chan HC, Ahmad RB, et al. A comparison between the effectiveness of a gamified approach with the conventional approach in point-of-care ultrasonographic training. BMC Med Educ. 2020;20(1):263.

    Article  Google Scholar 

  13. Diemer G, Jaffe R, Papanagnou D, Zhang XC, Zavodnick J. Patient safety escape room: a graduate medical education simulation for event reporting. MedEdPORTAL. 2019;15:10868.

    Article  Google Scholar 

  14. Zhang XC, Lee H, Rodriguez C, Rudner J, Chan TM, Papanagnou D. Trapped as a group, escape as a team: applying gamification to incorporate team-building skills through an ‘escape room’ experience. Cureus. 2018;10(3):e2256.

    PubMed  PubMed Central  Google Scholar 

  15. Zhang XC, Diemer G, Lee H, Jaffe R, Papanagnou D. Finding the ‘QR’ to patient safety: applying gamification to incorporate patient safety priorities through a simulated ‘escape room’ experience. Cureus. 2019;11(2):e4014.

    PubMed  PubMed Central  Google Scholar 

  16. Schifferdecker KE, Reed VA. Using mixed methods research in medical education: basic guidelines for researchers. Med Educ. 2009;43:637–44.

    Article  Google Scholar 

  17. Padgett DK. Mixed methods, serendipity, and concatenation. In: Padgett DK, editor. The qualitative research experience. Wadsworth: Thomson Learning; 2004. p. 272–88.

    Google Scholar 

  18. Knowles MS. The modern practice of adult education: andragogy versus pedagogy: Association Press; 1970.

    Google Scholar 

  19. Sardi L, Idri A, Fernández-Alemán JL. A systematic review of gamification in e-health. J Biomed Inform. 2017;71:31–48.

    Article  Google Scholar 

  20. Gentry SV, Gauthier A, L'Estrade Ehrstrom B, Wortley D, Lilienthal A, Tudor Car L, et al. Serious gaming and gamification education in health professions: systematic review. J Med Internet Res. 2019;21(3):e12994.

    Article  Google Scholar 

  21. McCoy L, Lewis JH, Dalton D. Gamification and multimedia for medical education: a landscape review. J Am Osteopath Assoc. 2016;116(1):22–34.

    PubMed  Google Scholar 

  22. Ryan RM, Deci EL. Self-determination theory and the facilitation of intrinsic motivation, social development, and well-being. Am Psychol. 2000;55(1):68–78.

    Article  CAS  Google Scholar 

  23. Kusurkar RA, Ten Cate TJ, van Asperen M, Croiset G. Motivation as an independent and a dependent variable in medical education: a review of the literature. Med Teach. 2011;33(5):e242–62.

    Article  CAS  Google Scholar 

  24. Collins J. Medical education research: challenges and opportunities. Radiology. 2006;240(3):639–47.

    Article  Google Scholar 

  25. Betancourt JR. Cross-cultural medical education: conceptual approaches and frameworks for evaluation. Acad Med. 2003;78(6):560–9.

    Article  Google Scholar 

  26. Maudsley G. Mixing it but not mixed-up: mixed methods research in medical education (a critical narrative review). Med Teach. 2011;33(2):e92–104.

    Article  Google Scholar 

Download references

Acknowledgements

This research was supported (in whole or in part) by the HCA Healthcare and/or an HCA Healthcare-affiliated entity. The views expressed in this publication represent those of the author(s) and do not necessarily represent the official views of the HCA Healthcare or any of its affiliated entities.

Funding

None

Author information

Authors and Affiliations

Authors

Contributions

SG and JR contributed to study concept and design and worked on the acquisition of data; SG and JR contributed to analysis and interpretation of the data and drafted the manuscript; LG provided critical revision of the manuscript for important intellectual content. The authors read and approved the final manuscript.

Corresponding author

Correspondence to Shayne Gue.

Ethics declarations

Ethics approval and consent to participate

This study was conducted in accordance with the Declaration of Helsinki and exempt by the AdventHealth Orlando Institutional Review Board. The requirement for written informed consent was waived as the obtained data was de-identified.

Consent for publication

Not applicable

Competing interests

The authors declare that they have no competing interests.

Additional information

Publisher’s Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Supplementary Information

Additional file 1: Appendix A.

Game Design.

Additional file 2: Appendix B.

Play Mechanics and Scoring.

Additional file 3: Appendix C.

Pre-Intervention Survey.

Additional file 4: Appendix D.

Post-Intervention Survey.

Rights and permissions

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/. The Creative Commons Public Domain Dedication waiver (http://creativecommons.org/publicdomain/zero/1.0/) applies to the data made available in this article, unless otherwise stated in a credit line to the data.

Reprints and Permissions

About this article

Verify currency and authenticity via CrossMark

Cite this article

Gue, S., Ray, J. & Ganti, L. Gamification of graduate medical education in an emergency medicine residency program. Int J Emerg Med 15, 41 (2022). https://doi.org/10.1186/s12245-022-00445-1

Download citation

  • Received:

  • Accepted:

  • Published:

  • DOI: https://doi.org/10.1186/s12245-022-00445-1

Keywords

  • Gamification
  • Serious games
  • Medical education
  • Adult learning
  • Team-based learning