Board review course effect on resident in-training examination

Abstract

Background

The in-training examination is a national and yearly exam administered by the American Board of Emergency Medicine to all emergency medicine residents in the USA. The purpose of the examination is to evaluate a resident’s progress toward obtaining the fundamental knowledge to practice independent emergency medicine.

Aims

The purpose of this study was to determine the effects of a 40 hour board review lecture course on the resident in-training examination in emergency medicine.

Methods

A 40 hour board review lecture course was designed and implemented during the weekly 5 hour long resident conferences during the 8 weeks preceding the in-training examination date in 2006. Attendance was mandatory at the Accreditation Council for Graduate Medical Education (ACGME) standard of 70% or greater. A positive result was considered to be a 10% increase or greater in the resident’s individual national class percentile ranking among their national peers for their class year for the emergency medicine in-training examination. A resident was excluded from the study if there was no 2005 in-training examination score for self-comparison. The 95% confidence intervals (CI) were used to analyze the results.

Results

Of 16 residents, 1 (6.25%; 95% CI: 0–18%) showed a positive result of increasing their national class percentile ranking by 10% or greater. For the PGY2, one of the eight had a positive result (12.5%; 95% CI: 0–35.4%). For PGY3, no resident (0%; 95% CI: 0–35.4%) had a positive result.

Conclusions

A 40 hour board review lecture course has no positive effect on improving a resident’s in-training examination score.

Introduction

The American Board of Emergency Medicine (ABEM) in-training examination is a yearly national cognitive test administered to emergency medicine residents in USA to evaluate progress toward obtaining the fundamental knowledge to be independent emergency physicians. According to the ABEM website, residents with higher in-training examination scores have a higher likelihood of passing the initial qualifying examination to be certified emergency medicine physicians than residents with lower examination scores [1]. While there are many commercial weeklong board review lecture courses for the initial qualifying certification exam for emergency medicine, there is currently no review lecture course for the emergency medicine in-training examination. The author sought to determine the effects of implementing a 40 hour board review lecture course for resident in-training examination scores.

Methods

A 40 hour board review lecture course was designed and implemented during the weekly 5 hour long resident conferences during the 8 weeks preceding the in-training examination date in 2006. The review course was based on the topics outlined in the “model of clinical practice of emergency medicine” [2]. The course consisted of lecture reviews of test taking skills, trauma, pediatric emergencies, toxicological emergencies, cardiac emergencies, pulmonary emergencies, gastroenterology emergencies, neurology-psychiatry emergencies, endocrine emergencies, hematology-oncology emergencies, otorhinolaryngology-ophthalmology emergencies, genitourinary emergencies, obstetric-gynecology emergencies, orthopedic emergencies, dermatology-environmental emergencies, and emergency medicine potpourri given by the emergency medicine faculty at a university medical center. To accommodate the inclusion of the 40 hour review lecture series, the 4 hour monthly journal club was suspended for 10 months. Furthermore, attendance was mandatory at these lectures at the standard Accreditation Council for Graduate Medical Education (ACGME) guideline of 70% or greater [3]. A positive result was chosen to be a 10% increase or greater in the resident’s individual national class percentile ranking among their national peers for their class year for the emergency medicine in-training examination. A resident was excluded from the study if there was no 2005 in-training examination score for self-comparison. The 95% confidence intervals (CI) were used to analyze the results.

Results

Of 16 residents, 1 (6.25%; 95% CI: 0–18%) showed a positive result of increasing their national class percentile ranking by 10% or greater. For the PGY2, one of the eight had a positive result (12.5%; 95% CI: 0–35.4%). For PGY3, no resident (0%; 95% CI: 0–35.4%) had a positive result.

Discussion

Embarking on a successful career in emergency medicine requires a lifelong commitment to learning that starts during the residency years. The ABEM administers the in-training examination during residency, the initial qualifying certification examination post-residency, the recertification examination every 10 years as well as the yearly Lifelong Learning and Self-Assessment (LLSA) examination. In an effort to help individuals pass these examinations, many commercial review lecture courses have been developed for the board examination as well as the LLSA. Whether these courses are beneficial has not been formally studied using a rigorous scientific method. Based on the review courses advertisement brochures, it seems a very large percentage of their course participants pass the examination. Furthermore, some commercial review courses offer a money back guarantee and given the longevity of the course, one may surmise that more course participants pass the examination than fail the examination as the course continues to be profitable.

Because of the success of these courses, a 40 hour board review lecture course was designed and given to the residents prior to their in-training examination. Since this is a non-blinded study, one would expect a biased positive effect as the residents knew that the lectures were given to improve their individual in-training examination score. Interestingly, only 1 of 16 had a positive result of increasing their national class percentile ranking by 10% or greater in the study. Gillen’s study of a structured board review consisting of monthly reading assignments in an emergency medicine textbook followed by a graded multiple choice written examination showed no favorable effect on the ABEM in-training examination for emergency medicine PGY 2 and PGY 3 residents. Instead, the study found a decrease in the mean EM-2 institution program mean scores from 66.8 to 65.4 and mean EM-3 institution program mean score of 74.4 to 67.4 though these decreases were not statistically significant in the study [4]. However, the study did show an improvement in the intern class institution program mean score from 50.7 to 68.9. In the present study, national class year percentile ranking rather than mean scores was chosen because the mean scores tend to increase per PGY years and these mean scores differ yearly, while one would expect an individual’s national class percentile ranking to be more constant. For example, if a resident had a score of 70 which placed him at the 50 percentile ranking among his national peers, one would expect the resident to increase his score next year secondary to increased emergency medicine training, but more likely to remain at the 50 percentile ranking next year among his national class year peers who would have also improved with their similar increased training. Interns were excluded from this study because they did not have a previous in-training examination for self-comparison. Furthermore, the mean examination score for the emergency medicine intern class varies widely year to year in our institution.

Currently, there is no explanation for the lack of increase in the resident’s national class percentile ranking with the board review course. In other medical specialties, Wigton gave 40 1-h lectures on gastroenterology over a 1-month period to 24 internal medicine residents as well as had them take a multiple choice examination before and 3 weeks after the lecture series with mean scores increasing less than 10% [5]. Similarly, Bull et al. compared didactic lectures given by faculty to reviews prepared by residents from a major thoracic surgery textbook and found a statistically significant percentile increase of 31% with the resident reviews compared to 11% with faculty lectures (P < 0.05) [6]. Also, Wade and Kaminski compared the influence of different educational methods [textbooks, grand rounds, preceptor guidance, review courses, Surgical Education and Self-Assessment Program (SESAP)] on the in-training examination of surgical residents and found that independent study methods were the most effective pathway to successful examination performance [7].

Finally, several limitations of this study need to be recognized. First, the design of the 40 hour board review lectures has never been validated nor has the 10% increase expectation been validated. The author felt that the 10% increase was a reasonable goal given the time and effort devoted to designing a 40 hour emergency medicine review. A 5% increase was felt to be too low a goal given the time and effort expended on the review course, while a 20% increase was too unreasonable. Another limitation is that this study occurred at a single institution. Furthermore, this study involves a small sample size. Also, confounding factors such as lack of sleep and test anxiety may play a larger hidden role. Finally, there is no incentive for performance on the in-training examination as opposed to the certifying examination where passing the examination allows ABEM board certifications as well as avoids additional fees for retaking the examination.

Conclusion

A 40 hour board review lecture course had no positive effect on improving a resident’s in-service exam score.