Study design
This was an observational, retrospective, mixed-methods cohort study. Formative OSCEs have been conducted by the National Healthcare Group (NHG) EM residency programme in Singapore once or twice per year since 2013. Data were obtained from the results of formative OSCEs conducted from 2013 to 2019, and from a questionnaire distributed to NHG EM residents who had been in the programme during the study period.
Summative OSCE
Residents are required to sit for either the MMed or the MRCEM examination at any point during junior residency, which comprises the first 3 years of a 5-year residency programme, in order to progress to senior residency. Both the MMed and MRCEM examinations contain three parts. The first is a multiple-choice test, the second comprises short answer questions and the third is an OSCE. Applicants are required to pass the first two parts before sitting for the OSCE.
The MMed OSCE consists of 15 stations, with 12 min each for three resuscitation stations and 10 min each for the other stations. Each station is preceded by 2 min of reading time. The MRCEM OSCE consists of 18 stations (16 patient encounters and two rest stations), each 7 min long and preceded by 1 min of reading time. The MMed OSCE is mapped to the competencies of years 1-3 of the emergency medicine core curriculum, Singapore, whilst the MRCEM OSCE is mapped to the competencies of years 1-3 of the RCEM emergency medicine curriculum. Both curricula are similar, and expect competence in evaluating and managing adult and paediatric emergencies seen at a general emergency department. The content of both examinations is similar, and includes the following stations: history-taking (e.g. undifferentiated headache), physical examination (involving real patients with stable pathologies, e.g. valvular heart disease), communication (e.g. breaking bad news), procedures (e.g. performing toilet and suture on a manikin), resuscitation (e.g. cardiac arrest, major trauma) and others (e.g. mass disaster response and triage). The stations involve real patients, simulated patients, task trainers and manikins. Candidates are assessed by one assessor per station using station-specific checklists. Scores for various domains, as well as a global score, are given. If a simulated patient is present, scores are calibrated with input from the simulated patient.
The MMed OSCE is standard set via the modified Angoff method [20, 21]. Candidates are required to pass at least 11 out of 15 total stations, and two out of three resuscitation stations, to pass the examination. The MRCEM OSCE is standard set via the borderline regression method [22]. One standard error of measurement is added to the identified cut-off score to calculate the pass mark. Candidates are required to achieve the pass mark and pass at least one out of two resuscitation stations to pass the examination.
Formative OSCE
The formative OSCE is targeted at NHG EM junior residents who have yet to pass the summative OSCE. It is designed and blueprinted to be similar to the summative OSCE in format and core competencies tested. Each formative OSCE consists of 10-15 stations. The time allotted is the same as for the MMed OSCE, that is, 12 min for resuscitation stations and 10 min for the other stations, with 2 min of reading time prior to each station. The content of the stations, competencies expected and patient characteristics are designed to be similar to those of the summative OSCE. However, the formative OSCEs generally contain fewer stations than the summative OSCE, owing to logistical and manpower constraints.
The content of the stations changes in each OSCE. All cases are designed by the institution’s core faculty. The cases undergo two to three rounds of review and quality control by faculty, and standards are set via the modified Angoff method, with marking checklists and passing scores determined. Similar to the summative OSCE, residents are directly observed and scored by one assessor per station, using checklists that are designed to be similar to those of the summative OSCE, with input from the simulated patient if present. The assessors, who are members of the faculty, are trained on the assessment standards prior to the OSCE.
After the formative OSCE, a group feedback session for all participants is conducted, during which the assessors provide verbal feedback on each station. Residents are also shown their mark sheets, which contain their scores and written feedback from assessors for every station.
The number of participants in each formative OSCE varies depending on the number of residents who have yet to pass the summative OSCE. The number of formative OSCEs that a resident participates in prior to attempting the summative OSCE also varies as residents are given the freedom to decide at which stage of junior residency they attempt the summative OSCE. Residents can choose not to attend a formative OSCE if they are at work or on leave.
Selection of participants
All NHG EM residents during the study period were included. Residents who left the programme midway were excluded.
Data collection and processing
Data from scores for all formative OSCEs conducted during the study period were used. An online questionnaire using Google Forms was distributed via email to all current and past residents who were in the residency programme during the study period, and responses were collected over 1 month in November 2019. The data obtained from the questionnaire included whether and when residents had sat for and/or passed the summative OSCE. Residents’ perceptions of the usefulness of the formative OSCE were obtained on a 4-point Likert scale (1 = not useful, 4 = very useful), and through a mandatory free-text question on why they felt the formative OSCE was or was not useful.
An EM residency programme coordinator, who was not in the investigating team, distributed the questionnaires, collated the data, removed all identifiers and assigned identifying numbers to each participant before passing the data to the investigators. The programme coordinator checked the validity and accuracy of the data, and the investigators verified the internal consistency of the data.
Data analysis
We compared residents’ formative OSCE participation and scores with first-attempt summative OSCE success. We also compared other factors with first-attempt summative OSCE success, namely residency year, postgraduate year and residents’ perceptions of the usefulness of the formative OSCE. Categorical variables were analysed with Pearson’s chi-square and continuous and ordinal variables were analysed with Mann-Whitney U tests. Qualitative analysis of residents’ opinions on the usefulness of the formative OSCE was performed to triangulate the findings. All quantitative data were analysed using IBM SPSS Statistics version 23 (IBM Corp., Armonk, NY, USA).