Skip to main content

Developing an emergency medicine handoff tool: an electronic Delphi approach



Handoffs at the end of clinical shifts occur with high frequencies in the emergency department setting and they pose an increased risk to patients. There is a need to standardize handoff practices. This study aimed to use an electronic Delphi method to identify the core elements essential for an emergency department physician to physician handoff and propose a framework for implementation.


An electronic Delphi-style study with a national panel of board-certified emergency physicians in Saudi Arabia. The panel was conducted over four rounds. The first to identify elements relevant to the end of shift handoff and categorize them into domains, while the remaining three to score and debate individual elements.


Twenty-five board-certified emergency physicians from various cities and practice settings were enrolled. All panelists completed the entire Delphi process. Thirty-two elements were identified and classified into 4 domains. The top five rated handoff elements were patient identification, chief complaint history, clinical stability, working diagnosis, and consulting services involved. Panel scores showed convergence as rounds progressed and the final list of elements had a high-reliability score (Cronbach’s alpha 0.93).


This study yielded an itemized and ranked list of elements that are easy to implement and could be used to standardize patient handoffs by emergency physicians. While this study was conducted on an emergency medicine panel, the methods used may be adapted to develop standardized handoff frameworks that serve different disciplines or practice settings.


Handoffs are a daily part of the practice of emergency medicine (EM). Given the nature of shift work and the relatively high volumes of acute care visits, it is likely that emergency physicians hand off more patients than doctors in any other medical discipline [1]. In addition to handing off individual patients who are actively under the care of the outgoing physician, the emergency department (ED) end of shift handoffs often include the handoffs of some patients who are in waiting rooms, others who might be expected to arrive with ambulance services, patients who have been referred to admitting services but remain in the ED and those actively being treated in the ED. Many handover tools exist, but most are generic or have been developed for various specific disciplines or settings and therefore might not be practical or meaningful in the ED [2,3,4,5]. Techniques utilizing structured interviews and focus groups have been used to modify generic tools or create new tools for EM [6, 7]. Due to the association of handoffs with the increased risk of medical errors, both accreditation and professional agencies have called for improvement and standardization of the process [8, 9]. Nevertheless, a common standardized tool has not emerged, and handoffs remain prone to significant variations [10, 11]. The Delphi method is a well-established consensus-forming technique that has been used in various ways in the healthcare setting [12, 13]. We aimed to conduct a Delphi-style study with a panel of emergency physicians to generate and score a list of elements that may be required for the end of shift handoff.


We conducted an electronic Delphi-style study with a panel of emergency medicine consultants practicing in Saudi Arabia. At the time, this study was conducted the Saudi Commission for Health Specialties, the national licensing body, had 433 EM consultants registered. As previous literature suggests that Delphi panels with subject matter experts could be conducted with a relatively small number of experts [14, 15], we aimed for a panel of 20 to 30 consultant emergency physicians. Invitations to participate in the panel were sent to emergency physicians practicing in Saudi Arabia through the Saudi Arabian Society of Emergency Medicine. To capture providers unregistered with the Saudi Association of Emergency Medicine, open invitations were also sent to EM attending physicians through group emails and social media sites. All invitations contained a link to a web-based form to register as a panelist. To mitigate the loss of panelists during the study period, the form’s introduction detailed the expected panel process, including what would be expected of panelists and the amount of time that might be required. Panelists were required to be board-certificate in EM with no fewer than 3 years in practice as an EM attending physician. The Delphi process had 4 rounds (Fig. 1). In the first round, panelists were asked to provide itemized lists of all elements that they considered relevant for an end of shift handoff. The moderator compiled the lists, removed duplicates and organized the elements into general domains. In round 2, the compiled list of elements was sent to panelists in an electronic form for every panelist to score each element. As most elements provided by the panelists could be very important in any given specific situation, panel members were instructed to not score items on importance alone but rather on how frequently each element was required during handoffs, from 1 “rarely required” to 10 “always required.” Panelists were also able to add comments to support their positions. In round 3, each panelist received a spreadsheet showing the group’s average scores compared to the panelist’s own score for each element, along with the compiled comments or arguments shared by other panelists. Each panelist would then have a chance to change their score and/or comment on the group’s score to influence other panelists. In round 4, the average scores were recalculated based on the previous round’s results, and the comments were compiled and sent to each panelist in a spreadsheet similar to the one used in round 3; the panelists were given a final chance to revise their scores. Panel members were anonymous and unaware of each other’s identities throughout the study. The moderator, an emergency physician, did not contribute input to the panel. Emails and phone calls were used to communicate with panelists for follow up on missing input or clarifications on comments. Google forms, Google sheets, and Microsoft Excel spreadsheets were used to collect scores and comments in rounds 2 to 4.

Fig. 1

Study flow

The mean score for each element and pooled standard deviations for all elements were used as measures of central tendency and spread. We calculated the absolute difference in each element’s mean score between rounds and summed the differences to assess the amount of change in panel scores between rounds as a measure of the panel’s stability. The intraclass correlation coefficient (ICC) using a two-way random-effects model for consistency was also used to assess the panelist agreement in every round [16, 17]. Cronbach’s alpha was used to assess the reliability of the itemized list of handoff elements.

Statistical Package for Social Sciences (SPSS for Mac, Version 21; SPSS, Inc., Chicago, IL) was used for the analysis.


Registration was open for 6 weeks in early 2018. Twenty-nine EM physicians completed the registration form, and 4 were excluded because they had fewer than 3 years of experience as an EM attending physician. The 25 included panelists practiced in 16 different hospitals in 6 cities in Saudi Arabia, with an average of 7.9 years of post-board certification experience. All four rounds of the study were completed by the panel concluded over a period of 6 weeks. All panelists completed all four rounds of the study, and there were no dropouts.

Thirty-two specific handoff elements were identified and rated according to the described Delphi methods. Those were placed into the following four domains: nonclinical patient information, clinical patient information, patient course in the ED, and general ED status. The general ED status domain includes elements that are either general to the department or not specific to an individual patient being handed off. At the conclusion of the panel, the individual element mean scores ranged from 4.7 to 9.8 out of a maximum of 10 (Table 1). The top five rated elements were the chief complaint history, patient identification, clinical stability, working diagnosis, and consulting services involved.

Table 1 Final panel scores

The sum of absolute differences of elements means between rounds 2 and 3 was 15.8. Between rounds 3 and 4, the sum of absolute differences was only 3.48. The pooled standard deviations for all element scores in rounds 2, 3, and 4 were 2.1, 1.9, and 1.8, respectively. The panelists ICC for rounds 2, 3, and 4 were 0.23, 0.30, and 0.36, respectively. Reliability testing for the elements returned a Cronbach’s alpha of 0.93, 0.89, and 0.93 for rounds 2, 3, and 4 respectively.


This study demonstrates the use of the Delphi method in developing a discipline-specific set of handoff elements that could be utilized in the development of a handoff tool. Handoff tools have been commonly developed using focus groups or unstructured interviews (e.g., I-PASS [6, 18] and ABC [7]), but formal consensus development methods appear to have rarely been used for this purpose. The Delphi process is a formal consensus forming method that offers some key advantages for approaching handoffs in a specialized field [9]. Focus groups and brainstorming sessions are ideal for idea generation, but specialty experts may not require exploratory or idea-generating techniques as much as they require the prioritization of ideas that are typically shared and well known within the discipline. While group interviews and focus groups allow for better articulation of opinions, they may be dominated by stronger individuals or coalitions and the open format may inhibit some members from speaking freely [9]. They are also difficult to organize with larger groups or with members in different geographical areas. In contrast, the electronic Delphi is most practical for a relatively large group of subject matter experts who may not be able to meet in a specific time or location and the anonymity of Delphi panelists helps limit undue influence exerted by members who may be more senior in rank or more outspoken. In both I-PASS and ABC studies, the outcomes of staff interviews were thematic and non-discrete and needed to be further modified or refined by the facilitators into practical tools ready for implementation. In comparison, the Delphi method we describe yields a discrete list of items by design and the results could be directly used in a handoff tool when needed.

One of the limitations of the electronic Delphi method is the lack of face-to-face interactions that allow for groups to clarify ideas and shared understandings. This was noticed when some panelists on occasions gave an element a lower score only because they felt it was redundant and should be considered part of another element, while others considered the element at face value and scored it independently. Our design did not allow the group to revise the summarized list of elements produced after round 1. It is possible that an additional round of voting to select elements that needed to be broken down into several individual elements or individual elements that could have been combined might have added some value.

This study did not use a uni-dimensional scoring scale but rather a scale that combines the elements importance and frequency of use. Instead of asking panelists “how important is this element?” and “how frequently is this element used?”, we asked panelists about how frequently would a given element be on the important end of the scale “How frequently is this element required?”. While it could be argued that each of the two factors should have its own scale, we believe that the result of such an exercise would likely include elements with low importance and high frequency and others that are rare but important. Such a list is not readily usable and is impractical to implement as it would require further input from yet another subject matter expert to decide the weight and ranking of each factor. We opted to offload this cognitive exercise onto the panel of experts who, as a group, were most likely to be capable of balancing all factors involved in scoring elements in their own domain of expertise. This was supported by the high-reliability coefficient of the survey. However, much of the panel’s discussion revolved around how relevant a given element is in a handoff. Some elements may be critical in a given scenario but less important in another (e.g., “vital signs” in a critically ill patient vs. a young patient with a twisted ankle). A scoring scale that revolves around the element’s relevance (i.e., from “required only when directly relevant” to “required even when remotely relevant”) might offer a solution and may be explored in future studies.

Furthermore, with regard to the scoring structure, the frequency an element is required in a handoff is heavily influenced by the setting. It is possible that this might have skewed the overall scores towards the higher end of the scale because most of the panelists practiced in tertiary care teaching hospitals where more patients with more complex conditions require the provision of more details during handoffs (e.g., code status).

The panel’s ICCs were low, but this was expected given the relatively large number of panelists (25) and elements (32), leading to 800 assessments for each round. The intent was to examine the trend of the ICC measurements as rounds move forward. As rounds progressed, the panel’s ICC continued to increase showing increasing agreement on elements. The absolute sum of differences of the means and the pooled standard deviations both show clear convergence in round 4, also showing improved consensus.

Handoff tools serve as checklists to help reduce human errors. Although many handoff tools have been studied in healthcare settings [5, 6], most are generic and lacking in specificity and may not always be meaningful for providers working in a setting with specific expectations in a patient handoff. This study design addresses this issue by yielding a set of items specific to the discipline of the panelists. It also has practical appeal as the outcome’s format is easy to implement in checklists or forms on paper or electronic systems. It is also easy to replicate with different disciplines and on various scales from a single department within an institution to international professional groups and scientific societies.

We propose that the outcomes of a study conducted with the methods presented be used as a framework for developing standardized handoff tools. Professional groups may choose arbitrary cutoff values to design a handoff tool based on the results of such studies. These cutoff values could help decide which elements to include or which elements may be mandatory in a proposed handoff tool. Cutoff values may also be chosen on the basis of acuity for the area or department the handoff tool is intended for so that providers would have different handoff tools for different settings. For the purposes of illustration, we proposed upper and lower cutoff values for the elements in our study to facilitate a potential implementation in an electronic health record. In this example, elements above the upper cutoff value would be mandatory fields for all handoffs, those between the two cutoff values would be optional based on the context, and elements below the lower cutoff value would be excluded from the form (Fig. 2).

Fig. 2

A proposed final outcome of the handoff of an individual ED patient with a proposed lower cutoff line for inclusion of elements in a handoff tool and an upper cutoff line for mandatory elements


This study demonstrates how the electronic Delphi method was used in the development of an emergency medicine end-of-shift handoff tool using a panel of subject matter experts located in different locations and time zones. While the specific elements in this study may or may not be suitable for general use, the process yields an itemized and ranked list of elements that is easy to adopt on paper or electronic forms should it be implemented.

Availability of data and materials

The datasets used and/or analyzed during the current study are available from the corresponding author on reasonable request.


  1. 1.

    Australian Institute of Health and Welfare. Admitted patient care 2014–15: Australian hospital statistics. Health services series no. 68. Canberra: AIHW; 2016.

    Google Scholar 

  2. 2.

    Arora V, Johnson J. A model for building a standardized hand-off protocol. Jt Comm J Qual Patient Saf. 2006;32:646–55.

    Article  PubMed  Google Scholar 

  3. 3.

    Handoffs. Transitions of Care for Children in the Emergency Department. Pediatrics. 2016;138:e20162680.

    Article  Google Scholar 

  4. 4.

    Cheung D, Kelly J, Beach C, et al. Improving handoffs in the emergency department. Ann Emerg Med. 2010;55:171–80.

    Article  PubMed  Google Scholar 

  5. 5.

    Riesenberg L, Leitzsch J, Little B. Systematic review of handoff mnemonics literature. Am J Med Qual. 2009;24:196–204.

    Article  PubMed  Google Scholar 

  6. 6.

    Heilman J, Flanigan M, Nelson A, et al. Adapting the I-PASS handoff program for emergency department inter-shift handoffs. West J Emerg Med. 2016;17:756–61.

    Article  PubMed  PubMed Central  Google Scholar 

  7. 7.

    Farhan M, Brown R, Woloshynowych M, et al. The ABC of handover: a qualitative study to develop a new tool for handover in the emergency department. Emerg Med J. 2012;29:941–6.

    Article  PubMed  PubMed Central  Google Scholar 

  8. 8.

    Australasian College for Emergency Medicine. Guideline on Clinical Handover in the Emergency Department. Canberra: ACEM; 2015.

  9. 9.

    World Health Organization. Communication during patient hand-overs. Patient Safety Solutions. 2007;1:1–4.

  10. 10.

    Hern H, Gallahue F, Burns B, et al. Handoff practices in emergency medicine: are we making Progress? Acad Emerg Med. 2016;23:197–201.

    Article  PubMed  Google Scholar 

  11. 11.

    Riesenberg L, Leitzsch J, Massucci J, et al. Residents’ and attending physicians’ handoffs: a systematic review of the literature. Acad Med. 2009;84:1775–87.

    Article  PubMed  Google Scholar 

  12. 12.

    World Health Organization. WHO Handbook for Guideline Development. Geneva: World Health Organization; 2014.

    Google Scholar 

  13. 13.

    Hsu C, Sandford B. The Delphi technique: making sense of consensus. Pract Assess Res Eval 2007; 12. Accessed 13 Dec 2018.

  14. 14.

    Akins RB, Tolson H, Cole BR. Stability of response characteristics of a Delphi panel: application of bootstrap data expansion. BMC Med Res Methodol. 2005;5:37.

    Article  Google Scholar 

  15. 15.

    Thangaratinam S, Redman CW. The Delphi technique. Obstetric Gynaecol. 2005;7:120–5.

    Article  Google Scholar 

  16. 16.

    Bajpai R, Chaturvedi H. Evaluation of inter-rater agreement and inter-rater reliability for observational data: an overview of concepts and methods. J Indian Acad App Psychol. 2015;41:20–7.

    Google Scholar 

  17. 17.

    Koo T, Li M. A guideline of selecting and reporting Intraclass correlation coefficients for reliability research. J Chiropr Med. 2016;15:155–63.

    Article  PubMed  PubMed Central  Google Scholar 

  18. 18.

    Starmer AJ, Spector ND, Srivastava R, et al. I-PASS, a mnemonic to Stardardize verbal handoffs. Pediatrics. 2012;192(2):201–4.

    Article  Google Scholar 

Download references


We would like to acknowledge the emergency physicians of the panel, Drs. Abdullah Almarshed, Abdulmohsen Alsaawi, Abdulsalam Alzahrani, Abdussalam Alshehri, Ali Almajed, Anas Khan, Badr Aldawood, Basem Bahrani, Eman Alyousif, Faisal Almazroua, Ghada Bakhidar, Hamza Abutalib, Khalid Alaufi, Khalid Ateyyah, Mishal Almarshady, Mohamed Azzam, Mohammed Almutairi, Mohye Alwahhas, Mostafa Omaish, Nawfal Aljerian, Nayif Alsayari, Osama Kentab, Saeed Alzahrani, Saleh Alesa, and Shadi Almoziny. We would also like to acknowledge Ms. Najla Alassmi for her outstanding work in coordinating this project.


This study received no funding.

Author information




Both authors contributed to the conception and design of the work. KR moderated the panel, acquired and analyzed the data. Both authors contributed to writing the article and have read and approved the final manuscript.

Corresponding author

Correspondence to Khaled Alrajhi.

Ethics declarations

Ethics approval and consent to participate

The study was approved by our institutional review board.

Consent for publication

Not Applicable.

Competing interests

The authors declare that they have no competing interests.

Additional information

Publisher’s Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Open Access This article is distributed under the terms of the Creative Commons Attribution 4.0 International License (, which permits unrestricted use, distribution, and reproduction in any medium, provided you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made.

Reprints and Permissions

About this article

Verify currency and authenticity via CrossMark

Cite this article

Alrajhi, K., Alsaawi, A. Developing an emergency medicine handoff tool: an electronic Delphi approach. Int J Emerg Med 12, 37 (2019).

Download citation


  • Patient handoff
  • Patient safety
  • Emergency medicine
  • Delphi technique