Skip to main content
  • Original Research
  • Open access
  • Published:

A comparative analysis of risk stratification tools for emergency department patients with chest pain



Appropriate disposition of emergency department (ED) patients with chest pain is dependent on clinical evaluation of risk. A number of chest pain risk stratification tools have been proposed. The aim of this study was to compare the predictive performance for major adverse cardiac events (MACE) using risk assessment tools from the National Heart Foundation of Australia (HFA), the Goldman risk score and the Thrombolysis in Myocardial Infarction risk score (TIMI RS).


This prospective observational study evaluated ED patients aged ≥30 years with non-traumatic chest pain for which no definitive non-ischemic cause was found. Data collected included demographic and clinical information, investigation findings and occurrence of MACE by 30 days. The outcome of interest was the comparative predictive performance of the risk tools for MACE at 30 days, as analyzed by receiver operator curves (ROC).


Two hundred eighty-one patients were studied; the rate of MACE was 14.1%. Area under the curve (AUC) of the HFA, TIMI RS and Goldman tools for the endpoint of MACE was 0.54, 0.71 and 0.67, respectively, with the difference between the tools in predictive ability for MACE being highly significant [chi2 (3) = 67.21, N = 276, p < 0.0001].


The TIMI RS and Goldman tools performed better than the HFA in this undifferentiated ED chest pain population, but selection of cutoffs balancing sensitivity and specificity was problematic. There is an urgent need for validated risk stratification tools specific for the ED chest pain population.


Chest pain remains one of the most common presenting complaints in patients presenting to emergency departments (ED), with >100,000 patients being hospitalized each year in Australia with acute coronary syndromes (ACS) [1]. The morbidity, mortality and economic costs associated with this constitute a significant burden on the Australian health system [2]. In these patients, use of risk stratification tools to predict risk of death, cardiac complications and the pre-test probability of ACS has been demonstrated to aid clinicians in appropriate prioritizing of patients for investigations [3, 4] and assists in identification of those at higher risk who might benefit most from potent drug therapies or an early invasive therapeutic approach [510]. Risk stratification tools have further been shown to allow patients to be better informed of their prognosis [7], improve cost-effectiveness while minimizing unnecessary treatment complications [11] and reduce unnecessary admissions to inpatient monitored beds, without increasing complications, thereby potentially having a positive impact on access block [3, 6, 12, 13].

Recently, in Queensland, a suite of clinical pathways for the management of patients presenting to public hospitals with chest pain was developed [14], with application incentivized by practice improvement payments [15]. These pathways rely on risk stratification utilizing the National Heart Foundation of Australia risk stratification tool (HFA), perhaps the most prominent risk stratification utilized in Australia [16]. Although the individual components of this tool are evidence based and the tool was developed by consensus of an expert panel, it was designed for risk stratification of patients with ACS rather than for the undifferentiated ED chest pain population. There are conflicting data regarding its performance in ED chest pain populations [17, 18].

The aim of this prospective observational study was to compare the performance of three methods of risk stratification, namely, the National Heart Foundation of Australia risk stratification tool (HFA) [16], the Goldman score [6] and the Thrombolysis in Myocardial infarction (TIMI) risk score [19] for prediction of a composite outcome of major adverse cardiac events (MACEs) within 30 days of ED attendance.


This was a prospective, cohort study undertaken at a single tertiary referral ED. Patients presenting to the ED with non-traumatic chest pain during the preceding 48 h and aged >30 years were eligible for inclusion in the study. Exclusion criteria were the presence of a definitive non-ischemic cause for chest pain, isolated angina-equivalent symptoms, trauma-related chest pain, cardiac arrest on arrival to the ED, patients with ECG criteria for ST-elevation myocardial infarction (MI) on arrival to the ED and inability to provide informed consent.

Data collection occurred during the ED presentation on weekdays that a trained research nurse was available. Follow-up was undertaken by both review of medical records utilizing a standardized data collection tool and a phone call to the patient (or proxy if the patient was not contactable) employing a structured interview at 72 h and 30 days. A single emergency physician, blinded at the time to the patient outcomes, retrospectively undertook the risk stratification process using the prospectively collected data items and initial ED-acquired electrocardiogram.

All patients included in the study had their cardiac risk determined by each of three methods of risk stratification utilizing findings on presentation to the ED (see Table 1). The HFA [16] and Goldman tools classify patients into risk groups with nominal descriptors (e.g., high, low), while the TIMI risk tool derives a score out of seven [9, 17].

Table 1 Risk stratification tools

The primary outcome of interest was MACE within 30 days of ED presentation. MACE components were defined utilizing the American College of Cardiology Clinical Data Standards definitions [20] and included acute myocardial infarction (prevalent and incident), recurrent ischemia requiring urgent revascularization, cardiogenic shock, ventricular arrhythmia requiring emergent intervention or high-grade atrioventricular block requiring treatment, cardiac arrest and all-cause mortality.

For analysis, continuous variables with normal distribution were expressed as medians and interquartile ranges; categorical data were presented as percentages. Group differences in continuous and categorical variables are compared with Kruskal-Wallis and chi-square tests respectively. For each of the risk stratification methods, ROC curves were used to evaluate its predictive performance. Area under the curve (AUC) was utilized as a summary measure for diagnostic accuracy of the prediction tools across the gamut of risk groups [21], with 95% confidence intervals (CI). For the comparison of clinical performance of the risk scores, we chose to include AHA high risk, Goldman high risk and two cutoffs of the TIMI score (≥1 and ≥2). The latter were chosen pragmatically a priori and attempted to balance case discrimination and sensitivity. Inclusion of patients with a TIMI score of zero provides no case discrimination, while using a cutoff of ≥3 has been shown to have a sensitivity <60% [22].

For all comparisons a p-value of <0.05 was considered statistically significant. Statistical analysis was performed with Stata, version 10 (College Station, TX, USA). Power calculations were generated as follows: the AUC of the TIMI score has previously been demonstrated to be 0.6 [23], while the Goldman score has been shown to have an AUC of 0.9 [3, 6]. The performance of the HFA score was expected to be similar to that of the Goldman. Assuming a correlation between positive and negative groups of 0.4, a sample size of 400 patients was required to distinguish an AUC of 0.85 from one of 0.9, at a p value of 0.05 with an 80% power. An interim analysis was performed as approved study duration and funding were nearing their end. The study was terminated early as this interim analysis revealed a clearly statistically significant result. The institutional ethics committee approved the study.


Two hundred eighty-one patients were studied with 276 completing 30-day follow-up. The median age of the study group was 56 years (IQR 47.5-66), with the majority (61.5%) being male (Figure 1). Patient characteristics are summarized in Table 2. Of the 276 patients with 30-day follow-up, 39 (14.1%) had a MACE.

Figure 1
figure 1

Patient enrollment.

Table 2 Patient characteristics

The predictive performance of the risk stratification tools is shown in Figure 2. AUC for prediction of MACE was poorest for the HFA tool, with an AUC of 0.54 (95% CI 0.45-0.63). The TIMI risk score had the highest AUC of the three tools tested, with an AUC of 0.71 (95% CI 0.63-0.79), while the Goldman tool had an AUC of 0.67 (95% CI 0.57-0.77). The difference between the tools in predictive ability for MACE was highly significant (p = 0.0002). There was no statistically significant difference in performance between the Goldman tool and TIMI score. The sensitivity and specificity of the tools are summarized in Table 3.

Figure 2
figure 2

Predictive performance of risk stratification tools.

Table 3 Comparative performance of risk stratification tools

Utilizing the chi-square test to compare the AUC for the TIMI, Goldman and HFA tools, the difference between the tools in terms of predictive ability for MACE was found to be highly significant (chi2, N = 276, p <0.0001). Further analysis showed that the HFA tool performance was different from that of each of the other tools, which were similar in performance.


In this study of ED patients with chest pain, where no alternative non-ACS cause was apparent, none of the tools under investigation were ideal. Characteristics of an ideal tool for stratifying risk of chest pain in the ED population would be that it has high sensitivity for risk of MACE, specificity sufficient to enable feasible application and, from a patient perspective, limit exposure to unnecessary investigations or interventions. It would also have data elements that were non-subjective or, at least, had high interobserver reliability. The TIMI risk score had the best performance in stratifying risk for MACE. However, utilizing a cutoff of ≥1 to achieve a sensitivity of 97% resulted in a specificity of only 13%. A higher cutoff of >3 would be required to achieve a more acceptable specificity of 60%; however, the resultant sensitivity of 74% is unacceptable. The sensitivity of the Goldman score (69%) was insufficient to be useful in the ED population, even with the lowest possible cutoff, namely including all patients with a risk higher than ‘very low’ risk. The HFA risk stratification tool had an AUC of 0.54, with 95% confidence intervals (0.45-0.63) disturbingly encompassing an AUC of <0.5. Its sensitivity for MACE, using a cutoff to include all high-risk patients, was 100%, but clearly with a specificity of 8%, its feasibility in ED clinical practice is limited.

The use of an unstructured or individualized approach to ED assessment of chest pain has been shown to be associated with high resource utilization for patients with no coronary artery disease, while concurrently resulting in significant proportions of patients with ACS being missed [24]. Subsequently, emphasis has been placed on utilization of risk stratification tools. Unfortunately, studies employing risk stratification tools for chest pain in the ED setting often have two significant limitations. First, the tools employed have largely been designed for risk stratification of those with known ACS, usually patients admitted to the hospital. Utilizing these tools in the undifferentiated ED patient with chest pain may have a significant impact on both safety and efficiency. Second, many trials of chest pain risk stratification have tested the tools on ED populations where their chest pain is “thought to be of ischaemic origin.” However, as physician discrimination of ACS from other causes of chest pain has previously been shown to be poor, this may result in potentially flawed inferences [24].

The TIMI tool is widely reported as being utilized in the undifferentiated chest pain population in the ED, including as part of rapid diagnostic protocols [25]. In this study, TIMI had comparable sensitivity and specificity for MACE to that found in previously published studies [25], but again highlights that the selection of an appropriate cutoff, balancing sensitivity, specificity and clinical feasibility, is problematic.

Earlier studies identified a higher sensitivity for the Goldman tool in the ED setting than our results [3, 26]. This disparity may be accounted for by the fact that these studies differed in their inclusion criteria, with a focus on patients admitted to the hospital, including patients with ST elevation MI, and/or had much earlier follow-up for the endpoint of MACE. Additionally, both the definition of MI and the available cardiac biomarkers have changed considerably since these earlier studies.

The HFA risk stratification tool had an AUC of 0.54. Its high sensitivity for MACE (100%) came at the cost of a specificity of only 8.4%, reflecting classification of 93% of patients as being at “high risk.” This finding is concordant with a previously published study, which similarly questioned the HFA risk stratification tool’s suitability for use in this patient cohort [18]. If the HFA decision support tool [16] were applied to the patient population in this study, it would lead to 93% of ED chest pain patients being admitted to hospital wards and receiving treatments such as heparin. The weight of evidence suggests that the HFA risk stratification tool is inappropriate for use in chest pain pathways in the unselected ED chest pain population.

Other approaches to risk stratification of ED chest pain patients have recently been reported. The HEART score [4, 27] was developed in The Netherlands based on clinical experience and literature review rather than database methods. It has five components – history, ECG, age, risk factors and troponin level – which are each rated 0, 1 or 2 based on criteria. In a validation study, it was shown to have better discriminative performance than the GRACE and TIMI risk scores [27]. In multicentre validation, a low HEART score (≤3) had a 1.7% rate of MACE [28]. Of concern is that this score relies in part on subjective assessment of likelihood of ACS for which inter-rater data are scarce.

Another approach was taken by the GRACE investigators with the development of a score aimed at predicting the absence of MACE. The GRACE freedom from events score [29] was developed in an admitted chest pain cohort with likely ACS and has undergone limited external validation on admitted chest pain cohorts [30, 31]. No validation in an ED chest pain cohort has yet been published.

A similar approach has been taken in the development of the North American Chest Pain Rule (NACPR) [32]. It aims to identify low-risk ED chest pain patients suitable for early discharge. The rule consists of the absence of five predictors—ischemic ECG changes not known to be old, history of coronary artery disease, pain typical for ACS, initial or 6-h troponin level greater than the 99th percentile and age greater than 50 years. In internal validation, it was 100% sensitive (95% confidence interval 97.2% to 100.0%) and 20.9% specific (95% confidence interval 16.9% to 24.9%) for a cardiac event within 30 days, with 11% of patients being defined as low risk [32]. Its utility has been challenged in an external validation study and comparison to the HEART score [33]. The NACPR identified 4.4% (95% CI 3-6%) for early discharge with 100% (95% CI 98-100%) sensitivity for ACS, while the HEART score identified 20% (95% CI 18-23%) for early discharge with 99% (95% CI 97-100%) sensitivity for ACS. The low proportion of patients identified as low risk in a population with MACE of 22% is a serious threat to this score’s clinical utility. That said, the approach of trying to decide who is safe for discharge rather than identification of high risk is worthy of further exploration.

This study has some limitations that should be considered in interpreting the results. Bias in the verification of events is possible if the inability to contact patients in follow-up coincided with those having events. The potential for verification bias has been minimized by follow-up not only via chart review but also via admissions registry review and with the patient or proxy. While most data were collected prospectively, some were collected retrospectively and so are subject to potential data omission. Some of the data were reliant on patient self-report (e.g., of past history and risk factors). If available in the chart, this was verified; however, otherwise, no attempt was made to verify these data, reflecting the real-world ED clinical interaction. The study was conducted at a single site, and this may have impacted the external validity.


The TIMI risk stratification score appeared most suitable for use in an undifferentiated ED chest pain population, but selection of an appropriate cutoff is problematic. This study highlights the need for validated risk stratification tools for the ED chest pain cohort that examine not only their safety in terms of their sensitivity, but also their flow and efficiency impacts, as these factors have significant implications for safety for all ED patients.


  1. Hacker S, Madden R: Australia’s Health. Australian Institute of Health and Welfare. Canberra: AIHW Cat. No. AUS 44; 2004.

    Google Scholar 

  2. Begg S, Vos T, Barker B, et al.: The burden of disease & injury in Australia 2003. In Australian Institute of Health and Welfare. Canberra: AIHW; 2007.

    Google Scholar 

  3. Reilly BM, Evans AT, Schaider JJ: Impact of a clinical decision rule on hospital triage of patients with suspected acute cardiac ischemia in the emergency department. JAMA 2002, 288: 342–350. 10.1001/jama.288.3.342

    Article  PubMed  Google Scholar 

  4. Six AJ, Backus BE, Kelder JC: Chest pain in the emergency room: value of the HEART score. Netherlands Heart J 2008, 16: 191–196. 10.1007/BF03086144

    Article  CAS  Google Scholar 

  5. Lagerqvist B, Diderholm E, Lindahl B, et al.: FRISC score for selection of patients for an early invasive treatment strategy in unstable coronary artery disease. Heart 2005, 91: 1047–1052. 10.1136/hrt.2003.031369

    Article  PubMed Central  CAS  PubMed  Google Scholar 

  6. Goldman L, Cook F, Johnson PA, et al.: The need for intensive care in patients who come to emergency departments with acute chest pain. New Eng J Med 1996, 334: 1498–1504. 10.1056/NEJM199606063342303

    Article  CAS  PubMed  Google Scholar 

  7. Sabatine M, Antman E: The thrombolysis in myocardial infarction risk score in unstable angina/non ST-segment elevation myocardial infarction. J Am Coll Cardiol 2003, 41: S89-S95. 10.1016/S0735-1097(02)03019-X

    Article  Google Scholar 

  8. Sabatine MS, McCabe CH, Morrow DA, et al.: Identification of patients at high risk for death and cardiac ischemic events after hospital discharge. Am Heart J 2002, 143: 966–970. 10.1067/mhj.2002.122870

    Article  PubMed  Google Scholar 

  9. Pollack CV, Sites FD, Shofer FS, et al.: Application of the TIMI risk score for unstable angina and non-ST elevation acute coronary syndrome to an unselected emergency department chest pain population. Acad Emerg Med 2006, 13: 13–18. 10.1111/j.1553-2712.2006.tb00978.x

    Article  PubMed  Google Scholar 

  10. Yan AT, Yan RT, Tan M, et al.: Risk scores for risk stratification in acute coronary syndromes: useful but simpler is not necessarily better. Eur Heart J 2007, 28: 1072–1078. 10.1093/eurheartj/ehm004

    Article  PubMed  Google Scholar 

  11. Cakir B, Blue K: How to improve the management of chest pain: hospitalists and use of prediction rules. Southern Med J 2007, 100: 242–247. 10.1097/SMJ.0b013e31802f7f94

    Article  PubMed  Google Scholar 

  12. Durairaj L, Reilly B, Das K, et al.: Emergency Department admissions to inpatient cardiac telemetry beds: a prospective cohort study of risk stratification and outcomes. Am J Med 2001, 110: 7–11.

    Article  CAS  PubMed  Google Scholar 

  13. Hollander JE, Sites FD, Pollack CV, et al.: Identification of cardiac death and life-threatening ventricular dysrhythmias in low-risk patients with chest pain. Ann Emerg Med 2004, 43: 71–76. 10.1016/S0196-0644(03)00719-4

    Article  PubMed  Google Scholar 

  14. Clinical Practice Improvement Centre statewide cardiac pathways. 2009. [cited 2009 Jan 15]; Available from: . Accessed Febuary 11, 2014

  15. Centre CPI (Ed): Clinical Practice Improvement Payment - user guide In version 2.0 ed: Queensland Health 2009, 1–18.

  16. Aroney C, Aylward P, Kelly A-M, et al.: Guidelines for the management of acute coronary syndromes 2006. Med J Aust 2006,184(Suppl):S1-S29.

    Google Scholar 

  17. Cullen L, Greenslade J, Hammett CJ, et al.: Comparison of three risk stratification rules for predicting patients with acute coronary syndrome presenting to an australian emergency department. Heart Lung Circ 2013. epub ahead of print

    Google Scholar 

  18. Kelly AM: How useful are the heart foundation risk criteria for assessment of emergency department patients with chest pain? Emerg Med Australas 2012, 24: 260–265. 10.1111/j.1742-6723.2012.01536.x

    Article  PubMed  Google Scholar 

  19. Antman EM, Cohen M, Bernink P, et al.: The TIMI risk score for unstable Angina/Non-ST elevation MI: a method for prognostication and therapeutic decision making. JAMA 2000, 284: 835–842. 10.1001/jama.284.7.835

    Article  CAS  PubMed  Google Scholar 

  20. Cannon CP, Battler A, Brindis RG, et al.: American College of Cardiology key data elements and definitions for measuring the clinical management and outcomes of patients with acute coronary syndromes. A report of the American College of Cardiology Task Force on Clinical Data Standards (Acute Coronary Syndromes Writing Committee). J Am Coll Cardiol 2001, 38: 2114–2130. 10.1016/S0735-1097(01)01702-8

    Article  CAS  PubMed  Google Scholar 

  21. Zou KH, O’Malley AJ, Mauri L: Receiver-operating characteristic analysis for evaluating diagnostic tests and predictive models. Circulation 2007, 115: 654–657. 10.1161/CIRCULATIONAHA.105.594929

    Article  PubMed  Google Scholar 

  22. Hess EP, Agarwal D, Chandra S, et al.: Diagnostic accuracy of the TIMI risk score in patients with chest pain in the emergency department: a meta-analysis. CMAJ 2010, 182: 1039–1044. 10.1503/cmaj.092119

    Article  PubMed Central  PubMed  Google Scholar 

  23. de Araujo GP, Ferreira J, Aguiar C, et al.: TIMI, PURSUIT, and GRACE risk scores: sustained prognostic value and interaction with revascularization in NSTE-ACS. Eur Heart J 2005, 26: 865–872. 10.1093/eurheartj/ehi187

    Article  Google Scholar 

  24. Christenson J, Innes G, McKnight D, et al.: Safety and efficiency of emergency department assessment of chest discomfort. CMAJ 2004, 170: 1803–1807. 10.1503/cmaj.1031315

    Article  PubMed Central  PubMed  Google Scholar 

  25. Than M, Cullen L, Reid CM, et al.: A 2-h diagnostic protocol to assess patients with chest pain symptoms in the Asia-Pacific region (ASPECT): a prospective observational validation study. Lancet 2011, 377: 1077–1084. 10.1016/S0140-6736(11)60310-3

    Article  PubMed  Google Scholar 

  26. Goldman L, Cook EF, Johnson PA, et al.: Prediction of the need for intensive care in patients who come to the emergency departments with acute chest pain. New Eng J Med 1996, 334: 1498–1504. 10.1056/NEJM199606063342303

    Article  CAS  PubMed  Google Scholar 

  27. Backus BE, Six AJ, Kelder JC, et al.: A prospective validation of the HEART score for chest pain patients at the emergency department. Int J Cardiol 2013. doi: 10.1016/j.ijcard.2013.01.255. [Epub ahead of print]

    Google Scholar 

  28. Six AJ, Cullen L, Backus BE, et al.: The HEART score for the assessment of patients with chest pain in the emergency department: a multinational validation study. Crit Pathw Cardiol 2013, 3: 121–126.

    Article  Google Scholar 

  29. Brieger D, Fox KA, Fitzgerald G, et al.: Predicting freedom from clinical events in non-STelevation acute coronary syndromes: the Global Registry of Acute Coronary Events. Heart 2009, 95: 888–894. 10.1136/hrt.2008.153387

    Article  CAS  PubMed  Google Scholar 

  30. Kelly AM, Dabee P, Klim S, et al.: External validation of the GRACE Freedom from Events score. Heart Lung Circ 2012, 21: 582–585. 10.1016/j.hlc.2012.05.005

    Article  PubMed  Google Scholar 

  31. Soderholm M, Deligani MM, Choudhary M, et al.: Ability of risk scores to predict a low complication risk in patients admitted for suspected acute coronary syndrome. Emerg Med J 2012, 29: 644–649. 10.1136/emermed-2011-200328

    Article  PubMed  Google Scholar 

  32. Hess EP, Brison RJ, Perry JJ, et al.: Development of a clinical prediction rule for 30-day cardiac events in emergency department patients with chest pain and possible acute coronary syndrome. Ann Emerg Med 2012, 59: 115–125. 10.1016/j.annemergmed.2011.07.026

    Article  PubMed  Google Scholar 

  33. Mahler SA, Miller CD, Hollander JE, et al.: Identifying patients for early discharge: Performance of decision rules among patients with acute chest pain. Int J Cardiol 2012. doi: 10.1016/j.ijcard.2012.10.010. [Epub ahead of print]

    Google Scholar 

Download references

Author information

Authors and Affiliations


Corresponding author

Correspondence to Anne-Maree Kelly.

Additional information

Competing interests

Funding was provided for this project by the Queensland Emergency Medicine Research Fund. AMK is co-author of the Australasian guidelines for the management of acute coronary syndromes.

Authors’ contributions

EB had the concept for the study, all authors had input into study design, EB, TM and OT undertook analysis, all authors had input into interpretation of data, EB and AMK drafted the manuscript with all authors contributing to its refinement and approving the final manuscript.

Authors’ original submitted files for images

Below are the links to the authors’ original submitted files for images.

Authors’ original file for figure 1

Authors’ original file for figure 2

Rights and permissions

Open Access This article is distributed under the terms of the Creative Commons Attribution 2.0 International License ( ), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.

Reprints and permissions

About this article

Cite this article

Burkett, E., Marwick, T., Thom, O. et al. A comparative analysis of risk stratification tools for emergency department patients with chest pain. Int J Emerg Med 7, 10 (2014).

Download citation

  • Received:

  • Accepted:

  • Published:

  • DOI: