Challenging script concordance test reference standard by evidence: do judgments by emergency medicine consultants agree with likelihood ratios?

Ahmadi, Seyed-Foad; Khoshkish, Shahin; Soltani-Arabshahi, Kamran; Hafezi-Moghadam, Peyman; Zahmatkesh, Golara; Heidari, Parisa; Baba-Beigloo, Davood; Baradaran, Hamid R; Lotfipour, Shahram

doi:10.1186/s12245-014-0034-3

International Journal of Emergency Medicine

Table 2 Calculation of the score of a hypothetical test-taker

From: Challenging script concordance test reference standard by evidence: do judgments by emergency medicine consultants agree with likelihood ratios?

Disease/finding		Score of the categories
Disease/finding	Categorized LR	Very low (?2)	Low (?1)	Middle (0)	High (+1)	Very high (+2)
1. Meningitis
??1A. Presence of headache	Middle (0)	0	0	0.57	1	0.57
??1B. Absence of nausea/vomiting	Middle (0)	0	0.07	1	0.07	0
??1C. Presence of neck stiffness	Middle (0)	0	0.09	0.09	0.09	1
??1D. Presence of Brudzinski's sign	Middle (0)	0	0	0.22	0.44	1
??1E. Presence of Kernig's sign	Middle (0)	0	0	0.12	0.75	1
2. Myocardial infarction
??2A. Presence of chest pain with radiation to both arms	Very high (+2)	0	0	0.37	1	0.50
??2B. Absence of nausea/vomiting	Middle (0)	0	0	1	0.07	0
??2C. Presence of sharp chest pain or stabbing	Low (?1)	0.25	1	1	0.50	0.25
??2D. Presence of any ST segment elevation	High (+1)	0.10	0	0.10	0.30	1
??2E. Presence of any Q wave	High (+1)	0	0	1	1	1
3. Pneumonia (in a child)
??3A. Presence of retraction	Middle (0)	0	0	0.28	0.85	1
??3B. Absence of tachypnea	Middle (0)	1	0.60	0.80	0.60	1
??3C. Presence of crackles	High (+1)	0	0.16	0.33	1	1
??3D. Presence of grunting	High (+1)	0	0	0.71	1	0.42
??3E. Absence of fever	Middle (0)	0	0.71	1	0.28	0.14
4. Thoracic aortic dissection
??4A. Presence of history of hypertension	High (+1)	0	0.12	0.25	1	0.50
??4B. Presence of focal neurologic deficit	Very high (+2)	0	0	0.25	1	0.62
??4C. Absence of pulse deficit	Middle (0)	0.11	0.11	1	0.11	0.33
??4D. Absence of enlarged aorta/wide mediastinum in CXR	Low (?1)	0	1	0.40	0.80	0.80
??4E. Absence of sudden chest pain	Low (?1)	0.66	0.33	1	0.50	0
5. Appendicitis
??5A. Absence of anorexia	Middle (0)	0.25	0.50	1	0.12	0
??5B. Presence of guarding	High (+1)	0	0.12	0.37	1	0.37
??5C. Absence of rebound tenderness	Very low (?2)	0.16	1	0.66	0.33	0.33
??5D. Presence of psoas sign	High (+1)	0	0	0.83	1	0.66
??5E. WBC?=?12,000	Middle (0)	0	0.12	0.25	1	0.50
6. Congestive heart failure
??6A. Presence of third heart sound	Very high (+2)	0	0.11	0	0.55	1
??6B. Absence of cardiomegaly in CXR	Low (?1)	0.42	1	0.42	0.14	0.14
??6C. Presence of interstitial edema in CXR	Very high (+2)	0	0	0.12	0.75	1
??6D. Presence of atrial fibrillation in EKG	High (+1)	0	0.16	1	0.83	0.50
??6E. Presence of lateral EKG changes	High (+1)	0	0	1	0.66	0
SUM				21.73 (out of 30)

If a test-taker answered our script concordance test based on evidence-based likelihood ratios (LRs), and his/her answers were scored based on the judgments of this study's reference panel, the test-taker would get a score of 21.73 out of 30. For the above calculations, the numbers representing the judgments were categorized as very low, low, middle, high, and very high, similar to five-point Likert scales, using cutoff points of ?6, ?2, 2, and 6. Subsequently, the score of each category was calculated based on the judgments. Then, for LR of each finding, we identified its category (`categorized LR column) and its corresponding score (italicized number). Finally, we added up all italicized numbers. The calculation of the scores based on the judgments of the reference panel is explained elsewhere [16].

Back to article page

ISSN: 1865-1380

Contact us

General enquiries: journalsubmissions@springernature.com