Cognitive error in an academic emergency department
Benjamin H. Schnapp*, Jean E. Sun, Jeremy L. Kim, Reuben J. Strayer and Kaushal H. Shah
Received April 9, 2018; accepted June 18, 2018
Background: Medical error is a leading cause of death nationwide. While systems issues have been closely investigated as a contributor to error, little is known about the cognitive factors that contribute to diagnostic error in an emergency department (ED) environment.
Methods: Eight months of patient revisits within 72 h where patients were admitted on their second visit were examined. Fifty-two cases of confirmed error were identified and classified using a modified version of the Australian Patient Safety Foundation classification system for medical errors by a group of trained physicians.
Results: Faulty information processing was the most frequently identified category of error (45% of cases), followed by faulty verification (31%). Faulty knowledge (6%) and faulty information gathering (18%) occurred relatively infrequently. “Misjudging the salience of a finding” and “premature closure” were the individual errors that occurred most frequently (13%).
Conclusions: Despite the complex nature of diagnostic reasoning, cognitive errors of information processing appear to occur at higher rates than other errors, and in a similar pattern to an internal medicine service despite a different clinical environment. Further research is needed to elucidate why these errors occur and how to mitigate them.
Keywords: classification, cognitive, diagnosis, diagnostic error, emergency, error, information processing.
Medical errors were estimated in 2016 to cause 250,000 deaths per year in the US ; the rate of diagnostic error overall has been estimated at 10-15% , although reported rates are variable depending on the method used to estimate incidence . While a great deal of effort has been expended on the improvement of systems issues such as improving patient handoffs  and medication reconciliation , cognitive errors have proved more difficult to study and mitigate .
Previous work by Graber et al. showed that the majority of the cognitive errors identified among high-risk cases on an internal medicine service were errors of faulty synthesis (information processing and verification) and that relatively few errors were attributable to poor knowledge or data gathering . Though the diagnostic errors in an emergency department (ED) are likely to occur at around a similar rate as diagnostic errors in the rest of the health system , ED errors may be of a different character because of the fast pace  and frequent interruptions  that can lead to incomplete or unreliable gathering or transfer of information . One study identified an incomplete history and physical examination as a factor in 42% of ED malpractice claims .
The first step to reducing error in the ED is to understand the factors that contribute most frequently to medical error. One commonly examined source of potential medical errors are patient revisits, as a patient’s return to the ED within a short time frame suggests that an error may have occurred . The aim of this study was to determine the cognitive factors that contribute to error most frequently in the ED by examining revisits within 72 h.
Subjects and methods
The study was conducted at an adult ED in an urban academic public hospital in New York City with approximately 156,000 annual visits.
A revisit was defined for our study as two visits to the same ED within a 72 h window. Cases were excluded if the patient was under 18 or over the age of 89, the second visit was planned during the first visit (e.g. wound check follow-up), the patient was admitted on the first visit or if the patient was discharged on both visits. While it is possible that some of the excluded cases also contained instances of error, we felt that the most serious and important cases of error were likely to be found when the patient was discharged on the first visit and admitted on the second visit, as this suggests that the original disposition may have been made in error.
Cognitive error was defined as a delayed, incorrect or missed diagnosis due to an error in physician judgment as determined by information obtained later, a modification of the definition used by Graber et al. . Errors were classified using a modified version of the Australian Patient Safety Foundation classification system and abbreviated from the categorization described by Graber et al.  to focus only on the cognitive factors contributing to error, not on systems issues. This classification system describes four broad categories of factors which can lead medical decision-making to break down: faulty knowledge, faulty data gathering, faulty information processing and faulty verification. It also defines multiple subcategories for each of the factors. See Table 1 for the classification system used with descriptions of each factor.
As part of routine ED quality improvement measures, 72 h revisit case summaries are identified at our institution, with pertinent data included from the history, examination, laboratory results and imaging tests. If it remained unclear whether an error had occurred after review of the case summary, or further information was required to identify the type of error, the full medical record was examined and the original providers were contacted for further clarification when possible. Eight months of 72 h revisits from 2013 to 2014 were included in our analysis. Within the 8-month study period, 271 cases met the inclusion criteria and were examined.
A team of physicians, including two physicians with cognitive science experience, was trained on the modified Australian Patient Safety Foundation classification and its use over several hours prior to the start of the study. All team members assigned classifications to several example cases to ensure that all participants shared a similar understanding on how to use the classification system.
Identifying cognitive errors took place in two steps. In the first step, each case that met the inclusion criteria was examined by two inde-pendent reviewers to determine whether, in their clinical judgment, the case could have contained cognitive error as defined by our study criteria. Cases where the two reviewers disagreed were adjudicated by the full group. In the second step, all cases that were identified as possibly containing a cognitive error were examined in more detail by at least three members of the study team, who analyzed the details of each case and reached a consensus as a group as to whether an error occurred and if so, what categories of error occurred.
Data recording and analysis
Cases were categorized by error class and body system, as well as screened for known high-risk conditions that could predispose patients to medical error or poor outcomes, such as substance abuse or psychiatric illness. Cases were recorded in a secure database, accessible only to the researchers. Descriptive statistics were calculated using Numbers (Apple, Cupertino, CA, USA).
The study was approved by the Institutional Review Board at the participating hospital.
A total of 271 cases of revisits within 72 h met our inclusion criteria. A total of 131 (48%) cases were determined by both reviewers to contain no potential cognitive error; 140 (52%) were identified by at least one reviewer as containing a potential cognitive error and flagged for further review by the study team. Of the 140 cases where potential issues were identified, 52 cases (19%) were verified to represent instances of cognitive error as determined by group consensus (see Figure 1).
Within the 52 cases identified as containing a cognitive error, there were 120 cognitive factors identified as contributing to error (each case could be assigned more than one factor which contributed to the error). Among the four general categories of cognitive error, faulty information processing was the most common, representing 45% [95% confidence interval (CI) 36-54%] of the identified errors. Faulty verification was the next most frequently identified, representing 31% (95% CI 23-40%) of factors. Finally, factors of faulty data gathering and faulty knowledge occurred least commonly, representing 18% (95% CI 12-26%) and 6% (95% CI 3-12%) of errors, respectively. Table 2 represents the relative frequency of each of the four types of cognitive factors determined to contribute to error. The mean number of factors identified per case containing error was 2.3.
Of the 25 described specific cognitive factors which can contribute to error, the most commonly occurring were “misjudging the salience of a finding” and “premature closure”, which each represented 13% (95% CI 8-20%) of the identified types of error. In cases with confirmed error, the most common body system involved was hepatobiliary (e.g. missed cholecystitis), representing 19% of the cases, followed by pulmonary (e.g. pneumonia too sick to discharge), representing 13% of the cases; see Table 3 for the percentage of errors by body system. We also noted that a significant portion of the cases involved patients with a history of substance abuse (12% of cases) and patients with psychiatric illness and congestive heart failure (each 8% of cases); Table 4 shows the incidence of errors associated with risk factors identified in our study.
To the best of our knowledge, this is the first study to categorize and quantify the cognitive factors that contribute to errors in the ED. Our study shows that, similar to an inpatient internal medicine environment, the cognitive factors that contribute to error most often in 72 h revisits are faulty information processing and faulty verification of data.
Previously published literature from other medical disciplines shows mixed results regarding what types of cognitive errors are most common in diagnosis. An analysis of admitted patients found that errors of knowledge application were most frequent , similar to the results of this study. However, a study of primary care visits found a large number of errors involving history taking and examination of the patient  and a study of intensive care unit patients found that “failure to carry out the intended treatment” was the most frequent type of error . While some of these differences may be accounted for by the differences in clinical environments, the lack of an effective, comprehensive and agreed upon methodology for measuring cognitive error is likely a significant contributing factor to these discrepancies . The presence of trainees in an academic ED theoretically increases the likelihood that inadequate knowledge is contributing to error ; over half of the residents in one survey acknowledged that insufficient knowledge may have contributed to a recent error in their care . Despite trainees’ limited clinical experience, errors of data gathering and knowledge do not appear in our study to occur more frequently in the ED setting.
Past literature is also mixed on the question of what types of cardinal presentations are most likely to be prone to error and revisits. One study in Taiwan found that patients with abdominal symptoms were at highest risk to return , similar to the high prevalence of errors in return visits categorized as ‘hepatobiliary’ or ‘gastrointestinal’ in our study. A Dutch study found a similarly high rate of returns for abdominal pathology, but also a high rate of returns for patients with urinary symptoms , which our study did not find. Upper respiratory tract infections were the most common reason for a revisit in a Hong Kong ED study , and a study of physician-recalled errors found that the most frequently reported errors involved pulmonary problems . Our study did not calculate the distribution of complaints by system for all visits to the ED at our study site, but the rates of error found in our study for hepatobiliary complaints (19% of cases) and gastrointestinal complaints (10%) were significantly higher than the rate of ED visits nationally for all gastrointestinal complaints (6% of all visits) , suggesting that intra-abdominal complaints may be more vulnerable to diagnostic and cognitive error. While the poor localization of visceral pain lends biologic plausibility to this idea, cultural and health care delivery system differences may also play a role and more study is needed in this area.
Our study concurred with previous findings that patient factors can be a significant contributor to diag¬nostic and cognitive error . Specifically, patients with psychiatric disease , substance abuse  and congestive heart failure  are known to have a high frequency of ED revisits; diagnosis and ideal management may be difficult in these populations. In our study, errors in each of these at-risk populations appeared more frequently than the rate of ED visits nationally for these issues. Substance abuse was the source of 12% of errors identified but represented only 7% of ED visits nationally. Psychiatric disease was tied to 8% of errors in our study but represented only 4% of ED visits nationally. Congestive heart failure was related to 8% of errors but was the source of 3% of national ED visits, and patients with human immunodeficiency virus (HIV) were tied to 6% of errors, while making up only 0.4% of ED visits . However, the overall number of cases analyzed in this study is small, which may limit the generalizability of these conclusions.
A variety of techniques have been explored to attempt to reduce errors of information processing. Classically, two types of diagnostic thinking have been described, with System 1, or rapid pattern recognition, being vulnerable to error and System 2, or effortful logical reasoning, being a safety net which can catch errors . Consequently, some efforts at error reduction have focused on cognitive forcing strategies, designed to reorient clinicians to alternative diagnostic possibilities at pre-specified points in the workup (such as pop-up reminders in the electronic medical record). Unfortunately, while these interventions succeed in making physicians more deliberate, they may not reduce cognitive error [29-31], suggesting that System 2 may not be as effective as suggested at eliminating information synthesis problems. Additionally, experienced physicians may be able to effectively use System 1 thinking to make their workflow more efficient by rapidly categorizing patients without making errors . While some research shows that only additional clinical knowledge and experience reduces cognitive error , other evidence has suggested promise for guided reflection and cognitive forcing strategies [34, 35]. Ultimately, the effectiveness of interventions to reduce error may be context dependent; checklists have been shown to be effective in some contexts but not others . More research is still needed to determine what strategies may be most effective for reducing cognitive errors in the clinical environment.
There are several potential sources of bias for this study. Though many of our errors were attributed to either pre-mature closure or misjudging the salience of a finding, managing patients is a complex and dynamic interaction between knowledge and interpretation and it is possible that knowledge issues may affect information processing and verification. Additionally, while Croskerry and others have described many of the types of cognitive errors that occur  as well as the clinical situations in which they are most likely to be found, in practice, it can be difficult to categorize real errors that occur in the clinical setting as descriptions overlap and the error types lack strict criteria; we did not categorize distal causes of error in this analysis . Our definition of a revisit may have also systematically affected our results, as a missed important finding such fracture would be unlikely to have been admitted on their second visit.
The results of this study are also based on a retrospective review, which may have overestimated the incidence of errors, as the reviewers have the benefit of hindsight. In an effort to reduce hindsight bias and give maximum deference to providers, we attempted to include only cases of clear cognitive error. Our study is also limited by its single center nature, as patients may have returned and been admitted at other hospitals, as well as our documentation, as what was documented in the original medical record and the quality review case summaries may not fully reflect potential issues. As there is no gold standard for determining whether an error occurred or what cognitive factors may have contributed, we relied on group consensus, which can be subject to bias. Finally, as the classification system we used for cognitive factors contained more factors categorized as information processing and verification factors than those categorized as knowledge or data gathering factors, this may have contributed to the increased frequency that these factors were seen in our study.
Errors of information processing and verification were the most commonly identified errors in a study of patients with 72 h revisits to an academic ED. Patients with abdominal complaints were at highest risk for cognitive errors in diagnosis. Further standardization around describing and quantifying cognitive errors is needed to further elucidate how these errors impact care in the ED environment.
Acknowledgments: We thank Dr. Candice Cruz, Dr. Court¬ney Cassella, Dr. Angela Hua, Dr. Zara Mathew, Dr. Clark Owyang, Dr. Bradley Shy and Dr. Sumintra Wood for their invaluable assistance with completing this project.
Author contributions: All the authors have accepted responsibility for the entire content of this submitted manuscript and approved submission.
Research funding: None declared.
Employment or leadership: None declared.
Honorarium: None declared.
Competing interests: The funding organization(s) played no role in the study design; in the collection, analysis, and interpretation of data; in the writing of the report; or in the decision to submit the report for publication.
1. Makary MA, Daniel M. Medical error-the third leading cause of death in the US. Br Med J 2016;353:i2139.
2. Berner ES, Graber ML. Overconfidence as a cause of diagnostic error in medicine. Am J Med 2008;121:S2-23.
3. Graber ML. The incidence of diagnostic error in medicine. BMJ Qual Saf 2013;22(Suppl 2):ii21-7.
4. Riesenberg LA, Leitzsch J, Massucci JL, Jaeger J, Rosenfeld JC, Patow C, et al. Residents’ and attending physicians’ handoffs: a systematic review of the literature. Acad Med 2009;84:1775-87.
5. Mueller SK, Sponsler KC, Kripalani S, Schnipper JL. Hospital-based medication reconciliation practices: a systematic review. Arch Intern Med 2012;172:1057-69.
6. Zwaan L, Monteiro S, Sherbino J, Ilgen J, Howey B, Norman G. Is bias in the eye of the beholder? a vignette study to assess rec¬ognition of cognitive biases in clinical case workups. BMJ Qual Saf 2017;26:104-10.
7. Graber ML, Franklin N, Gordon R. Diagnostic error in internal medicine. Arch Intern Med 2005;165:1493-9.
8. Chellis M, Olson J, Augustine J, Hamilton G. Evaluation of missed diagnoses for patients admitted from the emergency department. Acad Emerg Med 2001;8:125-30.
9. Hamden K, Jeanmonod D, Gualtieri D, Jeanmonod R.
Comparison of resident and mid-level provider productivity in a high-acuity emergency department setting. Emerg Med J 2014;31:216-9.
10. Berg LM, Kallberg AS, Goransson KE, Ostergren J, Florin J, Ehrenberg A. Interruptions in emergency department work: an observational and interview study. BMJ Qual Saf 2013;22:656-63.
11. Stiell A, Forster AJ, Stiell IG, van Walraven C. Prevalence of information gaps in the emergency department and the effect on patient outcomes. Can Med Assoc J 2003;169:1023-8.
12. Kachalia A, Gandhi TK, Puopolo AL, Yoon C, Thomas EJ, Griffey R, et al. Missed and delayed diagnoses in the emergency department: a study of closed malpractice claims from 4 liability insurers. Ann Emerg Med 2007;49:196-205.
13. Calder L, Pozgay A, Riff S, Rothwell D, Youngson E, Mojaverian N, et al. Adverse events in patients with return emergency depart¬ment visits. BMJ Qual Saf 2015;24:142-8.
14. Zwaan L, de Bruijne M, Wagner C, Thijs A, Smits M, van der Wal G, et al. Patient record review of the incidence, consequences, and causes of diagnostic adverse events. Arch Intern Med 2010;170:1015-21.
15. Singh H, Giardina TD, Meyer AN, Forjuoh SN, Reis MD, Thomas EJ. Types and origins of diagnostic errors in primary care set-tings. JAMA Intern Med 2013;173:418-25.
16. Rothschild JM, Landrigan CP, Cronin JW, Kaushal R, Lockley SW, Burdick E, et al. The critical care safety study: the incidence and nature of adverse events and serious medical errors in intensive care. Crit Care Med 2005;33:1694-700.
17. Young JQ, Ranji SR, Wachter RM, Lee CM, Niehaus B, Auerbach AD. “July effect”: impact of the academic year-end changeover on patient outcomes: a systematic review. Ann Intern Med 2011;155:309-15.
18. Hobgood C, Hevia A, Tamayo-Sarver JH, Weiner B, Riviello R.
The influence of the causes and contexts of medical errors on emergency medicine residents’ responses to their errors: an exploration. Acad Med 2005;80:758-64.
19. Wu CL, Wang FT, Chiang YC, Chiu YF, Lin TG, Fu LF, et al. Unplanned emergency department revisits within 72 h to a secondary teaching referral hospital in Taiwan. J Emerg Med 2010;38:512-7.
20. van der Linden MC, Lindeboom R, de Haan R, van der Linden N, de Deckere ER, Lucas C, et al. Unscheduled return visits to a Dutch inner-city emergency department. Int J Emerg Med 2014;7:23.
21. Ng C, Chung C. An analysis of unscheduled return visits to the accident and emergency department of a general public hospital. Hong Kong J Emerg Med 2003;10:153-61.
22. Schiff GD, Hasan O, Kim S, Abrams R, Cosby K, Lamber BL, et al. Diagnostic error in medicine: analysis of 583 physician-reported errors. Arch Intern Med 2009;169:1881-7.
23. National Hospital Ambulatory Medical Care Survey: 2015 Emergency Department Summary Tables. Available at: https://www. cdc.gov/nchs/data/nhamcs/web_tables/2015_ed_web_tables. pdf. Accessed 14 Jun 2018.
24. Gandhi TK, Kachalia A, Thomas EJ, Puopolo AL, Yoon C, Brennan TA, et al. Missed and delayed diagnoses in the ambulatory setting: a study of closed malpractice claims. Ann Intern Med 2006;145:488-96.
25. van Nieuwenhuizen A, Henderson C, Kassam A, Graham T, Murray J, Howard LM, et al. Emergency department staff views and experiences on diagnostic overshadowing related to people with mental illness. Epidemiol Psychiatr Sci 2013;22:255-62.
26. Smith MW, Stocks C, Santora PB. Hospital readmission rates and emergency department visits for mental health and substance abuse conditions. Community Ment Health J 2015;51:190-7.
27. Chun S, Tu JV, Wijeysundera HC, Austin PC, Wang X, Levy D, et al. Lifetime analysis of hospitalizations and survival of patients newly admitted with heart failure. Circ Heart Fail 2012;5:414-21.
28. Stanovich KE, West RF. Individual differences in reasoning: implications for the rationality debate? In: Gilovich T, Griffin D, Kahneman D, editors. Heuristics and biases: the psychology of intuitive judgment. New York, NY: Cambridge University Press, 2002:421-40, Chapter xvi, 857 Pages.
29. Sherbino J, Kulasegaram K, Howey E, Norman G. Ineffectiveness of cognitive forcing strategies to reduce biases in diagnostic reasoning: a controlled trial. Can J Emerg Med 2014;16:34-40.
30. Monteiro SD, Sherbino J, Patel A, Mazzetti I, Norman GR, Howey E. Reflecting on diagnostic errors: taking a second look is not enough. J Gen Intern Med 2015;30:1270-4.
31. Sherbino J, Yip S, Dore KL, Siu E, Norman GR. The effectiveness of cognitive forcing strategies to decrease diagnostic error: an exploratory study. Teach Learn Med 2011;23:78-84.
32. Kovacs G, Croskerry P. Clinical decision making: an emergency medicine perspective. Acad Emerg Med 1999;6:947-52.
33. Monteiro SD, Sherbino JD, Ilgen JS, Dore KL, Wood TJ, Young ME, et al. Disrupting diagnostic reasoning: do interruptions, instructions, and experience affect the diagnostic accuracy and response time of residents and emergency physicians? Acad Med 2015;90:511-7.
34. Graber ML, Kissam S, Payne VL, Meyer AN, Sorensen A, Lenfestey N, et al. Cognitive interventions to reduce diagnostic error: a narrative review. BMJ Qual Saf 2012;21:535-57.
35. Lambe KA, O’Reilly G, Kelly BD, Curristan S. Dual-process cognitive interventions to enhance diagnostic reasoning: a systematic review. BMJ Qual Saf 2016;25:808-20.
36. Shimizu T, Matsumoto K, Tokuda Y. Effects of the use of differential diagnosis checklist and general debiasing checklist on diagnostic performance in comparison to intuitive diagnosis. Med Teach 2013;35:e1218-29.
37. Croskerry P. The importance of cognitive errors in diagnosis and strategies to minimize them. Acad Med 2003;78:775-80.
38. Croskerry P. Our better angels and black boxes. Emerg Med J 2016;33:242-4.