Skip to main content

Thank you for visiting nature.com. You are using a browser version with limited support for CSS. To obtain the best experience, we recommend you use a more up to date browser (or turn off compatibility mode in Internet Explorer). In the meantime, to ensure continued support, we are displaying the site without styles and JavaScript.

  • Article
  • Published:

Factors driving provider adoption of the TREWS machine learning-based early warning system and its effects on sepsis treatment timing

Nature Medicine volume 28, pages 1447–1454 (2022)Cite this article

Subjects

Abstract

Machine learning-based clinical decision support tools for sepsis create opportunities to identify at-risk patients and initiate treatments at early time points, which is critical for improving sepsis outcomes. In view of the increasing use of such systems, better understanding of how they are adopted and used by healthcare providers is needed. Here, we analyzed provider interactions with a sepsis early detection tool (Targeted Real-time Early Warning System), which was deployed at five hospitals over a 2-year period. Among 9,805 retrospectively identified sepsis cases, the early detection tool achieved high sensitivity (82% of sepsis cases were identified) and a high rate of adoption: 89% of all alerts by the system were evaluated by a physician or advanced practice provider and 38% of evaluated alerts were confirmed by a provider. Adjusting for patient presentation and severity, patients with sepsis whose alert was confirmed by a provider within 3 h had a 1.85-h (95% CI 1.66–2.00) reduction in median time to first antibiotic order compared to patients with sepsis whose alert was either dismissed, confirmed more than 3 h after the alert or never addressed in the system. Finally, we found that emergency department providers and providers who had previous interactions with an alert were more likely to interact with alerts, as well as to confirm alerts on retrospectively identified patients with sepsis. Beyond efforts to improve the performance of early warning systems, efforts to improve adoption are essential to their clinical impact and should focus on understanding providers’ knowledge of, experience with and attitudes toward such systems.

This is a preview of subscription content, access via your institution

Access options

Buy this article

  • Purchase on Springer Link
  • Instant access to full article PDF
Buy now

Prices may be subject to local taxes which are calculated during checkout

Fig. 1: Included study population by study question.

Similar content being viewed by others

Data availability

The data are not publicly available because they are from electronic health records approved for limited use by Johns Hopkins University investigators. Making the data publicly available without additional consent, ethical or legal approval might compromise patients’ privacy and the original ethical approval. To perform additional analyses using these data, researchers should contact A.W.W. or S.S. to apply for an institutional review board-approved research collaboration and obtain an appropriate data-use agreement.

Code availability

The TREWS early warning system described in this study is available from Bayesian Health. The underlying source code is proprietary intellectual property and is not available. Code for the primary statistical analyses can be found at https://github.com/royadams/henry_et_al_2022_code.

References

  1. Bates, D. W., Saria, S., Ohno-Machado, L., Shah, A. & Escobar, G. Big data in health care: using analytics to identify and manage high-risk and high-cost patients. Health Aff. 33, 1123–1131 (2014).

    Article  Google Scholar 

  2. Topol, E. J. High-performance medicine: the convergence of human and artificial intelligence. Nat. Med. 25, 44–56 (2019).

    Article  CAS  PubMed  Google Scholar 

  3. Liang, H. et al. Evaluation and accurate diagnoses of pediatric diseases using artificial intelligence. Nat. Med. 24, 443–448 (2019).

    Google Scholar 

  4. Bauer, M. et al. Automation to optimise physician treatment of individual patients: examples in psychiatry. Lancet Psychiat. 6, 338–349 (2019).

    Article  Google Scholar 

  5. Beam, A. L. & Kohane, I. S. Big data and machine learning in health care. JAMA 319, 1317–1318 (2019).

    Article  Google Scholar 

  6. Castaneda, C. et al. Clinical decision support systems for improving diagnostic accuracy and achieving precision medicine. J. Clin. Bioinforma. https://doi.org/10.1186/s13336-015-0019-3 (2015).

  7. Henry, K. E., Hager, D. N., Pronovost, P. J. & Saria, S. A targeted real-time early warning score (TREWScore) for septic shock. Sci. Transl. Med. 7, 299ra122 (2015).

    Article  PubMed  Google Scholar 

  8. Churpek, M. M., Adhikari, R. & Edelson, D. P. The value of vital sign trends for detecting clinical deterioration on the wards. Resuscitation 102, 1–5 (2016).

    Article  PubMed  PubMed Central  Google Scholar 

  9. Amland, R. C. & Sutariya, B. B. Quick sequential (sepsis-related) organ failure assessment (qSOFA) and St John sepsis surveillance agent to detect patients at risk of sepsis: an observational cohort study. Am. J. Med. Qual. 33, 50–57 (2018).

    Article  PubMed  Google Scholar 

  10. Tomašev, N. et al. A clinically applicable approach to continuous prediction of future acute kidney injury. Nature 572, 116–119 (2019).

    Article  PubMed  PubMed Central  CAS  Google Scholar 

  11. Sittig, D. F. et al. Grand challenges in clinical decision support. J. Biomed. Inform. 41, 387–392 (2008).

    Article  PubMed  Google Scholar 

  12. Abramoff, M. D., Tobey, D. & Char, D. S. Lessons learned about autonomous AI: finding a safe, efficacious, and ethical path through the development process. Am. J. Ophthalmol. 214, 134–142 (2020).

    Article  PubMed  Google Scholar 

  13. Khairat, S. et al. Reasons for physicians not adopting clinical decision support systems: critical analysis. JMIR Med. Inform. https://doi.org/10.2196/medinform.8912 (2018).

  14. Kelly, C. J., Karthikesalingam, A., Suleyman, M., Corrado, G. & King, D. Key challenges for delivering clinical impact with artificial intelligence. BMC Med. 17, 1–9 (2019).

    Article  CAS  Google Scholar 

  15. Celi, L. A., Fine, B. & Stone, D. J. An awakening in medicine: the partnership of humanity and intelligent machines. Lancet Digit. Health 1, e255–e257 (2019).

    Article  PubMed  PubMed Central  Google Scholar 

  16. Mertz, L. From Annoying to appreciated: turning clinical decision support systems into a medical professional’s best friend. IEEE Pulse 6, 4–9 (2015).

    Article  PubMed  Google Scholar 

  17. Greenes, R. A. et al. Clinical decision support models and frameworks: seeking to address research issues underlying implementation successes and failures. J. Biomed. Inform. 78, 134–143 (2018).

    Article  PubMed  Google Scholar 

  18. Wright, A. et al. Analysis of clinical decision support system malfunctions: a case series and survey. J. Am. Med. Inform. Assoc. 23, 1068–1076 (2016).

    Article  PubMed  PubMed Central  Google Scholar 

  19. Ruppel, H. & Liu, V. To catch a killer: electronic sepsis alert tools reaching a fever pitch? BMJ Qual. Saf. https://doi.org/10.1136/bmjqs-2019-009463 (2019).

  20. Downing, N. L. et al. Electronic health record-based clinical decision support alert for severe sepsis: a randomised evaluation. BMJ Qual. Saf. 28, 762–768 (2019).

    Article  PubMed  PubMed Central  Google Scholar 

  21. Giannini, H. M. et al. A machine learning algorithm to predict severe sepsis and septic shock. Crit. Care Med. https://doi.org/10.1097/ccm.0000000000003891 (2019).

  22. Moja, L. et al. Effectiveness of a hospital-based computerized decision support system on clinician recommendations and patient outcomes: a randomized clinical trial. JAMA Netw. Open 2, 1–16 (2019).

    Article  Google Scholar 

  23. Khan, S. et al. Improving provider adoption with adaptive clinical decision support surveillance: an observational study. JMIR Hum. Factors 6, 1–10 (2019).

    Article  Google Scholar 

  24. Mann, D. et al. Impact of clinical decision support on antibiotic prescribing for acute respiratory infections: a cluster randomized implementation trial. J. Gen. Intern. Med. 35, 788–795 (2020).

    Article  PubMed  PubMed Central  Google Scholar 

  25. Kwan, J. L. et al. Computerised clinical decision support systems and absolute improvements in care: meta-analysis of controlled clinical trials. BMJ https://doi.org/10.1136/bmj.m3216 (2020).

  26. Escobar, G. J. et al. Automated identification of adults at risk for in-hospital clinical deterioration. N. Engl. J. Med. 383, 1951–1960 (2020).

    Article  PubMed  PubMed Central  Google Scholar 

  27. Schaefer, K. E., Chen, J. Y. C., Szalma, J. L. & Hancock, P. A. A meta-analysis of factors influencing the development of trust in automation: implications for understanding autonomy in future systems. Hum. Factors 58, 377–400 (2016).

    Article  PubMed  Google Scholar 

  28. Cai, C. J. et al. Human-centered tools for coping with imperfect algorithms during medical decision-making. Proc of the CHI Conference on Human Factors in Computing Systems, pp. 1–14 (2019).

  29. Gaube, S. et al. Do as AI say: susceptibility in deployment of clinical decision-aids. NPJ Digit. Med. https://doi.org/10.1038/s41746-021-00385-9 (2021).

  30. Jacobs, M. et al. Designing AI for Trust and Collaboration in Time-Constrained Medical Decisions: A Sociotechnical Lens. Proc of the Conference on Human Factors in Computing Systems (CHI), pp. 1–14 (2021).

  31. Shortliffe, E. H. & Sepúlveda, M. J. Clinical decision support in the era of artificial intelligence. JAMA 10025, 9–10 (2018).

    Google Scholar 

  32. Kumar, A. et al. Duration of hypotension before initiation of effective antimicrobial therapy is the critical determinant of survival in human septic shock. Crit. Care Med. 34, 1589–1596 (2006).

    Article  PubMed  Google Scholar 

  33. Seymour, C. W. et al. Time to treatment and mortality during mandated emergency care for sepsis. N. Engl. J. Med. 376, 2235–2244 (2017).

    Article  PubMed  PubMed Central  Google Scholar 

  34. Liu, V. X. et al. The timing of early antibiotics and hospital mortality in sepsis. Am. J. Respir. Crit. Care Med. 196, 856–863 (2017).

    Article  PubMed  PubMed Central  Google Scholar 

  35. Adams, R. et al. Prospective, multi-site study of patient outcomes after implementation of the TREWS machine learning-based early warning system for sepsis. Nat. Med. https://doi.org/10.1038/s41591-022-01894-0 (2022).

  36. Henry, K. E., Hager, D. N., Osborn, T. M., Wu, A. W. & Saria, S. Comparison of automated sepsis identification methods and electronic health record–based sepsis phenotyping: improving case identification accuracy by accounting for confounding comorbid conditions. Crit. Care Explor. https://doi.org/10.1097/cce.0000000000000053 (2019).

  37. Saria, S. & Henry, K. E. Too many definitions of sepsis: can machine learning leverage the electronic health record to increase accuracy and bring consensus? Crit. Care Med. https://doi.org/10.1097/CCM.0000000000004144 (2020).

  38. Singer, M. et al. The third international consensus definitions for sepsis and septic shock (sepsis-3). JAMA 315, 801–810 (2016).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  39. Levy, M. M., Evans, L. E. & Rhodes, A. The surviving sepsis campaign bundle: 2018 update. Crit. Care Med. 46, 997–1000 (2018).

    Article  PubMed  Google Scholar 

  40. Hospital Inpatient - Specifications Manuals - Sepsis Resources. QualityNet https://qualitynet.cms.gov/inpatient/specifications-manuals/sepsis-resources. (2021)

  41. Rhee, C. et al. Prevalence, underlying causes, and preventability of sepsis-associated mortality in US acute care hospitals. JAMA Netw. Open 2, e187571 (2019).

    Article  PubMed  PubMed Central  Google Scholar 

  42. Adams, R. et al. 1405: assessing clinical use and performance of a machine learning sepsis alert for sex and racial bias. Crit. Care Med. 50, 705 (2022).

    Article  Google Scholar 

  43. Evans, I. V. R. et al. Association between the New York sepsis care mandate and in-hospital mortality for pediatric sepsis. JAMA https://doi.org/10.1001/jama.2018.9071 (2018).

  44. Guy, J. S., Jackson, E. & Perlin, J. B. Accelerating the clinical workflow using the sepsis prediction and optimization of therapy (SPOT) tool for real-time clinical monitoring. NEJM Catal. Innov. Care Deliv. https://doi.org/10.1056/CAT.19.1036 (2020).

  45. Perlin, J. B. et al. 2019 John M. Eisenberg patient safety and quality awards: SPOTting sepsis to save lives: a nationwide computer algorithm for early detection of sepsis: innovation in patient safety and quality at the national level (Eisenberg award). Jt. Comm. J. Qual. Patient Saf. 46, 381–391 (2020).

    PubMed  Google Scholar 

  46. McCoy, A. & Das, R. Reducing patient mortality, length of stay and readmissions through machine learning-based sepsis prediction in the emergency department, intensive care unit and hospital floor units. BMJ Open Qual. https://doi.org/10.1136/bmjoq-2017-000158 (2017).

  47. Shimabukuro, D. W., Barton, C. W., Feldman, M. D., Mataraso, S. J. & Das, R. Effect of a machine learning-based severe sepsis prediction algorithm on patient survival and hospital length of stay: a randomised clinical trial. BMJ Open Respir. Res. https://doi.org/10.1136/bmjresp-2017-000234 (2017).

  48. Wong, A. et al. External validation of a widely implemented proprietary sepsis prediction model in hospitalized patients. JAMA Intern. Med. 48109, 1–6 (2021).

    Google Scholar 

  49. Sendak, M. P. et al. Real-world integration of a sepsis deep learning technology into routine clinical care: implementation study. JMIR Med. Inform. 8, e15182 (2020).

    Article  PubMed  PubMed Central  Google Scholar 

  50. Khairat, S., Marc, D., Crosby, W. & Al Sanousi, A. Reasons for physicians not adopting clinical decision support systems: critical analysis. JMIR Med. Informatics https://doi.org/10.2196/medinform.8912 (2018).

  51. Kitzmiller, R. R. et al. Diffusing an innovation: clinician perceptions of continuous predictive analytics monitoring in intensive care. Appl. Clin. Inform. https://doi.org/10.1055/s-0039-1688478 (2019).

  52. Abramoff, M. D., Lavin, P. T., Birch, M., Shah, N. & Folk, J. C. Pivotal trial of an autonomous AI-based diagnostic system for detection of diabetic retinopathy in primary care offices. NPJ Digit. Med. https://doi.org/10.1038/s41746-018-0040-6 (2018).

  53. Ginestra, J. C. et al. Clinician perception of a machine learning–based early warning system designed to predict severe sepsis and septic shock. Crit. Care Med. https://doi.org/10.1097/ccm.0000000000003803 (2019).

  54. Topiwala, R., Patel, K., Twigg, J., Rhule, J. & Meisenberg, B. Retrospective observational study of the clinical performance characteristics of a machine learning approach to early sepsis identification. Crit. Care Explor. https://doi.org/10.1097/cce.0000000000000046 (2019).

  55. Carspecken, C. W., Sharek, P. J., Longhurst, C. & Pageler, N. M. A clinical case of electronic health record drug alert fatigue: consequences for patient outcome. Pediatrics 131, e1970–e1973 (2013).

    Article  PubMed  Google Scholar 

  56. Bansal, G. et al. Beyond accuracy: the role of mental modelsin human–AI team performance. Proc. AAAI Conference on Human Computation and Crowdsourcing 7, 2–11. (2019).

  57. Jacobs, M. et al. How machine learning recommendations influence clinician treatment selections: the example of the antidepressant selection. Transl. Psychiatry https://doi.org/10.1038/s41398-021-01224-x (2021).

  58. Lee, J. D. & See, K. A. Trust in automation: designing for appropriate reliance. Hum. Factors 46, 50–80 (2004).

    Article  PubMed  Google Scholar 

  59. Hoff, K. A. & Bashir, M. Trust in automation: integrating empirical evidence on factors that infuence trust. Hum. Factors 57, 407–434 (2015).

    Article  PubMed  Google Scholar 

  60. Murphy, E. V. Clinical decision support: effectiveness in improving quality processes and clinical outcomes and factors that may influence success. Yale J. Biol. Med. 87, 187–197 (2014).

    PubMed  PubMed Central  Google Scholar 

  61. Mann, D. et al. Adaptive design of a clinical decision support tool: what the impact on utilization rates means for future CDS research. Digit. Heal. 5, 1–12 (2019).

    Google Scholar 

  62. Henry, K. E. et al. Human-machine teaming is key to AI adoption: clinicians’ experiences with a deployed machine learning system. NPJ Digit. Med. https://doi.org/10.1038/s41746-022-00597-7 (2022).

  63. Rhee, C. et al. Infectious Diseases Society of America position paper: recommended revisions to the national severe sepsis and septic shock early management bundle (SEP-1) sepsis quality measure. Clin. Infect. Dis. 72, 541–552 (2021).

    Article  PubMed  Google Scholar 

  64. Rhee, C. et al. Diagnosing sepsis is subjective and highly variable: a survey of intensivists using case vignettes. Crit. Care 20, 1–8 (2016).

    Article  Google Scholar 

  65. Rhee, C., Dantes, R. B., Epstein, L. & Klompas, M. Using objective clinical data to track progress on preventing and treating sepsis: CDC’s new ‘Adult Sepsis Event’ surveillance strategy. BMJ Qual. Saf. https://doi.org/10.1002/sca.20201 (2018).

  66. Rhee, C. et al. Sepsis surveillance using adult sepsis events simplified eSOFA criteria versus sepsis-3 sequential organ failure assessment criteria∗. Crit. Care Med. 47, 307–314 (2019).

    Article  PubMed  PubMed Central  Google Scholar 

  67. Jordan, M. I. & Jacobs, R. A. Hierarchical mixtures of experts and the EM algorithm. Neural Comput. 6, 181–214 (1994).

    Article  Google Scholar 

  68. Soleimani, H., Hensman, J. & Saria, S. Scalable joint models for reliable uncertainty-aware event prediction. IEEE Trans. Pattern Anal. Mach. Intell. 40, 1948–1963 (2018).

    Article  PubMed  Google Scholar 

  69. Schulam, P. & Saria, S. Can you trust this prediction? Auditing pointwise reliability subsequent to training. In Proc. of the 22nd International Conference on Artificial Intelligence and Statistics, 89:1022-1031 (2019).

  70. Pierson, E., Cutler, D. M., Leskovec, J., Mullainathan, S. & Obermeyer, Z. An algorithmic approach to reducing unexplained pain disparities in underserved populations. Nat. Med. 27, 136–140 (2021).

    Article  CAS  PubMed  Google Scholar 

  71. Subbaswamy, A. & Saria, S. From development to deployment: dataset shift, causality, and shift-stable models in health AI. Biostatistics 21, 345–352 (2020).

    PubMed  Google Scholar 

  72. Finlayson, S. G. et al. The clinician and dataset shift in artificial intelligence. N. Engl. J. Med. 385, 283 (2021).

    Article  PubMed  PubMed Central  Google Scholar 

  73. Subbaswamy, A., Schulam, P. & Saria, S. Preventing failures due to dataset shift: learning predictive models that transport. In Proc. of the International Conference on Artificial Intelligence and Statistics, PMLR 89:3118-3127 (2019).

  74. Henry, K., Wongvibulsin, S., Zhan, A., Saria, S. & Hager, D. Can septic shock be identified early? Evaluating performance of a targeted real-time early warning score (TREW score) for septic shock in a community hospital: global and subpopulation performance. Am. J. Respir. Crit. Care Med. 195, 7016 (2017).

    Google Scholar 

  75. Peltan, I. D. et al. ED door-to-antibiotic time and long-term mortality in sepsis. Chest 155, 938–946 (2019).

    Article  PubMed  PubMed Central  Google Scholar 

  76. Vincent, J. L. et al. The SOFA (sepsis-related organ failure assessment) score to describe organ dysfunction/failure. Intensive Care Med. 22, 707–710 (1996).

    Article  CAS  PubMed  Google Scholar 

  77. Knaus, W. A., Draper, E. A., Wagner, D. P. & Zimmerman, J. E. APACHE II: a severity of disease classification system. Crit. Care Med. 13, 818–829 (1985).

    Article  CAS  PubMed  Google Scholar 

  78. Metcalfe, D. et al. Charlson and Elixhauser coding mapping. BMC Med. Res. Methodol. 19, 1–9 (2019).

    Google Scholar 

  79. Seabold, S. & Perktold, J. statsmodels: econometric and statistical modeling with Python. Proc. 9th Python in Science Conference (2010).

  80. Norton, E. C., Miller, M. M. & Kleinman, L. C. Computing adjusted risk ratios and risk differences in Stata. Stata J. 13, 492–509 (2013).

    Article  Google Scholar 

Download references

Acknowledgements

The authors thank Y. Ahmad, A. Zhang, M. Yeo and Y. Karklin whose work significantly contributed to early iterations of the development of the deployed system. We also thank R. Demski, K. D’Souza, A. Kachalia, A. Chen and clinical and quality stakeholders who contributed to tool deployment, education and championing the work. The authors gratefully acknowledge the following sources of funding: the Gordon and Betty Moore Foundation (award 3926), the National Science Foundation Future of Work at the Human-technology Frontier (award 1840088) and the Alfred P. Sloan Foundation research fellowship (2018). This information or content and conclusions are those of the authors and should not be construed as the official position or policy of, nor should any endorsements be inferred by the NSF the US Government.

Author information

Author notes

  1. These authors contributed equally: Katharine E. Henry, Roy Adams, Cassandra Parent.

Authors and Affiliations

  1. Department of Computer Science, Johns Hopkins University, Baltimore, MD, USA

    Katharine E. Henry & Suchi Saria

  2. Malone Center for Engineering in Healthcare, Johns Hopkins University, Baltimore, MD, USA

    Katharine E. Henry, Roy Adams & Suchi Saria

  3. Department of Psychiatry and Behavioral Science, Johns Hopkins School of Medicine, Baltimore, MD, USA

    Roy Adams

  4. Department of Biomedical Engineering, Johns Hopkins University, Baltimore, MD, USA

    Cassandra Parent

  5. Health Informatics, University of California, San Francisco, San Francisco, CA, USA

    Hossein Soleimani

  6. Howard County General Hospital, Columbia, MD, USA

    Anirudh Sridharan & Robert C. Linton II

  7. Department of Quality Improvement, The Johns Hopkins Hospital, Baltimore, MD, USA

    Lauren Johnson, Maureen Henley, Katrina Houston & Anushree R. Ahluwalia

  8. Department of Medicine, Johns Hopkins School of Medicine, Baltimore, MD, USA

    David N. Hager, Sara E. Cosgrove, Edward S. Chen, Albert W. Wu & Suchi Saria

  9. Suburban Hospital, Bethesda, MD, USA

    Andrew Markowski

  10. Department of Emergency Medicine, Johns Hopkins School of Medicine, Baltimore, MD, USA

    Eili Y. Klein & Mustapha O. Saheed

  11. Department of Medicine, The Johns Hopkins Hospital, Baltimore, MD, USA

    Sheila Miranda

  12. Department of Health Policy and Management, Johns Hopkins Bloomberg School of Public Health, Baltimore, MD, USA

    Albert W. Wu & Suchi Saria

  13. Department of Epidemiology, Johns Hopkins Bloomberg School of Public Health, Baltimore, MD, USA

    Albert W. Wu

  14. Department of International Health, Johns Hopkins Bloomberg School of Public Health, Baltimore, MD, USA

    Albert W. Wu

  15. Bayesian Health, New York, NY, USA

    Suchi Saria

Authors
  1. Katharine E. Henry
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Roy Adams
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Cassandra Parent
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Hossein Soleimani
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Anirudh Sridharan
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Lauren Johnson
    View author publications

    You can also search for this author in PubMed Google Scholar

  7. David N. Hager
    View author publications

    You can also search for this author in PubMed Google Scholar

  8. Sara E. Cosgrove
    View author publications

    You can also search for this author in PubMed Google Scholar

  9. Andrew Markowski
    View author publications

    You can also search for this author in PubMed Google Scholar

  10. Eili Y. Klein
    View author publications

    You can also search for this author in PubMed Google Scholar

  11. Edward S. Chen
    View author publications

    You can also search for this author in PubMed Google Scholar

  12. Mustapha O. Saheed
    View author publications

    You can also search for this author in PubMed Google Scholar

  13. Maureen Henley
    View author publications

    You can also search for this author in PubMed Google Scholar

  14. Sheila Miranda
    View author publications

    You can also search for this author in PubMed Google Scholar

  15. Katrina Houston
    View author publications

    You can also search for this author in PubMed Google Scholar

  16. Robert C. Linton II
    View author publications

    You can also search for this author in PubMed Google Scholar

  17. Anushree R. Ahluwalia
    View author publications

    You can also search for this author in PubMed Google Scholar

  18. Albert W. Wu
    View author publications

    You can also search for this author in PubMed Google Scholar

  19. Suchi Saria
    View author publications

    You can also search for this author in PubMed Google Scholar

Contributions

K.E.H., R.A., C.P., E.S.C., A.W.W. and S.S. contributed to the initial study design and preliminary analysis plan. S.S. led the development and deployment efforts for the TREWS software. K.E.H., H.S., A.S., R.C.L., L.J., M.H., S.M., D.N.H., A.W.W. and S.S. contributed to the system development and deployment. K.E.H., R.A., C.P., E.Y.K., S.E.C., A.R.C., E.S.C., D.N.H., A.W.W. and S.S. contributed to the review and analysis of the results. All authors contributed to the final preparation of the manuscript.

Corresponding authors

Correspondence to Albert W. Wu or Suchi Saria.

Ethics declarations

Competing interests

Under a license agreement between Bayesian Health and the Johns Hopkins University, K.E.H., S.S. and Johns Hopkins University are entitled to revenue distributions. Additionally, the University owns equity in Bayesian Health. This arrangement has been reviewed and approved by the Johns Hopkins University in accordance with its conflict-of-interest policies. S.S. also has grants from Gordon and Betty Moore Foundation, the National Science Foundation, the National Institutes of Health, Defense Advanced Research Projects Agency, the Food and Drug Administration and the American Heart Association; she is a founder of and holds equity in Bayesian Health; she is the scientific advisory board member for PatientPing; and she has received honoraria for talks from a number of biotechnology, research and health-tech companies. This arrangement has been reviewed and approved by the Johns Hopkins University in accordance with its conflict-of-interest policies. D.N.H. discloses salary support and funding to his institution from the Marcus Foundation for the conduct of the vitamin C, thiamine and steroids in sepsis trial. S.E.C. declares consulting fees from Basilea for work on an infection adjudication committee for an S. aureus bacteremia trial. The other authors declare no competing interests.

Peer review

Peer review information

Nature Medicine thanks Derek Angus, Melanie Wright and the other, anonymous, reviewer(s) for their contribution to the peer review of this work. Primary Handling editor: Michael Basson, in collaboration with the Nature Medicine team.

Additional information

Publisher’s note Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Extended data

Extended Data Fig. 1 Retrospective predictive performance of the TREWS model.

Performance of the TREWS model on retrospective data. Figure (a) shows the receiver operating characteristic curve and Figure (b) shows the sensitivity–PPV curve (also referred to as the precision-recall curve).

Extended Data Fig. 2 Annotated screenshot of the TREWS interface.

Annotated screenshot of the TREWS provider evaluation page. Annotations show the main provider actions: reviewing the alert explanation, indicating whether the patient has a suspected source of infection and reviewing sources of organ dysfunction.

Extended Data Table 1 Time from alert to first antibiotic order among retrospectively identified sepsis patients
Full size table
Extended Data Table 2 TREWS alert volume per day during the study period including re-alerts and alerts flagging patients who are candidates for escalation
Full size table
Extended Data Table 3 Population characteristics
Full size table
Extended Data Table 4 Model features
Full size table

Supplementary information

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Henry, K.E., Adams, R., Parent, C. et al. Factors driving provider adoption of the TREWS machine learning-based early warning system and its effects on sepsis treatment timing. Nat Med 28, 1447–1454 (2022). https://doi.org/10.1038/s41591-022-01895-z

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1038/s41591-022-01895-z

This article is cited by

Search

Advanced search

Quick links

Nature Briefing

Sign up for the Nature Briefing newsletter — what matters in science, free to your inbox daily.

Get the most important science stories of the day, free in your inbox. Sign up for Nature Briefing