Intended for healthcare professionals

CCBYNC Open access
Research

Electronic health record alerts for acute kidney injury: multicenter, randomized clinical trial

BMJ 2021; 372 doi: https://doi.org/10.1136/bmj.m4786 (Published 18 January 2021) Cite this as: BMJ 2021;372:m4786

Opinion

The challenge of minimal risk in e-alert trials

  1. F Perry Wilson, associate professor1 2,
  2. Melissa Martin, research associate1 2,
  3. Yu Yamamoto, statistician1 2,
  4. Caitlin Partridge, data scientist3,
  5. Erica Moreira, data scientist3,
  6. Tanima Arora, postdoctoral researcher1 2,
  7. Aditya Biswas, data scientist1 2,
  8. Harold Feldman, professor4,
  9. Amit X Garg, professor5,
  10. Jason H Greenberg, assistant professor2 6,
  11. Monique Hinchcliff, associate professor7,
  12. Stephen Latham, professor8,
  13. Fan Li, assistant professor9,
  14. Haiqun Lin, professor10,
  15. Sherry G Mansour, instructor1 2,
  16. Dennis G Moledina, assistant professor1 2,
  17. Paul M Palevsky, professor11,
  18. Chirag R Parikh, professor12,
  19. Michael Simonov, instructor2,
  20. Jeffrey Testani, associate professor13,
  21. Ugochukwu Ugwuowo, postdoctoral researcher1 2
  1. 1Department of Medicine, Section of Nephrology, Yale University School of Medicine, New Haven, CT, USA
  2. 2Clinical and Translational Research Accelerator, Department of Medicine, Yale University School of Medicine, New Haven, CT, USA
  3. 3Joint Data Analytics Team, Yale School of Medicine, New Haven, CT, USA
  4. 4Department of Epidemiology and Biostatistics and the Center for Clinical Epidemiology and Biostatistics, Perelman School of Medicine at the University of Pennsylvania, Philadelphia, PA, USA
  5. 5Department of Epidemiology and Biostatistics and Department of Medicine, Division of Nephrology, Schulich School of Medicine & Dentistry, Western University, ON, Canada
  6. 6Department of Pediatrics, Yale University School of Medicine, New Haven, CT, USA
  7. 7Department of Medicine, Section of Rheumatology, Allergy and Immunology, Yale University School of Medicine, New Haven, CT, USA
  8. 8Yale Interdisciplinary Center for Bioethics, Yale Law School, New Haven, CT, USA
  9. 9Department of Biostatistics, Yale School of Public Health, New Haven, CT, USA
  10. 10Rutgers University Biomedical and Health Sciences, Newark, NJ, USA
  11. 11Medicine and Clinical & Translational Science, University of Pittsburgh School of Medicine and Renal Section, VA Pittsburgh Healthcare System, Pittsburgh, PA, USA
  12. 12Department of Medicine, Division of Nephrology, John Hopkins Medicine, Baltimore, MD, USA
  13. 13Department of Internal Medicine, Section of Cardiology, Yale University School of Medicine, New Haven, CT, USA
  1. Correspondence to: F Perry Wilson, 60 Temple St, Suite 6C, New Haven, CT 06410, USA francis.p.wilson{at}yale.edu (or @fperrywilson on Twitter)
  • Accepted 16 October 2020

Abstract

Objective To determine whether electronic health record alerts for acute kidney injury would improve patient outcomes of mortality, dialysis, and progression of acute kidney injury.

Design Double blinded, multicenter, parallel, randomized controlled trial.

Setting Six hospitals (four teaching and two non-teaching) in the Yale New Haven Health System in Connecticut and Rhode Island, US, ranging from small community hospitals to large tertiary care centers.

Participants 6030 adult inpatients with acute kidney injury, as defined by the Kidney Disease: Improving Global Outcomes (KDIGO) creatinine criteria.

Interventions An electronic health record based “pop-up” alert for acute kidney injury with an associated acute kidney injury order set upon provider opening of the patient’s medical record.

Main outcome measures A composite of progression of acute kidney injury, receipt of dialysis, or death within 14 days of randomization. Prespecified secondary outcomes included outcomes at each hospital and frequency of various care practices for acute kidney injury.

Results 6030 patients were randomized over 22 months. The primary outcome occurred in 653 (21.3%) of 3059 patients with an alert and in 622 (20.9%) of 2971 patients receiving usual care (relative risk 1.02, 95% confidence interval 0.93 to 1.13, P=0.67). Analysis by each hospital showed worse outcomes in the two non-teaching hospitals (n=765, 13%), where alerts were associated with a higher risk of the primary outcome (relative risk 1.49, 95% confidence interval 1.12 to 1.98, P=0.006). More deaths occurred at these centers (15.6% in the alert group v 8.6% in the usual care group, P=0.003). Certain acute kidney injury care practices were increased in the alert group but did not appear to mediate these outcomes.

Conclusions Alerts did not reduce the risk of our primary outcome among patients in hospital with acute kidney injury. The heterogeneity of effect across clinical centers should lead to a re-evaluation of existing alerting systems for acute kidney injury.

Trial registration ClinicalTrials.gov NCT02753751.

Introduction

Acute kidney injury—an abrupt decline in the kidney’s filtration function—is a common occurrence in patients in hospital and is associated with nearly a 10-fold increase in inpatient mortality.123 In survivors, acute kidney injury is associated with increased risks of chronic kidney disease, end stage kidney disease, and mortality.45 Despite the strong association with important outcomes, the syndrome (which is asymptomatic) is often missed in the course of clinical care; it occurs in roughly 15% of patients in hospital and yet is recorded in the electronic health record in fewer than half of patients with acute kidney injury.678910 Best practices for acute kidney injury, such as routine urinalysis and measurement of urine output, occur infrequently, and even repeat measurement of creatinine is not universal. These deficiencies provide a reason for alerting providers about the presence of acute kidney injury.11

With the assumption that increased recognition of acute kidney injury will improve care of these patients and thus improve clinical outcomes, many health systems in the United States, and the National Health Service in the United Kingdom, have introduced an electronic alert for acute kidney injury as part of routine clinical care.121314 Data on the efficacy of these alerts are limited, however, and definitive evidence of their benefit for patient outcomes remains absent. Additionally, as acute kidney injury is a heterogeneous condition, lacking proven strategies for its mitigation, an alert might prompt actions that are unnecessary or even harmful.

The effectiveness of alerting probably depends on a combination of patient, provider, and disease factors, and might differ in different care environments, thus requiring rigorous evaluation. We conducted a multicenter randomized clinical trial of usual care versus usual care plus pop-up acute kidney injury alerts delivered through an electronic health record. The alert was designed to reach multiple providers and was implemented using an identical protocol across a diverse set of hospitals. We hypothesized that alerts would influence provider behavior and improve clinical outcomes.

Methods

Extended details of the methods are shown in a supplementary web appendix.

Trial design and oversight

A detailed description of the trial design and rationale has been previously published.15 Briefly, we conducted a patient level, parallel group, randomized controlled trial of health record electronic alerts versus usual care in adults with acute kidney injury at six hospitals in the Yale New Haven Health System in Connecticut and Rhode Island, US. The trial was conducted in accordance with the principles of the Declaration of Helsinki, and two institutional review boards associated with the six study hospitals approved the study, which was deemed minimal risk. Patients were enrolled, and a requirement for informed consent was waived, as the alert was deemed unlikely to affect patient welfare and informing patients of their diagnosis of acute kidney injury would contaminate the usual care group. An external data and safety monitoring board performed four independent assessments during the trial, including one formal interim analysis at 50% recruitment. These analyses were not stratified by hospital. Hospitals were given guidelines to terminate the study if the P value associated with interim analysis was less than or equal to 0.001 for efficacy or less than or equal to 0.005 for harm. The hospitals also had the authority to terminate the study of their own accord regardless of those recommendations. This trial was registered with clinicaltrials.gov under registration number NCT02753751 on 25 April 2016, before study initiation and patient enrolment.

Patients

Inpatient adults aged 18 or older with acute kidney injury, defined by the Kidney Disease: Improving Global Outcomes (KDIGO) criteria as an increase in creatinine 0.3 mg/dL (26.5 μmol/L) within 48 hours or 1.5 times the lowest measured creatinine within the previous seven days of admission to hospital, were eligible.16 Owing to missing and inaccurate urine output data, urine output was not used to classify acute kidney injury. Patients with a history of end stage kidney disease (based on International Classification of Disease, ninth and 10th revision (ICD-9 and ICD-10) codes, a dialysis order in the past year, or an initial hospital creatinine greater than 354 μmol/L were excluded automatically by the detection algorithm. We subsequently excluded patients whose admission date was before the inception of alerts at a study hospital, whose first alert occurred after hospital discharge, who had been enrolled in a previous study, and who were enrolled during a two week period in which alerting ceased owing to an upgrade of the electronic health record system (fig S1).

Randomization and masking

Eligible patients were identified using an acute kidney injury detection algorithm built into our best practice alert within the Epic electronic health record system. When the chart is opened, the algorithm automatically assesses the patient record for inclusion and exclusion criteria. If all criteria are met, the patient is automatically enrolled into the trial and randomized. Simple randomization was performed within Epic using an internal random number rule with concealed allocation, as our electronic health record system cannot perform permutated block randomization or import external randomization lists. Randomization was performed at the patient rather than the provider level, as inpatients at participating hospitals are often cared for by many providers who change throughout their hospital stay. All study investigators were blinded to patient randomization status until enrollment was complete.

Intervention

The intervention was an automated, electronic, pop-up alert which fired whenever the patient’s electronic chart (Epic Systems, Verona, WI) was opened (fig 1). Alerts were displayed only to individuals who had authority to change or enter new orders on behalf of the patient—hereafter referred to as “providers”—which included interns, residents, fellows, attending physicians, nurse practitioners, and physician’s assistants. Alerts were displayed each time the chart was opened, provided the patient continued to meet criteria for acute kidney injury. If the provider agreed or disagreed with the presence of acute kidney injury, the alert was suppressed for 48 hours for that provider. If multiple providers used the electronic health record to care for the same patient, each of them separately received the alert whenever they opened the patient chart. Patients randomized to the usual care group generated “silent” alerts, which did not display to providers but were tracked, allowing the study team to observe which type of providers opened the chart and how often.

Fig 1
Fig 1

Screenshot of the acute kidney injury (AKI) alert. Creatinine 1 mg/dL=88.42 μmol/L.

The alert contained an option to add acute kidney injury to the patient’s problem list, and also a link to an acute kidney injury order set (supplemental fig S2); the set included options for blood and urine testing and kidney imaging but were limited to tests and procedures considered minimal risk (that is, intravenous fluid administration was not included). Providers at each of the study hospitals received education about acute kidney injury and the alert system before study roll out during departmental conferences. The educational program explained the definitions of acute kidney injury, the functioning of the alert, and the ethics of the study but did not provide suggested practices of care.

Outcomes

The primary outcome was a composite of inpatient acute kidney injury progression (defined as an increase in acute kidney injury stage), receipt of dialysis, or death within 14 days of randomization. Prespecified secondary outcomes included the components of the primary outcomes and frequency of various practices of care for acute kidney injury. These practices included administration of contrast, fluids, or a nephrotoxic agent (a non-steroidal anti-inflammatory drug, angiotensin converting enzyme inhibitor/angiotensin receptor blocker, or aminoglycoside), ordering a urinalysis, documentation of acute kidney injury, monitoring of creatinine and urine output, and ordering a kidney consultation. An assessment of each hospital’s alert effects was also prespecified.

Statistical analysis

We present descriptive statistics as median (interquartile range) for continuous variables and proportions for categorical variables. For the primary analysis, and all comparisons of categorical variables between the intervention and control group, we used the Mantel-Haenszel test, accounting for each hospital site as a stratum. We used the Mantel-Haenszel approach to obtain the pooled relative risks across hospital strata without adjusting for other baseline factors.

As a sensitivity analysis, we used modified Poisson generalized estimating equations with a robust variance estimator to present the relative risk estimates, adjusting for the following characteristics assessed at the time of randomization: age, sex, race, creatinine, blood urea nitrogen, white blood cell count, heart rate, respiratory rate, systolic and diastolic blood pressure, chronic heart failure, hypertension, diabetes, malignancy, intensive care unit status, modified sequential organ failure assessment (mSOFA) score, Elixhauser comorbidity score, and hospital.17 We used the Van Elteren test to compare continuous variables across the intervention groups, accounting for hospital strata.

To compare time to event between study groups, we used Cox proportional hazard regression with intervention as the independent variable, stratified by hospital, with censoring at 14 days after randomization. For individual hospital analyses, Kaplan-Meier curves were generated and log rank tests were used. Patients discharged before 14 days without an outcome of interest were assumed to be free of that outcome at 14 days. Death was treated as a censoring event in analyses where death was not the outcome. We used Schoenfeld residuals to examine the proportional hazards assumption in Cox models; there were no violations. We specifically amended the protocol to include a test for site heterogeneity on 4 May 2018—that is, two months after the initial patient enrollment and before we had started alerting in any of the hospitals in which adverse outcomes were ultimately detected. This new protocol was based on ongoing executive committee calls and the hypothesis that the alert effect might differ by hospital. No data were used to drive this decision.

Our preliminary data suggested that 24.5% of patients would experience the composite outcome. We considered a relative 20% reduction in this outcome (to 19.6) was clinically significant and would represent an acceptable threshold to justify adding an interruptive alert to the clinician workflow. A sample size of 2512 in each arm of the study achieves 90% power to detect this degree of change, but given the potential for contamination across study arms (whereby clinicians “learn” to identify and care for acute kidney injury over the course of the trial), which would tend to bias the results toward the null, we inflated the sample size by 20% to a total of 6030. Although this sample increase does not deal with the bias toward the null, it does allow for increased detection of a smaller effect size.

Statistical analyses were conducted in Stata version 15.1 (College Station TX), SAS version 9.4 (SAS Institute, Cary, NC), and R (RStudio version 1.2.5033 (R version 3.5.3), Boston, MA), and P≤0.04 was considered statistically significant for the primary outcome (to account for the interim analysis). We do not report P values for secondary and exploratory analyses, except for safety outcomes, but report 95% confidence intervals in all cases. The widths of the 95% confidence intervals have not been adjusted for multiplicity, and inferences drawn from these might not be reproducible.

Patient and public involvement

No patients were involved in setting the research question or the outcome measures, nor were they involved in developing plans for recruitment, design, or implementation of the study. No patients were asked to advise on interpretation or writing up of results. Patient involvement was not sought owing to the limited public understanding of acute kidney injury and a lack of definitive treatment.

Results

From 29 March 2018 to 14 December 2019, 6030 individuals met enrollment criteria and were randomized (supplemental fig S1). Those patients generated a total of 226 316 alerts (including silent alerts in the usual care arm), which reached 3632 unique providers, a median (interquartile range) of 7 (7-13) for each patient. Characteristics of the study population at the time of randomization are shown in table 1 (stratified analyses, supplemental tables 1-6). The median (interquartile range) age of patients was 71 (59-82) years, 2882 (48%) were female, and 946 (16%) self-identified as black. The majority (n=4564 (76%)) were admitted to a medical service, and 1923 (32%) were in an intensive care unit at the time of randomization.

Table 1

Patient characteristics at the time of randomization. Data are presented as median (interquartile range) or count (percent)

View this table:

Patients randomized to acute kidney injury alerts were more likely to receive an order for intravenous fluids (38.4% alert arm v 34.8% usual care arm, absolute difference 3.8% (95% confidence interval 1.4% to 6.2%)), urinalysis (17.0% v 14.9%, 1.9% (0.1% to 3.7%)), and to have serum creatinine measured (87.2% v 85.2%, 1.8% (0.1% to 3.6%)) within 24 hours of randomization (table 2; stratified analyses, supplemental tables 7-12). Randomized patients were also more likely to have documentation of acute kidney injury in the problem list (70.0% alert arm v 63.0% usual care arm, absolute difference 7.0% (95% confidence interval 4.6% to 9.3%)) during the hospital admission. No significant differences were found in the frequency of intravenous contrast administration or recording of urine output between the arms at 24, 48, or 72 hours after randomization.

Table 2

Process outcomes stratified by study arm. Unless otherwise specified, outcomes reflect proportion completed within 24 hours of randomization. Results for usual care and alert groups are shown as number (percent) unless otherwise specified; absolute differences are shown as percent (95% confidence interval) unless otherwise specified

View this table:

Primary outcome: progression of acute kidney injury, dialysis, or death

The primary outcome occurred in 653 (21.3%) of 3059 patients with an alert and in 622 (20.9%) of 2971 patients receiving usual care (relative risk 1.02, 95% confidence interval 0.93 to 1.13, P=0.67; table 3 and stratified analyses supplemental tables 13-18). No evidence of a change was seen in the relative effect across intervention groups over time (P=0.90 for interaction). Time to event analyses for the composite outcome and for each component and each hospital are shown in supplemental figures S3-S6 and were of similar magnitude.

Table 3

Primary and secondary outcomes of interest.* Results for usual care and alert groups are shown as number (percent) unless otherwise specified; absolute difference is shown as percent (95% confidence interval) except for duration of AKI and maximum AKI stage achieved, in which odds ratios (95% confidence interval) are reported

View this table:

Assessment of heterogeneity of treatment effect

Evidence of heterogeneity of treatment effect was found across the six hospitals (P=0.05; fig 2). Some sites suggested a trend towards a protective effect of alerts, such as hospital 3 (relative risk 0.80, 0.62 to 1.04, P=0.09), but others showed potential harm from alerts. In a post hoc analysis, this heterogeneity was not accounted for by patient characteristics, but by whether the hospitals were stratified as “teaching” or “non-teaching.” In particular, there appeared to be harm from alerts at the two non-teaching hospitals in the study (n=765), where 24.3% of patients in the alert arm compared with 16.3% in the usual care arm met the primary outcome (relative risk 1.49, 1.12 to 1.98, P=0.006). No such effect was found at the four teaching hospitals (0.97, 0.87 to 1.07, P=0.54), with P=0.006 for interaction comparing teaching with non-teaching hospitals.

Fig 2
Fig 2

Primary and secondary outcome events, stratified by hospital type. Error bars are 95% confidence intervals of the observed proportion of events. AKI=acute kidney injury

The differences in the primary outcomes observed in the non-teaching versus teaching hospitals seemed to be driven by death as opposed to dialysis or progression of the acute kidney injury. We noted evidence of heterogeneity of the association between alerts and death across the enrolled hospitals (P=0.05). Again, alerts were associated with a significantly higher risk of death at 14 days in the non-teaching hospitals, where 59 (15.6%) patients in the alert group versus 33 (8.6%) patients receiving usual care met this outcome. The relative risk of death was 1.82 (1.22 to 2.72, P=0.003) in the non-teaching hospitals compared with a relative risk of 0.89 (0.74 to 1.06, P=0.18) at the teaching hospitals (P=0.001 for interaction).

Secondary outcomes and process measures

The frequency of kidney consultations was similar between the two arms. Within 14 days of randomization, 23.8% of the alert group and 23.9%of the usual care group had received a consultation (absolute difference 0.1% (95% confidence interval −2.1% to 2.2%)). No evidence of heterogeneity of this effect was seen across the study hospitals (P>0.99). Similarly, in time to event analysis, no increased rate of kidney consultations was found across the two arms (hazard ratio 1.00 (95% confidence interval 0.90 to 1.11). No significant difference was seen in the rate of discharge to hospice overall or in the non-teaching hospitals.

Subgroup analyses

Prespecified subgroup analyses are shown in figure 3. The effect of alerting was largely similar between medical and surgical patients, ICU and non-ICU patients, by age, race, and sex, and across baseline creatinine values.

Fig 3
Fig 3

Prespecified subgroup analyses show similar alert effect across a diverse array of patient characteristics. Diamonds reflect relative risk, with bars showing 95% confidence interval. Creatinine 1 mg/dL=88.42 μmol/L. ICU=intensive care unit

Sensitivity and post hoc analyses

We examined the adjusted relative risk of the primary outcome accounting for a variety of baseline factors individually and stratified into teaching and non-teaching hospitals (supplemental table 19). The adjusted relative risks were broadly similar, although the adjusted risk of death in the non-teaching hospitals attenuated to 1.58 (95% confidence interval 1.08 to 2.31) and in the teaching hospitals to 0.85 (0.72 to 1.01).

We also performed a formal mediation analysis to determine if certain actions taken after alerting might have mediated the adverse outcomes in the non-teaching hospitals (supplemental table 20).18 The use of intravenous fluid (in drip or bolus form, in high chloride or more physiologic form) did not appear to mediate the adverse effect of the alert. Other proxies of volume overload (respiratory rate, oxygen saturation, and the change in these parameters) similarly did not show significant effect mediation. Exposures to medication, kidney consultations, and the percent of alerts that went to attending versus other providers also showed no significant effect mediation. Finally, the burden of other electronic health record alerts (such as those for sepsis or interactions between drugs) did not appear to mediate the deleterious effects of alerts for acute kidney injury.

We also examined the risks of two components of our primary outcome, death and progression of acute kidney injury, in the period two years before the start of this trial in each hospital (fig S7). Broadly, these risks were similar to those of the control group in the randomized portion of the study. For example, two years before our study started, the risk of death at 14 days after acute kidney injury in the non-teaching hospitals was 8.6%, similar to the risk of death for the control group of 8.6%, and significantly lower than the risk of death of 15.6% for the intervention group.

Discussion

Principal findings

In this multicenter randomized trial of patients with acute kidney injury, the provision of an electronic health record alert to providers had no effect on the risk of progression of acute kidney injury, dialysis, or death. Substantial heterogeneity of effect was seen across study hospitals, however, with signals for harm emerging in the two non-teaching hospitals. We cannot be sure whether the observed differences in effect were due to the presence of teaching services or other practices that might differ between the hospitals in our study. The fact that some process measures (such as orders for intravenous fluids and urinalysis) occurred more often in the alert arm suggests that the alerts were being noticed by providers and that they could change provider behavior, but the magnitude of these changes was relatively small.

Reconciliation of those findings with the fact that the risk of death was higher in the alert arm in the non-teaching institutions is difficult. This increased risk might have been due to random chance, although the P value for the alert:death association of 0.003 suggests that chance is unlikely. Furthermore, adjustment for a variety of baseline factors did not ablate the observed association, suggesting that imbalance in baseline covariates is insufficient to explain the association. Additionally, in our prior, single center, acute kidney injury alert trial (conducted at a medical center not involved in this study), we noted a numerically higher, although statistically insignificant, risk of death in the alert versus the usual care arm. The risk of death at 14 days in that trial was 7.1% in the usual care group and 7.7% in the alert group (P=0.58).19 To date, that study is the only previous trial of acute kidney injury alerts that employed randomization at the patient level. Finally, we examined historical death rates of acute kidney injury in the study hospitals and found them to be quite similar to the rates seen in the usual care arm of our trial. Taken together, these data suggest that there might be true harm associated with the provision of alerts for acute kidney injury in certain situations. In view of the secondary nature of these analyses, the results should be considered hypothesis generating, and any attribution to the “non-teaching” nature of the hospital is premature.

As alerts of this type are in widespread clinical use, we sought to determine if any factor downstream of randomization drove these results. We speculated that an increase in fluid administration in the alert group might lead to harm through volume overload, but accounting for fluid administration did not attenuate the observed effect. We further considered that certain types of providers (eg, medical doctors v physician’s assistants) might respond to acute kidney injury differently, but there was no indication that the effect changed according to the type of provider receiving the alert. Furthermore, we assessed whether alert fatigue might be to blame for poorer outcomes in the alert group, but accounting for the frequency of other electronic health record alerts did not change the observed association with the alert for acute kidney injury. Possibly, while no single factor mediated the adverse effect in every patient, a constellation of factors resulted in the observed effect. We are left without a satisfying unifying explanation for the potential harm. Alerts might distract providers from other, more clinically important, patient care. Alternatively, in the larger teaching institutions, systems exist to protect patients from certain harms (like inappropriate drug dosing), which are not present in the smaller hospitals (regardless of academic status). Finally, we should consider that alerts could compel providers to do something, even if it is not clinically indicated, perhaps owing to fears of medicolegal consequences if they do not act.

Comparisons with other studies

Alerts of all kinds, from electronic to auditory to visual, have proliferated in medical care with the best of intentions under assumptions that risk is minimal.20 Notably, alerting systems for acute kidney injury have already been widely adopted across various health systems. The published literature suggests that alerts are often effective in changing processes of care, but their effect on clinical efficacy has been mixed.21 For example, a stepped wedge randomized trial of 24 059 episodes of acute kidney injury in the UK found a decreased length of stay, but no change in overall mortality risk between the control and intervention periods.22 In contrast, a before/after study of a clinical decision support system for acute kidney injury among 64 512 patients found a significant reduction in mortality risk from 10.2% to 9.4% from before to after the intervention.23 This design could not account for secular trends in the care of patients with acute kidney injury, although the mortality risk in patients with non-acute kidney injury did not change over the study period.

This trial cannot show a clear efficacy of alerts, but the phenomenon of “alert fatigue”—a tendency to disregard even important alerts owing to their abundance—is increasingly recognized.24252627 The economic costs of alerts might be minimal after the initial alert build, but the costs of provider attention, burnout, and potential overtreatment might be substantial and clearly deserve additional study.28 This trial was conducted under a waiver of informed consent, based on the prevailing assumption that informational alerts for acute kidney injury pose minimal risk to the patient and the recognition that obtaining the consent of those randomized to the control group would invalidate the study results. Unfortunately, for the alert deployed in this study, the assumption of minimal risk proved incorrect. These results do illustrate, however, that clinical trials of seemingly benign interventions, such as alerts, require rigorous evaluation and should not be implemented without robust evidence of safety and efficacy.

Strengths and limitations of this study

Limitations of the study include randomization at the patient level, allowing for providers potentially to learn how to better identify acute kidney injury throughout the course of the trial. This contamination would tend to bias the results toward the null hypothesis that the alerts did not affect the primary outcome, but an analysis accounting for the duration that the alerts had been in effect at each hospital found no evidence of this. Furthermore, our previous research has shown that patient level factors, such as baseline creatinine value and sex, contribute to alert benefit.29 Additionally, the alert was largely informational. Although the alert provided a link to an acute kidney injury order set, it provided no patient-specific recommendations (such as identifying current nephrotoxic drug treatment)—a key feature of the successful alert for pediatric acute kidney injury piloted by the nephrotoxic injury negated by just-in-time action (NINJA) program, and by certain “care bundles” for acute kidney injury that have been used in the UK.2230 The heterogeneity of acute kidney injury might require more personalized interventions. Furthermore, the alert was sent only to certain care providers, notably excluding nurses and pharmacists, who might specifically benefit from receipt of the alert. The alert did not use urine output criteria, as many ward patients do not have reliable estimates of urine output. This might have meant that the study population was not fully representative of all patients with acute kidney injury. Finally, the alert was conducted across six hospitals within a single large health system in northeastern US, limiting generalizability. However, the hospitals represent a diverse mix of patients and care models, suggesting that these results might apply more broadly.

The strengths of the study include its large size with an ability to detect clinically meaningful changes in hard endpoints, its enrollment across a diverse group of hospitals, and the use of a fully automated enrollment and randomization system that could pave the way for future trials of alerting and clinical decision support systems.

Conclusions

In this large, randomized, multicenter study, we found that, overall, the provision of an automated, electronic alert for acute kidney injury had a modest effect on process measures and no effect on the risk of progression of acute kidney injury, dialysis, or death. Substantial heterogeneity of effects across hospitals was found, with the concerning finding of a significantly increased risk of death in the alert group in non-teaching hospitals. This study argues against the implementation of informational alerts for acute kidney injury and for a reconsideration of the alerts currently used.

What is already known on this topic

  • Acute kidney injury occurs in 15% of patients in hospital and is associated with poor outcomes, yet fewer than 50% of these results are documented in the electronic health record, and best practices for acute kidney injury are performed inconsistently

  • Limited data from randomized trials exist to illustrate the clinical benefit of alerts for acute kidney injury or demonstrate their efficacy on patient outcomes

What this study adds

  • This randomized controlled trial found only a modest effect of electronic alerts on process methods for acute kidney injury and no overall effect on the risks of death, dialysis, or disease progression in patients in hospital with acute kidney injury

  • The heterogeneity of effect of the alert across hospitals argues for a more rigorous evaluation of safety and efficacy of both current and future alerting systems for acute kidney injury

Acknowledgments

We thank the Joint Data Analytics Team at Yale for their help in data acquisition and Epic integration of our acute kidney injury detection algorithm and alert.

Footnotes

  • Contributors: The corresponding author attests that all listed authors meet authorship criteria and that no others meeting the criteria have been omitted. All authors meet ICMJE recommendations for authorship: Have made substantial contributions to the conception or design of the work (FPW, MM, YY, TA, HF, AXG, JHG, MH, SL, SGM, DGM, PMP, CRP, MS, JT, UU), or to the acquisition, analysis, or interpretation of data for the work (FPW, MM, YY, CP, EM, TA, AB, HF, AXG, JHG, MH, SL, FL, HL, SGM, DGM, PMP, CRP, MS, JT, UU); drafted the work or revised it critically for important intellectual content (all authors); gave final approval of the version to be published (all authors); agreed to be accountable for all aspects of the work in ensuring that questions related to the accuracy or integrity of any part of the work are appropriately investigated and resolved (all authors). FPW is the guarantor.

  • Funding: This work was supported by grants NIH P30DK079310 and R01 DK113191 to FPW. The funding sources had no involvement in study design; in collection, analysis, and interpretation of data; in writing of the manuscript; or in the decision to submit for publication. All authors have full access to all data in the study and take responsibility for the integrity of the data and the accuracy of its analysis.

  • Competing interests: All authors have completed the ICMJE uniform disclosure form at www.icmje.org/coi_disclosure.pdf and declare the following support for the submitted work: PMP is a consultant for Baxter and receives grant support from Dascena and BioPorto. AXG is supported by the Dr Adam Linton Chair in Kidney Health Analytics and a Clinician Investigator Award from the Canadian Institutes of Health Research. HF is a consultant for the Kyowa Kirin Corporation, and is the editor in chief of the American Journal of Kidney Disease. JT reports grants and personal fees from Sequana Medical, grants and personal fees from BMS, personal fees from Astra Zeneca, personal fees from Novartis, grants and personal fees from five laboratories, personal fees from Cardionomic, personal fees from Bayer, grants and personal fees from Boehringer Ingelheim, personal fees from MagentaMed, grants from Otsuka, personal fees from Renalguard, grants and personal fees from Sanofi, grants and personal fees from FIRE1, grants from Abbott, personal fees from WL Gore, and personal fees from Windtree therapeutics outside the submitted work. SGM receives grant support from the American Heart Association and the Patterson Trust Fund. CRP is on the advisory board of RenalytixAI and owns equity in the same; serves on the data and safety monitoring board of Genfit Pharma and is supported by NIH grants R01085757 and UO1DK-082185. MH has received consulting fees from Abbvie. DGM receives grant support from the NIH/NIDDK (K23DK117065). FPW reports grant support from NIDDK R01DK113191 and P30DK079210.

  • Ethical approval: This research was approved by the Yale School of Medicine institutional review board (ID 1604017596) and the Bridgeport Hospital institutional review board (IDs 051802, 041801, 071808) under a waiver of informed consent.

  • Data sharing: A deidentified participant dataset with an associated data dictionary can be found at https://doi.org/10.5061/dryad.4f4qrfj95.

  • The corresponding author affirms the honest, accurate, and transparent account of the reported research and that no important or relevant aspects of the study have been omitted. Any discrepancies from the study as originally planned have been explained.

  • Dissemination to participants and related patient and public communities: There are no plans to disseminate the results of the research to individual study participants, but the results of this study will be disseminated as medical manuscripts. An associated press release will be made, and all manuscripts will be publicly available on PubMed Central. The study data have been made publicly available at the above website.

  • Provenance and peer review: Not commissioned; externally peer reviewed.

http://creativecommons.org/licenses/by-nc/4.0/

This is an Open Access article distributed in accordance with the Creative Commons Attribution Non Commercial (CC BY-NC 4.0) license, which permits others to distribute, remix, adapt, build upon this work non-commercially, and license their derivative works on different terms, provided the original work is properly cited and the use is non-commercial. See: http://creativecommons.org/licenses/by-nc/4.0/.

References