Elsevier

American Heart Journal

Volume 254, December 2022, Pages 102-111
American Heart Journal

Trial Designs
Development and validation of an automated algorithm for end point adjudication for a large U.S. national registry

https://doi.org/10.1016/j.ahj.2022.08.006Get rights and content

Background

Clinical events committee (CEC) evaluation is the standard approach for end point adjudication in clinical trials. Due to resource constraints, large registries typically rely on site-reported end points without further confirmation, which may preclude use for regulatory oversight.

Methods

We developed a novel automated adjudication algorithm (AAA) for end point adjudication in the National Cardiovascular Data Registry Left Atrial Appendage Occlusion (LAAO) Registry using an iterative process using CEC adjudication as the “gold standard.” A ≥80% agreement rate between automated algorithm adjudication and CEC adjudication was prespecified as clinically acceptable. Agreement rates were calculated.

Results

A total of 92 in-hospital and 127 post-discharge end points were evaluated between January 1, 2016 and June 30, 2019 using AAA and CEC. Agreement for neurologic events was >90%. Percent agreement for in-hospital and post-discharge events was as follows: ischemic stroke 95.7% and 94.5%, hemorrhagic stroke 97.8% and 96.1%, undetermined stroke 97.8% and 99.2%, transient ischemic attack 98.9% and 98.4% and intracranial hemorrhage 100.0% and 94.5%. Agreement was >80% for major bleeding (83.7% and 90.6%) and major vascular complication (89.1% and 97.6%). With this approach, <1% of site reported end points require CEC adjudication. Agreement remained very good during the period after algorithm derivation.

Conclusions

An AAA-guided approach for end point adjudication was successfully developed and validated for the LAAO Registry. With this approach, the need for formal CEC adjudication was substantially reduced, with accuracy maintained above an 80% agreement threshold. After application specific validation, these methods could be applied to large registries and clinical trials to reduce the cost of event adjudication while preserving scientific validity.

Section snippets

Methods

This study was funded by the American College of CardiologyNCDR and the National Heart, Lung and Blood Institute grants R56HL142765 and R01HL142765. The authors are solely responsible for the design and conduct of this study; all study analyses, the drafting and editing of the paper and its final contents.

Results

Between January 1, 2016 and June 30, 2019, a total of 219 end points including 92 in-hospital peri-procedural and 127 follow-up end points reported by the 585 sites in the LAAO Registry underwent adjudication using the AAA and CEC methods. With an iterative approach, agreement >80% was achieved for all end points, though agreement varied (Table I).

Concordance between the 2 adjudication methods was >90% for all neurologic end points. Percent agreement for peri-procedural and follow-up (through 2

Discussion

We demonstrated that an AAA can be developed to adjudicate major adverse events in a large-scale national registry with a robust data quality program. Validation against traditional CEC adjudication showed very good concordance with >90% agreement for all neurologic event types and >80% agreement for major vascular and major bleeding complications. Importantly, agreement exceeded the pre-specified 80% agreement threshold, making the AAA driven approach to end point ascertainment sufficient for

Conclusion

In conclusion, we developed a novel AAA which, in the context of a robust data quality program, can accurately adjudicate clinical events in a large, national registry. AAA development enhanced the registry data quality processes. Although AAA development identified some data combinations that require greater scrutiny, CEC adjudication was infrequently needed. An AAA based approach to end point ascertainment has the potential to significantly reduce costs and maintain data quality with

Funding

This study was funded by the American College of Cardiology (ACC) National Cardiovascular Data Registry (NCDR) and the National Heart, Lung and Blood Institute (NHLBI) grants R56HL142765 and R01HL142765.

Disclosures

Dr Curtis reported an institutional contract with the American College of Cardiology for his role as Senior Scientific Advisor of the NCDR and equity interest in Medtronic. Dr Freeman reports salary support from the ACC NCDR and the NHLBI and consulting/advisory board fees from Boston Scientific, Medtronic, Janssen Pharmaceuticals, Biocynetics/Pacemate and Biosense Webster. Dr Friedman has received: research support from Boston Scientific, Biosense Webster, Merit Medical, and Abbott; consulting

References (11)

There are more references available in the full text version of this article.

Cited by (0)

View full text