Developing an Unbiased Machine Learning Tool for Prediction of Acute Coronary Syndrome

Abstract Significance: Racial and sex disparities in the diagnosis and care of acute coronary syndrome (ACS) patients are well documented. As machine learning algorithms (MLA) become more common in healthcare settings, it is imperative to ensure that these methods do not contribute to disparities through biased predictions or differential accuracy across racial and sex groups. Research Question: Can a MLA be trained to be more accurate and less biased than commonly used risk stratification systems for ACS prediction? Prior Work: The research team developed a preliminary gradient boosted tree model for myocardial infarction (MI) prediction using retrospective data from electronic health records. On a hold-out test set, the algorithm classifier attained an area under the receiver operating characteristic curve (AUROC) value of 0.92 when tested for the detection of MI at any point during a patient’s hospital stay. Other prior work by the research team involved development of a MLA to minimize bias in inpatient mortality predictions between White and non-White patient groups. The model was found to be unbiased as measured by the equal opportunity difference (EOD = 0.016, p = 0.204) and outperformed commonly used severity scoring systems MEWS, SAPS-II, and APACHE in respect to bias and accuracy. Specific Aims: In Aim 1, an unbiased model for early ACS prediction will be developed. Preprocessing the MLA training data will remove aspects of the data that reflect systemic health inequities while maintaining the aspects of the data that reflect relevant patient measurements and outcomes. Assessment of equal opportunity difference (EOD) and the Zemel statistic will provide a means to evaluate the MLA’s ability to operate without sex or racial bias. In Aim 2, the model’s performance will be compared to three commonly used ACS risk stratification scores. Evaluating model performance and bias against these systems will allow for comparison of the unbiased MLA to the current ACS standard of care. Methods: Aim 1: An ACS prediction algorithm that will be demonstrated to be unbiased when comparing performance accuracy on White vs. non-White and male vs. female emergency department patients will be developed. The model’s performance will be assessed with regard to the EOD and Zemel statistic, which measure the difference in false negative results and average predicted risk, respectively, between White and non-White and male and female patients under the null hypothesis of no difference. Aim 2: Model performance will be compared to modified versions of three other commonly used ACS risk stratification scores: the Global Registry of Acute Coronary Events (GRACE) score; the Platelet glycoprotein IIb/IIIa in Unstable angina: Receptor Suppression Using Integrilin (eptifibatide) Therapy (PURSUIT) score; and the Thrombolysis in Myocardial Infarction (TIMI) score, some of which have been shown to perform differentially across gender and race. EOD and the Zemel statistic will also be assessed as a measure of bias for the MLA, GRACE, PURSUIT and TIMI scores. Future Directions: The MLA will be implemented in live hospital settings for prospective evaluation.Narrative Race- and sex-related differences in the diagnosis and treatment of acute coronary syndrome (ACS) are well documented and can influence patient outcomes. Conventional risk stratification scoring systems used to guide resource allocation and treatment decisions may reflect racial and sex bias in their design and, in some cases, in their implementation. In response to the need for unbiased ACS clinical decision support tools, it is proposed herein to use machine learning to develop and validate an unbiased ACS prediction tool on the basis of electronic health record data to promote accurate and equitable recognition of ACS.