Machine Learning Improves Accuracy of Risk Predictions and Improves Operational Effectiveness


“Predictive analytics allows our team to work smarter, and more efficiently and leads to better targeted outreaches and outcomes for our patients.”

– Mary P. Leahy, MD
CEO, Bon Secours Charity Health System


The Centers for Medicare and Medicaid Services (CMS) readmission penalties are a significant concern for healthcare organizations, with over 2,500 hospitals being penalized each year, resulting in CMS withholding more than $500 million in payments.

For Westchester Medical Center Health Network (WMCHealth), hospital readmissions carried more than financial consequences. Care managers had to use multiple systems and time-consuming, manual processes to identify recently discharged patients at risk for readmission. These processes limited the effectiveness of the care management team, as care managers lost valuable time searching patient records for data needed to prioritize their workload and choose the right interventions.

To address this problem, the data analytics teams at WMCHealth and network member Bon Secours Charity Health System (BSCHS) leveraged artificial intelligence and machine learning to develop a more accurate readmission risk prediction model that would enable care managers to use their time coordinating and engaging with patients more effectively. Results include:

  • A risk prediction model that is 17 percent more accurate than widely used readmission risk models in identifying patients at high-risk and low-risk for readmission within 30-days.
  • Care managers obtain follow-up appointments faster, usually within seven days, and connect patients with the services needed to prevent unnecessary visits to the emergency department and readmissions to the hospital.
  • 1,327 hours per year saved, freeing up care managers to spend more time with patients.


Nationally, more than 2,500 hospitals face readmission penalties from CMS each year, with CMS withholding more than $500 million in payments.1  While CMS’ Hospital Readmission Reduction Program decreased CMS’ spending on readmissions by $9 billion by 2014, the rate of hospital readmissions only dropped 0.1 percent on average between 2013 and 2016—hospital readmissions remain a challenge.2

Many organizations and EMRs use the LACE index (length of stay, acuity of admission, Charlson Comorbidity Index, and the number of emergency department visits in the preceding six months) to predict which patients are at the highest risk for readmission. The index was developed using data from 4,812 patients admitted to 11 hospitals in Canada between October 2002 and July 2006.3 While the LACE index is widely used, it was developed using data from middle-aged Canadian patients who did not have serious comorbidities. Therefore, critics have questioned the validity of the LACE index in its applicability to broader patient populations.4,5

BSCHS is a three-hospital health system in the lower Hudson Valley of New York and a member of WMCHealth. WMCHealth has evolved since 2014 from a single tertiary care hospital and trauma facility into a 1,700-bed healthcare system with ten hospitals on eight campuses spanning 6,200 square miles of the Hudson Valley. Given the financial implications of readmissions and the concerns about the accuracy of the LACE index for predicting their occurrences within its patient population, WMCHealth examined its processes to see where they could be improved.


To deal with the challenges associated with effectively managing a growing population of patients with chronic diseases, care managers at BSCHS would attempt to identify which of its recently discharged patients were at highest risk for readmission, engaging with the patients to coordinate care.

Efforts to identify risk were complicated by the multiple systems that care managers had to use, and the time-consuming, manual processes required to identify recently discharged patients at risk for readmission. As a result, care managers were less effective, spending time searching patient records for the data they needed to prioritize their workload and choose the right interventions for their patients.

Seeking a more efficient process, BSCHS attempted to automate patient identification, generating a daily report to work from; however, this report did not provide complete information on discharged patients. There was a risk-scoring feature, but it was more reflective of mortality (with a focus on comorbidities) rather than the risk of readmission, which BSCHS was attempting to pinpoint. Furthermore, the generic algorithm failed to take into consideration the characteristics of the BSCHS patient population.

BSCHS needed a different mechanism for identifying patients at risk for readmission, as its labor-intensive process was inefficient and negatively impacted care managers’ patient engagement opportunities.


BSCHS and WMCHealth data analytics team partnered with Health Catalyst, using the Health Catalyst® Data Operating System (DOS™) Platform,™, and™ to develop a more accurate readmission risk prediction model that would enable its care managers to optimize their time spent coordinating care and engaging with the right patients.

This new solution incorporated the historical data on 54,000 patient discharges and 6,000 inpatient readmissions within 30-days of inpatient discharge to train the machine learning platform and to identify specific variables significant to the BSCHS patient population.

The team used an open source machine learning package to compare several algorithms tuned over many hyperparameters to arrive at the final predictive model, which was developed using the random forest algorithm.

After running the hospital’s historical data through the machine learning algorithm, the team used the analytics platform to leverage billing and clinical data to perform trials to validate the model’s predictions against various patient cohorts, which also ensured utility across sub-groups. In the final readmission risk prediction model, 24 variables were included.

When the team was confident that the risk predictions were as accurate as possible, risk scores were supplied to the care managers as part of the Health Catalyst Community Care Advanced Application.

The analytics application enables BSCHS care managers to fill critical gaps in chronic and preventative care through the creation of a new readmission reduction module, making valuable information available to care managers within an application and workflow they routinely use to complete their work. Care managers can easily access lists of recently discharged patients, and their readmission risk score.

To make the patient lists and risks scores more actionable, BSCHS paired them with EMR data that also informs care managers about interventions. Care managers are now able to review the discharge list, risk scores, and supporting data elements in a single visualization.


By utilizing its analytics platform, in coordination with a machine learning algorithm, BSCHS has built a knowledgeable and accurate model that best reflects its patient population. This has resulted in:

  • A risk prediction model that is 17 percent more accurate in identifying patients at high-risk and low-risk for readmission within 30-days. This was driven by an eight percent increase in true positives (actual readmissions correctly classified as “high-risk”) and a 30 percent decrease in false positives (actual non-readmissions incorrectly classified as “high risk”).
    • The increase in accuracy was measured in relation to BSCHS’ existing EMR risk model and the LACE index. Seventy-two percent of the machine learning model’s predictions were accurate, compared to 62 percent of the LACE scores and 61 percent of the EMR scores. The difference is attributed to two factors: the machine learning model used more variables than the other two models, and it better reflects the characteristics of the BSCHS patient populations. The other risk models use generic algorithms that are not trained on data for a specific population.
    • Using three months of retrospective data not used during model training, the machine learning model correctly classified 432 more discharges than the LACE model and 460 more than the EMR model.
  • Discharge lists and risk scores that provide care managers with objective data they can use to prioritize their daily patient outreach. Care managers utilize the information on high-risk and moderate-risk patients in conversations with primary care providers and specialists. As a result, follow-up appointments occur faster, usually within seven days, and patients are connected with the services needed to prevent unnecessary visits to the emergency department and hospital readmissions.
  • 1,327 hours per year saved. Care managers can now spend more time with the right patients, without spending hours performing manual chart review prior to engaging with them.


WMCHealth plans to expand the use of the risk prediction information to additional locations and to improve the machine learning model further. The team also intends to integrate social determinants of health data into the predictive model, continuing to improve population health management and patient outcomes.


  1. Advisory Board. (2017). 2,573 hospitals will face readmission penalties this year. Is yours one of them? Retrieved from
  2. Castellucci, M. (2017). Results of CMS’ readmissions program has hospitals, experts questioning its purpose. Modern Healthcare. Retrieved from
  3. Canadian Medical Association Journal. (2010). New tool to predict early death or hospital readmission. ScienceDaily. Retrieved from
  4. Cotter, P. E., Bhalla, V. K., Wallis, S. J., & Biram, R. W. (2012). Predicting readmissions: Poor performance of the LACE index in an older UK population. Age Ageing, 41(6), 784–789. Retrieved from pubmed/22644078
  5. Tan, S. Y., Low, L. L., Yang, Y., & Lee, K. H. (2013). Applicability of a previously validated readmission predictive index in medical patients in Singapore: A retrospective study. BMC Health Services Research, 13, 366. Retrieved from


Loading next article...