Placement Success Predictor: Using Site-Customized Machine Learning Models to Predict the Best Level of Care Placement for Each Child's Behavioral Health Needs

ABSTRACT Hundreds of thousands of children (669,799) were confirmed victims of maltreatment in the United States in 2017. In that same year, 34% of the 442,733 children in foster care had been in more than one placement and 11% were in a group home or institution. Out-of-home placement decisions have extremely high stakes for the present and future well-being of these children because some placement types, and multiple placements, are associated with poor outcomes. But which children require --and, more importantly, would benefit-- from a placement in residential care? Decision-making support tools currently used by states to recommend specific level-of-care (LOC) placements for children do not maximize the rich data and innovative methodological approaches that are being explored in other fields like medicine. In addition, structured decision making (SDM) has been used to guide decisions about risk in child welfare settings but, in comparison to predictive modelling, SDM is limited by the use of a smaller group of factors to make recommendations. Outcome Referrals, Inc. has employed sophisticated machine learning techniques over the past 10 years to risk-adjust behavioral health outcome data for clients using their current characteristics. We have evidence of the predictive validity of this approach for generating risk-adjusted scores via the PCORI-funded RCT in which clients were matched with clinicians who had previous clients who did well in treatment. With the assistance of NIH SBIR funding, we plan to improve the success rates of children in the child welfare system with an innovative, scientifically- derived tool called “Placement Success Predictor” to guide level-of-care decision-making. This product will use machine learning algorithms to predict individualized outcomes for children and adolescents in a particular placement type. During the Phase I project, we developed and validated machine learning models to predict the probability of each youth’s success within placement types and conducted usability testing of a prototype to assess potential barriers to implementation. The likelihood of success recommendations provided by the machine learning models developed for the “Placement Success Predictor” clinical decision-making support tool were associated with improved youth well-being (i.e., those children that were placed in the Placement Success Predictor-recommended placement had four times higher success rates). Potential future users found the prototype of this tool easy to use. The machine learning models appear to distinguish clients who will do well in various placement types and suggest that only 10% of clients would be successful in higher-cost level of care. With the Family First Prevention Act, states are now required to pay the average $88,000 per year to keep a child in residential care if that high level of care is not authorized. We submit this tool could help improve overall outcomes and reduce healthcare costs when and if high-cost residential care options are judiciously provided to children who are most likely to benefit from them. In Phase II we propose to develop this flexible program to generate placement recommendations per client based on automatically updated custom, site-specific machine learning prediction models then assess its efficacy and effectiveness.