Placement Success Predictor: Using Site-Customized Machine Learning Models to Predict the Best Level of Care Placement for Each Child's Behavioral Health Needs

ABSTRACTHundreds of thousands of children (669,799) were confirmed victims of maltreatment in the United States in 2017; in that same year, of the 442,733 children in foster care, 34% had been in more than one placement and 11% were in a group home or institution. Out-of-home placement decisions have extremely high stakes for the present and future well-being of these children because some placement types, and multiple placements, are associated with poor outcomes. With the Family First Prevention Act, states will be required to pay the average $88,000 per year to keep a child in residential care if that high level of care is not authorized. But which children require --and, more importantly, would benefit-- from a placement in residential care? Decision-making support tools currently used by states to recommend specific level of care (LOC) placements for children do not maximize the rich data and innovative methodological approaches that are being explored in other fields like medicine. In addition, structured decision making (SDM) has been used to guide decisions about risk in child welfare settings but, in comparison to predictive modelling, SDM is limited by the use of a smaller group of factors to make recommendations. Outcome Referrals, Inc. has employed sophisticated machine learning techniques over the past 10 years to risk-adjust behavioral health outcome data for clients using baseline characteristics. Initial models predicted more than 30% of the outcome variance (i.e., it was possible to predict 30% of the variance in how depressed a client would be at follow-up). The next model improved that prediction to more than 50%, and our latest model has increased this to an average of 71%. With the assistance of Phase I NIH SBIR funding, we plan to improve the success rates of children in the child welfare system with an innovative, scientifically-derived product called “Placement Success Predictor.” To guide level-of-care decision-making, this product will use site-customized, machine learning algorithms to predict the likelihood of an adolescent having a good outcome in a particular placement type in a specific community. We have preliminary evidence supporting the feasibility of developing these models based on work supported by the Duke Endowment Foundation. During this six-month Phase I project, we propose to 1) validate these preliminary machine learning models by applying them to new client data from our partner behavioral health organization, 2) explore options for sharing results of these models to facilitate their use in practice (e.g., aggregate predictions across different domains in a weighted way), 3) assess key stakeholder satisfaction with a new prototype, and 4) develop and test customized models for multiple placement types with a state-wide child welfare and juvenile justice dataset.PROJECT NARRATIVEHundreds of thousands of children were confirmed victims of maltreatment in the United States in 2017; in that same year, of the 442,733 children in foster care, approximately one out of 10 were placed in a group home or institution. Out-of-home placement decisions have extremely high stakes for the present and future well-being of these vulnerable children because some placements, and multiple placements, are associated with poor outcomes. The likelihood of success recommendations provided by the proposed “Placement Success Predictor” tool will help placement staff and administrators identify the best placement setting for each child using machine-learning statistics to predict the child’s chances of success in each potential treatment setting.