Who Will Seize the Child Abuse Prediction Market?

By Darian Woods

Lisa Mayrose knew Florida’s Department of Children & Families needed to overhaul how it investigated phone calls reporting beaten and neglected children.

“We had a rash of child deaths,” Mayrose, the regional managing director of the department in Tampa, said in an interview with The Chronicle of Social Change.

The deaths were concentrated in Hillsborough County, where there were nine child homicides between 2009 and 2012. At the time, child protective services were contracted out to private youth services agency Hillsborough Kids. This contract was worth $65.5 million a year.

So in 2012, the department made changes. It commissioned a comprehensive analysis of the data behind the child deaths that were concentrated in Hillsborough County. Hillsborough Kids lost out on the $65.5 million contract and went into liquidation. A private youth services agency, Eckerd Youth Alternatives, was selected by the department to take care of approximately 2,900 abused children in Hillsborough County. The next year, Florida Governor Rick Scott boosted funding for new social workers. Perhaps most radically, a new decision-making tool called Rapid Safety Feedback was introduced in the county.

Rapid Safety Feedback uses — in the parlance of big data crunchers and, increasingly, social scientists — predictive analytics to prioritize calls of suspected child abuse.

Predictive analytics in child protective services means assigning suspected abuse cases to different risk levels based on characteristics that have been found to be linked with child abuse. These risk levels can automatically revise as administrative data is updated. Administrative data may be as simple as school reports or could delve deeper into other information that the state holds: the parents’ welfare checks, new criminal offenses or changing marital status.

Combining predictive analytics with more investigators seems to be producing results in Hillsborough County. According to Eckerd, who also holds contracts in Pasco and Pinellas counties, since it took over the contract in 2012, the quality of reviews has improved 30 percent. There is a significant increase in completed documentation by caseworkers. There have also been zero child homicides in the county since the handover.

Florida’s use of predictive analytics to better prioritize suspected child maltreatment is on the frontier of child protective services, and it is a frontier that is increasingly competitive.

A battle for better child maltreatment risk prediction is heating up around the United States and the world. The world’s largest privately held software company, the SAS Institute, has turned its attention to child wellbeing. Leading researchers are turning towards predictive analytics as a tool for child protective services.

Following success with reduced child fatalities in Florida, Eckerd is donating its new predictive tool, Rapid Feedback Mechanism, to Connecticut’s Department of Children and Families for one year. States such as Alaska and Maine are also working with Eckerd to improve their decision-making for children under investigation.

Texas is sharing data between its Department of Family and Protective Services and the Department of State Health Services, finding that linked data can predict risks of child fatalities. Both Los Angeles County and Allegheny County in Pennsylvania are investigating ways that predictive analytics can be used to prevent child maltreatment. Jurisdictions as far away as New Zealand are also navigating new territory in how administrative welfare data could be used to inform how child abuse calls are screened.

The child welfare predictive analytics pond is still relatively small, with a few key companies and non-profit agencies competing for contracts and hegemony in what promises to be a lucrative market.

Eckerd’s Rapid Safety Feedback in Florida 

The report that led to Eckerd developing the Rapid Safety Feedback tool analyzed calls to the Florida Abuse Hotline between 2007 and 2013. The facts were striking.

At a lower level, boys had a 47 percent higher risk of death than girls. Seventy-five percent of all child deaths occurred before the age of three. The children who had died were often in homes where substance abuse was a problem. Many of the parents were abused as children. Another risk factor was a boyfriend of the mother in the house.

Putting these factors into a new tool helped case workers prioritize.

“Previously there might have been a call about a child with a black eye, and the case would close if the investigator didn’t find any bruises,” Mayrose said. “Now we treat the family like a moving picture and really look at the experience of the child within the family.”

In 2014, the department also boosted the number of social workers on the front line. This allowed an additional 171 child protective investigators to be hired to spend more time with high-risk families. This was nearly a 20 percent increase and allowed Florida, through the Department of Children & Families’ contract with Eckerd, to change the way it dealt with suspected child abuse.

“If the family falls into a high-risk category, we’re going to heighten the case for a more experienced worker, and they are going to independently read the case,” said Eckerd’s Senior Quality Director, Bryan Lindert.

There have not been any child homicides in the Tampa area since the changes. Although, given the rarity of these events, a lapse in child deaths could be as much anomaly as anything else.

“I never try to claim causality,” Lindert said.

Because resources and attention also were increased when Rapid Safety Feedback was implemented, it is hard to tease out exactly what impact Rapid Safety Feedback has had. Nevertheless, other states are working with Eckerd to bring this kind of change to their child protective services. Eckerd is currently working with Connecticut, Alaska and Maine to apply Rapid Safety Feedback to their child safety investigations. Further counties and states such as Oklahoma and Nevada are also discussing partnering with Eckerd.

The Department of Children & Families in Florida is positive about how data-driven decision-making has improved child protective services in its state.

Regional Managing Director Lisa Mayrose said that data could be used to save children’s lives even beyond child protective services. She said that more child drownings in certain ZIP codes could allow community training and prevention to focus in these areas.

“If we’re not using predictive analytics to drive our attention, then we’re not using our resources effectively,” she said.

A Short History of Statistics in Child Welfare

Using data to solve social problems is not new. In President Herbert Hoover’s 1930 address to the White House Conference on Child Health and Protection, he shed light on the role of data.

“Statistics can well be used to give emphasis to our problem,” Hoover said. “Out of 45 million children … 6 million are improperly nourished … 1 million have defective speech, [and] 675,000 present behavior problems.”

President Lyndon Johnson’s War on Poverty heralded vast new sums for the research of social problems in the 1960s.

“Johnson gave so much balance to the numbers guys,” said author Joe Flood.

Flood’s 2010 book, The Fires, documents the rise of the fields of operations research, game theory and systems analysis and how lessons from mathematics and defense were applied to solving issues of poverty and urban problems in the 1960s and 70s.

Predictive analytics has its roots in these fields applying statistical techniques to social problems, but it is a much faster, operationalized, and real-time version of the kind of statistics Hoover was talking about. Instead of simply identifying 6 million improperly nourished children, the child protective services sector has the potential to predict, with stomach-knotting accuracy, who those 6 million children will be ahead of time.

Recognizing this potential, large private firms want a stake.

The Statistical Analysis System (SAS) Institute

The common engine behind the work in Florida, Connecticut, and Los Angeles is the database software called Statistical Analysis System (SAS).

The SAS Institute is not a traditional child protective services partner. SAS began in 1976 as researchers at the North Carolina State University sought to improve agriculture crop yields. With its software used as the backbone for large datasets all over the world, the SAS Institute is now the planet’s largest privately held software company. In 2014, it made over $3 billion in revenue.

Business analytics is a $14.4 billion a year business globally, dominated by market leaders like SAP, Oracle, and IBM. SAS faces strong competition from these companies. There is even a free, open source software package similar to SAS called R, which draws away paying business.

In 2009, The New York Times reported on the company’s prospects.

“We know we have to change — no question about it,” SAS senior vice president Jim Davis said in the article.

So while the company has traditionally focused on business and academia, SAS is promoting the use of its software to solve social problems like child abuse, and has been expanding its connections to state and local government.

“We believe we can truly change outcomes for kids,” said Kay Meyer, a principal industry consultant in SAS’s state and local government practice division.

According to its website, SAS’s state and local government practice brings in $100 million in revenue each year. Government is a growth industry for SAS: this practice has been growing at double-digit rates every year for the last five years. $100 million is a large amount of money; in state and local government information and communications technology contracting, the potential for growth is even greater.

The SAS Institute aims to lead social analytics, and is pursuing what it sees as a wave of opportunities in state and local government.

“It’s hard to quantify at this point,” Meyer said. “But as these issues have become more and more well-known to the public, the greater the public outcry for solutions.”

As SAS expands this role, it is hiring leading experts in the state and local government sector, including professionals from child protective services. Before moving to SAS, Meyer worked for North Carolina’s state government where she was the program director for the North Carolina Statewide Data Integration program. SAS has also recently recruited Will Jones, an expert with the title Child Well Being Specialist.

Until April this year, Jones worked as chief of programs at Eckerd Youth Alternatives. Seeing the potential for predictive analytics, he moved to SAS.

“We’re at the very beginning of utilization of predictive analytics,” Jones said. “I think fifteen years down the line, I’d like to see every state provider and every private provider using a tool.”

Jones believes predictive analytics could help prevent child deaths in foster care, assist with foster children aging out of care, and even steer vulnerable youth away from committing crime if the problem is identified early. Jones suggested that doing the job better might also have fiscal benefits for state and local government.

“The answer may not be that we need more social workers,” Jones said. “I’m not a big believer in more workers as a standalone solution.”

While faster processing power and cheaper data storage means that the quantity of data available in public officials’ hands is larger than ever, some warn that predictive analytics based on real-time data, while shinier, may be no stronger than existing decision-support tools such as the long-time leader in this now-hot market of risk assessment: Structured Decision Making.

The Old vs. the New: Structured Decision Making vs. Predictive Analytics

Structured Decision Making (SDM) is used in jurisdictions in over 20 states, and it is widely considered the industry benchmark. That position may change as new child abuse screening tools are competing fiercely to be used across the country.

“When I see predictive analytics that’s more robust than what we’ve done, I’ll say that’s got some promise. So far I haven’t seen anything that can outperform what we’ve got,” said Raelene Freitag.

Freitag is the director of the National Council on Crime & Delinquency’s Children’s Research Center, which produces the SDM.

SDM is, in essence, a form. It is a list of prompts like: “Primary Caretaker has Historic or Current Alcohol or Drug Problem” (1 point,) “Current housing is physically unsafe” (1 point,) and “Domestic Violence in the Household in the Past Year” (2 points.) These tally up to a final score of risk. The final score could mean the difference between children staying with their family or being taken into care.

“It’s designed for key decision moments,” Freitag said.

This tool is a static snapshot; unlike predictive analytics it is not updated in real time. While not making use of real-time administrative data to the same extent as predictive analytics, one advantage of SDM is that it has been thoroughly researched.

A 2000 review of decision-support tools in child protective services by the National Council on Crime & Delinquency finds support for actuarial-based systems like SDM. This was reinforced in 2004 by Director of Evaluation at Alameda County Social Services Agency, Will Johnson, who tested the validity of Structured Decision Making as used in the largely urban county east of San Francisco. Some studies have been less favorable, most notably in regards to SDM’s implementation in South Australia and Queensland, Australia.

Freitag believes the less positive reviews from Australia were due to poor training and not the tool itself.

“We didn’t know the first thing about implementation science when we went into Queensland,” she said. “Training is a good first step, but without coaching you won’t get practice change.”

Decision-making Across the United States

Across states, attitudes towards child protection vary. One challenge for any tool like SDM is how its recommendations integrate into state legislation and cultural expectations of when a family should be investigated further after a call comes in.

A new Children’s Bureau report, released in January 2015, shows vast differences in how many calls are further investigated across states. The report shows that the proportion of child maltreatment cases screened out of the system varied from zero in Illinois, New Jersey and North Dakota to 74 percent in Vermont, 71 percent in Minnesota and 83 percent in South Dakota. That means that for every 10 calls going to South Dakota’s Child Protective Services, only two will be investigated further.

Freitag said that there are many factors that explain this difference, some of which can be addressed by tools like SDM.

“What do people in the community call in about? You will see some variation in that,” she said. “Some people are loathe to invite government intervention. Some other people are more into punitive responses to bad parenting.”

There are also changes in screening practices over time, especially after a child fatality.

To address cultural, legislative and procedural differences among states, whenever SDM is brought to a new jurisdiction, the Children’s Research Center brings together a work group consisting of a majority of line staff as well as supervisors and managers. This allows new guidance to accompany SDM depending on the legislation in the new jurisdiction.

Discretionary over-ride is also possible, although this is rare.

“There’s always a balance between professional judgment and what the tool says,” Freitag said. 

Social Workers and Statisticians

Joe Flood’s book, The Fires, mentioned earlier, describes how the RAND Corporation’s partnership with New York City leaders obsessed with statistics created an imbalance of perspectives and suggests that this was associated with the burning of the homes of hundreds of thousands of New York residents.

“When you have by-the-gut traditionalists and spreadsheet jocks with thick glasses not talking to each other, you have a problem,” Flood said.

Flood is optimistic that applying statistics to child protection can open up new insights that have previously been left hidden. The problem comes when the knowledge and judgment of fieldworkers is ignored.

“I’ve heard people saying we’re moving from the age of Big Data to the age of smart analysis,” he said. “But that’s the age we should have been in all along.”

One of the Hardest Jobs to Imagine

Social workers must often choose between two traumas: the trauma of removing a child from their family or the trauma of continued abuse. It’s a decision that some argue should not be left to human judgment alone.

“We need to use science to demonstrate what is working, what is data and research telling us,” said Child Well Being Specialist at SAS, Will Jones.

The decision to remove a child from a home is one of fraught trade-offs, cultural complexity and risk assessment.

Massachusetts Institute of Technology Associate Professor Joseph Doyle finds that Illinois children who remained in the home fared better on a range of indicators. The children who were not taken into foster care had fewer arrests, higher earnings, lower teen-pregnancy rates and lower unemployment in later life compared with children in similar home situations who were placed in foster care. Yet many children will be at risk of further abuse if left in the home.

“Being a case worker is one of the hardest jobs I can imagine,” SAS industry consultant Kay Meyer said.

Even in the most egregious cases of child abuse, where the decision is clear that a child should be taken out of a home, the removal can still be traumatic for the child. Freitag, of the National Council on Crime & Delinquency, recalled an episode earlier in her career that illustrated this tension.

As a patrol officer in Milwaukee, she was called into an apartment at 2 AM. There was no food in the kitchen but spoiled milk and cockroach-infested Cheerios. She had to keep walking to prevent the mice and cockroaches from crawling up her legs. Five children and babies were alone. This was the worst case of child maltreatment that Freitag had seen.

“I remember how the kids screamed when they were taken from their home,” Freitag said.

It was not just the abject conditions of the house and the children that give Freitag reason to recall the painful details from that night. It was what happened unexpectedly later that week at the court house: When the mother appeared for her charge of neglect, her children came running down the hall to embrace her. They missed her and wanted to be with her.

“We oughtn’t get into positions where we argue either for child rescue or child preservation,” Freitag said. “What we need to realize is that either could be the right answer for this child in this moment in time.”

Decision-support tools offer assistance to social workers and child abuse investigators. Supplementing professional judgment and point-in-time tools like SDM with predictive analytics may not be a panacea, but it could help. Bryan Lindert, a senior quality director at Eckerd, sees the picture holistically.

“We put a lot of emphasis on the tool, and not the artisan who’s using the tool,” Lindert said. “No matter how good your paints are, what matters is the decision made by the artist.”

Testing the Test

Whichever tool used for the prevention of child abuse ends up dominating the market, there is still the question about how to judge whether it has been effective.

“Just because there’s new data or new findings, that doesn’t change the standards by which any new tool should be judged, and these need not be lost in the excitement around predictive analytics,” said Director of Research at the National Council of Crime and Delinquency, Jesse Russell.

Russell, whose organization created, researches and actively promotes Structured Decision Making, set out four criteria for testing new tools.

According to Russell, tools should be tested for validity, reliability, equitability and utility. Validity means that a tool should first get the answers as close to right as possible. Second, it should be able to be done consistently: it should not matter whether the prediction was done on a Tuesday or a Thursday, in winter or summer. Third, the tool should explicitly address equity concerns: does this tool embed racial or socioeconomic inequities or does it alleviate them? Finally, the answers have to be useful in practice.

“If you don’t have the capacity to change practice, then it’s not helpful,” Russell said.

With the child protective services sector on the cusp of a new world of decision-support tools, Russell considers the dialogue between front-line workers, managers, statisticians and politicians to be constructive.

“Everyone is at the table,” Russell said. “The mood is generally one of excitement.”

The standard for child protective services decision-making is rapidly changing. Since 2013, Eckerd’s Rapid Safety Feedback has been spreading across the country. Researchers and departments are linking administrative data with child protective services information at a scale that has not been possible until now. SAS’s statistical software is now underpinning many risk-assessment tools, and the company is expanding into the child protective services realm. For now, Structured Decision Making maintains its position as the standard tool.

The discussions happening now among policymakers, statisticians, social workers and communities about the future of child protective services decision-making appear to be held in good faith overall.

These conversations are only going to get tougher as big tech companies, sensing fresh markets, ramp up competition with SAS, as communities debate state surveillance, as communities of color learn whether these tools will improve or exacerbate higher child removal rates, and as social worker judgment starts to clash with statistical output. One high-profile child death may derail everything.

For the sake of the child whose life depends on the right decision—to stay with the family or to be taken into foster care—keeping those conversations open will be critical.

Darian Woods is a Master of Public Policy candidate 2016 at the Goldman School of Public Policy at University of California, Berkeley. He is an editor at the School’s PolicyMatters Journal. This piece was written as part of the graduate course, Journalism for Social Change.

Written By Chronicle Of Social Change

Who Will Seize the Child Abuse Prediction Market? was originally published @ The Chronicle of Social Change and has been syndicated with permission.