The Future of Ethical Decision Making in Child Welfare

July 12, 2019 | Chris Scharenbroch

Twenty-five years ago, decisions in child protection were made by social workers without the aid of standardized tools. These decisions were often based on workers’ perceptions of a family’s circumstances. Data were not always used or even collected, and agencies struggled to promote consistent and equitable decision making.

Recognizing that structuring decisions could be an improvement over clinical judgment alone, Evident Change pioneered the use of actuarial risk assessment in child protection.

From the beginning, we knew that decision making in child welfare had to balance the goal of protecting children with the protection of people’s civil liberties. Another layer of complexity was added by the extremely high-stress environment in which child protection decision making happens. Thus, we learned quickly that improving decisions is far more complicated than simply offering a risk-based tool.

To positively affect decision making, we needed to understand the full continuum of child welfare practice and policy. Today, decades beyond the beginnings of our work in child protection, Evident Change staff have gained global experience and expertise in research, social work, administration, and training. Perhaps most importantly, we strive to be experts in facilitating conversations with child welfare agencies as they work to balance resources, priorities, and liabilities with equity values.

Because child welfare is so complex, we created the Structured Decision Making^® (SDM) risk assessment as more than an automated, purely algorithmic tool in order to support ethical and effective decision making.

The Purpose of SDM^® Risk Assessment

The SDM® risk assessment is designed to identify the group of families most in need of an agency’s services. Though it might seem simple, figuring out which families to serve is incredibly complex. It requires child welfare agencies to balance service opportunities with civil liberties; risk classification with agency values; and agency programming with agency resources. Evident Change has learned over time that tool developers alone should not force answers to these questions; agencies and their stakeholders must be at the table.

Getting It Right

Matching a decision-support tool to purpose requires us to think beyond predictive accuracy. From the onset, Evident Change understood that risk assessments developed from systems rife with disproportionality would forever need checks and balances. This meant developing an equity value standard, which we strive to apply to risk assessment and all our work. In short, this standard helps ensure that children and families, regardless of their race/ethnicity, have similar likelihoods of returning to the system. As we continue to learn, our standard evolves.

Decision Support That Makes a Difference

Staff must understand and trust the tool they use if it is to successfully support decision making. There is evidence that automated black-box tools are simply not followed because workers don’t understand or trust the information. If workers don’t trust the tool, they miss its purpose. If workers can’t learn from the tool, it offers no support in casework.

We believe that everyone—families, staff, community, administrators, researchers, and subject-matter experts—should collectively determine policy standards and application. Data and analytics must also be available and shared to understand how the tool impacts different populations.

For all this to be possible, a risk assessment must be transparent. We believe this is the only viable future for ethical risk assessment. We also believe that the SDM risk model—along with our staff and processes for development, implementation, validation, and continuous quality improvement—is the model best positioned to help agencies to do this work.

What Makes the SDM Risk Assessment Different? People

SDM risk assessment comes with trade-offs that agencies must consider regarding staff resources, but its benefits are what sets the SDM risk assessment apart from automated statistical models.

Evident Change works with agency staff to help develop and test the assessment. This allows us to incorporate their subject-matter expertise and gives them ownership of their tool.

Social workers complete the assessment in person so they are prepared to communicate its meaning to families. This also puts workers in the position to make informed overrides to the tool when necessary. Our analytics show that the risk classification is usually overridden between 5% and 10% of the time. These numbers reflect a reasonable rate of agreement with the tool, but also demonstrate that staff are using their own clinical judgment appropriately.

The SDM risk assessment creates the advantage of giving social workers the ability to take in information that would not be captured by an automated model. Families’ lives are complex, and it takes a trained social worker to observe and integrate the many factors at play.

Because people complete the SDM risk assessment (as opposed to an independently operating tool), agencies must invest in their staff by providing ongoing professional development and training. This kind of investment in workers is also a decision based on values. While training requires agency resources, the result is that workers can increase their skills in making difficult decisions, benefitting children and families as well as the agency.

We believe the staff resources necessary to use SDM risk assessment are offset by these value-adds to the utility and validity of the tool.

How Do We Know SDM Risk Assessment Works?

Twenty studies in peer-reviewed journals, as well as numerous validation studies in agencies using the SDM risk assessment, have shown that the SDM risk tool performs well. This includes research conducted in the United States and in countries abroad.

The performance of a classification tool like the SDM risk assessment can be analyzed in many ways, and no one method can tell us everything we need to know. A widely accepted standard for measuring performance leverages two metrics: 1) distribution (how many families are in each risk level), and 2) discrimination (how well the risk assessment works to classify families). We also use other metrics to examine the effectiveness of the tool, but we focus on distribution and discrimination because in our experience, they are the best measures to pursue a tool that is transparent, fair, equitable, and useful.

When a risk assessment can transparently and reliably identify families at highest risk of returning to the system, agencies can use that information to guide decisions about opening cases and providing services. A robust body of research confirms this is true of the SDM risk assessment.

The Future of Ethical Risk Assessment

Risk impacts families. Our curriculum will help staff have conversations with families about the risk assessment. We want their voices and experiences to be a part of the risk system. To include this voice, it will be even more critical going forward to have risk levels, risk items, and risk definitions that are transparent and reasonable, so that everyone has a common understanding of what risk means.

We believe the future of decision making in child welfare requires more transparency, not less. Helping families understand the system has been demonstrated to be a key factor in improved outcomes. A future where workers and families have even less understanding of the case process than they already do is a future headed in the wrong direction.

The staff of Evident Change are deeply committed to children and families. Every day in our work and conversations about research, practice, or policy, there is a common thread: How does this affect children, parents, people? Our bottom line will always be what supports the best decision making to benefit people involved in systems. Every child deserves to grow up happy and healthy; every family deserves respectful and effective services. For us, that means accurate, equitable, transparent risk assessment.
 

Chris Scharenbroch is associate director of research analytics at Evident Change.