/ Uncategorized / By

Clarifying the Role of Risk Assessment

March 10, 2014 | Chris Baird, Board Member, NCCD

baird

The recent National Council on Crime and Delinquency (NCCD) evaluation of risk assessment instruments used in juvenile justice was unique on two counts. First, it was uncommonly comprehensive, testing eight different assessment models using data from 10 agencies. In addition, other researchers, including representatives of firms that market a number of the models tested, served on an advisory board that had both a role in framing the study and an opportunity to respond to NCCD findings.

The recent National Council on Crime and Delinquency (NCCD) evaluation of risk assessment instruments used in juvenile justice was unique on two counts. First, it was uncommonly comprehensive, testing eight different assessment models using data from 10 agencies. In addition, other researchers, including representatives of firms that market a number of the models tested, served on an advisory board that had both a role in framing the study and an opportunity to respond to NCCD findings.

It is no secret that NCCD has questioned the value of many recent developments in risk assessment. Our concerns were first published in 2009 in A Question of Evidence: A Critique of Risk Assessment Models Used in the Justice System. The results of the current study added to our concerns and provide the opportunity for additional commentary.

Risk assessment can be invaluable, separating offenders into risk groups with very different rates of recidivism. This allows agencies to devote their limited resources to high-risk individuals and optimize their prospects for improving outcomes. But risk assessment is not precise. It can assign an individual to a risk group with, for example, a historical 60% rate of recidivism; but it cannot, with any precision, identify individual offenders who will and will not re-offend within each risk group.

Virtually all fields face this same conundrum. In a recent New York Times opinion piece, Paul Kalanithi discussed the problem physicians face when presenting information to patients. Using Kaplan-Meier survival curves, doctors know how long, on average, a patient at a specific stage of disease will live; they cannot tell an individual where he/she is in the curve. Thus doctors are increasingly reluctant to estimate survival times for individuals. Kalanithi concludes, “It’s impossible, irresponsible even, to be more precise than you can be accurate.”

Yet, over the last 10 to 15 years, the terminology associated with risk assessment in the justice field has shifted to imply precision. Risk classification has become risk prediction. Needs are now “criminogenic,” implying causation rather than simple correlation. Correlations are often referred to as “effect size” and relationships gleaned from group data are used to drive case planning for individual offenders. Some models (the YLS-CMI is an example) claim they address “responsivity;” that is, they match treatment recommendations to offender characteristics and learning styles. Clearly, nothing in these systems provides such guidance, and prior research indicates that effectively addressing responsivity requires more in-depth analysis of the individual than these models offer. In effect, all of the above terms imply far more than what can be legitimately concluded from current risk assessment models.

The real question is this: “Is this trend toward terminology that implies greater precision simply a matter of semantics or does it represent a more significant problem?“ For years now, workers have been told to concentrate on a select number of needs defined as criminogenic. However, needs identified as criminogenic are simply those correlated (albeit modestly) with recidivism. But a general relationship to recidivism does not mean the need is related to criminal behavior of an individual. In fact, most correlations between needs and recidivism are so limited that concluding that a specific need plays no role in the delinquent behavior of an individual may well be the safer assumption. Plugging risk assessment directly into case planning protocols represents a significant and unsupported leap in logic, conflating the roles of group data and assessment of individual needs. More problematically, this trend can lead to misuse of resources and ineffective case management.

Any earnest desire to identify needs to drive case planning efforts requires a move beyond general statistical inference to determine which needs are truly important to address for each case. Prior research has taken on this challenge (e.g., the I-Level system and Strategies for Juvenile Supervision). Both of these efforts indicate that far more analysis is required than simple identification of needs.

It is time to temper both the claims and expectations that accompany most “generation 3 and 4” risk instruments, to clarify the appropriate role of risk assessment, and to engage other methods for identifying needs that should be addressed for each individual.

Chris Baird served as executive vice president of NCCD, directing the Midwest office, from 1985 through 2010. Baird retired in 2013 and now serves on the NCCD Board of Directors. He has designed risk assessment, classification, and case management systems for child welfare, adult probation and parole, and juvenile justice systems. He developed and managed the National Institute of Corrections Model Probation and Parole program, which was implemented in 31 state agencies and hundreds of county probation departments throughout the United States. From 1990 to 1997, he directed NCCD’s Children’s Research Center, which developed risk assessment and decision-making systems used in child protection services for more than 50 state and county agencies in the United States and Australia. He and colleagues wrote a comprehensive evaluation of the system in Michigan assessing its impact on subsequent abuse and neglect. He directed and authored a national study funded by the Office of Child Abuse and Neglect (OCAN) that compared CPS risk assessment systems in four jurisdictions. He has authored numerous journal articles and other publications on research, program development, and management issues in child welfare, juvenile justice, and corrections.