/ Uncategorized / By

Risk Assessment, Will You Be My Valentine?

February 12, 2015 | Colleen Kerwin, Researcher

Colleen Kerwin

It’s that time of year again—love is in the air and reservations are being made at all the posh restaurants in town. Time to put on that favorite dress or suit and have a special evening with your significant other (or a fun single friends’ night out). The trouble is, it’s been a while since you’ve worn this favorite article of clothing and it doesn’t seem to be fitting quite right. Time to go to the tailor.

The truth is, risk assessments are just like your favorite dress or suit. At one point they fitted their population perfectly, but over time people—and populations—change. Just as it’s necessary to tailor your favorite dress or suit from time to time, it is necessary to “tailor” risk assessments.

An ill-fitting outfit can really undermine a person’s confidence, distracting her or him from whomever they are interacting with as they worry about their appearance. Ill-fitting risk assessments can just as easily undermine an agency’s ability to properly serve clients. Using a risk assessment that is not finely tuned can cause problems such as misappropriated services and interventions, unmanageable workload, and inequitable decision making.

So how do you know if your favorite dress or suit doesn’t fit? It seems like a silly question, because you just know. You look at the hems and they are too high or too low. The waist is baggy or tight. Just as your favorite clothing has its essential points that must fit, risk assessments have three essential measures of fit: distribution, accuracy and equity.

Distribution

Distribution is the grouping produced by the risk assessment. It can be examined by calculating the percentage of the population classified into each risk level. A distribution that fits will be functional as it relates to practice. A distribution should not be lopsided or have some groupings that are too small or too large. If a risk assessment produces a classification that is not informative or reasonable with respect to the availability of intervention or services, it does not fit.

Accuracy

Accuracy refers to how well the risk assessment classifies clients by their likelihood of a recurrence. To measure it, we must choose a data outcome to examine. For example, risk assessments used in juvenile justice are designed to classify clients by their likelihood of subsequent referrals/arrests, petitions, adjudications, and placements. When a risk assessment is accurate, low-risk clients will, on average, experience less recurrence than moderate-risk clients; moderate-risk clients will experience less recurrence than high-risk clients.

Equity

Equity is the degree to which the risk assessment performs without bias across different subgroups of the population. To look at equity, we use the measures of distribution and accuracy to gauge whether populations with similar rates of recurrence are distributed similarly by risk level regardless of race/ethnicity, gender, or location. Ideally, classifications of low, moderate, and high risk will have similar distributions and accuracy across subgroups. If the risk assessment is not performing equitably, clients in different sub-groups with similar risk of recurrence may not be classified in the same risk level.

By checking these three measures you can tell if your risk assessment is in need of tailoring to its current population. This process is sometimes referred to as “norming.”

When you have decided your dress or suit does not fit and needs to be tailored, what do you do? You take it to someone with the expertise to fix it. It’s the same with your risk assessment. Once you have decided the distribution is off or the equity needs to be improved, you present your risk assessment and measures to someone with the expertise to tailor, or norm, your risk assessment.

To norm a risk assessment, researchers study a cohort of clients, using a risk assessment, and compare them by how often they return to the system. Statistical techniques are used to explore potential revisions or modifications to better fit the assessment. Through this process, the risk scoring may be adjusted or risk factors may be added, dropped, or re-weighted based on contribution to the classification. This process is repeated until a valid assessment instrument with appropriate distribution, discrimination, and equity is developed.

Tailors provide the technical skills to make your clothing fit, but without communicating with you they would have no idea what to fix. When norming a risk assessment, there must be communication between the researchers and a stakeholders group. Although the norming process is informed by data analytics, any modifications or revisions to the assessment should be vetted by an expert stakeholders group. The stakeholders oversee the implementation and use of the risk assessment in practice. This expert group determines how to best implement the modifications and revisions to reflect local policy.

Your favorite dress or suit has gone to the tailor and your confidence in its ability to turn heads has never been higher. Taking the same time and care to periodically examine and norm your risk assessment can produce the same confidence in your assessment’s ability to properly serve clients.

So consider a slightly unconventional valentine this year, and give your risk assessment the TLC it deserves.