Highlights:
- This report discusses findings from a well-conducted randomized controlled trial (RCT) of program services funded by the Department of Justice’s Second Chance Act (SCA) Adult Demonstration Program, aimed at facilitating the re-entry of prisoners into the community.
- The RCT found that the SCA services had no significant effect on the primary outcome of reincarceration over the 30 months following random assignment (60 percent of the SCA group were reincarcerated versus 59 percent of the control group, a difference that was not statistically significant).
- However, the SCA-funded services included a diverse array of efforts (e.g., case management, employment assistance, substance abuse treatment). It is very possible that some were effective but their effect was diluted by others that were ineffective or harmful, resulting in an average effect near zero. Such a pattern occurs in other policy areas.
- We therefore propose evidence-based reforms to SCA, similar to those we’ve advanced in other policy areas, that focus on a central goal: identifying a subset of specific re-entry strategies rigorously shown to produce important effects on recidivism and other key outcomes.
- In a world where most attempts to make progress fail and a few succeed, such evidence-based reform is essential if we hope to improve the lives of justice-involved and other at-risk populations.
- A thoughtful comment from the lead study author, and our brief rejoinder, follow the main report.
In this report, we discuss new findings from an evaluation of program services funded by the U.S. Department of Justice’s (DOJ) Second Chance Act (SCA) Adult Demonstration Program. Under the SCA, DOJ has awarded approximately $700 million since 2009 to state and local agencies and community organizations to help individuals returning to the community after incarceration. The Adult Demonstration Program is among the largest of SCA’s grant programs. In 2010, DOJ commissioned an RCT to evaluate the effectiveness of SCA services delivered by seven of the 15 state or local agencies that received funding in 2009—the first year of the program.[i] The seven sites had approximately two years to develop and implement the SCA services, after which the RCT got underway.
This was a high-quality, clearly-reported RCT, and its findings are disappointing. Our full two-page review of this study and its findings is linked here. The following is a short overview:
This was a well-conducted RCT that measured the effects of services funded by the SCA Adult Demonstration Program to facilitate the re-entry of prisoners into the community. The study randomly assigned 966 adults being released from prisons and jails in seven program sites to either a treatment group that was offered SCA-funded re-entry services or to a control group that was not (but still could access other available services in the community). The SCA services varied across the study sites and included case management, employment assistance, substance abuse treatment, cognitive behavioral therapy, housing assistance, and other supportive services.
The study found that SCA services at the seven sites had no significant effect on the pre-specified primary outcome of reincarceration over the 30 months following random assignment. Specifically, 60 percent of the SCA group were reincarcerated versus 59 percent of the control group, a difference that was not statistically significant. The study found significant effects—some adverse, some positive—on a few exploratory outcomes (e.g., an increase in the number of re-arrests over the 30 months, and an increase in workforce earnings in the eighth quarter after random assignment). However, such findings are only suggestive in nature as they could have appeared by chance due to the study’s measurement of numerous exploratory outcomes.[ii]
Source: D’Amico and Kim 2018 (link).
We highlight these findings not just because of their importance for re-entry programs and policy, but because they point to what we believe is a central (but fixable) flaw in government’s approach to funding social programs more generally. The SCA is typical of many federal social programs, in that it awards grants to state and local agencies to expand or supplement a diverse array of existing services in those jurisdictions. This is a logical approach if one assumes that many of these services are effective, in which case the federal grants—by supplementing such services—should lead to positive outcomes.
Unfortunately, that’s not the world we live in. We live in a world where most program services and strategies (“interventions”), when rigorously evaluated, are found not to produce the hoped-for improvements in participants’ lives. In an earlier Straight Talk report, we described this as the “800-pound gorilla” that stymies efforts to make progress across many fields (e.g., social policy, medicine, business):
The bottom line is that it is harder to make progress than commonly appreciated… The pattern of disappointing effects for most rigorously-evaluated programs—along with findings of important positive effects for a few—is compelling and transcends multiple fields. It needs to be taken seriously.
Thus, a government program such as SCA that funds a potpourri of state and local interventions is likely to have an average effect that is small or zero, since most interventions in the potpourri are not efficacious. The SCA RCT findings of disappointing effects are the latest of many such findings from RCTs of large federal social programs that fund a diverse group of state and local interventions.
However, we believe it is important not to misinterpret the SCA RCT findings as showing that no SCA-funded interventions were effective. It is very possible that some SCA-funded interventions (e.g., particular substance abuse or mental health treatments) were effective, but their effect was diluted by others that were ineffective or even harmful, resulting in an average program effect that was near zero. An earlier Straight Talk report provided a concrete example of this phenomenon: RCT evidence shows that charter schools as a general educational strategy do not, on average, improve student achievement compared to the regular public schools in their jurisdiction. However, some specific charter school interventions, such as the Knowledge is Power Program (KIPP), have been shown in high-quality RCTs to produce much better achievement outcomes than their regular school counterparts.
So, what’s an alternative federal approach to funding re-entry services that might lead to greater success? We suggest starting with the hard reality that we as a society don’t yet know, with a degree of confidence, what actually works to improve recidivism rates and other key outcomes for prisoners who are re-entering the community. High-quality RCTs in this area—including the SCA study, a Department of Labor-commissioned RCT of various employment-focused re-entry strategies, and RCTs of specific re-entry interventions (e.g., case management, transitional jobs)—have generally found disappointing effects. There are a few instances of promising RCT findings (e.g., Amity Prison Therapeutic Communities, Reentry Modified Therapeutic Community), but these studies have limitations such as single-site settings, small samples, and/or short-term follow-ups. Thus, there is not yet an example of a re-entry intervention—analogous to KIPP in the charter school field—whose evidence base provides confidence that it would produce meaningful effects if implemented faithfully on a wider scale.
Thus, as we have proposed in other policy areas, we suggest reforming SCA program and evaluation funding to focus on a central goal: building a body of re-entry interventions that are backed by strong, replicated evidence of meaningful effects on recidivism and other key outcomes. In an earlier Straight Talk report, we outlined one possible strategy—“tiered evidence”—for achieving this goal. It would involve, among other things, identifying re-entry interventions with especially promising initial evidence (such as the two cited above), and commissioning replication RCTs to see if the promising effects can be reproduced on a larger scale in new settings.
In an encouraging development, it appears that DOJ may be moving toward a strategy like this in its most recent funding announcement for the SCA Adult Re-entry Program, which incentivizes prospective grantees to adopt re-entry strategies with a promising evidence base and rigorously evaluate them. The devil, of course, is in the details, and it remains to be seen whether evidence will play a meaningful or pro forma role in DOJ’s funding decisions under this program. We hope it is meaningful role, as a clear departure from the traditional federal spending approach is plainly needed.
Response provided by Ronald D’Amico, the lead study author
We want to thank the Straight Talk reviewers for their commentary. While we agree with many of its points, we first want to raise three important clarifying remarks that put this review’s findings into context. First, the evaluation was of just seven SCA programs out of hundreds that have been funded by DOJ. It would be a mistake to generalize from these seven to SCA as a whole. Second, the seven grantees that we studied were among the very earliest to receive funding. Since then, DOJ tightened grant requirements to emphasize the importance of evidence-based programming and enhanced technical assistance during a required planning stage; both strategies were designed to ensure that subsequent waves of grantees developed programs grounded in what works. We don’t know if these changes improved program effectiveness, but they might have. Third, it’s important to consider the counterfactual. It’s not correct to say that these seven SCA programs were not effective, only that they were not more effective than existing reentry programming. States and local agencies had other funding they used to provide reentry services to the control group. Also, the seven grantees’ SCA programs often represented tweaks to what the grantees were already doing rather than wholesale changes. The SCA grants certainly enhanced grantees’ capacities, and to this extent the funds were helpful, but in most cases, SCA programs weren’t different enough from the counterfactual for marginal effects to appear.
That said, we agree with the central premise of this review. Although DOJ required the seven SCA grantees in this evaluation to use evidence on what works in designing their programs, the grantees were too different from each other in how they used their funds to draw clear conclusions about what works and what doesn’t; this evaluation was not a test of a well-defined program model for reentry planning. Moreover, the grantees and service providers were very different types of agencies, which included departments of corrections, sheriff’s offices, health and human services agencies, and CBOs. Finally, the types of individuals that grantees targeted were very different from one another—some grantees enrolled just women while others focused primarily on men, and some worked with prison populations while others focused on jails. Unfortunately, sample sizes were not nearly adequate to tease out these separate effects. As the reviewers note, “SCA is typical of many federal social programs, in that it awards grants to state and local agencies to expand or supplement a diverse array of existing services in those jurisdictions.” It is very difficult to build up a knowledge base this way. Future research needs to move in the direction that the reviewers suggest—we need to be using rigorous methods to test the effectiveness of specific, well-defined interventions in specific contexts. Moreover, we need to ensure that these interventions are implemented well, with fidelity to the model. This is an ambitious research agenda, but the only one that holds promise for significantly advancing our understanding of how to design effective social programs.
—Ronald D’Amico, President Emeritus, Social Policy Research Associates
Rejoinder by the LJAF Evidence-Based Policy team
We appreciate the lead author’s thoughtful comments and agree with them. Based on the input, we revised the first paragraph of our report to make clearer that the RCT evaluated the effectiveness of SCA services delivered by seven of the 15 state or local agencies that received funding in the first year of the program—i.e., 2009. We also noted that the seven sites had two years to develop and implement the SCA services before the evaluation got underway, so as to enable a fair test of the SCA services at these sites. (The report text shown above is the revised version.)
In addition, we agree with the author’s point that the RCT measured the impact of the SCA services compared to the usual re-entry services received by the control group. In our full two-page evidence summary, we describe the study’s findings on the differences in services received by the two groups as follows:
The study found that members of the SCA group received significantly more reentry-related services than the control group, including case management assistance, employment assistance, cognitive behavioral therapy, housing assistance, and outpatient substance abuse treatment. Differences between the two groups in the proportion of individuals receiving these services ranged from 11 to 20 percentage points, and were statistically significant.
Thus, the study’s central finding is that the added services funded by SCA did not produce the hoped-for effects on the primary outcome of reincarceration.
We thank the lead author for his engagement in this colloquy and helpful comments.
References:
[i] DOJ selected the seven study sites because the Department believed these sites were best able to participate in a rigorous evaluation. Thus, the study was designed to estimate the impact of SCA services at these sites rather than across all SCA grantees.
[ii] For each outcome that a study examines, there is roughly a one in 20 chance that the test for statistical significance will produce a false-positive result when the program’s true effect is zero. Thus, a study that measures numerous exploratory outcomes is likely to produce some false-positive findings.