Highlights:
- New findings from a large randomized controlled trial (RCT) of Nevada’s Reemployment and Eligibility Assessment (REA) program for unemployment claimants show that this low-cost program—$250 per participant—produced earnings gains of 15 percent ($8,460) over three years, and net government savings.
- These findings reproduce highly positive results of an earlier RCT of the Nevada program conducted in the aftermath of the Great Recession, indicating that the program’s effects are generalizable.
- The REA results illustrate that impact size often depends more on the specific program model (Nevada REA) than on the general approach (reemployment programs), as other rigorously-tested REA programs with different features have produced small or no impacts.
- Thus far, the federal and state governments have missed key opportunities to replicate and expand the Nevada REA program, so as to potentially improve the lives of millions of American workers.
- Our nation faces massive unemployment in the wake of the COVID-19 pandemic. Nevada REA would be an ideal strategy for policy officials to implement and rigorously test on a large scale once our nation is able to turn attention to reemploying the millions of dislocated workers.
In this report, we highlight positive new findings from a large replication RCT of Nevada’s Reemployment and Eligibility Assessment (REA) program for unemployment insurance (UI) claimants. The new findings—sizable earnings gains, sustained over at least three years—reproduce the highly positive results of an earlier RCT of the Nevada program conducted under very different economic conditions (the immediate aftermath of the Great Recession). The successful replication provides evidence for the generalizability of the program’s effects, and suggests that this low-cost program—roughly $250 per participant—would be an ideal strategy to implement and rigorously test on a larger scale once our nation is able to turn attention to reemploying the millions of workers dislocated by the pandemic.
An initial RCT launched in 2009 found that Nevada’s REA program—a low-cost innovation in the UI system—produced earnings gains of 18 percent and net government savings. Nevada’s REA program is a mandatory program for new UI claimants, delivered by trained staff at One-Stop Career Centers across the state, who provide each of the following services during a single interview session:
- A UI eligibility review to verify that the claimant is eligible for benefits (e.g., has been actively seeking employment) and to prevent overpayment;
- Labor market information (e.g., regarding job openings, wage trends);
- Development of an individual reemployment plan; and
- Provision of reemployment services (e.g., job search and resume assistance, job match against automated labor exchanges, referral to job training).
The program is inexpensive, costing less than $250 per participant. A high-quality, U.S. Department of Labor (DOL)-sponsored RCT of the program, launched in 2009 with a sample of nearly 33,000 UI claimants, found sizable impacts—namely (i) a $2,611 (18 percent) increase in wage earnings per participant over the 1.5 years after study entry, and (ii) net government savings (after accounting for the program’s cost) of approximately $604 per participant due to reduced UI payments over the two years after study entry. Our summary of the study, which was conducted by Impaq International, is posted here.[1]
These RCT findings were exceptional. As we have discussed in prior Straight Talk reports [a,b,c], surprisingly few social programs, innovations, and strategies evaluated in high-quality RCTs are found to produce the hoped-for improvements in participants’ lives—a pattern that also occurs in other fields like medicine and business where RCTs are common. Indeed, as we discuss below, even in the field of REA programming, RCTs of other versions of REA with substantively different components have found small or no positive impacts.
Based on these exceptional findings, we funded a replication RCT in Nevada. The new RCT reports similarly impressive impacts: 15 percent earnings gain over three years, and net government savings. Impaq is conducting the replication RCT, which is still ongoing. A key aim of the study is to determine whether the findings of the initial Nevada RCT, which took place in the immediate aftermath of a severe economic recession (2009-2011), could be reproduced in a much healthier economy and labor market (2014-2018 for the interim study report [2]).
The newly-released interim report, based on a randomized sample of over 91,000 UI claimants in Nevada, shows impacts that are remarkably similar to the first study: (i) a $8,460 (15 percent) increase in earnings per participant over the three years after study entry; and (ii) net government savings (after accounting for the program’s cost) of about $200 per participant from reduced UI payments during the year after study entry.[3]
This is the latest in a series of successful replications of credible RCT evidence of sizable impacts, illustrating the value of such evidence in identifying programs that reliably move the needle on important outcomes. Other successful replications of sizable RCT impacts that we have recently discussed include the ASAP program for low-income college students, Per Scholas job training for low-income individuals, KIPP public charter schools, and Riverside GAIN for welfare recipients (link, see pages 1-2). Relatedly, the Institute of Education Sciences’ review of 49 high-quality impact evaluations conducted under the U.S. Department of Education’s Investing in Innovation (i3) Fund found that the impacts of programs with the strongest prior evidence often replicated in subsequent RCTs.
The REA results illustrate that impact size often depends more on the specific program model (Nevada REA) than on the general approach (reemployment programs), as other rigorously-tested REA programs with different features have produced small or no impacts. Over the past 15 years, DOL has sponsored RCTs of various REA programs in nine states in addition to Nevada. These other versions of REA were substantively different from the Nevada program and from each other, consistent with the high level of flexibility allowed under DOL’s REA grants.
These studies all found far smaller impacts for the other versions of REA.[4] For example, the most recent RCT, launched in 2015, evaluated REA programs in Indiana, New York, Washington, and Wisconsin, with a combined sample of nearly 300,000 UI claimants. Across the four states, the study found that REA increased earnings by just 2 percent in the first year after study entry, and by 0.7 percent in the second year.[5] The earnings impacts were also very small in each of the four states analyzed separately. (Features of the Nevada program that may account for its superior impacts are discussed in the endnote.[6])
This sensitivity of impact findings to the specific program model is an important pattern that occurs across diverse areas of social policy. As we have discussed in a previous Straight Talk report, for example, well-conducted RCTs have found that KIPP charter schools (middle and elementary) produce sizable, sustained gains in student academic achievement. By contrast, the average impact of charter schools generally—based on a large, federally-sponsored RCT of 36 over-subscribed U.S. charter middle schools using a wide array of program models—appears to be close to zero. We have also highlighted a similar pattern in well-conducted RCTs of early childhood home visiting programs: Most are found to produce disappointing impacts while a few, such as the Nurse-Family Partnership, are truly effective.
Thus far, the federal and state governments have missed key opportunities to replicate and expand the Nevada REA program, so as to potentially improve the lives of millions of American workers. For example, DOL did not seize the opportunity, in launching the most recent REA evaluations in 2015, to fund a replication of the Nevada REA model in other states to see if the blockbuster Nevada impacts found in the 2009 DOL RCT could be reproduced across multiple settings and on a larger scale. Instead, the 2015 evaluation tested four other state REA programs which, as noted above, produced negligible earnings impacts.
Congress renewed DOL’s REA grant authority in 2018 and, to its credit, directed that states use at least 25 percent of their funds starting in 2023 (rising to 50 percent by 2027) for program approaches “with a high or moderate causal evidence rating that show a demonstrated capacity to improve employment and earnings outcomes for program participants.”[7]
However, DOL’s guidance on the new legislation provides that a state’s REA program will qualify as meeting this evidence standard if credible impact studies show statistically significant effects, even if those effects are trivial in size and therefore of little practical or policy significance. This “statistical significance” criterion is particularly problematic in the case of REA because many of the RCTs of state REA programs had very large samples, and were therefore capable of detecting very small effects as statistically significant—including, for example, 1 to 3 percent earnings impacts in Indiana, Washington, and New York. As a result, under DOL’s criterion, REA programs with such small effects will have the same priority for funding as the Nevada program that produced earnings gains of 15 to 18 percent.[8]
Our nation faces massive unemployment in the wake of the COVID-19 pandemic. The Nevada REA program would be an ideal strategy for policy officials to implement and rigorously test on a large scale once our nation can turn attention to reemploying the millions of dislocated workers. We have a rare opportunity to stop guessing at how to help workers who have lost their jobs, and to advance an approach that could truly improve their lives. Would the program prove effective under such extreme economic conditions? It produced sizable impacts in an earlier time of economic upheaval—the aftermath of the Great Recession—and we should find out if it can do so again.
Response provided by the lead study author
We invited the lead study author, Marios Michaelides, to provide written comments on our report. He appreciated the opportunity to respond and did not have comments to add.
References:
[1] Our posted summary reports dollar amounts that are inflation-adjusted to 2017, and therefore differ slightly from the amounts reported in this Straight Talk report, which are not inflation-adjusted.
[2] The study will continue to measure earnings outcomes through the first quarter of 2020.
[3] The study found that the reduction in UI payments on the original claim was $457 per person over the one-year period following study entry. We calculated the net government savings by subtracting the program cost ($250 per person) from this amount. Note that this is just the savings from the original UI claim over a one-year period. Additional government savings from reduced UI receipt may continue to accrue in subsequent years; the study will measure and report on any such savings in future follow-up reports.
[4] Eileen Poe Yamagata, Jacob Benus, Nicholas Bill, Hugh Carrington, Marios Michaelides, and Ted Shen. “Impact of the Reemployment and Eligibility Assessment Initiative.” Impaq International, June 2011. Jacob Benus, Eileen Poe-Yamagata, Ying Wang, and Etan Blass. “Reemployment and Eligibility Assessment (REA) Study FY 2005 Initiative: Final Report.” Impaq International, March 2008. Jacob A. Klerman, Correne Saunders, Emily Dastrup, Zachary Epstein, Douglas Walton, Tara Adam, with Burt Barnow, “Evaluation of impacts of the Reemployment and Eligibility Assessment (REA) Program: Final report.” Abt Associates, November 2019.
[5] The first year’s impact was statistically significant; the second year’s was not.
[6] Nevada’s program differed from the programs evaluated other states in that (i) Nevada required UI claimants to participate in both the eligibility assessment and reemployment services, whereas some other states required only the assessment (and encouraged, but did not require, the services); and (ii) the Nevada interviewers typically provided the eligibility assessment and reemployment services seamlessly during the same interview session, whereas in the other states interviewers typically referred claimants to a separate office or organization for the reemployment services, and such services were often not delivered. Yamagata et. al., 2011 (see reference 4) theorize that these features of the Nevada program may help explain its superior outcomes.
[7] Bipartisan Budget Act of 2018, Public Law 115-223, section 30206. Also, in 2015, Congress changed the name of the program to Reemployment Service and Eligibility Assessment (RESEA).
[8] The DOL guidance actually provides that a state’s REA program will qualify as meeting the evidence requirement if credible impact studies show statistically significant effects on employment and duration of UI benefits (despite the statutory language, the guidance says nothing about earnings). DOL’s latest RCT found that Indiana, Washington, and New York’s REA programs each produced statistically significant, though extremely small, effects on these outcomes.