Highlights:
- Replication of promising findings from initial randomized controlled trials (RCTs) is critical to building evidence that policymakers and program operators can rely on.
- We discuss findings of replication RCTs of two programs that showed the potential, in initial RCTs, to increase graduation rates of low-income community college students—Accelerated Study in Associate Programs (ASAP) and Stay the Course (STC).
- The replication RCTs produced strikingly different results.
- The replication RCT of ASAP in Ohio found a large, 16 percentage point increase in graduation at three years, highly similar to the findings of the original RCT of ASAP at the City University of New York.
- By contrast, the replication RCT of STC, conducted within the same community college system as the original RCT, failed to find any significant positive effects on graduation.
- Colleges that implement ASAP faithfully can be reasonably confident that they will improve graduation outcomes of low-income students. This unfortunately cannot yet be said for STC.
As funders of randomized controlled trials (RCTs), our primary goal is to identify programs and strategies that can meaningfully improve people’s lives, and, importantly, that can be shown to do so in more than one study across different settings. Because many promising programs with positive results in an initial RCT are unable to reproduce the results in a subsequent study, positive replication is critical to having confidence that broader expansion of a program is warranted. We’ve previously described successfully replicated programs as the “promised land” for evidence-based policymaking, since this kind of evidence of effectiveness can (i) rule out the possibility that a program’s observed impacts were the result of a statistical fluke and (ii) demonstrate that the program can work in different places and/or time periods. So, we are always on the lookout for programs with promising findings from a single RCT that would be good candidates for a replication trial.
In recent years, two such programs in higher education that came to our attention were the Accelerated Study in Associate Programs (ASAP) developed by the City University of New York (CUNY), and Stay the Course (STC) implemented at Tarrant County College in Fort Worth, Texas. These programs aim to increase graduation rates for low-income students. Both programs, in different ways, provide more intensive and individualized coaching and counseling than is ordinarily available to students, and both require participating students to meet minimum enrollment requirements. In addition, ASAP offers financial assistance to students. The per-student cost of ASAP is somewhat higher than that of STC.[1]
In initial RCTs, both programs showed the potential to improve important postsecondary outcomes. Specifically, in an RCT with a sample of 896 students at three CUNY campuses, ASAP had been shown to increase three-year graduation rates by a statistically significant 18 percentage points, from 22 percent in the control group to 40 percent in the treatment group. Meanwhile, an RCT of STC with a sample of 869 students at one Tarrant County College campus did not find statistically significant improvements in graduation rates for the full sample of students, but the RCT results suggested that the program had a positive graduation effect of roughly the same size as ASAP’s for the subset (22%) of students assigned to the treatment group who actually participated in the program.
Based on these two pieces of promising prior evidence, we helped fund subsequent RCTs that sought to replicate the initial findings for both programs. In the case of ASAP, this second study included 1,501 low-income students at three community college campuses in Ohio. For STC, the second study of the program took place at two Tarrant County College campuses and had a sample of 261 students.[2] In providing funding to support these RCTs, we hoped to move both programs closer to “strong, replicated evidence” – that is, the type of evidence that would give other colleges around the country confidence that, if they were to faithfully implement these programs, they would meaningfully improve college graduation rates for low-income students.
As described below, this turned out to be true for one program but not the other.
The replication trial of ASAP in Ohio was a big success. Specifically, the second RCT found that 35 percent of the treatment group completed a degree by the end of three years compared to 19 percent of the control group – a 16 percentage point positive effect that was statistically significant and very close to the effect found in the initial RCT at CUNY. These are the largest impacts on degree completion that we are aware of in findings from high-quality RCTs.
By contrast, the replication trial of STC found no significant impact on degree completion at the end of three years (38% of the treatment group had earned a degree vs. 39% of the control group) nor on the percent of students still enrolled at the three-year mark (50% of the treatment group vs. 53% of the control group). This suggests that the initial positive effect found in the first RCT may have been due to chance (as opposed to a true effect), or that, even if true, the positive effect was sensitive to other conditions that were different in the second RCT (such as the location, time period, or policy environment in which the program was implemented). Either explanation implies that other colleges considering implementing STC cannot yet be confident that it will lead to better graduation outcomes.
Notably, the final report on the second STC RCT examines possible explanations for its disappointing findings, including factors present during that second RCT that might not apply elsewhere. While this analysis is useful, additional RCTs are the only way to produce strong evidence on the circumstances in which STC does or does not improve student graduation rates. Fortunately, the researchers who have conducted the STC RCTs are working to conduct replication RCTs on other college campuses, which will hopefully identify conditions under which STC can be effective.
For the country to improve college completion for low-income students, policy makers need evidence they can rely on. This requires successful replication of initially promising results. In this regard, the difference between the replicable results of ASAP and the so far not-replicated STC results are clear. Colleges that implement ASAP can be reasonably confident that they will improve student graduation rates if they implement the program faithfully. Based on the current evidence, this unfortunately cannot yet be said for STC.
Response provided by the study authors
We invited the lead study authors, Cynthia Miller for the ASAP replication RCT and James Sullivan for the STC replication RCT, to provide written comments on our report. They said that they appreciated the chance to respond and did not have comments to add.
References:
[1] The total three-year cost of ASAP services in Ohio and at CUNY is approximately $5,500 and $10,300 respectively, compared to $4,300 for STC. This is the direct cost of program services, and does not include additional costs associated with program participants enrolling in more classes than control group students.
[2] Although the second study included fewer students than the first, it had a greater ability to detect an impact on graduation (if such an impact existed) because 87 percent of treatment group members in the second study participated in the program versus only 22 percent in the first RCT. The higher rate in the second RCT was achieved by randomly assigning only students who expressed an interest in STC and consented, whereas the first RCT randomly assigned all STC-eligible students.