Highlights:
- This report discusses new findings from a well-conducted randomized controlled trial (RCT) of Tennessee’s voluntary prekindergarten program for low-income children.
- This study is the first large RCT of a state-funded pre-k program and one of only two such studies ever conducted of a public preschool program in the United States. As such, it provides uniquely credible evidence in the area of early childhood education.
- While the study found positive short-term effects on child achievement (at the end of the pre-k year), these effects dissipated as children entered elementary school and turned modestly negative by third grade. At the third-grade follow-up, the control group scored significantly higher in math and science achievement than the pre-k group.
- We offer possible reasons for the adverse effects, and suggest that the program be reformed by incorporating evidence-based funding criteria aimed at greatly improving its effectiveness over time.
- A thoughtful comment from the study authors follows the main report.
In this report we discuss newly-published findings from a randomized controlled trial (RCT) of Tennessee’s voluntary prekindergarten (pre-k) program for low-income children (Lipsey, Farran, and Durkin 2018). We are highlighting this study for two reasons. First, the effectiveness of state and local pre-k programs is a topic of high policy importance. Approximately 28 percent of the nation’s four-year-olds are enrolled in pre-k programs funded by states, municipalities, or school districts—a number that has grown rapidly over time (Chaudry and Datta 2017)—and policy officials often tout pre-k as a powerful tool for closing school achievement gaps between minorities and whites and increasing earnings later in life (e.g., Executive Office of the President 2015).
Second, this study provides uniquely credible evidence on the topic. It is the first large RCT of a state-funded pre-k program, and one of only two such studies ever conducted of public preschool programs—the other being the national RCT of the federal Head Start program. Other studies of public or private preschool programs have had weaknesses that limit the reliability of their findings, such as lack of random assignment (e.g., Oklahoma universal pre-k, Chicago Child-Parent Centers) or small samples and imperfect randomization (e.g., Perry Preschool Project, Abecedarian Project).
What did the Tennessee study find? Like the Head Start RCT, it found positive effects on student achievement at the end of the pre-k year (e.g., their ability to identify letters and words), but these effects dissipated as children entered elementary school and—in the case of Tennessee—turned modestly negative by third grade, with the control group outperforming the pre-k group in math and science achievement. Here’s a brief overview of the newly-published third-grade findings in Tennessee:
Program description. Tennessee’s voluntary pre-k (VPK) program is a statewide program that enrolls four-year-old children from low-income families. State funding for the program is approximately $87 million per year. The program provides a minimum of 5.5 hours of instructional time per day, five days per week. Classes have a maximum of 20 students and are taught by state-licensed teachers using one of 22 curricula approved by the Tennessee Department of Education. Based on classroom observations and other measures, the program’s quality appears to be fairly typical of state pre-k programs around the country.
Study design. The study randomly assigned 3,131 eligible children who applied for admission at one of 79 oversubscribed VPK programs across the state to either (i) a program group that was offered admission, or (ii) a control group that was not (but could access other available child and family services in the community). In the study’s primary and most reliable analysis,[i] student achievement and other outcomes were measured in third grade using state educational records (e.g., scores on Tennessee state tests).
Findings. At the end of third grade, the study found statistically-significant adverse effects on student math and science achievement. In math, the VPK group scored 0.12 standard deviations lower than the control group, which equates to roughly 13 percent less growth in math achievement than would be expected in the third grade year.[ii] In science, the VPK group scored 0.09 standard deviations lower than the control group, which equates to roughly 23 percent less growth in science achievement than would be expected in the third grade year.[iii]
The study found no significant effects on reading achievement in third grade, or on school attendance, grade retention, or disciplinary infractions measured from kindergarten through third grade. The study found that VPK students were identified as needing special education services for speech/language impairment or learning/intellectual disabilities at a slightly higher rate than control group students (13.3 percent versus 10.6 percent in the third grade year).[iv]
Study quality. This was a well-conducted RCT. Sample attrition was relatively low (e.g., third-grade test scores were obtained for approximately 80 percent of both the VPK group and the control group). Both groups were also highly similar in their demographic characteristics—an indication that random assignment was successful. The study appropriately kept sample members in their original, randomly-assigned groups when conducting the primary analysis of VPK’s effects (per an “intention-to-treat” approach) even though some members of the VPK group did not enroll in the program and some members of the control group did.[v] The study’s analysis appropriately accounted for the fact that random assignment was carried out within each of 79 VPK programs across the state.
So what might account for the program’s adverse effects? One possibility is that, contrary to conventional wisdom, some children may be better off academically if—instead of attending public pre-k—they stay at home at age four (which is where two-thirds of the sample children who did not enroll in VPK ended up). Another possible explanation is suggested by the fact that the VPK group had a higher rate of special education placements than the control group from kindergarten through third grade. This higher rate appears, in part, to be a result of children in the VPK group receiving special education designations during their pre-k year that carried over into elementary school. It is thus possible that VPK results in the identification of some pre-K children for special education who are just developing more slowly but who would otherwise catch up by kindergarten. Such identification may then lead to lower educational expectations and levels of instruction for these children. In addition to the above explanations, there may be others that could plausibly account for the study’s findings.
What lessons can we draw from these findings? Our thoughts here are similar to those we offered on the disappointing Head Start RCT findings in an earlier Straight Talk report, and we would refer readers to the relevant section of that report for a more complete discussion. But as a brief summary:
- Tennessee’s VPK program, much like Head Start, gives local VPK sites effective wide latitude in how to implement their programs, resulting in a diversity of approaches;
- It is likely that a subset of local sites and approaches are effective, but their impact is offset by others that are ineffective or harmful (studies suggest that, in Head Start, this is indeed the case); and
- The above findings and observations, we believe, underscore the need to reform programs such as VPK and Head Start by incorporating (i) rigorous evaluations aimed at identifying the subset of local approaches that are effective, and (ii) once such approaches are identified, strong incentives or requirements for other local program sites to adopt and faithfully implement them on a larger scale.
In a world where, as we have frequently noted, most attempts to make progress fail and a few succeed, spending as usual in pre-k and other areas without a clear focus on evidence about what works is unlikely to solve the nation’s problems. Reform is needed.
Comment provided by study authors Dale Farran and Mark Lipsey
We believe that most people will agree that society has an obligation to prevent or ameliorate the harmful effects of poverty on children. U.S. policy makers have shown little interest in prevention, but have been broadly supportive of preschool as an approach to amelioration. Head Start has this same mission, but a movement emerged in the 1980s to provide pre-k under the supervision of the public schools, with further expansion beginning in the early 2000s. However, the only research evidence available to guide states on whether such programs would be effective came from small, boutique researcher-driven programs implemented 40-50 years previously. Although the state programs were not like these earlier experimental projects, the data on their long-term effectiveness was proclaimed loudly and frequently to justify state funding.
In 2008 we worked closely with the Tennessee Department of Education to craft a strong experimental design that would assess the effectiveness of the TN Voluntary Pre-K program (TNVPK). Other than the Head Start Impact study, this would become the only randomized control study of a scaled-up public pre-k program.
Our initial results supported the immediate effectiveness of pre-k; children in the program performed better at the end of pre-k than control children, most of whom had stayed home. The press, the public, and our colleagues relished these findings. But ours was a longitudinal study and the third grade results told a different story. Not only was there fade out, but the pre-k children scored below the controls on the state achievement tests. Moreover, they had more disciplinary offenses and none of the positive effects on retention and special education that were anticipated.
Those findings were not welcome. So much so that it has been difficult to get the results published. Our first attempt was reviewed by pre-k advocates who had disparaged our findings when they first came out in a working paper – we know that because their reviews repeated word-for-word criticisms made in their prior blogs and commentary. We are grateful for an open-minded editor who allowed our recent paper summarizing the results of this study to be published (after, we should note, a very thorough peer review and 17 single-spaced pages of responses to questions raised by reviewers). We are also appreciative of the objective assessment and attention to detail represented in the Straight Talk review.
It is, of course, understandable that people are skeptical of results that do not confirm the prevailing wisdom, but the vitriol with which our work has been greeted is beyond mere scientific concern. Social science research can only be helpful to policy makers if it presents findings openly and objectively, even when unwelcome.
We share with our colleagues a commitment to the goal of providing a better life for poor children. Blind commitment to one avenue for attaining that goal, however, is unnecessarily limiting. If pre-k is not working as hoped and intended, we need to roll up our sleeves and figure out what will work, with solid research to guide that effort.
References
[i] The study also analyzed outcomes for a subsample of children whose parents consented to annual individual student assessments. However, a high proportion of parents did not provide such consent, and non-consent rates were higher in the control group than the VPK group, potentially undermining the equivalence of the two groups in key characteristics and reducing confidence in the findings (since the analysis may no longer be the “apples-to-apples” comparison that one would expect of a large RCT). The study’s findings for this consented subsample were consistent with those for the full RCT sample.
[ii] The average annual gain in math achievement for U.S. students during third grade on seven nationally normed tests is 0.89 standard deviations (Lipsey et. al. 2012). The difference in math achievement between VPK and control group students of 0.12 standard deviations equates to 13 percent of this annual gain.
[iii] The average annual gain in science achievement for U.S. students during third grade on seven nationally normed tests is 0.48 standard deviations (Lipsey et. al. 2012). The difference in science achievement between VPK and control group students of 0.11 standard deviations equates to 23 percent of this annual gain.
[iv] The study’s measurement of multiple outcomes at the third-grade follow-up raises the possibility of false findings of statistically-significant effects, since each test for statistical significance has roughly a one in 20 chance of producing a false finding if the program’s true effect is zero. However, when we applied a statistical adjustment (Benjamini-Hochberg) for the study’s measurement of multiple outcomes, the adverse effect on math achievement remained statistically significant (p<0.05) and the adverse effect on science achievement came close to statistical significance (p<0.10), indicating that these are unlikely to be false findings. The effect on special education placements was no longer statistically significant after this adjustment, which somewhat reduces confidence that it is a true effect.
[v] Among children randomly assigned to the VPK group, 87 percent actually enrolled in VPK (13 percent did not). Among children randomly assigned to the control group, 34 percent ended up enrolling in VPK—primarily because they had been waitlisted and were admitted to the program when a space opened up. This rate of “no-shows” in the VPK group and “cross-overs” in the control group means that the program’s effect on those children who actually attended VPK is substantially larger (i.e., worse) than the effects that we describe in the main text for the full randomized sample—perhaps nearly twice as large.