Highlights:
- Conditional cash transfer (CCT) programs, which have been implemented in more than 60 countries, provide cash payments to poor families contingent on their meeting certain conditions such as using preventive healthcare, keeping children enrolled in school, or engaging in employment or training. The goal is to encourage families to invest in activities that help them break the cycle of poverty.
- We summarize a well-conducted randomized controlled trial (RCT) of a U.S. CCT program—Family Rewards 2.0. The study found that the program did not produce the hoped-for effects on most key outcomes (g., child education, parental employment) during a two to four-year period.
- This study was more credible than some earlier RCTs of CCTs in that it measured key outcomes through both participant self-reports and objective data (educational and wage records). CCT studies that rely primarily on self-reports may overstate program effects, given the incentives for treatment group members (but not controls) to report positive outcomes.
- CCT programs do not yet show promise as a way to reduce long-term poverty in the United States.
- The study team’s response, and our brief rejoinder, follow the main report.
Conditional cash transfers (CCTs) are rapidly expanding as a poverty-reduction strategy around the world. Following the publication of positive findings from a randomized controlled trial (RCT) launched in 1998 to evaluate Mexico’s CCT program (Progresa/Oportunidades), more than 60 countries have adopted CCT programs. Such programs vary in their specific features, but all share the common element of providing cash payments to poor families that are contingent on meeting certain conditions, such as using preventive healthcare, keeping children enrolled in school, and participating in employment, educational, or training activities. The goal is to encourage families to invest in their own development so as to break the cycle of poverty.
CCT programs are most prevalent in low- and middle-income countries, but in the mid-2000s, New York City launched a CCT program called Opportunity NYC—Family Rewards and engaged the research firm MDRC to evaluate the program in a large RCT. The main evaluation reports, which presented outcomes three to six years after random assignment, found that the program did not produce significant effects on most major educational, health, or workforce outcomes [1][2]. However, the reports suggested ways the program might be modified to become more effective. Based on these suggestions, MDRC launched a revised CCT program called Family Rewards 2.0 in 2011 in the Bronx (New York) and Memphis, Tennessee, and embedded an RCT to measure the new program’s effectiveness.
This Straight Talk post highlights the findings from the RCT of Family Rewards 2.0, as reported by MDRC last year. The study found the program to be generally well-implemented but—like its predecessor Opportunity NYC—it produced largely disappointing effects on the targeted outcomes. The following is our brief overview of the study, and our full three-page evidence summary is linked here.
This was a well-conducted RCT of Family Rewards 2.0, a conditional cash transfer program for low-income families with at least one child entering ninth or 10th grade. Over about a three-year period, the program provided each participating family with cash rewards that were contingent on meeting certain goals related to children’s school performance, the family’s use of preventive health care, and parents’ employment. The sample comprised 2,456 families in the Bronx (New York) and Memphis, Tennessee. At follow-up two to four years after random assignment, the study found that the program produced no significant effects on children’s educational outcomes and either no significant effects or modest adverse effects on parents’ employment and earnings (depending on whether employment/earnings were measured with self-reports or official state records, respectively). The program produced a modest, statistically-significant increase in household income during the program period, but this is likely just due to the cash rewards that the families received during that time (i.e., it’s a mechanical result of program participation). The program also produced a small, statistically-significant positive effect on parents’ self-reported health during the program period. On average, the program cost $13,459 per family, 48 percent of which was paid directly to families as cash rewards.
We chose to highlight this study for two reasons. First, it helps illustrate a pattern that we believe is widely under-appreciated in the policy community: Most programs—even many that are thoughtfully designed and well implemented—are found not to produce the hoped-for effects when rigorously evaluated in well-conducted RCTs. (As we have noted elsewhere, this pattern also occurs in other fields, such as medicine and business.) That’s why we believe rigorous evaluations like this are so essential. Without them, governments will fund many ineffective programs, and will have no valid way to identify the subset of programs that do produce important effects and should be expanded to improve people’s lives on a large scale.
Second, in an important respect the MDRC study was more credible than some of the earlier RCTs of CCTs, including the initial Mexico study in the 1990s [1][2], in that it did not rely primarily on self-reports to measure the key outcomes.[i] Unlike such earlier studies, the MDRC study used both self-reports and objective measures such as school district records on student outcomes and state unemployment insurance (UI) data on parents’ employment and earnings.
Self-reported data are potentially problematic in CCT studies because treatment group members are incentivized by the program to report positive outcomes in education, employment, and other areas in order to increase their receipt of cash benefits. Even though the treatment group’s responses to a research survey do not affect their cash benefits, they may still put a positive gloss on their responses to be consistent with their earlier reports, or they might wonder if a positive response on the research survey might somehow affect their future cash benefits. The control group, by contrast, has not been incentivized to report positive outcomes. This differential incentive between the treatment and control groups to report positive outcomes could well result in overstated findings in CCT studies that rely primarily on self-reported outcomes.
Barnow and Greenberg 2015 provides evidence that similar incentives for treatment group members to misstate their employment and earnings in the U.S. negative income tax RCTs resulted in inaccurate study findings for outcomes measured through self-reports as opposed to state records. [ii]
Another reason self-reports might produce inflated findings in CCT studies is that treatment group members may feel grateful for the large cash benefits they received and want to show the researchers that they put the money to productive use by improving their or their families’ educational or workforce outcomes. This phenomenon—a form of “social desirability bias”—could cause treatment group members to overstate their outcomes relative to the control group members, resulting in exaggerated program effects.
Either or both of these phenomena may help explain why both of MDRC’s CCT studies found that the employment and earnings effects as measured through self-reports were more favorable than the same effects measured through state UI records—substantially so in the Opportunity NYC study, modestly so in Family Rewards 2.0 study. (There are other possible explanations for the discrepant findings based on self-reports versus UI records, as discussed in MDRC 2013, pp. 175-177.)
The bottom line is that CCT programs in the United States—at least the two versions that have been rigorously tested on a sizable scale—do not yet show promise as a way to break the cycle of poverty.
Response provided by Cynthia Miller, lead study author
The study team appreciates your characterization of the study as well implemented and credible. We put significant effort into helping our partners implement the program, in order to provide a fair test of the CCT idea. We also agree that it is important to rely on multiple data sources when measuring the program’s effects. However, it may be too strong to discount the survey data entirely and to assert that the records data are more reliable. While it is true that there are several potential types of bias that may exist with survey data, it is not clear how important these are in practice. It is true, for example, that a CCT program group may have more incentive to report positive outcomes, particularly for rewarded activities. However, program group parents in our first CCT study, Opportunity NYC—Family Rewards, did not report that their children were doing better in school than control group parents, even though school performance was a rewarded activity. These findings matched those found in the school district records data. (In the second CCT study, Family Rewards 2.0, we did not analyze parents’ reports on their children’s school performance.) As another example, we did find a reduction in reported material hardships in the first test of Family Rewards, and it is difficult to imagine that the program group had an extra incentive to report fewer hardships, such as food insecurity.
In addition, when considering employment, the records data (UI records) do miss certain types of work –the types of jobs that are relatively prevalent among our typical study samples. As we noted in the reports, about a third of the jobs reported on the survey but not in UI data were in child care. A fair number were also in “health care support,” which is probably capturing home health aide employment. Thus, we feel that a good study needs to rely on both sources of data and that each has strengths and weaknesses.
Rejoinder by LJAF Evidence-Based Policy Team
We appreciate the study team’s thoughtful response, and agree that both types of outcome data—self-reports and administrative records—are valuable and complementary. Indeed, we believe a key strength of this study was its use of both data types.
As discussed in our report, we believe that in CCT studies, the findings based on administrative records may be more reliable than those based on self-reports due to the program’s incentives for treatment group members (but not controls) to self-report positive outcomes. The idea that program incentives may cause inaccurate findings for self-reported outcomes is supported by evidence from the U.S negative income tax RCTs, and in the U.S. CCT studies would help explain the pattern of more favorable employment/earnings effects based on self-reports versus administrative records.
The study team in its response acknowledges this potential source of bias in the self-reported outcomes but says it did not appear to affect the Opportunity NYC study’s’ findings on children’s school performance, as the effects based on parental reports matched those based on school records. We agree with this point. Our take-away is that the hypothesis that program incentives cause overstated findings for self-reported outcomes has greater support in the employment and earnings data than in the educational data. One reason we would encourage future CCT studies to measure key outcomes through both self-reported and objective measures is that doing so would help shed additional light on the reliability of the different types of outcome data.
References
[i] While the RCT of Mexico’s CCT program used self-reports to measure most of its key outcomes (e.g., school enrollment, child labor, parental employment, food expenditures, and health), it did assess child height—an indicator of nutrition—through anthropometric measurements taken during the household interview, and found a positive effect on that outcome for preschool children.
[ii] In the negative income tax RCTs, members of the treatment group were incentivized to understate (rather than overstate) their employment and earnings, because doing so would result in larger cash payments under the program. Barnow and Greenberg provide evidence that this caused the RCTs to overstate the negative income tax’s adverse effect on employment and earnings in the self-reported data as compared to state UI data.