Highlights:
- We review a recent large randomized controlled trial (RCT) of the Paycheck Plus program, which tested the effects of expanding the Earned Income Tax Credit (EITC) for low-income, single adults without dependent children. The EITC is a refundable tax credit that subsidizes workers’ wage earnings.
- The study’s overview of “Main Findings,” cited in the New York Times and elsewhere, reports positive study results, such as an increase in after-bonus earnings (earnings after accounting for taxes and the Paycheck Plus bonus) and a reduction in severe poverty.
- Our concern: The Main Findings overview reports only positive results and omits other primary study findings that cast real doubt on the program’s effectiveness as a poverty-fighting tool—e.g., no significant effects on total household income, worker earnings, or the overall poverty rate. In addition, the positive finding on severe poverty is best viewed as only preliminary, as it could well be a false-positive that appeared by chance due to the study’s measurement of effects on numerous outcomes.
- Such selective reporting—a common problem in the evaluation literature—could easily lead policy officials to promote the Paycheck Plus approach in the mistaken belief it has been rigorously shown to increase income and reduce poverty.
- The lead researcher’s response to our concerns, and our rejoinder, follow the main report.
MDRC, a leading social policy research organization, recently published its final report on an RCT of the Paycheck Plus demonstration in New York City.* The Paycheck Plus RCT was designed to test the effects of significantly expanding the Earned Income Tax Credit (EITC) for low-income, single adults without dependent children. The EITC is a federal anti-poverty program that provides a refundable tax credit to low-income Americans to subsidize their wage earnings.[i] Under current law, the benefit available to single adults without dependent children is much smaller than that available to families with children. Paycheck Plus used state and philanthropic funds to increase the maximum benefit for these single workers from $510 to $2,000 per year, and expand eligibility to those earning up to $30,000 per year (versus about $15,000 under current law) so that more workers would qualify for the benefit.
The study randomly assigned nearly 6,000 low-income single adults without dependent children in New York City to (i) a treatment group that was eligible for the Paycheck Plus benefits, or (ii) a control group that was only eligible for the usual (smaller) EITC; and measured outcomes for the two groups over a three-year period. MDRC’s September 2018 study report provides the following overview of the “main findings”:
- Paycheck Plus increased after-bonus earnings (earnings after accounting for taxes and the bonus) and reduced severe poverty.
- The program modestly increased employment rates. Positive effects on employment were concentrated among women and the more disadvantaged men in the study.
- Providing individuals with information about employment services may increase the employment effects of Paycheck Plus.
- Paycheck Plus led to an increase in tax filing rates and the use of Volunteer Income Tax Assistance sites for tax preparation.
- The program also led to an increase in child support payments among noncustodial parents.
The findings are consistent with other research on the federal EITC, indicating that an effective work-based safety net program can increase incomes for vulnerable and low-income individuals and families while encouraging and rewarding work.
So, what’s wrong with this overview? Following an all-too-common pattern in the evaluation literature, it selectively reports positive study findings while omitting other key findings that cast real doubt on the program’s effectiveness as a poverty-fighting tool. Specifically, the study’s primary, pre-registered outcomes were “income, earnings, poverty, child-support payments, criminal-justice-system involvement, and family formation” (see study registration). Taking these outcomes in order, the study found:
- No significant effect on total annual household income (such income averaged $21,303 for the treatment group versus $21,527 for the control group, as measured near the end of the study[ii]).
- No significant effect on worker earnings not including the Paycheck Plus bonus (workers in the treatment group earned an average of $12,716 per year over the study period, versus $12,560 for the control group[iii]).
- No significant effect on poverty (49.4 percent of the treatment group had household income below the poverty line versus 50.0 percent of the control group, as measured near the end of the study).
- Suggestive positive effects on child support payments by noncustodial parents (a statistically significant increase of $48 per month in the second year of the study, but no significant effects in the first or third years).
- No significant effects on criminal justice involvement (e.g., 18.8 percent of former prisoners in the treatment group were arrested over the study period versus 16.0 percent of former prisoners in the control group).
- No significant effects on most measures of family formation, except for a significant adverse effect on the percent “living with a partner” (10.6 percent for the treatment group versus 13.3 percent for the control group, as measured near the end of the study).
The only one of the above effects mentioned in the MDRC overview is the positive effect on child support payments.
It may seem paradoxical that Paycheck Plus increased workers’ after-bonus earnings, as noted in the MDRC overview (i.e., earnings after accounting for taxes and the Paycheck Plus bonus), but had no significant effect on total household income or poverty rates. A possible explanation is that the Paycheck Plus bonus paid to workers had an adverse effect on income from other household members, causing them to work a bit less or (as noted in the last bullet above) disrupting household relationships and causing them to leave the household. The study findings provide tentative support for this explanation: Annual income from other household members was $900 lower in the treatment group than the control group in (roughly) the last year of the study[iv]—an effect that approached statistical significance (p=0.08) and approximately offset the increase in workers’ after-bonus earnings.
In short, we believe that MDRC’s overview of the study’s main findings—as set out in both the study report and the MDRC press release—is not a balanced presentation, and could easily lead policy officials to adopt or expand the Paycheck Plus approach in the mistaken belief it has been rigorously shown to increase income and reduce poverty. Indeed, New York Times columnist David Leonhardt cites the overview’s rosy findings in calling for an expansion of the EITC (in a piece entitled “How To Cut Poverty Now”). The New York City Mayor’s Office of Economic Opportunity similarly echoes the overview’s one-sided summary, as does an article by former Congressional Budget Office Director Douglas Holtz-Eakin.
Consistent with Nobel laureate Richard Feynman’s entreaty to graduating Caltech students in 1974, we believe researchers have a special responsibility to present study findings in an impartial manner, and should bend over backward to do so. There is another, ongoing RCT of Paycheck Plus in Atlanta that has not yet reported results. Our hope is that the findings are strongly positive but, even if not, that they are reported in a balanced and accurate way.
Response provided by Cynthia Miller, the lead study author
Thank you for the opportunity to respond to your review.
It is incorrect to state that the key findings are misreported, and that “the findings cast doubt on the intervention’s effectiveness as a poverty-fighting tool.”
First, the main report and, more importantly, the Executive Summary present the full set of findings on household income and poverty. Both documents report and emphasize: 1) modest positive effects on employment, found from three different sources –administrative tax data, UI data, and survey data; 2) earnings effects that are small, positive but not significant; and 3) an increase in after-bonus earnings, no increase in overall household income, and a reduction in severe poverty.
Second, severe poverty is part of poverty. As noted in the Executive Summary and the main report, the program did not systematically move individuals from below the poverty threshold to above it, but it did lead to a shift from extreme poverty (less than 50 percent of the threshold) to less extreme poverty (50 percent to 100 percent of the threshold). As noted in the report, Paycheck Plus provides a credit that is much less than the federal EITC for families with children, which has been found to reduce the number of families below 100 percent of the threshold. The plateau region of the Paycheck Plus bonus, however, starts at about 50 percent of the poverty line for a single individual. In addition, most of the bonus’s effects on work and earnings were concentrated at the lower end of the earnings distribution.
A few other responses:
1. Employment is a primary outcome. In the study registry, employment was inadvertently left out of the list of primary outcomes, but it is listed in the first sentence of the description of the intervention: “The evaluation will test the potential impact of an enhanced Earned Income Tax Credit (EITC) on income, employment, and earnings for participating single adults.”
2. The IRS data only became available to the study after it began. For that reason, after-bonus earnings was included as one measure of income. It doesn’t capture all sources of income, as the survey measure does, but it is also not as noisy of a measure and does not suffer from recall biases. As noted in the Executive Summary and main report, the effect on average income (as reported on the survey) is not statistically significant, most likely because small (but statistically insignificant) reductions in income at the top of the distribution offset increases at the bottom of the distribution. The increases at the bottom led to a reduction in severe poverty.
In sum, the findings provide evidence, consistent with prior non-experimental research, that an expanded bonus can increase work and potentially reduce poverty. Effects will also be assessed for the Atlanta replication, which will be pre-specified on the AEA registry, focusing on the same set of primary and secondary outcomes and subgroups. Eventually, a pooled analysis (for New York and Atlanta combined) will provide the most statistical power to examine the program’s effects.
Rejoinder by the LJAF Evidence-Based Policy team
We appreciate the lead author’s response, but believe she does not address our central concern: The overview of “Main Findings” that introduces the study report and is reproduced in MDRC’s press release and web page on the study only reports positive findings. No mention is made of the disappointing findings for primary outcomes such as total household income, worker earnings, overall poverty rates, criminal justice involvement, and family formation.
The lead author’s response does not address this concern about selective reporting in the Main Findings overview, and merely notes that “the main report and, more importantly, the Executive Summary present the full set of findings on household income and poverty.” We don’t believe this is sufficient. An overview of main findings is supposed to summarize the study’s bottom-line results; and, for the many readers who do not have time to review the nine-page executive summary or 95-page full report, it will often be their only exposure to the study findings. How many readers of the New York Times article on Paycheck Plus who click on the link to MDRC’s Main Findings overview will then go on to download and carefully read the executive summary and main report? Some may but many will not. Those who do not may mistakenly assume that the Main Findings overview is a balanced, impartial presentation rather than a one-sided accounting.
On a separate item—in the lead author’s response, she agrees that the study found no effect on the overall poverty rate, but notes that the study found (i) a significant decrease in severe poverty (i.e., income below 50 percent of the poverty threshold) and (ii) a corresponding significant increase in moderate poverty (i.e., income between 50 and 100 percent of the threshold).
We agree that this finding would be promising if valid, but note that it is only suggestive and not yet reliable under accepted scientific standards. The reason is that the study measured numerous primary outcomes—including the six that were pre-registered and, based on the lead author’s response, four others (employment, after-bonus earnings, severe poverty, moderate poverty). This is problematic because, for each outcome that a study measures, there is roughly a one in 20 chance that the test for statistical significance will produce a false-positive result when the program’s true effect is zero. So if, as in this case, a study has many primary outcomes, the chance of false-positives is substantial. The established method of correcting for this problem—described, for example, in the latest FDA draft guidance—is to apply a statistical adjustment across the study’s primary outcomes. When we do so,[v] the effects on severe and moderate poverty are no longer statistically significant and thus best regarded as preliminary. We also note that the study found a suggestive adverse effect on participants’ self-reported material hardship,[vi] and no significant effect on food insecurity,[vii] adding further uncertainty to the idea that Paycheck Plus reduced severe poverty.
We thank the lead author for engaging in this colloquy, and hope it is helpful to readers.
References:
* The Laura and John Arnold Foundation (LJAF) did not provide funding support for this research, but—for purposes of disclosure—we note that LJAF provided funding support for outreach to state and federal policymakers and practitioners on the problem, the intervention, and early findings of the study. LJAF is also providing partial funding support for a separate RCT of Paycheck Plus in Atlanta.
[i] Also, 29 states and the District of Columbia administer their own versions of the EITC program.
[ii] The study measured household income, poverty, criminal justice involvement, via a survey administered approximately 32 months (2.7 years) after random assignment.
[iii] The study measured earnings and child support payments over the three-year study period using administrative data (IRS tax records and New York City child support records, respectively).
[iv] This effect on income from other household members is not included in MDRC’s study report; we learned of it through email correspondence with the lead study author.
[v] We used the Benjamini-Hochberg method of controlling the false-discovery rate.
[vi] Specifically, 51 percent of the treatment group reported at least one material hardship in the past 12 months versus 48 percent of the control group (p=0.08).
[vii] Specifically, 24 percent of the treatment group reported that they sometimes or often did not have enough food to eat versus 23 percent of the control group. This difference was not statistically significant.