The Friedman Foundation may declare that the new Brookings Institute report on school vouchers uses the ‘Gold Standard’ in research but certainly the foundation’s founder wouldn’t have won his Nobel Prize using such methods.
Yes, Milton Friedman, the father of school choice, would have lauded the finding that school vouchers improve college-going rates for black students, although no other student groups. But he would have known enough not to anoint such research as the ‘Gold Standard’. Not that the author’s did a sloppy job, not at all, it was actually a well done study. The fact of the matter is there were significant limitations to this study, as is common in education research, that prevents this study from reaching the ‘Gold Standard’ status and making the broad conclusion that vouchers should be expanded to improve college enrollment rates.
To reach such a standard groups of students would need to be randomly assigned a voucher and then compare the college enrollment rates of those students who received a voucher to those students who did not. By doing so isolates the impact of the voucher from other factors that may impact whether a student goes on to enroll in college. In this study they examined the college enrollment rates of New York City students who took part in a lottery to receive a $1,000 voucher to attend just about any private school in New York City through a private voucher program in the late 1990’s. To achieve randomization—the ‘Gold Standard’– they compared the college enrollment rates of those students in the lottery that were offered a voucher to those students in the lottery who were not offered a voucher.
Such comparison would meet the ‘Gold Standard’ since the only difference between the two groups would be whether they received a voucher or not. However, this is not necessarily the case. Due in part to the fact that 22 percent of the students ‘offered’ a voucher never actually used the voucher to attend a private school. This could be for a couple of reasons:
- Since the voucher didn’t cover the full cost of tuition only those students who had more motivated families that could afford to pay the difference actually used their voucher.
- The students may not have met the admission standards or other requirements for the private schools they were interested in attending.
- The students and their families may have just decided their current public school was still the best fit for the student.
Do we know what impact these differences had on the results? No. But once you have such differences in the groups being compared the study no longer meets the ‘Gold Standard’ since these and other differences could account for any differences in college enrollment rates between the two groups.
Just because the study doesn’t meet the ‘Gold Standard’ for research doesn’t mean the study should be ignored. However, many voucher proponents including the authors seem to be blind to the these limitations and make broad claims about the impact vouchers have on future college enrollments. This study may provide some insight into whether vouchers improve college going rates for low-income black students but it certainly does not prove this is the case. Due to some of the limitations, the study actually created more questions than it answered. Future research needs to address these questions before we know for sure what impact vouchers have on college enrollments:
- Why didn’t all the students who were offered a voucher use their voucher to attend a private school?
- Is there a difference in college enrollment rates between students who used their voucher and those who were offered a voucher but did not use their voucher?
- Why did voucher students stop using their voucher while they were still eligible to receive it? Did they remained in public schools?
- Were students more likely to enroll in college the longer they received a voucher?
- Was there a difference in the quality of high schools attended by students who were offered a voucher than the high schools attended by students who were not offered a voucher?
- Why do vouchers improve the college enrollment rates of low-income black students but not low-income Hispanic students?
- Why would a student who was offered a voucher but never used it be more likely to enroll in college than a student who wanted a voucher but was never offered one?
Without answers to these questions the study adds little insights into what impact expanding voucher programs would have on future college enrollment rates. Unfortunately, since this study failed to meet the Gold Standard policymakers are left with more questions than answers. – Jim Hull
For more insights on the limitations of this study check out the blog post at School Board News Today.