Learn About: 21st Century | Charter Schools | Homework
Home / Edifier


The EDifier

January 26, 2012

Working through further misconceptions about value-added measures

Filed under: Growth Models,Teacher evaluation,teachers,Uncategorized — rstandrie @ 4:53 pm

In previous posts, I’ve taken issue with many of the criticisms leveled by Diane Ravitch and Valerie Strauss, who disputed the findings from the recent study by Harvard and Columbia economists. That study found students who had teachers with high value-added scores were more successful later in life than students who had teachers with lower value-added scores.

Since I believe many of the criticisms were based on common misconceptions about value-added measures, let’s keep going:

Criticism 6: The study predominately studied teachers and students in the 1990’s when there was no test-based accountability. Value-added measures as well as future student outcomes may differ if consequences were attached to test scores since teachers may focus more on raising scores than providing richer instruction and would likely negatively impact students’ future outcomes.

Response 6: Basing the study on teachers and students from the 1990’s was necessary in order to examine the impact of teachers on elementary students ten years after they graduated from high school. So the findings on the long-term impact are actually based on very recent data.

 It is also true that the results may differ if high stakes were placed on the test results.  Keep in mind, however, that if the data did indeed come from New York City (as Ravitch and Stauss assume), test-based accountability had been in place since the late 1980’s, and that accountability was particularly focused on low-performing schools enrolling predominately poor students.  Even if the data is not from New York City, the fact that the district was testing students and collecting so much data shows it is likely there was some sort of test-based accountability in place at the time.

Yet the study found the impact of having a high value-added teacher was the same for students of all income levels. So even the poorest students, who were much more likely to attend schools where teachers were under pressure to raise test scores, benefited as much from high value-added teachers as higher-income students, who were less likely to attend a school that faced any accountability pressure.

 Criticism 7: The study doesn’t show that value-added can accurately identify individual teachers as effective or ineffective.

Response 7: As I stated previously, no one seriously argues that that individual teachers should be evaluated using value-added measures alone. I am not aware of any proposed or current teacher evaluation systems that have value-added scores account for more than 50 percent of a teacher’s overall evaluation. As the Center for Public Education report Building a Better Evaluation System states, value-added scores can be an effective tool in accurately identifying effective and ineffective teachers, but they should be used within the context of a comprehensive evaluation system that includes observations and other qualitative measures of a teacher’s performance.

Tomorrow I’ll finish the list of misconceptions and criticisms by Strauss and Ravitch. – Jim Hull





January 25, 2012

More misconceptions about value-added measures

Filed under: Growth Models,Teacher evaluation,teachers,Uncategorized — rstandrie @ 4:17 pm

Yesterday I responded to some common criticisms of value-added measures to evaluate teachers. As I mentioned yesterday, these criticisms have been used by critics such as Diane Ravitch and Valerie Strauss to  dispute the findings from the recent study by Harvard and Columbia economists. That study found students who had teachers with high value-added scores were more successful later in life than students who had teachers with lower value-added scores. While many, including me, believed that the study provides strong evidence that value-added measures can be an effective tool in identifying effective teachers, Ravitch and Strauss took exception to such a conclusion.

To be clear, neither Ravitch nor Strauss is questioning the report’s methodology or its findings.  As Ravitch notes “The problems of the study are not technical, but educational.” However, I believe that the criticisms they raise stem more from common misconceptions about value-added scores and how they are used. Here are some additional criticisms, with my responses.

Criticism 3: Teachers in subjects not subject to testing, such as art, music, physical education, and many times social studies do not have value-added scores.

Response 3: I took on this issue in the Center’s Building a Better Evaluation System report. It is a limitation of value-added models, but there are systems in place that still are able to evaluate teachers in non-tested subjects. One technique is to use the school’s value-added score in place of the individual teacher’s value-added score, since all teachers contribute to the success of the school. The school’s value-added score, combined with other measures like observations, would provide a more accurate measure of teacher effectiveness. Not even the most ardent support of value-added scores believes that a decision about an individual teacher should be made based on a value-added score alone.

Criticism 4: It is no wonder higher performing students are more likely to go to college.

Response 4: The key word here is ‘higher.’ If the study found that previously high-performing students were more likely to go on to college, there’d be little to talk about. But it didn’t. The study found a student who had a teacher with a high value-added score was more likely to go to college than if that same student had had an average teacher, and much more likely than if that student had had a teacher with a low value-added score.

Criticism 5: The impact of highly effective teachers on student incomes is relatively small at $250 per year.

Response 5: The number that Ravitch and Strauss cite is the impact of having one high value-added teacher for one year and having average teachers for all other years. Much like compound interest, having more than one highly effective teacher really starts to pay off, especially when you compare it to the impact of poor teachers. For instance, having four highly effective teachers instead of four teachers with low value-added scores would improve a student’s future income by $2,000 per year. That’s not chump change! And the income difference was calculated at age 28 – just before the income gap between college and non-college students really starts to widen.

Keep reading tomorrow for the final set of criticisms. – Jim Hull





Answering the critics: Misconceptions about value-added measures

Filed under: Growth Models,Teacher evaluation,teachers — Jim Hull @ 10:04 am

Neither education historian Diane Ravitch or Washington Post blogger Valerie Strauss are fans of using value-added measures to evaluate teachers.  [Note: value-added is a statistical term describing the measure of a teacher’s impact on their student’s academic growth – see our report for a further explanation.] Both Ravitch and Strauss are particularly upset with the attention given a recent study on value-added measures, which I wrote about earlier this month.

Apparently, Ravitch and Strauss do not believe, as I do, that the results are that big of a deal. They argue that despite the study’s positive results, using value-added measures to evaluate teachers is a bad idea. Their criticisms pretty much capture the general consensus of value-added critics. But many of these criticisms, though well-intended, are based on misunderstandings of value-added measures, especially when used in teacher evaluation formulas.

In the next few posts, I’ll examine the merits of common criticisms of value-added measures that Ravitch, Strauss and others have highlighted, and point out the misconceptions.

Criticism 1: Studies have shown value-added measures to be unreliable, invalid, and unfair.

Response 1: This is an overstatement. Yes, there are several rigorous studies showing that this is the case, but only if you use a single value-added score to evaluate an individual teacher.

Nobody is seriously proposing to use value-added measures this way. There is no teacher evaluation system I am aware of that even proposes using a value-added score for more than 50 percent of a teacher’s total evaluation. At least half of a teacher’s evaluation would be based on qualitative measures such as principal and peer observations – which, by the way, correlate highly with value-added scores. Other systems propose using statistical techniques that make value-added scores more reliable, such as averaging a teacher’s scores over multiple years.

Keep in mind, too, that although value-added measures are not perfect they are better at identifying the true effectiveness of teachers than the teacher evaluation systems in place now as I show in our report Building a Better Evaluation System.

Criticism 2: Teachers would avoid teaching the most challenging students and avoid teaching in the most challenging schools and districts if teachers were evaluated using value-added.

Response 2: Value-added measures were designed specifically to combat this problem. Yes, previous attempts to evaluate teachers using quantitative measures did result  in teachers avoiding challenging positions. However, value-added measures more accurately isolate a teacher’s impact on students’ test scores by explicitly taking into consideration students’ prior achievement. This means, for instance, that teachers who teach low-performing students are compared to other teachers of low-performing students. In addition, value-added measures are based on the amount of growth students make in a year – not their overall score at the end of the year, as previous methods did.

Strauss adds that value-added can’t possibly measure a teacher’s true effectiveness, since 22 percent of children are in poverty and that poverty is strongly correlated to student achievement. I guess she is assuming that value-added doesn’t take into account a student’s socioeconomic status, but this is untrue. Value-added measures account for all student characteristics, including poverty level. Strauss is correct there is a strong correlation between poverty and a student’s achievement level – that is, a student’s achievement at one point in time. But there is little correlation between poverty and achievement growth — the change in student achievement over time. And value-added measures are based on achievement growth, not level. It’s this focus on growth that makes value-added measures so valuable – and why you should come back tomorrow to read more answers to the criticisms about value added.

– Jim Hull





January 9, 2012

Are value-added measures accurate?

Filed under: Growth Models,Report Summary,Teacher evaluation — Jim Hull @ 11:43 am

It’s just one study, but a new study from Harvard and Columbia economists provides powerful evidence that value-added measures can be an accurate tool in evaluating the effectiveness of teachers. As CPE found in its report, Building a Better Evaluation System, although value-added measures are not perfect, they can be valuable tools in identifying highly effective teachers. This study provides even stronger evidence that this is indeed the case.

The report doesn’t dismiss the limitations of value-added that I discussed in my report: most importantly, that value-added scores tend to fluctuate from year to year, although even one year’s worth of value-added data can be useful. However, the Harvard/Columbia report points out that teacher value-added scores averaged over three or four years are accurate measures of a teacher’s true effectiveness. They were able to determine this in two ways:

1) By finding that when a previously identified highly effective teacher transferred to a new school, that school’s achievement improved within the grade the teacher was assigned. Conversely, when a highly effective teacher left a school, the school’s achievement fell in the grade the teacher previously taught.  Furthermore, the change in scores in both instances matched the change predicted based on teacher’s value-added score.

2) Students of highly effective teachers had better life outcomes than students who had average effective teachers. Students of highly effective teachers were less likely to be teenage parents and more likely to go on to college as well as earn higher wages and save more for retirement.  

These are very important findings, especially considering the authors actually set out to show that value-added measures didn’t work – that they had more to do with student motivation or principal selection or other factors outside the control of teachers, as the authors stated in this New York Times article. Yet, they proved themselves wrong. The data not only showed that teachers matter, but that they matter a great deal to their students’ long-term outcomes. They also showed that value-added measures can be an effective and reliable tool administrators and policymakers can use to ensure all students have access to good teachers. – Jim Hull





October 31, 2011

Merit pay revisited- Is Denver’s pay for performance a model plan?

Filed under: Growth Models,Pay for Performance,teachers — Tags: , , — Mandy @ 4:19 pm

Although it remains a controversial issue, merit pay has long since evolved from the days when test scores were the single factor in determining whether a teacher would get paid for performance.  Nowadays a number of school districts across the country have developed multi-pronged plans aimed at equitably rewarding teachers for their accomplishments.  Nonetheless, the question still remains: Is there actually a way to fairly reward a professional who deals with the advancement of human capital?  No plan is perfect, but one district might have come close.

In 2009, The Center took a look at merit pay and made mention of Denver’s ProComp Pay for Performance plan.  Now, a three year study, conducted by Dan Goldhaber and Joe Walch of the Center for Education Data and Research, has come out.  The study was conducted between the fall of 2006 and spring of 2010 on Denver’s ProComp plan. Denver Public Schools (DPS) requires all teachers who were hired in 2006 or later to be a part of the ProComp plan and gives veteran teachers the choice whether to opt in or not.  ProComp offers teachers four opportunities to receive bonuses, which include:

  1. Knowledge and Skills: Teachers may earn pay for completing one professional development unit per year (and can bank extra PDU’s), getting advanced  degrees and licenses, and can even receive tuition and student loan reimbursement (50 to 65 percent received this pay)
  2. Comprehensive Professional Evaluation: Based on principal evaluations, which are every 1 to 3 years (5 to 14 percent received this pay)
  3. Market Incentives: Aimed at teachers who work in hard-to-serve schools and/0r in hard-to-staff subject areas, as reviewed by school demographics and market supply (35 to 65 percent received this pay)
  4. Student Growth: Teachers set up student growth objectives, based on what they expect students to learn, which are approved by the principal (example: I expect x number of students to  exceed expectations in Reading on the Colorado Student Assessment Program (CSAP).) (70 to 80 percent received this pay)

The study suggests that the ProComp plan made teachers feel more supported and in turn, allowed them to more consistently meet their goals (Robles 2011).  In fact, between 2006 and 2010, 15 percent of the non-ProComp teachers even switched over to join the plan after seeing the positive results ProComp had on their schools and colleagues.  Not only has ProComp made the teaching profession more attractive, Goldhaber and Walch conclude that:

  • There were significant learning gains across grades and subjects;
  • The benefits of tracking data and evaluating educators spread from ProComp teachers to the entire district;
  • There was an expectation that the program would cause a negative atmosphere between team members but the opposite actually occurred and role models were bred;
  • ProComp teachers’ students had larger than expected gains on the state assessment.

Skeptics argue that these rewards focus more on classroom instruction than student test achievement and that ProComp is inconsistent with the value-added approach. Goldhaber and Walch point out that, “whether this is good or bad is clearly a normative question” but that “overall, ProComp has had a positive effect.”  They also suggest that states might want to consider investing in similar programs, especially for their Race to the Top objectives. Yesenia Robles of the Denver Post notes that ProComp has helped propel infrastructure reforms to change recruitment practices and enhance methods of data gathering.  She goes on to point out that the difference between non-ProComp and ProComp teachers’ student growth objectives are comparable to the difference between a first and second year teacher’s.   Her article, DPS Teacher-Pay System Likely Boosting Student Achievement, Study Finds, also points out that Denver Public Schools has retained 160 more teachers per year since 2006 and that 80 percent of all DPS teachers currently participate in the program.   Robles notes that, “The ProComp system is already in the process of changing with the implementation of the district’s evaluation-and-support system, known as LEAP, now being tested in 94 percent of DPS schools.”  Right now it is still too early to tell if ProComp can survive these alterations. 

ProComp is an even-handed, well-formed pay for performance plan that other districts can use as a model and will hopefully emulate.  The research shows that ProComp was not only received well by DPS teachers but most significantly, student success consistently progressed. –M. Newport

(To see whether similar pay for performance plans have been successful, check out this ECS report.)





Older Posts »
RSS Feed