Learn About: 21st Century | Charter Schools | Homework
Home / Edifier

The EDifier

July 5, 2012

Poverty Not the Reason Teachers Labeled Ineffective

It’s no secret that most teachers are not fans of being evaluated based on student test scores.  This is true even though, according to the findings of an upcoming teacher survey from EdSector, the majority of teachers believe they should receive pay increases if they consistently receive outstanding evaluations from their principals . The survey also found that support for performance pay fell to less than a third of teachers if pay increases were based on student test scores. So teachers don’t mind being paid based on their performance but they don’t want their performance determined by test scores.

So why is this the case? Why are teachers so against being evaluated, even in part, based on how their students perform on standardized tests? Of course, there is no single answer but EdSector provides insights into the minds of many teachers. They highlight one teacher who said:

While she loved her job, if her pay depended on student test scores, she “would be crazy” to continue working in her school, which has a student body that is 90 percent low-income and speaks 18 different languages.

She says this because she, like many teachers, believes that standardized tests do not accurately measure the progress low-income and other traditionally disadvantaged students make during a given school year. Essentially, many teachers, just like the one highlighted, believe that their evaluations scores would suffer if they taught a class mostly made up of students from disadvantaged backgrounds.

I certainly don’t blame teachers for having such concerns. There have been many attempts to pay teachers based on their students’ performance that did penalize teachers for working with disadvantaged students.

However, times have changed and statisticians have developed tools to take on this problem. For example, value-added models have been designed to specifically isolate the impact a teacher has on their students no matter their students’ background or previous achievement. Value-added models are not perfect but they do a pretty good job of determining if a student would be better off with one specific teacher instead of another.

Despite the use of value-added models, teachers still believe they would essentially be penalized for having a class of disadvantaged students.  Even researchers who are critical of using value-added data to evaluate teachers believe teachers of disadvantaged students are penalized when they are evaluated based on value-added data.

However, findings from a recent report from the CALDER Center run counter to such criticism. The report found that a teacher’s evaluation score does not suffer if they teach more disadvantaged students. Specifically, CALDER found that teachers in North Carolina that transferred from a low-poverty school to a high-poverty school actually received higher value-added scores in the high-poverty school than they had in the low-poverty school.

Such finding provides strong evidence that teachers of disadvantaged students are not penalized if they are evaluated using high quality value-added data. Even so, as the Center’s report Building a Better Evaluation System argues, no teacher should be evaluated based only on student test scores – or any single measure for that matter. For a teacher evaluation system to yield both accurate results and provide information to help all teachers improve, multiple measures should be used, including student test scores. – Jim Hull

October 31, 2011

Merit pay revisited- Is Denver’s pay for performance a model plan?

Although it remains a controversial issue, merit pay has long since evolved from the days when test scores were the single factor in determining whether a teacher would get paid for performance.  Nowadays a number of school districts across the country have developed multi-pronged plans aimed at equitably rewarding teachers for their accomplishments.  Nonetheless, the question still remains: Is there actually a way to fairly reward a professional who deals with the advancement of human capital?  No plan is perfect, but one district might have come close.

In 2009, The Center took a look at merit pay and made mention of Denver’s ProComp Pay for Performance plan.  Now, a three year study, conducted by Dan Goldhaber and Joe Walch of the Center for Education Data and Research, has come out.  The study was conducted between the fall of 2006 and spring of 2010 on Denver’s ProComp plan. Denver Public Schools (DPS) requires all teachers who were hired in 2006 or later to be a part of the ProComp plan and gives veteran teachers the choice whether to opt in or not.  ProComp offers teachers four opportunities to receive bonuses, which include:

  1. Knowledge and Skills: Teachers may earn pay for completing one professional development unit per year (and can bank extra PDU’s), getting advanced  degrees and licenses, and can even receive tuition and student loan reimbursement (50 to 65 percent received this pay)
  2. Comprehensive Professional Evaluation: Based on principal evaluations, which are every 1 to 3 years (5 to 14 percent received this pay)
  3. Market Incentives: Aimed at teachers who work in hard-to-serve schools and/0r in hard-to-staff subject areas, as reviewed by school demographics and market supply (35 to 65 percent received this pay)
  4. Student Growth: Teachers set up student growth objectives, based on what they expect students to learn, which are approved by the principal (example: I expect x number of students to  exceed expectations in Reading on the Colorado Student Assessment Program (CSAP).) (70 to 80 percent received this pay)

The study suggests that the ProComp plan made teachers feel more supported and in turn, allowed them to more consistently meet their goals (Robles 2011).  In fact, between 2006 and 2010, 15 percent of the non-ProComp teachers even switched over to join the plan after seeing the positive results ProComp had on their schools and colleagues.  Not only has ProComp made the teaching profession more attractive, Goldhaber and Walch conclude that:

  • There were significant learning gains across grades and subjects;
  • The benefits of tracking data and evaluating educators spread from ProComp teachers to the entire district;
  • There was an expectation that the program would cause a negative atmosphere between team members but the opposite actually occurred and role models were bred;
  • ProComp teachers’ students had larger than expected gains on the state assessment.

Skeptics argue that these rewards focus more on classroom instruction than student test achievement and that ProComp is inconsistent with the value-added approach. Goldhaber and Walch point out that, “whether this is good or bad is clearly a normative question” but that “overall, ProComp has had a positive effect.”  They also suggest that states might want to consider investing in similar programs, especially for their Race to the Top objectives. Yesenia Robles of the Denver Post notes that ProComp has helped propel infrastructure reforms to change recruitment practices and enhance methods of data gathering.  She goes on to point out that the difference between non-ProComp and ProComp teachers’ student growth objectives are comparable to the difference between a first and second year teacher’s.   Her article, DPS Teacher-Pay System Likely Boosting Student Achievement, Study Finds, also points out that Denver Public Schools has retained 160 more teachers per year since 2006 and that 80 percent of all DPS teachers currently participate in the program.   Robles notes that, “The ProComp system is already in the process of changing with the implementation of the district’s evaluation-and-support system, known as LEAP, now being tested in 94 percent of DPS schools.”  Right now it is still too early to tell if ProComp can survive these alterations. 

ProComp is an even-handed, well-formed pay for performance plan that other districts can use as a model and will hopefully emulate.  The research shows that ProComp was not only received well by DPS teachers but most significantly, student success consistently progressed. –M. Newport

(To see whether similar pay for performance plans have been successful, check out this ECS report.)

July 22, 2011

New York City stops pay for performance

The New York Times is reporting that a “program that distributed $56 million in performance bonuses to teachers and other school staff members over the last three years will be permanently discontinued” because a study by the RAND corporation found that “the bonuses had no positive effect on either student performance or teachers’ attitudes toward their jobs.”

Before you react, consider this encouraging quote from the article: 

“The results add to a growing body of evidence nationally that so-called pay-for-performance bonuses for teachers that consist only of financial incentives have no effect on student achievement….Teachers also reported that improving as teachers and seeing their students learn were bigger motivators than a bonus, [the researchers] said.”

In an arena where attacks can often be cloaked under the phrase “It’s for the sake of the children,” it’s refreshing to see tangible evidence that children’s learning truly is a powerful motivator. And some of the other reasons given by the researchers for the program’s ineffectiveness, such as not understanding how the awards were determined, also point out that the logistical can be just as important (and far less divisive to fix) than the ideological.

In fact, several of the results of the RAND study seem to correlate with the Center’s earlier research review, “Promise or peril? Teacher pay for performance,” which pointed out that it is often how these programs are developed and structured that makes a difference.

So if your state or district is considering pay for performance, educate yourself first. There’s no use spending money on something that doesn’t work. –Rebecca St. Andrie

Filed under: Pay for Performance,teachers,Uncategorized — Tags: , , — rstandrie @ 10:01 am

January 7, 2011

What does baseball have to do with evaluating teachers?

“Look on the back of their baseball card.”

That’s the common reply from baseball managers and general managers around the major leagues when asked about what kind of production they expect from a player in an upcoming season.

It is a reference to the career stats of baseball players, which appear on the back of their baseball cards. Most often, at the end of the season a player’s stats will be more or less as their career stats.

This basic assumption is so accepted in Major League Baseball that teams offer tens of millions of dollars to those players with the strongest career stats, while players with the weakest stats tend to disappear after a few years.  The stakes don’t get much higher than that.

But in education, it’s still rare for similar high stakes decisions about school personnel (teachers, principals, and other administrators) to be made using data such as student test scores. Critics argue that evaluating educators primarily based on student test scores would be unfair. That even the most complex statistical models designed to isolate educators’ impact, such as value-added growth models, are not reliable enough to base high-stakes decisions such as salaries or tenure.

Now, these critics argue correctly that there is only a moderate correlation between a teacher’s past performance, as measured by value-added models, and their future effectiveness. For example, a Brookings report on value-added models stated that the correlation between test-based measures of teacher effectiveness between one school year and the next is between .30 and .40 (1.0 would mean a perfect correlation and 0.0 would be no correlation). This is a low to moderate correlation in statistical terms. Sounds like the critics have a strong point.

However, Brookings also points out when comparing that correlation to statistics used in other professions, such as Major League Baseball, the correlation doesn’t sound very weak.

Take batting average—one of the most widely used statistics to evaluate a player. The between-season correlation for batting averages for Major League Baseball players is just .36. Yet, a team will pay a player millions for hitting .300 and cut a player for hitting .250.  

Of course batting average is just one statistic teams use to evaluate a player and value-added scores are just one statistic that could be used to evaluate educators, but both can be key measures to determine how effective they will be in the future if used correctly.  

Do teams sometimes make the wrong decision by paying millions for the player who hit .300 last season while cutting the player who hit .250? Yes, it does happen. But overall, the teams are better off signing the players who had the strongest stats on back of his baseball card. Our schools could be more effective if they kept the educators with the strongest stats as well.– Jim Hull

Filed under: Growth Models,Pay for Performance,teachers — Tags: , , — Jim Hull @ 12:50 pm

April 8, 2010

D.C. teachers get pay for performance

Yesterday, the Washington Post announced that, after two years of negotiation, D.C. Schools Chancellor Michelle Rhee and the Washington Teachers’ Union had reached tentative agreement on a new contract.

In that contract was an agreement for “a voluntary pay-for-performance program that will allow teachers to earn annual bonuses for student growth on standardized tests and other measures of academic success.”

I’m sure the program has yet to be developed, but two things are encouraging about that sentence. For one, it talks about student growth, not simply student test scores. Second, since it’s a voluntary program, it’s obvious that the program looks for teacher buy-in.

Pay-for-performance plans are a hot topic right now. The two features I mentioned above are two of the few things we know about what makes them successful. But how much more do we know about whether they really work? Read our full report, “Promise or peril? Teacher pay for performance.–Rebecca St. Andrie

RSS Feed