Learn About: 21st Century | Charter Schools | Homework
Home / Edifier


The EDifier

January 25, 2012

Answering the critics: Misconceptions about value-added measures

Filed under: Growth Models,Teacher evaluation,teachers — Jim Hull @ 10:04 am

Neither education historian Diane Ravitch or Washington Post blogger Valerie Strauss are fans of using value-added measures to evaluate teachers.  [Note: value-added is a statistical term describing the measure of a teacher’s impact on their student’s academic growth – see our report for a further explanation.] Both Ravitch and Strauss are particularly upset with the attention given a recent study on value-added measures, which I wrote about earlier this month.

Apparently, Ravitch and Strauss do not believe, as I do, that the results are that big of a deal. They argue that despite the study’s positive results, using value-added measures to evaluate teachers is a bad idea. Their criticisms pretty much capture the general consensus of value-added critics. But many of these criticisms, though well-intended, are based on misunderstandings of value-added measures, especially when used in teacher evaluation formulas.

In the next few posts, I’ll examine the merits of common criticisms of value-added measures that Ravitch, Strauss and others have highlighted, and point out the misconceptions.

Criticism 1: Studies have shown value-added measures to be unreliable, invalid, and unfair.

Response 1: This is an overstatement. Yes, there are several rigorous studies showing that this is the case, but only if you use a single value-added score to evaluate an individual teacher.

Nobody is seriously proposing to use value-added measures this way. There is no teacher evaluation system I am aware of that even proposes using a value-added score for more than 50 percent of a teacher’s total evaluation. At least half of a teacher’s evaluation would be based on qualitative measures such as principal and peer observations – which, by the way, correlate highly with value-added scores. Other systems propose using statistical techniques that make value-added scores more reliable, such as averaging a teacher’s scores over multiple years.

Keep in mind, too, that although value-added measures are not perfect they are better at identifying the true effectiveness of teachers than the teacher evaluation systems in place now as I show in our report Building a Better Evaluation System.

Criticism 2: Teachers would avoid teaching the most challenging students and avoid teaching in the most challenging schools and districts if teachers were evaluated using value-added.

Response 2: Value-added measures were designed specifically to combat this problem. Yes, previous attempts to evaluate teachers using quantitative measures did result  in teachers avoiding challenging positions. However, value-added measures more accurately isolate a teacher’s impact on students’ test scores by explicitly taking into consideration students’ prior achievement. This means, for instance, that teachers who teach low-performing students are compared to other teachers of low-performing students. In addition, value-added measures are based on the amount of growth students make in a year – not their overall score at the end of the year, as previous methods did.

Strauss adds that value-added can’t possibly measure a teacher’s true effectiveness, since 22 percent of children are in poverty and that poverty is strongly correlated to student achievement. I guess she is assuming that value-added doesn’t take into account a student’s socioeconomic status, but this is untrue. Value-added measures account for all student characteristics, including poverty level. Strauss is correct there is a strong correlation between poverty and a student’s achievement level – that is, a student’s achievement at one point in time. But there is little correlation between poverty and achievement growth — the change in student achievement over time. And value-added measures are based on achievement growth, not level. It’s this focus on growth that makes value-added measures so valuable – and why you should come back tomorrow to read more answers to the criticisms about value added.

– Jim Hull





January 19, 2012

Evaluating teachers: look at the data, please

Filed under: Teacher evaluation,teachers,Uncategorized — rstandrie @ 3:55 pm

A new report by the Measures of Effective Teaching project and supported by the Bill & Melinda Gates Foundation looks into the accuracy of classroom observations, as compared to and combined with other things such as value-added measures. The results are too long to discuss in a blog post, but one point caught my eye.

Much of the report compared the “predictive power” of different evaluations. That is, it looked at how likely any given type of evaluation (classroom observations, rigorous student surveys, value-added measures, or a combination), was to predict student gains on state tests in math and reading.

The graph that caught my eye compared the predictive value of a combination of these evaluations vs. the predictive value of graduate degrees or years of experience, the two measures used most often in determining raises and layoffs. For graduate degrees and experience, the predictive value was slight; but for the combined evaluation measures, the predictive value was noticeably more significant. Students who had the top 25% of teachers (as identified by the combined evaluation) gained roughly between one and five more months’ worth of learning than those who had teachers in the bottom 25%.

The point is this: a combined evaluation, using multiple measures of a teacher’s effectiveness including value-added measures, was more effective in identifying the teachers who boosted students learning than any of the traditional measures such as experience or degrees. And the report showed elsewhere that a combined evaluation was more effective than any single evaluation measurement.

My takeaway? Please, let’s use data in evaluating teachers. It doesn’t just have to be numbers — for instance, those rigorous student surveys did a good job, too. But as we argued in Building a Better Evaluation System, value-added models have a place in evaluating teachers as part of a system of multiple measures.

It’s time to build a better evaluation system. It won’t be perfect, but the evidence is mounting that it will be better than what we’ve got. –Rebecca St. Andrie

 





November 17, 2011

Teacher Evaluations- The Chicago Approach

Filed under: instruction,Teacher evaluation,teachers — Mandy @ 10:01 am

In an era where the stakes are high and many districts are considering linking evaluations to merit, it is important that these evaluations are accurate and that observers are properly trained.  A recent study from the University of Chicago Consortium on School Research, called Rethinking Teacher Evaluation in Chicago, found that teachers in Chicago with the highest principal observation ratings were the same teachers whose students showed the greatest gains in learning. This correlation suggests that principals in Chicago are able to “distinguish between strong and weak teaching” to “capture factors that matter for student learning”(Chicago Teacher Evaluation Pilot Shows Promise for Fairly, Accurately Evaluation Teachers).

So how are these principals in Chicago able to be so spot on with their evaluations?  According to the study, their secret weapon is the Chicago’s Excellence in Teaching program, which was piloted in 44 schools in 2008 and has since expanded (Urban Education Institute, 2011). This program uses the Charlotte Danielson Framework for Teaching as its model. The elements observed in Framework for Teaching observations are much more exhaustive than those found on your average check sheet. Below are the four domains upon which Framework for Teaching observations are based: (Charlotte Danielson Framework for Teaching)

Domain 1: Planning and Preparation
  • Demonstrating Knowledge of Conten and Pedagogy Demonstrating
  • Knowledge of Students
  • Setting Instructional Outcomes
  • Demonstrating Knowledge of Resources
  • Designing Coherent Instruction
  • Designing Student Assessments
Domain 2: The Classroom Environment
  • Creating an Environment of Respect and Rapport
  • Establishing a Culture for Learning
  • Managing Classroom Procedures
  • Managing Student Behavior
  • Organizing Physical Space
Domain 3: Instruction
  • Communicating with Students
  • Using Questioning and Discussion
  • Techniques
  • Engaging Students in Learning
  • Using Assessment in Instruction
  • Demonstrating Flexibility and Responsiveness
Domain 4: Professional Responsibilities
  • Reflecting on Teaching
  • Maintaining Accurate Records
  • Communicating with Families
  • Participating in a Professional
  • Community
  • Growing and Developing Professionally
  • Showing Professionalism

Some of the study’s findings include:

1. Students with the greatest growth on test scores were in classrooms where teachers received the highest ratings on their Framework for Teaching evaluations. Conversely, students who showed the least growth were in classrooms where teachers received the lowest ratings.

2. Principals and trained observers gave consistent ratings (based on watching the same teacher conduct the same lesson). This evidence of fairness might be something for other districts to consider when using classroom observations to make hiring, firing and promotion decisions.

3. Conferences between teachers and principals became more thoughtful and less subjective than in the past. The conversations were centered on instructional practice.

4. More than half of the principals were positive about the Framework for Teaching approach, follow-up conferences, and the professional development that they received.

5. Overall, 72% of principal ratings were found to be consistent and fair. The other 28% were either too high or too low.

6. Although principals and teachers were generally positive about the evaluation system and conferences, many principals were not equipped to have deep discussions about teaching practice.

“These findings have important policy implications for states and districts across the country working to implement evaluation systems that include classroom observations” (Urban Education Institute, 2011). This pilot program is not perfect, but it does give principals a better eye for what works in the classroom and gives teachers a clearer definition of “effective teaching” while providing them with ”feedback on how they stack up on those criteria.” 

Chicago’s Excellence in Teaching program is still a work in progress, with an obvious need for more professional development and ongoing support for principals and teachers. However, it is a huge forward step for other districts to possibly consider as a model. According to the Illinois State Superintendent, Christopher A. Koch, “This study shows that we’re moving in the right direction” and “shows the observation methods we’re moving toward are valid and reliable measures of solid teaching.”

–M. Newport





November 9, 2011

Response: Are Teachers Overpaid?

Filed under: funding,Public education,teachers — Tags: , — Mandy @ 10:31 am

Last week, The Heritage Foundation’s Jason Richwine and Andrew Biggs addressed why they feel that teachers are overpaid. Since teachers are more likely to hold another job to pay the bills than any other profession in America (according to a study conducted by the Bureau of Labor Statistics (BLS study), I disagree.  

Below you’ll find some of my responses to Richwine and Biggs:

 “A teacher who receives a given salary for nine months of work is clearly better compensated than someone who earns the same salary for a full year’s work.”

  • Response: The Organization for Economic and Cooperative Development (OECD) found that American teachers spend more hours per year working than do teachers in any other country (Education at a Glance 2011 - OECD).  On paper, teachers work about 7.5 hour days with 30 minute lunch breaks, teach for 180 days, and work for 195 days. However, the reality is that teachers have homework. According to the BLS study, 79 % of American public school teachers begin their workday at 7am and 51% conduct another full day’s worth of work over the weekend. Teachers get 10 weeks off in the summer, in addition to holiday breaks, but this time is often used for professional development workshops, setting up classrooms, adjusting to a change in assignment or grade level, and once again, planning.

“After full accounting, benefits for teachers are shown to be significantly more generous than those paid to employees of large private-sector establishments.”

  • Response: Comparing benefits is complex since each state handles (each district as a matter of fact) perks very differently. However, the report did not take these differences into consideration when comparing private and teacher benefits.

“Public school teachers earn less in wages…than non-teachers with the same level of education….The wage gap disappears when both groups are matched on cognitive ability rather than on years of education.”

  • Response: Just as in business, where the ability to communicate and network is as valuable as academic ability, effective teachers have abilities that go beyond the SAT scores used by this report. However, testing out this theory would require increasing, not decreasing, teacher salaries. Research into shortages of STEM teachers, for instance, shows that students with higher SAT scores and GPAs head into the higher-paying, private sector jobs available.

“Reducing teacher compensation…could be put toward classroom materials…”

  • Response: According to last year’s study by the National School Supply and Equipment Association (NSSEA), “public school teachers in the United States spend more than $1.33 billion on school supplies and instructional materials” (NSSEA – Publications: Industry Reports). On average, teachers spend $552 on supplies per year (currently less with the suffering economy), assuming that the average PTA is spending $936 per class. Teacher spending can actually triple in cases where the PTA does not spend as much.  As someone who worked at a school (that didn’t have paper for a period of time) and at a teacher resource store, I have witnessed teachers not even think twice about paying out of pocket to compensate for a lack of resources. Reducing teacher salaries would decrease classroom supplies, since teachers are often the ones that pay for them.

In conclusion, America has grown more and more focused on international tests that compare our students to those of other industrialized countries. We are inspired by these countries’ high math scores, work ethic, and use of technology.  This inspiration has spurred reforms such as holding our teachers more accountable, but we have yet to be inspired by their treatment of teachers. According to the OECD’s study, teachers in other industrialized countries make an average of 117% more than American public school teachers and, in Korea, teachers make a whopping 221% more.

If we even have to address whether our teachers are being overpaid, we have failed (at the perception of what the job is and should be) and are far from where we need to be. –Mandy Newport





November 7, 2011

Filling the teacher gap

Filed under: Teacher evaluation,teachers — Tags: , — Patte Barth @ 11:14 am

Baby boomers comprise half of the current teaching force. Over the next ten years, schools will lose all of them to retirement.  That’s a lot of teachers to replace.  Yet at present, about half of new teachers leave the profession before their first five years are up.  If nothing changes, the nation’s schools could well find themselves scrambling to put adults in our classrooms at the same time they’re trying to guarantee every student is taught to higher standards by a highly effective teacher.

The National Association of State Boards of Education offers its answer to the conundrum in their new report, Gearing up: Creating a systemic approach to teacher effectiveness. The current teacher policy discussion is dominated by a push for greater teacher accountability including calls to make it easier to dismiss ineffective teachers. So it’s refreshing to see NASBE address the issue from recruitment to preparation to retention. Evaluation is also key. But while a lot of proposed evaluation plans can seem like punishment to teachers, the emphasis here is on evaluation as a tool for continuous improvement.

 The more notable recommendations include:

  • “Significant” exposure to clinical experiences, or what many of us know as student teaching.  NASBE cites several good examples of intensive programs, such as Professional Development Schools and Teacher Residencies. Understanding that these can be costly, they encourage schools and universities to at minimum make sure teacher candidates are paired with highly effective mentors or cooperating teachers, and that the experiences are integrated into teachers’ preparation from the very beginning.
  • Rigorous and comprehensive assessments that new teachers must pass to complete the program. The assessments must certify that teachers are able to teach students to be college and career-ready.
  • Multi-year induction programs. In order to reduce the high attrition rate of new teachers, NASBE urges districts to develop a multi-year induction process that will provide support for teaching novices as they adjust to the classroom. We would add that schools should also rethink assignment policies that allow veteran teachers to claim the cushiest teaching spots, leaving inexperienced teachers with the most challenging classrooms — a match up that serves neither the fledgling teacher nor her students.
  • Evaluation systems designed for the purpose of improving instruction. As we have written many times, the value of value-added data is its power to isolate the effect of programs and practices so that educators can learn from what’s working and correct what isn’t.  This message seems to get lost when teachers don’t trust the data, how it will be used, and who’s using it.  Taking a lesson from our friends at the Data Quality Campaign, it’s better to use data as a flashlight than a hammer.

 NASBE has found real-life examples of each of the recommendations that make for good reading.  A pdf of the executive summary can be downloaded here.

Also check out the Center’s reports on teacher recruitment and building better evaluation systems.  – Patte Barth





« Newer Posts — Older Posts »
RSS Feed