Learn About: 21st Century | Charter Schools | Homework
Home / Edifier


The EDifier

October 17, 2013

Seeing results takes time

As one of the first districts to make student achievement a major component of a comprehensive teacher evaluation system, Washington, DC is now seeing some of the benefits according to a new study from the National Bureau of Economic Research. The NBER authors find that since DC implemented its teacher evaluation systems called IMPACT, 4 percent of the lowest rated teachers were dismissed and 30 percent of minimally effective (IMPACT’s second lowest category) voluntarily left the district. Of course, losing lower performing teachers is only good for students if they are replaced by more effective teachers. The good news is that according to the study this is what is happening in DC.

However, as I argued in CPE’s latest report, Trends in Teacher Evaluation , in order for IMPACT and similar evaluation systems to truly be effective they need to focus on improving and supporting all teachers and not just concentrate on identifying and dismissing a small percentage of continuously low-performing teachers. While IMPACT focuses more on dismissing low-performing teachers than most evaluation systems, it does provide all teachers with supports, such as instructional coaches, to continuously improve their performance as well as provide financial incentives to retain high performing teachers. The evidence is less clear on the effect of DC’s new evaluation system in these areas.

Critics point to the fact that DC test scores have remained relatively flat since the implementation of IMPACT to counter claims of the evaluation system’s success. Although I argue any good evaluation system should result in increased student achievement, such results take time, especially when looking at a large district like DC. There are no silver bullets in education. This is true for the new teacher evaluations systems being implemented in over 40 states. They will not improve student achievement overnight. It will take time to accurately identify teachers who shouldn’t be in the classroom and replace them with more effective teachers. And as CPE’s recent report on teacher professional development found, it takes time for instructional supports to change teacher practices as well.

As such, it is not surprising DC hasn’t seen a significant increase in student achievement as of yet. However, this doesn’t mean that DC should take a wait and see approach. District leaders need to continually listen to teachers, administrators, and even students to get their feedback on how IMPACT should be improved and make adjustments as necessary just as they have done in the past. For districts in any state to get the greatest impact, district leaders need to continuously update their evaluation systems to not only ensure they are accurately evaluating teachers but are providing teachers timely feedback and the support they need to implement the best instructional practices. Teacher evaluation systems should never be static but be living policies that adjust with the ever changing needs of the district and their teachers, and eventually have a positive impact on student achievement. – Jim Hull

Filed under: Teacher evaluation,teachers — Tags: — Jim Hull @ 3:33 pm





August 20, 2013

Wrong about standardized tests

1335-1243972165NX9TA couple weeks ago I blogged that it seemed there were as many defenders of standardized testing as there were defenders of Alex Rodriguez and Lance Armstrong. My perception was influenced by the many blogs and news articles I continually read as well as the number of conversations I have had with researchers, policymakers, and educators about dangers of standardized testing. Of course, these are anecdotal and not based on any hard data which is why I was completely wrong in assuming standardized tests had few supporters.

In fact, standardized tests have significant support from a group within the education community that I failed to recognize, parents. Students may be the ones actually receiving an education but in reality parents are the true consumers of education which make their opinion extremely valuable, certainly much more valuable than any blogger, journalist, researcher, or policy analyst (including yours truly).

Which is why the results from a recent poll conducted by the Associated Press- NORC Center for Public Affairs Research on parent attitudes of the quality of education in the United States are so intriguing. While the perception is that standardized tests are as popular as serial cheaters, a large majority of parents support the use of standardized testing. Nearly three out of four parents believe it is very important or extremely important that their child’s school regularly assess whether or not their child is meeting statewide expectations for their grade level. Perception is also that students are just taking too many tests, yet just 26 percent of parents believe this to be true. Another perception is that students in low-performing schools are being inundated with standardized tests yet parents of children in low-performing schools are more likely to state their child takes too few standardized tests than parents in higher-performing schools.

Contrary to popular belief, parents believe standardized tests provide an accurate measure of their child’s ability and the quality of the school their child attends. In fact, 75 percent of parents say that standardized tests are an accurate or somewhat accurate measure of their child’s performance while 69 percent of parents believe they are an accurate or somewhat accurate measure of their child’s school. This is likely why over 90 percent of parents believe standardized tests should be used to identify areas where students need extra help and why over 80 percent of parents believe they should be used to ensure all students meet grade-level standards. The majority (60 percent) of parents also believe that standardized tests should be used to evaluate teachers although they also believe teachers should be evaluated based on classroom observations and student feedback as well.

If perception was indeed reality then standardized testing would only be popular among policymakers and so-called reformers who have forced standardized tests on public schools without any regard to those who are impacted most. While the poll shows a lack of support by teachers, a large majority of parents do in fact support the use of standardized testing for a variety of reasons. They see it is a valuable tool to help improve their child’s performance as well as determine how their child stacks up to grade-level standards.  Of course, standardized tests are just one tool and other measures should be used to evaluate students and schools as well. However, we shouldn’t throw the baby out with the bath water so we should be continually working to find the right balance of standardized tests and other measures to evaluate and improve the performance of all our students and schools and end the most outrageous perception of all, that our public schools are failing. – Jim Hull

Filed under: Assessments,Parents,Public education,Testing — Tags: , , — Jim Hull @ 2:21 pm





July 12, 2013

Greater flexibility in teacher assignments has its benefits

Research has consistently shown that the most disadvantaged students are less likely to have an effective teacher than their more advantaged peers. This is not only true across districts but within districts as well. Some have blamed this phenomenon—at least in part—on policies that prevent school leaders from replacing ineffective teachers and have argued that if principals were given the flexibility to move an ineffective teacher to another school within the district (called forced transfer) more disadvantaged students would have access to high quality teachers.

To date, there has been little research on whether such flexibility would lead to the intended outcomes. But a recent study found in fact that flexibility can lead to greater teacher effectiveness in low-performing urban schools. The study is not definitive since it is based on only one school district (Miami-Dade). But it found that principals in low-performing schools identified low-performing teachers– as measured by their value-added scores and number of days the teacher was absent—for transfer and were able to replace those teachers with more effective teachers in terms of both test scores (value-added scores) and fewer teacher absences. While the less effective teachers were more likely to be transferred to a much higher performing school with fewer disadvantaged students, unfortunately they were no more effective in their new school. Of course this leads to bigger discussion about what districts should do with ineffective teachers which is a topic for another post.

Such findings give credence to the theory that greater district flexibility in teacher assignments will lead to a more equitable distribution of teachers so disadvantaged students can get their fair share of effective teachers. In the long-term such a policy can have a big effect on narrowing achievement gaps as research has consistently shown the tremendous impact effective teachers have on student achievement.—Jim Hull

Filed under: Achievement Gaps,research,Teacher evaluation,teachers — Tags: , , — Jim Hull @ 4:07 pm





June 20, 2013

A false comparison

I have a lot of respect for Dr. Linda Darling-Hammond (LDH). She is a wonderful advocate for teachers and public schools in general. She not only highlights the strengths of our teachers and our public schools but is continually finding methods to improve them as well. In fact, she is one of the reasons I got into education policy research.

However, from time to time when she is advocating for teachers and our public schools she makes false comparisons that one wouldn’t expect from a respected researcher. She made such a comparison in her critique of the recent National Center on Teacher Quality’s report card on schools of education which found the vast majority of schools of education to be of poor quality.

LDH took exception to the findings and pointed out several flaws in the report card’s methodology and data collection. However, her criticism that the report card did not connect schools of education to the actual effectiveness of their graduates, while a fair point, was not backed by sufficient evidence. Specifically she stated:

In this study, the highest-achieving states on the National Assessment of Educational Progress (NAEP)— including Massachusetts, Vermont, New Hampshire, Maine, New Jersey, and Minnesota — all got grades of C or D, while low-achieving Alabama got the top rating from NCTQ.  It is difficult to trust ratings that are based on criteria showing no relationship to successful teaching and learning.”

As the former president of the American Education Research Association (AERA) she knows that just because Massachusetts’ students score higher on NAEP than students in Alabama it doesn’t mean teachers in Massachusetts are more effective than teachers in Alabama. While teachers have a significant impact on student achievement and can close achievement gaps, student background factors like their socioeconomic status are highly correlated to a student’s overall achievement level. This is the rationale behind measuring student’s annual growth, which has less to do with their background characteristics and more to do with the effectiveness of their teacher. Since Alabama has many more students from disadvantaged backgrounds than Massachusetts, Vermont, New Hampshire, Maine, New Jersey, and Minnesota, it is possible that students in Alabama made greater gains while in school than students in the high-performing states but they just started school at a lower level so had lower overall achievement scores.

LDH knows a more accurate way to compare the effectiveness of teachers in Alabama to their peers in other states like Massachusetts is by using a value-added model that measures a teacher’s effectiveness based on the achievement growth of their students while controlling for student background characteristics. Value-added measures may not be perfect but they provide a much more accurate comparison than judging the quality of a state’s teachers by their overall NAEP scores.

While I agree with LDH that schools of education should be evaluated based on the effectiveness of their graduates once they are in the classroom, such data is not possible at the moment. In fact, schools of education have not exactly embraced such attempts to evaluate them as such. If schools of education want to be more fairly evaluated they should do it themselves and be more transparent about how effective they really are. Taxpayers are sending millions of dollars in the form of subsidized loans and loan forgiveness programs to these schools with no evidence of a return on their investment. With nearly half of teachers leaving the profession within five years and research consistently showing the lack of effectiveness of most first-year teachers, the evidence we do have shows teachers are not prepared for their first year in the classroom. Unfortunately, we have relied on teachers ‘learning by doing’ instead of our schools of education adequately preparing them to be effective their first day in the classroom.

Most of our teachers are very good but most didn’t start off that way. If all our current teachers were nearly as effective in their first year as in their fifth year, the U.S. would have one of the most effective teaching forces in the world.– Jim Hull

Filed under: Growth Models,NAEP,Public education,Teacher evaluation,teachers — Tags: — Jim Hull @ 9:22 am





June 11, 2013

A teacher’s misconceptions of using test scores to evaluate teachers

I certainly understand teachers’ angst when it comes to being evaluated based on their student test scores. Being evaluated is not a fun experience but when that evaluation is also based on the performance of others it makes the process that much more stressful. To make matters worse there appears to be a number of misconceptions regarding exactly how test scores are used to evaluate teachers.

USA Today editorial by Alexandria, VA high school teacher Patrick Welch is an example of some of the misconceptions many teachers have about using tests to evaluate them. Mr. Welch’s observations are far from unique. I have heard similar misconceptions from other teachers and even researchers which makes identifying such misconceptions all the more important.

New teacher evaluation systems more accurate than previous evaluations systems

First of all, it must be recognized that there is no such thing as a perfect evaluation system in any profession. So while even the best evaluation systems will certainly identify some truly effective teachers as ineffective it doesn’t mean teachers shouldn’t be evaluated at all. However, evaluation systems should be designed to minimize the chances of such occurrences.  As explained in our Building a Better Evaluation report, there are a number of tools available to make evaluations more accurate.

Test scores can accurately measure a teacher’s performance when used with other measures

Mr. Welch cites research showing the inaccuracies of using test scores to evaluate teachers. For example, some teachers are identified as effective using one test but identified as ineffective when based on a different test. While the research he cites is sound, they do not tell the full story because they are based on using one test to measure a teacher’s effectiveness. In no state that requires teachers to be evaluated based on their impact on student achievement is a teacher ever evaluated based on one measure. In fact, every state requires at least one measure of instructional practice such as classroom observations to make up at least half a teacher’s evaluation. Even in states where half of the teacher’s evaluation is based on student achievement measures, those states require multiple measures of student achievement, not just results from the state assessment.

Teachers are not penalized for teaching lower-performing students

Keep in mind when evaluating teachers based on measures of student achievement it doesn’t mean that teachers are evaluated based on their students’ overall scores. Instead teachers are evaluated on the growth of their students’ test scores from one year to the next. So teachers are evaluated based on the growth their students made while in their teacher’s class.

This is something Mr. Welch appears to misunderstand. This is typified by his story about how he improved the test scores from his Advanced Placement class by encouraging lower-performing students to dropout. He also quotes Jesse Rothstein who said of ranking teachers based on test scores “rewards or penalizes teachers for the kids they teach, not just for how well they do it.”

Fortunately for teachers and students teaching lower-performing students will not likely hurt a teacher’s evaluation. A well designed teacher evaluation system will not penalize teachers for teaching the hardest to teach students. Even if student test scores are a significant factor in the teacher’s evaluation. That is because teachers are evaluated based on the growth their students make from year to year and not on their overall achievement.

Truly rigorous evaluations will assess student growth using a value-added model (VAM) that takes into account each students prior achievement and other background factors to determine if they made as much or more growth than if those students had an average teacher. By making such comparisons, teachers are evaluated based on the quality of their instruction not on the students assigned to their class that particular year. So while Mr. Welch would be correct to worry about being evaluated on his students’ overall test scores, this is not what is happening across the country— although a report CPE will release later this summer finds teachers in some states are more likely to be evaluated accurately than in other states.

Recent research shows teachers can be evaluated accurately

While no evaluation system is perfect and all states should continually update their systems to make them more accurate, the Measures of Teacher Effectiveness (MET) study shows that combining measures of student growth with measures of instructional practice to evaluate teachers provides a reasonably accurate measure of a teacher’s true effectiveness. And when the results are looked at over multiple years they are a more accurate measure.

So while it is understandable that Mr. Welch and many teachers across the country are apprehensive about being evaluated in part on their student test scores, such anxiety should be put at ease knowing they are being evaluated based on the growth their students make and not on the overall performance. And those teachers who are evaluated using value-added scores should have even less anxiety as they provide the most accurate measure of a teacher’s true effectiveness especially when combined with measures of instructional practice. – Jim Hull

Filed under: Teacher evaluation,teachers — Tags: , — Jim Hull @ 11:34 am





Older Posts »
RSS Feed