Learn About: 21st Century | Charter Schools | Homework
Home / Edifier


The EDifier

November 17, 2016

What does “evidence-based” mean?

The Every Student Succeeds Act requires schools to use “evidence-based interventions” to improve schools.  The law also includes definitions of what evidence means, and recent guidance from the Department of Education has provided additional clarification on what passes as “evidence-based.”  Mathematica has also put out a brief guide on different types of data that have similar categories as the Department of Education, but also provide explanations for data we may see in the media or from academic researchers that do not qualify as hard data but can still help us understand policies and programs.

ESSA Evidence

What follows is a brief summary of what qualifies as “evidence-based” starting with the strongest first:

Experimental Studies:  These are purposefully created experiments, similar to medical trials, that randomly assign students to treatment or control groups, and then determine the difference in achievement after the treatment period.  Researchers also check to make sure that the two groups are similar in demographics.  This is considered to be causal evidence because there is little reason to believe the two similar groups would have had different outcomes except for the effect of the treatment.  Studies must involve at least 350 students, or 14 classrooms (assuming 25 students per class) and include multiple sites.

Quasi-experimental Studies:  These still have some form of comparison group, which may be between students, schools, or districts that have similar demographic characteristics.  However, even groups that seem similar on paper may still have systematic differences, which makes evidence from quasi-experimental studies slightly less reliable than randomized studies.  Evidence from these studies are often (but not always) considered to be causal, though experiment design and fidelity can greatly affect how reliable these conclusions are across other student groups.  Studies must involve at least 350 students, or 14 classrooms (assuming 25 students per class) and include multiple sites.

Correlational Studies: Studies that result in correlational effects can’t necessarily prove that a specific intervention caused students in a particular program to have a positive/negative effect.  For example, if Middle School X requires all teachers to participate in Professional Learning Communities (PLCs), and they end up with greater student improvement than Middle School Y, we can say that their improved performance was correlated with PLC participation.  However, there could have also been other changes at the school that truly caused the improvement, such as greater parental participation, so we cannot say that the improvement was caused by PLCs, but that further study should be done to see if there is a causal relationship.  Researchers still have to control for demographic factors; in this example, Middle School X and Middle School Y would have to be similar in both their teacher and student groups.

With all studies, we also have to consider who was involved and how the program was implemented.  A good example of this is the class-size experiment performed in Tennessee in the 1980s.  While their randomized control trial found positive effects of reducing class size by an average of seven students per class, when California reduced class sizes in the 1990s they didn’t see as strong of effects.  Part of this was implementation – reducing class sizes means hiring more teachers, and many inexperienced, uncertified teachers had to be placed in classrooms to fill the gap, which could have reduced the positive effect of smaller classes.  Also, students in California may be different than students in Tennessee; while this seems less likely for something like class size, it could be true for more specific programs or interventions.

An additional consideration when looking at evidence is not only statistical significance (whether or not we can be certain that the effect of a program wasn’t actually zero, using probability), but the effect size.  If an intervention has an effect size of 0.01 standard deviations* (or other units), it may only translate to the average student score changing a fraction of a percentage point.  We also have to consider if that effect is really meaningful, and if it’s worth our time, money, and effort to implement, or if we should look for a different intervention with greater effects.  Some researchers would say that an effect size of 0.2 standard deviations is the gold standard for really making meaningful changes for students.  However, I would also argue that it depends on the cost, both of time and money, of the program.  If making a small schedule tweak could garner 0.05 standard deviations of positive effect, and cost virtually nothing, then we should do it.  In conjunction with other effective programs, we can truly move the needle for student achievement.

School administrators should also consider the variation in test scores.  While most experimental studies report on the mean effect size, it is also important to consider how high- and low-performing students fared in the study.

Evidence is important and should guide policy decisions.  However, we have to keep in mind its limitations and be cautious consumers of data to make sure that we’re truly understanding how the study was done to see if its results are valid and can translate to other contexts.

 

*Standard deviations are standardized units used to help us compare programs, considering that most states and school districts use different tests.  The assumption is that most student achievement scores follow a bell curve, with the average score being at the top of the curve.  In a standard bell curve, a change of one standard deviation for a student at the 50th percentile would bump him/her up to 85th percentile, or down to the 15th percentile, depending on the direction of the change.  A report of the effect size of a program typically indicates how much the mean of the students who participated in the program changed from the previous mean or changed from the group of students who didn’t receive the program.

Filed under: CPE,Data,ESSA — Tags: , — Chandi Wagner @ 3:39 pm





November 2, 2016

Thoughts on nuance and variance

As we approach the 2016 general election, I’ve heard public officials, family, and friends make very clear statements regarding which side of the aisle they support.  Yet, I find it hard to believe that the average American falls in line 100% with either political party, or supports every word and tenet of a particular public policy.  We are nuanced people.  Very few issues are as black-and-white as we’d like them to be.  Here’s a guide for things to consider when considering your stance on a particular issue, candidate, or political party, put in the context of educational issues.

  1. Most issues have an “it depends” clause.

With the onslaught of information available today, it makes sense that we want answers that are black-and-white.  The reality, though, is that there’s gray area for most policies and practices.  We also have to balance our ideological values with evidence.  Charter school proponents may believe in free-market values and choice to improve public schools through vouchers and charter schools, but I haven’t seen widespread evidence that choice in and of itself actually improves academic achievement or long-term outcomes in significant ways.  Yes, there are individual students who have benefited, but there are also individual students who have lost out.  Charter school opponents claim that taking away publicly-elected oversight through school boards is detrimental to the public’s ability to provide free and quality education to all.  Yet, the reality is that some public schools have dismal records, and charter or private schools have sometimes had success with the same students.  We have to acknowledge that we all want good things for our kids, and then use the evidence to figure out what that looks like without demonizing the other side.

  1. Most policies rely heavily on the quality of their implementation to be successful.

Common Core seems to be a prime example of this.  Two-thirds of Americans are in support of some sort of common standards across the country.  Yet, barely half of Americans are in support of Common Core.  Support for both questions have dwindled significantly from about 90% of public support in 2012.  Even presidential candidate Hillary Clinton has called the roll-out of Common Core “disastrous,” despite supporting them overall.

CommonCore

Source: http://educationnext.org/ten-year-trends-in-public-opinion-from-ednext-poll-2016-survey/

They were implemented quickly in many states, often without the curriculum materials or professional development to help teachers succeed in teaching the new standards.  While support for Common Core seems to be leveling off with teachers, who are most familiar with them, several states have repealed or are considering repealing the Common Core.  The new state standards that have been written in South Carolina and Indiana are extremely similar to the Common Core, which means that it may not be the concept or content that people disagree with so much as how they were implemented and the ensuing political backlash.

 

  1. Statistics usually tell us about an average (the typical student) but variance is also important.

Charter schools are a prime example of this.  On average, they have similar student achievement outcomes as traditional public schools.  But, there are schools that outperform their counterparts and schools that woefully underperform.  We have to think about those schools, too.

This is also clear in school segregation.  The average black student in the U.S. attends a school that is 49% black, 28% white, 17% Latino, 4% Asian, and 3% “Other,” but that doesn’t mean that every black student has this experience.  At the edges of the spectrum, however, 13% of U.S. public schools are over 90% black and Latino, while 33% of schools are less than 10% black and Latino.  To understand the reality, we need to look at the variety of students’ experiences (known in statistic-speak as “variance”) not just the average.

  1. There’s always room for improvement. “Fixing” a policy may mean making adjustments, not abandoning it altogether.

Student assessments under No Child Left Behind (2001) resulted in the narrowing of curriculum.  But, we also learned more about disadvantaged student groups and have continued closing the achievement gap for students of color.  Should we throw out testing altogether? Some would say yes, but most Americans say no.  Graduation rates, college enrollment, and achievement scores have all increased since NCLB passed in 2001.  What we can do is improve on student assessments.  Adjusting consequences for students, teachers, and schools could result in less narrowing of curriculum and subjects taught.  Involving more well-rounded tests that encourage creative and critical thinking would help teachers emphasize these skills in class.  Continued improvement in data use can help teachers and school administrators adjust their practices and policies to see continued student growth.  States have the power to make some of these changes under the new Every Student Succeeds Act without dismantling gains made under No Child Left Behind.






September 28, 2016

How do we measure the immeasurable— and should we?

We address what we assess. I’ve never cared so much about how far I walked until I bought a Fitbit and saw that my friends apparently walk 15 miles a day.  The same is true of schools.

Under No Child Left Behind (NCLB), we began assessing our students’ math, reading, and science abilities, and test scores improved.  While some of that growth may have been due to teachers teaching to the test or students adapting to standardized assessments, we should still acknowledge that having stronger data about achievement gaps has helped us build the argument for greater equity in education.

The Every Student Succeeds Act (ESSA) adds a new, non-academic factor to school accountability in response to the over-emphasis on tested subjects that many schools experienced under NCLB.  States have to determine what their accountability plan will include, and policy wonks are chiming in with research and cautionary tales.  It seems that we can all agree that the non-academic factor should be equitable (not favoring particular student groups), mutable (able to be changed), measurable (we have to be able to put some sort of ranking or number on it), and important to student growth and learning (or else, who cares?).  So far, I haven’t heard any consensus come out of the field on what this could look like.

SEL

The reality is that states may even want to consider testing out several different variables to see what the data tells them.  The non-academic variable could be minimally weighted until states are sure that their data is reliable, both ensuring that schools aren’t penalized for faulty data and that schools don’t try to game the new system.  States may also choose to use multiple indicators to ensure that pressure isn’t exerted on one lone factor.  States also have to keep in mind that children develop at different ages.  While chronic absenteeism is a problem for students of all ages, first-graders may differ in their abilities to self-regulate their emotions, based on gender and age.

A group of CORE districts in California have been testing a “dashboard” of metrics for several years, and are offering their strategy to the entire state, as documented by Stanford’s Learning Policy Institute.  Forty percent of a school’s rating is based on social and emotional learning indicators, including measures of social-emotional skills; suspension/expulsion rates; chronic absenteeism; culture/climate surveys from students, staff, and parents; and English learner re-designation rates.  The other 60% is based on academic performance and growth.

The reality is that our students need more than just math and reading.  They need to learn how to interact with others who are different from themselves.  They need to be able to creatively problem solve.  They need to think critically about the world around them.  Good teachers have been teaching their students these skills for decades; now we just have to make sure that all students have these enriching opportunities.

Filed under: Accountability,CPE,ESSA — Tags: — Chandi Wagner @ 8:00 am





April 14, 2016

What’s different about ESSA?

What’s Different about ESSA?

The Elementary and Secondary Education Act of 1965 (ESEA) created the starting point for equity-based education reforms. It established categorical aid programs for specific subgroups that were at-risk of low academic achievement. “Title I” comes from this act- it created programs to improve education for low-income students. No Child Left Behind (NCLB) was a reauthorization of ESEA which gave more power to the federal government to ensure that all students received an equitable education and that standardized testing was the vehicle to assess high-standards for schools.

In 2015, the Every Student Succeeds Act (ESSA) again reauthorized ESEA and changed much of the language and policies of NCLB. At its foundation, the law gave a lot of decision-making power back to the states. Although state’s still need to have high-standards, test their students, and intervene in low-performing schools, the state’s themselves will have the power to determine the “how”.

This table below provides the key differences between NCLB and ESSA and was compiled from several sources (listed at the bottom) which provide a great deal more detail and specifics for those interested in learning more.

 

ESSA Table

 

-Breanna Higgins

 

Sources:

http://www.ncesd.org/cms/lib4/WA01000834/Centricity/Domain/52/GeneralNCLB%20vs%20ESSA%20Comparison%20-%20Title%20I-Federl%20Programs.pdf

http://neatoday.org/2015/12/09/every-student-succeeds-act/

http://all4ed.org/essa/

http://www.ascd.org/ASCD/pdf/siteASCD/policy/ESEA_NCLB_ComparisonChart_2015.pdf

Filed under: Accountability,CPE,ESSA — Tags: , — Breanna Higgins @ 1:10 pm





January 20, 2016

ESSA Gives More Power to the States

The Every Student Succeeds Act, ESSA, is the newest federal legislation to improve national education systems. This act replaces the heavy-hand of NCLB and places more emphasis on states to do the heavy lifting. There was a lot of criticism of state implementation of NCLB (some of the weaknesses and frustrations around the law may have been more the fault of implementation than the law itself) and now the states will need to take on more responsibility over innovation in policy-creation, testing, and accountability along with the compliance role they have been doing for years.

The state and local education agencies will need to reflect on and improve their own staff and capacity to succeed in this important work. Education agencies have become increasingly political in recent years and the average tenure of state chiefs is only 3.2 years. This tenuous environment and rapid shifts in leadership make it more difficult for agencies to complete long-term goals and for staff to have a coherent sense of direction.

In addition to changing leadership, the recession lessened the staff numbers in most education departments, leaving less employees to monitor the same numbers of schools, students, and federal funds/programs. Despite the upturn in the economy, EdWeek reports that staff numbers have not increased and has led staff members to be overstretched and to work on programs where they have little experience.

The point of understanding these staffing problems is that they will be exacerbated as ESSA demands more of the states. States finally have the decision-making power that they have been longing for, but an important question is: do they have the capacity to follow through? We can hope that as states gain power, they will also be able to hire qualified employees who can devise policies that are best for their state. They need experts to transform their lowest performing schools and groups of students, to create or revise accountability systems for schools, create or adopt academic standards (Common Core is an option here but it not required), and update school performance measures to include a school quality characteristic. These initiatives all require experts to take the lead in creating and implementing the policies, as well as to evaluate their effectiveness.

Local education systems should be aware of coming changes and work with states and schools to bridge the gaps in implementation of new policies. The more state and local systems can cooperate and communicate, the better chance policies have of being honestly implemented and becoming a success. –Breanna Higgins

Filed under: Accountability,CPE,ESSA,Public education — Breanna Higgins @ 10:47 am






RSS Feed