Learn About: 21st Century | Charter Schools | Homework
Home / Edifier

The EDifier

December 2, 2013

10 questions to understanding PISA results

The big day is almost upon us. Tomorrow the results from the 2012 Program for International Student Assessment (PISA) will be released. The rhetoric pertaining to the quality of our public schools is certainly going to be amplified tomorrow, with critics lamenting how the results show our public schools are in dire straits while others will argue the results are meaningless. To help you understand what the PISA results actually signify, the Center for Public Education has answered 10 key questions about what PISA actually measures and what the results mean for our public schools.

1. What is PISA?

The Program for International Student Assessment (PISA) is an assessment of reading, math, and science literacy given every three years to 15-year-old students in public and private schools in about 65 countries. The international institution Organisation for Economic Cooperation and Development (OECD) coordinates the development and administration of PISA worldwide while the U.S. Department of Education’s National Center for Education Statistics (NCES) conducts the assessments in the U.S.

Unlike most state assessments that measure how much knowledge a student has acquired, PISA is designed to measure how well students can apply their knowledge to real-world situations. To measure such skills, the test items on PISA are primarily “constructed response,” meaning the test-taker has to write their answers to the questions, and there are few multiple-choice items. U.S. students typically do not perform as well on open-ended, constructed response items. This is one reason many states are adopting new standards, including the new Common Core State Standards, which are intended to emphasize how well students can solve problems and think critically based on the concepts, topics and procedures they have learned.

2. Why are PISA results important?

PISA is one of the few tools we have to compare the outcomes of high school students internationally.  PISA provides valuable information on how prepared high school students are for postsecondary success whether in the workplace, career training, or higher education.

3. Is the U.S. ranking on PISA negatively impacted because unlike other countries the U.S. educates and tests all its students?

No, this used to be true several decades ago, but is no longer the case. Every industrialized country now educates all their students, including language minority, special needs and low-performing students. Every country that participates in PISA must adhere to strict sampling rules to ensure the country’s results are nationally representative of all 15-year-old students. Indeed, the decision to test secondary students at age 15 was made in part because young people at that age are still subject to compulsory schooling laws in most participating nations, which provides more assurance that PISA will capture the broadest sample.

4. Where does the U.S. really rank on PISA?

In 2009, 30 countries had higher mathematics scores than the U.S. but just 23 of these countries significantly outperformed the U.S. Because only a sample of each nation’s students participate in PISA, much like political polls, each country’s score has a margin of error. This means that the score is actually an estimate of how the country would perform if every 15-year-old took PISA. In science, 21 countries had higher scores than the U.S., but only 18 scored significantly higher; in reading, while 16 countries scored higher, just nine countries significantly outperformed the U.S.

OECD reports statistically significant differences in performance between nations, which is a more accurate way to look at PISA rankings than a straight listing of average scores.

5. Does PISA measure the effectiveness of public school systems?

Not completely, for three reasons: 1) PISA results are representative of the performance of all 15-year-olds in participating countries including those  attending private schools; 2) PISA makes no attempt to isolate schools from outside factors such as poverty or high proportions of non-native language speakers that may have an impact on  performance —such factors are important to include in the mix when evaluating the effectiveness of each country’s schools; and 3) No single measure can incorporate every outcome we expect from our public schools. To gain a better perspective of the overall effectiveness of educational systems, you should consider multiple measures. NSBA’s Center for Public Education’s Data First Data Center is a good resource to get you started when examining public schools in the U.S.

6. How does the U.S. stack up on other international measures?

The U.S. fares much better on other international assessments.  U.S. 4th and 8th graders performed among the top 10 countries in both math and science on the most recent Trends in Mathematics and Science Study, which was administered to more than 60 countries (TIMSS, 2011). Moreover, only four countries outperformed U.S. 4th graders in reading on the 2011 Progress on International Reading Literacy Study (PIRLS). Finally, U.S. students led the world in civics in 1999, the last year the CivEd was given. As of 2009, the nation’s 15-year-old students did not compare as well on PISA, especially in math and science. However, the U.S. performed better in reading by scoring among the “top 10.”

7. Has the U.S. shown improvement on PISA?

The U.S. saw a slight improvement in math scores between 2006 and 2009. It wouldn’t be surprising if such gains continued in 2012 as U.S. high school students continue to take more rigorous math courses. It is important to point out that the U.S. has demonstrated improvements on other measures since PISA was first given in 2000. U.S. 4th and 8th graders made among the greatest gains in math between 1995 and 2011 on TIMSS. The U.S. also made dramatic gains in on-time graduation rates by improving from 67 percent in 2000 to 75 percent in 2010 according to Education Week. Even on the National Assessment of Education Progress (NAEP), U.S. 4th and 8th graders have shown significant progress between 2000 and 2013, although high school students are not showing the same gains. The lack of progress on PISA appears to be the exception rather than the rule in terms of international comparisons.

8. How should the results be used?

We need to get beyond seeing PISA as a horse race by fixating on whether the U.S. finishes in win, place, or show. Instead, we need to see PISA results as an opportunity to assess if best practices in teaching and learning in other countries can also work for secondary schools here in the U.S. For example, we should  look at how much time other countries give teachers for professional development, how much they pay their teachers, how much time teachers spend in the classroom, how much flexibility exists at the local level, how special needs students are taught, and how much time students spend in school. Answers to these and others questions could be instructive for U.S. educators and policymakers. While PISA gives us an opportunity to learn from other countries it is important to keep in mind that just because a high-performing or high-gaining country does something does not mean it will work in U.S. schools.

9. Does poverty affect the U.S. performance on PISA more than in other countries?

Many analysts observe that poverty has a greater impact on student performance in the U.S. than elsewhere. For one thing, the U.S. has the highest child poverty rates among industrialized countries. For another, students in the U.S. who live in poverty tend to have less access to resources that research consistently shows impact student achievement, including highly effective teachers, access to rigorous curriculum, and high quality pre-k programs. Yet, poverty is just one of several factors that affect the standing of the U.S. In comparing the performance of top students around the world—where poverty is likely less of a factor—America’s top students still do not compare well to their peers in other countries. For example, in 2009 19 countries’ top students (scoring in the top 10 percent) outperformed the U.S.’s top students in science on PISA.

10. Are PISA results a precursor of America’s future economic competitiveness?

Our high school graduates’ preparation for postsecondary success certainly has some impact on the future economic competitiveness of the U.S. However, as stated in question 5, PISA is just one measure of high school students’ college and career readiness. In addition, many factors besides K-12 schooling contribute to the economic competitiveness of the U.S. and every other country, including, for example, a country’s monetary and fiscal policies. But for a country to maximize its economic output it needs a well-educated society which would lead to lower unemployment rates and less demand for government services. Stanford University Economist Eric Hanushek estimates that if the U.S. had scored 50 points higher on PISA in 2000 by 2015 GDP would be 4.5 percent higher than currently projected. Such an increase is the equivalent to the total expenditures on U.S. K-12 schools in 2015. Keep in mind, however, this does not mean that if the U.S. doesn’t improve on PISA that GDP will decline when our current high school graduates enter the workforce. However, it does show that education does affect future economic outcomes.

One Response to “10 questions to understanding PISA results”

  1. […]   10 Questions to Understanding PISA Results from the Center for Public Education  –  here […]

Leave a Reply

RSS Feed