Last week the Web site Global Report Card (GRC) was launched by the George W. Bush Presidential Center. It enables the public to compare their school district’s academic performance in math and reading to that of students in 25 developed countries around the world, including top-achieving Finland, Canada, Japan, and Singapore.
Although the Web site is easy to use, actually making such comparisons is not. There are significant limitations in making fair comparisons of districts across states, never mind across countries. However, Jay Greene and Josh McGee, who created the GRC, have postulated that their comparisons of Boston to Finland (for example) are fair and reliable.
You can put me in the skeptical camp on this one. Not only are they comparing results across countries, they are doing it across grade levels as well. For U.S. school districts, they use scores from state assessments from all tested grades, which is grades 3 through 8 and 10th grade in most states. Other countries’ results are based on the international assessments in which they participated, which would at most include 4th, 8th and 10th grade in math and 4th and 10th grade in reading.
Keep in mind different assessments with significantly different purposes and given in different years were used in different grades and subjects. For example, 4th and 8th grade math scores are derived from TIMSS, which is designed specifically to measure how well students have learned what they were expected to be taught in school. Tenth grade reading and math scores come from PISA, which measures how well students can apply their math and reading knowledge to real life problems, no matter if they attained that knowledge in school or not. To add even more complexity to the comparisons, not all 25 countries participated in each of the assessments at each of the grade levels. Hence, districts’ results on their state assessments across multiple grade levels are compared to each country’s results across different assessments that not all comparable countries took part in.
You may remember I was skeptical in a post earlier this year of another report that compared the U.S. to other countries and that comparison was based on one grade level, in one year, on one assessment for each country. And then countries were only compared to U.S. states who had only taken one assessment, in one grade level, in one year. A far more straightforward comparison than the GRC, yet still statistically questionable.
Both report cards, however, attempt to make important comparisons that — if fair and reliable — would provide valuable information on how our students compare to their peers in other countries. Yet, we don’t know how reliable the comparisons actually are, especially at the district level, where smaller districts appear to have a distinct advantage over larger districts with similar demographics.
However, maybe the GRC with all its question marks will lead to accurate international comparisons at both the state and district levels. Because it really is an important question to answer whether our students in our best districts are as prepared as students in the highest performing countries. The answer could have a tremendous impact on the focus of our education reform efforts. – Jim Hull