“Today’s students are digital natives. Their schools are usually equipped with computers, and digital tools are an integral part of the learning environment.” This reality has become a driving force behind the U.S. Department of Education’s push to transition the National Assessment of Educational Progress (NAEP) or the Nation’s Report Card from paper and pencil to digitally based assessments (DBA). The “good” news is that DBA can assist educators in gathering information about how students use the digital tools and studying that information to improve curriculum and instruction in K-12 classrooms. But the “bad” news is that we are unclear as to what extend DBA might affect students’ test scores.
What information has been gathered by the digitally-based NAEP writing assessments?
The information collected by the computer-based NAEP writing assessment tells us how students used digital writing, editing and reviewing tools. These tools include keyboard use, Thesaurus, Spell check, and Text-to-speech. In general, students who used the digital editing and reviewing tools appropriately scored higher in the writing assessment.
As an editing tool, Thesaurus can help students enhance or improve their writing. The data show that students who accessed the thesaurus more frequently scored higher on average than those who used it less often (Figure 1).
Figure 1: Percentage of students and average scale scores for grade 8 and 12 writing, by how often students accessed the thesaurus in the NAEP writing assessment (Data source: U.S. Department of Education)
Not every digital tool that was used frequently in the NAEP writing assessment is positively correlated with the writing scores of students. As a proofreading tool, Spell checker can assist students in correcting typos and often misspelled words. Students, in both grade 8 and grade 12, who did not use or too often used the spell-check tool, scored lower than students who used the tool appropriately (i.e. engaged in the action between 1-10 times) (Figure 2).
Figure 2: Percentage of students and average scores for 8th and 12th grades writing, by how often students used the spell-check tool in the NAEP writing assessment (Image cited from the U.S. Department of Eduation)
Students who used the text-to-speech (TTS) tool more frequently scored lower on average than those who engaged in this action less frequently. In the 2011 NAEP writing assessments, students could use TTS to hear the writing prompt read aloud by a speech synthesizer program but not to write their responses. The information collected by the DBA shows that students with disabilities and students who were identified as English language learners (ELLs) were more likely to use TTS frequently (i.e. 3 times or more) during the writing assessments.
Recent research shows that TTS, a read-aloud tool being widely implemented in public schools to assist students to improve their reading comprehension skills, may assist students with reading difficulties. Yet, it is unclear how effective TTS can be in helping students, particularly students in special education or ELLs, to improve their writing skills. We found that, on average, students with disabilities or ELLs scored approximately 40 points lower than their peers without disabilities or non-ELLs across the board (Figure 3).
Figure 3: Percentage of students and average scale scores for grade 8 and 12 writing, by how often students used the text-to-speech feature for the prompt in the NAEP writing assessment: Disability status of student (including those with 504 plan), Status as English Language Learner (Data source: U.S. Department of Education)
What factors may contribute to the performance gap in digitally-based writing assessments?
In the digital age, a social context relevant to learning and assessment is the great and varied differences in access to and competence in technology among students. According to the data, students, both eighth and twelfth graders, who did not use the spell-check digital tool and also reported not having a computer at home scored the lowest in the writing assessment. In a sense, lack of access to computer may contribute to insufficient competence in using technology. By contrast, students who reported having computers at home, on average, performed better than students who reported not having computers at home.
Familiarity with digital editing tools certainly benefits students in the computer-based writing assessment. The data show that among eighth graders, the highest writing scores were achieved by students who not only used the spell-check tool properly but also reported having access to the Internet when they were taught writing. In contrast, students who did not use the spell-check tool or didn’t have access to Internet when taught to write scored the lowest. Our finding suggests that it is important for teachers to routinely demonstrate the use of online digital tools during writing instruction.
The NAEP data also reveal that some classroom practices are positively related to students’ writing scores. On average, twelfth-grade students who did writing assignments every day or almost every day scored the highest, regardless of whether they used paper or a computer. Among the eighth graders, the group that performed the best in the writing assessment were students who were asked by their teachers to frequently use spell check. These results indicate that to help students perform well in digitally-based writing assessments, teachers should give students writing assignments daily and educate students on the use of digital editing tools.
In brief, our data analysis shows that students’ access to and competence in technology matter in this digital age. Yet, what matters most is for students to practice writing every day and for teachers to instruct them in the best and appropriate use of available digital tools.