Assessment of writing ability in secondary education

Comparison of analytic and holistic scoring systems for use in large-scale assessments

Abstract

Although writing is an important subject of language teaching in secondary education, it is often neglected in large-scale assessments. We report results of a study with 1,365 German high school students in Grade 8 that was conducted in the context of national monitoring of educational achievement. Student responses on seven different persuasive and informative writing tasks were evaluated with two different scoring systems: (i) analytic scoring with 14 dichotomous criteria capturing specific aspects of content, text structure, and language usage, and (ii) holistic scoring based on a comprehensive rating scale similar to the NAEP Holistic Scoring Guide accompanied by semi-holistic scales for content, style (i.e., language usage and organization), and language correctness. We inspected both scoring systems in terms of inter-rater and intra-rater reliability, dimensionality on the construct level, and convergence of the scoring results. Special attention is given to the generalizability of the findings across different writing tasks and text genres. The results showed better reliability for the holistic and semi-holistic scales than for most of the analytic criteria. For both scoring systems, content and structural aspects were closely associated whereas language correctness was a clearly distinct dimension of writing ability. Both scoring systems measured the same latent construct.