A review of three large-scale datasets critiquing item design, data collection, and the usefulness of claims

Issues of validity and usefulness of three large-scale longitudinal data sets are reviewed in this chapter. The Trends in International Mathematics and Science Study (TIMSS), the National Assessment of Educational Progress (NAEP), and the Educational Longitudinal Study of 2002 (ELS:2002) are compared and contrasted with respect to differences in sampling frame, internal and external validity, and especially construct validity of assessment items. Conclusions about the usefulness of large-scale secondary data analysis show that the reviewed assessments have been critical for determining inequities of opportunity for gender, ethnicity, socioeconomic status, and across national boundaries. They have also been useful for researchers examining the effectiveness of curricular policy on student learning. Moreover, some stakeholders have used the results as evidence that a nation’s future GDP is predicted by the outcome on TIMSS, and that students need more mathematical knowledge and skills to compete in a world that has an ever increasing rate of technological expansion. Though longitudinal, the duration of the studies presents a problem, as none follow students’ mathematical abilities or development for any length of time (e.g., early childhood into adulthood), and few studies from large-scale assessments shed light onto the kinds of pedagogy or curricular tasks that positively impact student learning. Lastly, threats to validity for large-scale studies are critiqued, and shown to be underreported in the literature.