Scale development with small samples: a new application of longitudinal item response theory

Purpose: Measurement development in hard-to-reach populations can pose methodological challenges. Item response theory (IRT) is a useful statistical tool, but often requires large samples. We describe the use of longitudinal IRT models as a pragmatic approach to instrument development when large samples are not feasible. Methods: The statistical foundations and practical benefits of longitudinal IRT models are briefly described. Results from a simulation study are reported to demonstrate the model’s ability to recover the generating measurement structure and parameters using a range of sample sizes, number of items, and number of time points. An example using early-phase clinical trial data in a rare condition demonstrates these methods in practice. Results: Simulation study results demonstrate that the longitudinal IRT model’s ability to recover the generating parameters rests largely on the interaction between sample size and the number of time points. Overall, the model performs well even in small samples provided a sufficient number of time points are available. The clinical trial data example demonstrates that by using conditional, longitudinal IRT models researchers can obtain stable estimates of psychometric characteristics from samples typically considered too small for rigorous psychometric modeling. Conclusion: Capitalizing on repeated measurements, it is possible to estimate psychometric characteristics for an assessment even when sample size is small. This allows researchers to optimize study designs and have increased confidence in subsequent comparisons using scores obtained from such models. While there are limitations and caveats to consider when using these models, longitudinal IRT modeling may be especially beneficial when developing measures for rare conditions and diseases in difficult-to-reach populations.