The Performance of Multivariate Methods for Two-Group Comparisons with Small Samples and Incomplete Data

Keenan A. Pituch, Megha Joshi, Molly E. Cain, Tiffany A. Whittaker, Wanchen Chang, Ryoungsun Park, Graham J. McDougall

Research output: Contribution to journalArticlepeer-review

2 Scopus citations


In intervention studies having multiple outcomes, researchers often use a series of univariate tests (e.g., ANOVAs) to assess group mean differences. Previous research found that this approach properly controls Type I error and generally provides greater power compared to MANOVA, especially under realistic effect size and correlation combinations. However, when group differences are assessed for a specific outcome, these procedures are strictly univariate and do not consider the outcome correlations, which may be problematic with missing outcome data. Linear mixed or multivariate multilevel models (MVMMs), implemented with maximum likelihood estimation, present an alternative analysis option where outcome correlations are taken into account when specific group mean differences are estimated. In this study, we use simulation methods to compare the performance of separate independent samples t tests estimated with ordinary least squares and analogous t tests from MVMMs to assess two-group mean differences with multiple outcomes under small sample and missingness conditions. Study results indicated that a MVMM implemented with restricted maximum likelihood estimation combined with the Kenward–Roger correction had the best performance. Therefore, for intervention studies with small N and normally distributed multivariate outcomes, the Kenward–Roger procedure is recommended over traditional methods and conventional MVMM analyses, particularly with incomplete data.

Original languageEnglish (US)
Pages (from-to)704-721
Number of pages18
JournalMultivariate Behavioral Research
Issue number5
StatePublished - Sep 2 2020


  • Kenward–Roger correction
  • missing data
  • multivariate multilevel model
  • small samples

ASJC Scopus subject areas

  • Statistics and Probability
  • Experimental and Cognitive Psychology
  • Arts and Humanities (miscellaneous)


Dive into the research topics of 'The Performance of Multivariate Methods for Two-Group Comparisons with Small Samples and Incomplete Data'. Together they form a unique fingerprint.

Cite this