Hypothetical case replacement can be used to quantify the robustness of trial results

Kenneth A. Frank, Qinyun Lin, Spiro Maroulis, Anna S. Mueller, Ran Xu, Joshua M. Rosenberg, Christopher S. Hayter, Ramy A. Mahmoud, Marynia Kolak, Thomas Dietz, Lixin Zhang

    Research output: Contribution to journalArticlepeer-review

    16 Scopus citations


    Objectives: We apply a general case replacement framework for quantifying the robustness of causal inferences to characterize the uncertainty of findings from clinical trials. Study design and setting: We express the robustness of inferences as the amount of data that must be replaced to change the conclusion and relate this to the fragility of trial results used for dichotomous outcomes. We illustrate our approach in the context of an RCT of hydroxychloroquine on pneumonia in COVID-19 patients and a cumulative meta-analysis of the effect of antihypertensive treatments on stroke. Results: We developed the Robustness of an Inference to Replacement (RIR), which quantifies how many treatment cases with positive outcomes would have to be replaced with hypothetical patients who did not receive a treatment to change an inference. The RIR addresses known limitations of the Fragility Index by accounting for the observed rates of outcomes. It can be used for varying thresholds for inference, including clinical importance. Conclusion: Because the RIR expresses uncertainty in terms of patient experiences, it is more relatable to stakeholders than P-values alone. It helps identify when results are statistically significant, but conclusions are not robust, while considering the rareness of events in the underlying data.

    Original languageEnglish (US)
    Pages (from-to)150-159
    Number of pages10
    JournalJournal of Clinical Epidemiology
    StatePublished - Jun 2021


    • Case replacement
    • Clinical importance
    • Fragility
    • Randomized controlled trials
    • Robustness of findings
    • Statistical significance

    ASJC Scopus subject areas

    • Epidemiology


    Dive into the research topics of 'Hypothetical case replacement can be used to quantify the robustness of trial results'. Together they form a unique fingerprint.

    Cite this