Quantifying and addressing the prevalence and bias of study designs in the environmental and social sciences

Alec P. Christie; David Abecasis; Mehdi Adjeroud; Juan C. Alonso; Tatsuya Amano; Alvaro Anton; Barry P. Baldigo; Rafael Barrientos; Jake E. Bicknell; Deborah A. Buhl; Just Cebrian; Ricardo S. Ceia; Luciana Cibils-Martina; Sarah Clarke; Joachim Claudet; Michael D. Craig; Dominique Davoult; Annelies De Backer; Mary K. Donovan; Tyler D. Eddy; Filipe M. França; Jonathan P.A. Gardner; Bradley P. Harris; Ari Huusko; Ian L. Jones; Brendan P. Kelaher; Janne S. Kotiaho; Adrià López-Baucells; Heather L. Major; Aki Mäki-Petäys; Beatriz Martín; Carlos A. Martín; Philip A. Martin; Daniel Mateos-Molina; Robert A. McConnaughey; Michele Meroni; Christoph F.J. Meyer; Kade Mills; Monica Montefalcone; Norbertas Noreika; Carlos Palacín; Anjali Pande; C. Roland Pitcher; Carlos Ponce; Matt Rinella; Ricardo Rocha; María C. Ruiz-Delgado; Juan J. Schmitter-Soto; Jill A. Shaffer; Shailesh Sharma; Anna A. Sher; Doriane Stagnol; Thomas R. Stanley; Kevin D.E. Stokesbury; Aurora Torres; Oliver Tully; Teppo Vehanen; Corinne Watts; Qingyuan Zhao; William J. Sutherland

doi:10.1038/s41467-020-20142-y

Quantifying and addressing the prevalence and bias of study designs in the environmental and social sciences

Alec P. Christie, David Abecasis, Mehdi Adjeroud, Juan C. Alonso, Tatsuya Amano, Alvaro Anton, Barry P. Baldigo, Rafael Barrientos, Jake E. Bicknell, Deborah A. Buhl, Just Cebrian, Ricardo S. Ceia, Luciana Cibils-Martina, Sarah Clarke, Joachim Claudet, Michael D. Craig, Dominique Davoult, Annelies De Backer, Mary K. Donovan, Tyler D. EddyFilipe M. França, Jonathan P.A. Gardner, Bradley P. Harris, Ari Huusko, Ian L. Jones, Brendan P. Kelaher, Janne S. Kotiaho, Adrià López-Baucells, Heather L. Major, Aki Mäki-Petäys, Beatriz Martín, Carlos A. Martín, Philip A. Martin, Daniel Mateos-Molina, Robert A. McConnaughey, Michele Meroni, Christoph F.J. Meyer, Kade Mills, Monica Montefalcone, Norbertas Noreika, Carlos Palacín, Anjali Pande, C. Roland Pitcher, Carlos Ponce, Matt Rinella, Ricardo Rocha, María C. Ruiz-Delgado, Juan J. Schmitter-Soto, Jill A. Shaffer, Shailesh Sharma, Anna A. Sher, Doriane Stagnol, Thomas R. Stanley, Kevin D.E. Stokesbury, Aurora Torres, Oliver Tully, Teppo Vehanen, Corinne Watts, Qingyuan Zhao, William J. Sutherland

Research output: Contribution to journal › Article › peer-review

48 Scopus citations

Abstract

Building trust in science and evidence-based decision-making depends heavily on the credibility of studies and their findings. Researchers employ many different study designs that vary in their risk of bias to evaluate the true effect of interventions or impacts. Here, we empirically quantify, on a large scale, the prevalence of different study designs and the magnitude of bias in their estimates. Randomised designs and controlled observational designs with pre-intervention sampling were used by just 23% of intervention studies in biodiversity conservation, and 36% of intervention studies in social science. We demonstrate, through pairwise within-study comparisons across 49 environmental datasets, that these types of designs usually give less biased estimates than simpler observational designs. We propose a model-based approach to combine study estimates that may suffer from different levels of study design bias, discuss the implications for evidence synthesis, and how to facilitate the use of more credible study designs.

Original language	English (US)
Article number	6377
Journal	Nature communications
Volume	11
Issue number	1
DOIs	https://doi.org/10.1038/s41467-020-20142-y
State	Published - Dec 2020
Externally published	Yes

ASJC Scopus subject areas

General Chemistry
General Biochemistry, Genetics and Molecular Biology
General Physics and Astronomy

Access to Document

10.1038/s41467-020-20142-y

Cite this

Christie, A. P., Abecasis, D., Adjeroud, M., Alonso, J. C., Amano, T., Anton, A., Baldigo, B. P., Barrientos, R., Bicknell, J. E., Buhl, D. A., Cebrian, J., Ceia, R. S., Cibils-Martina, L., Clarke, S., Claudet, J., Craig, M. D., Davoult, D., De Backer, A., Donovan, M. K., ... Sutherland, W. J. (2020). Quantifying and addressing the prevalence and bias of study designs in the environmental and social sciences. Nature communications, 11(1), Article 6377. https://doi.org/10.1038/s41467-020-20142-y

Christie, AP, Abecasis, D, Adjeroud, M, Alonso, JC, Amano, T, Anton, A, Baldigo, BP, Barrientos, R, Bicknell, JE, Buhl, DA, Cebrian, J, Ceia, RS, Cibils-Martina, L, Clarke, S, Claudet, J, Craig, MD, Davoult, D, De Backer, A, Donovan, MK, Eddy, TD, França, FM, Gardner, JPA, Harris, BP, Huusko, A, Jones, IL, Kelaher, BP, Kotiaho, JS, López-Baucells, A, Major, HL, Mäki-Petäys, A, Martín, B, Martín, CA, Martin, PA, Mateos-Molina, D, McConnaughey, RA, Meroni, M, Meyer, CFJ, Mills, K, Montefalcone, M, Noreika, N, Palacín, C, Pande, A, Pitcher, CR, Ponce, C, Rinella, M, Rocha, R, Ruiz-Delgado, MC, Schmitter-Soto, JJ, Shaffer, JA, Sharma, S, Sher, AA, Stagnol, D, Stanley, TR, Stokesbury, KDE, Torres, A, Tully, O, Vehanen, T, Watts, C, Zhao, Q & Sutherland, WJ 2020, 'Quantifying and addressing the prevalence and bias of study designs in the environmental and social sciences', Nature communications, vol. 11, no. 1, 6377. https://doi.org/10.1038/s41467-020-20142-y

@article{fcd606ad24fe435da26797effc0177ed,

title = "Quantifying and addressing the prevalence and bias of study designs in the environmental and social sciences",

abstract = "Building trust in science and evidence-based decision-making depends heavily on the credibility of studies and their findings. Researchers employ many different study designs that vary in their risk of bias to evaluate the true effect of interventions or impacts. Here, we empirically quantify, on a large scale, the prevalence of different study designs and the magnitude of bias in their estimates. Randomised designs and controlled observational designs with pre-intervention sampling were used by just 23% of intervention studies in biodiversity conservation, and 36% of intervention studies in social science. We demonstrate, through pairwise within-study comparisons across 49 environmental datasets, that these types of designs usually give less biased estimates than simpler observational designs. We propose a model-based approach to combine study estimates that may suffer from different levels of study design bias, discuss the implications for evidence synthesis, and how to facilitate the use of more credible study designs.",

author = "Christie, {Alec P.} and David Abecasis and Mehdi Adjeroud and Alonso, {Juan C.} and Tatsuya Amano and Alvaro Anton and Baldigo, {Barry P.} and Rafael Barrientos and Bicknell, {Jake E.} and Buhl, {Deborah A.} and Just Cebrian and Ceia, {Ricardo S.} and Luciana Cibils-Martina and Sarah Clarke and Joachim Claudet and Craig, {Michael D.} and Dominique Davoult and {De Backer}, Annelies and Donovan, {Mary K.} and Eddy, {Tyler D.} and Fran{\c c}a, {Filipe M.} and Gardner, {Jonathan P.A.} and Harris, {Bradley P.} and Ari Huusko and Jones, {Ian L.} and Kelaher, {Brendan P.} and Kotiaho, {Janne S.} and Adri{\`a} L{\'o}pez-Baucells and Major, {Heather L.} and Aki M{\"a}ki-Pet{\"a}ys and Beatriz Mart{\'i}n and Mart{\'i}n, {Carlos A.} and Martin, {Philip A.} and Daniel Mateos-Molina and McConnaughey, {Robert A.} and Michele Meroni and Meyer, {Christoph F.J.} and Kade Mills and Monica Montefalcone and Norbertas Noreika and Carlos Palac{\'i}n and Anjali Pande and Pitcher, {C. Roland} and Carlos Ponce and Matt Rinella and Ricardo Rocha and Ruiz-Delgado, {Mar{\'i}a C.} and Schmitter-Soto, {Juan J.} and Shaffer, {Jill A.} and Shailesh Sharma and Sher, {Anna A.} and Doriane Stagnol and Stanley, {Thomas R.} and Stokesbury, {Kevin D.E.} and Aurora Torres and Oliver Tully and Teppo Vehanen and Corinne Watts and Qingyuan Zhao and Sutherland, {William J.}",

note = "Publisher Copyright: {\textcopyright} 2020, The Author(s).",

year = "2020",

month = dec,

doi = "10.1038/s41467-020-20142-y",

language = "English (US)",

volume = "11",

journal = "Nature communications",

issn = "2041-1723",

publisher = "Nature Publishing Group",

number = "1",

}

TY - JOUR

T1 - Quantifying and addressing the prevalence and bias of study designs in the environmental and social sciences

AU - Christie, Alec P.

AU - Abecasis, David

AU - Adjeroud, Mehdi

AU - Alonso, Juan C.

AU - Amano, Tatsuya

AU - Anton, Alvaro

AU - Baldigo, Barry P.

AU - Barrientos, Rafael

AU - Bicknell, Jake E.

AU - Buhl, Deborah A.

AU - Cebrian, Just

AU - Ceia, Ricardo S.

AU - Cibils-Martina, Luciana

AU - Clarke, Sarah

AU - Claudet, Joachim

AU - Craig, Michael D.

AU - Davoult, Dominique

AU - De Backer, Annelies

AU - Donovan, Mary K.

AU - Eddy, Tyler D.

AU - França, Filipe M.

AU - Gardner, Jonathan P.A.

AU - Harris, Bradley P.

AU - Huusko, Ari

AU - Jones, Ian L.

AU - Kelaher, Brendan P.

AU - Kotiaho, Janne S.

AU - López-Baucells, Adrià

AU - Major, Heather L.

AU - Mäki-Petäys, Aki

AU - Martín, Beatriz

AU - Martín, Carlos A.

AU - Martin, Philip A.

AU - Mateos-Molina, Daniel

AU - McConnaughey, Robert A.

AU - Meroni, Michele

AU - Meyer, Christoph F.J.

AU - Mills, Kade

AU - Montefalcone, Monica

AU - Noreika, Norbertas

AU - Palacín, Carlos

AU - Pande, Anjali

AU - Pitcher, C. Roland

AU - Ponce, Carlos

AU - Rinella, Matt

AU - Rocha, Ricardo

AU - Ruiz-Delgado, María C.

AU - Schmitter-Soto, Juan J.

AU - Shaffer, Jill A.

AU - Sharma, Shailesh

AU - Sher, Anna A.

AU - Stagnol, Doriane

AU - Stanley, Thomas R.

AU - Stokesbury, Kevin D.E.

AU - Torres, Aurora

AU - Tully, Oliver

AU - Vehanen, Teppo

AU - Watts, Corinne

AU - Zhao, Qingyuan

AU - Sutherland, William J.

PY - 2020/12

Y1 - 2020/12

N2 - Building trust in science and evidence-based decision-making depends heavily on the credibility of studies and their findings. Researchers employ many different study designs that vary in their risk of bias to evaluate the true effect of interventions or impacts. Here, we empirically quantify, on a large scale, the prevalence of different study designs and the magnitude of bias in their estimates. Randomised designs and controlled observational designs with pre-intervention sampling were used by just 23% of intervention studies in biodiversity conservation, and 36% of intervention studies in social science. We demonstrate, through pairwise within-study comparisons across 49 environmental datasets, that these types of designs usually give less biased estimates than simpler observational designs. We propose a model-based approach to combine study estimates that may suffer from different levels of study design bias, discuss the implications for evidence synthesis, and how to facilitate the use of more credible study designs.

AB - Building trust in science and evidence-based decision-making depends heavily on the credibility of studies and their findings. Researchers employ many different study designs that vary in their risk of bias to evaluate the true effect of interventions or impacts. Here, we empirically quantify, on a large scale, the prevalence of different study designs and the magnitude of bias in their estimates. Randomised designs and controlled observational designs with pre-intervention sampling were used by just 23% of intervention studies in biodiversity conservation, and 36% of intervention studies in social science. We demonstrate, through pairwise within-study comparisons across 49 environmental datasets, that these types of designs usually give less biased estimates than simpler observational designs. We propose a model-based approach to combine study estimates that may suffer from different levels of study design bias, discuss the implications for evidence synthesis, and how to facilitate the use of more credible study designs.

UR - http://www.scopus.com/inward/record.url?scp=85097492820&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=85097492820&partnerID=8YFLogxK

U2 - 10.1038/s41467-020-20142-y

DO - 10.1038/s41467-020-20142-y

M3 - Article

C2 - 33311448

AN - SCOPUS:85097492820

SN - 2041-1723

VL - 11

JO - Nature communications

JF - Nature communications

IS - 1

M1 - 6377

ER -

Quantifying and addressing the prevalence and bias of study designs in the environmental and social sciences

Abstract

ASJC Scopus subject areas

Access to Document

Other files and links

Fingerprint

Cite this