Written Justifications are Key to Aggregate Crowdsourced Forecasts

Saketh Kotamraju; Eduardo Blanco

Written Justifications are Key to Aggregate Crowdsourced Forecasts

Saketh Kotamraju, Eduardo Blanco

Engineering, Ira A. Fulton Schools of (IAFSE)

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution

Abstract

This paper demonstrates that aggregating crowdsourced forecasts benefits from modeling the written justifications provided by forecasters. Our experiments show that the majority and weighted vote baselines are competitive, and that the written justifications are beneficial to call a question throughout its life except in the last quarter. We also conduct an error analysis shedding light into the characteristics that make a justification unreliable.

Original language	English (US)
Title of host publication	Findings of the Association for Computational Linguistics, Findings of ACL
Subtitle of host publication	EMNLP 2021
Editors	Marie-Francine Moens, Xuanjing Huang, Lucia Specia, Scott Wen-Tau Yih
Publisher	Association for Computational Linguistics (ACL)
Pages	4206-4216
Number of pages	11
ISBN (Electronic)	9781955917100
State	Published - 2021
Event	2021 Findings of the Association for Computational Linguistics, Findings of ACL: EMNLP 2021 - Punta Cana, Dominican Republic Duration: Nov 7 2021 → Nov 11 2021

Publication series

Name	Findings of the Association for Computational Linguistics, Findings of ACL: EMNLP 2021

Conference

Conference	2021 Findings of the Association for Computational Linguistics, Findings of ACL: EMNLP 2021
Country/Territory	Dominican Republic
City	Punta Cana
Period	11/7/21 → 11/11/21

ASJC Scopus subject areas

Language and Linguistics
Linguistics and Language

Cite this

Kotamraju, S., & Blanco, E. (2021). Written Justifications are Key to Aggregate Crowdsourced Forecasts. In M.-F. Moens, X. Huang, L. Specia, & S. W.-T. Yih (Eds.), Findings of the Association for Computational Linguistics, Findings of ACL: EMNLP 2021 (pp. 4206-4216). (Findings of the Association for Computational Linguistics, Findings of ACL: EMNLP 2021). Association for Computational Linguistics (ACL).

Written Justifications are Key to Aggregate Crowdsourced Forecasts. / Kotamraju, Saketh; Blanco, Eduardo.
Findings of the Association for Computational Linguistics, Findings of ACL: EMNLP 2021. ed. / Marie-Francine Moens; Xuanjing Huang; Lucia Specia; Scott Wen-Tau Yih. Association for Computational Linguistics (ACL), 2021. p. 4206-4216 (Findings of the Association for Computational Linguistics, Findings of ACL: EMNLP 2021).

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution

Kotamraju, S & Blanco, E 2021, Written Justifications are Key to Aggregate Crowdsourced Forecasts. in M-F Moens, X Huang, L Specia & SW-T Yih (eds), Findings of the Association for Computational Linguistics, Findings of ACL: EMNLP 2021. Findings of the Association for Computational Linguistics, Findings of ACL: EMNLP 2021, Association for Computational Linguistics (ACL), pp. 4206-4216, 2021 Findings of the Association for Computational Linguistics, Findings of ACL: EMNLP 2021, Punta Cana, Dominican Republic, 11/7/21.

Kotamraju S, Blanco E. Written Justifications are Key to Aggregate Crowdsourced Forecasts. In Moens MF, Huang X, Specia L, Yih SWT, editors, Findings of the Association for Computational Linguistics, Findings of ACL: EMNLP 2021. Association for Computational Linguistics (ACL). 2021. p. 4206-4216. (Findings of the Association for Computational Linguistics, Findings of ACL: EMNLP 2021).

Kotamraju, Saketh ; Blanco, Eduardo. / Written Justifications are Key to Aggregate Crowdsourced Forecasts. Findings of the Association for Computational Linguistics, Findings of ACL: EMNLP 2021. editor / Marie-Francine Moens ; Xuanjing Huang ; Lucia Specia ; Scott Wen-Tau Yih. Association for Computational Linguistics (ACL), 2021. pp. 4206-4216 (Findings of the Association for Computational Linguistics, Findings of ACL: EMNLP 2021).

@inproceedings{9c502a316eb14e3aa8c2fa5727438de3,

title = "Written Justifications are Key to Aggregate Crowdsourced Forecasts",

abstract = "This paper demonstrates that aggregating crowdsourced forecasts benefits from modeling the written justifications provided by forecasters. Our experiments show that the majority and weighted vote baselines are competitive, and that the written justifications are beneficial to call a question throughout its life except in the last quarter. We also conduct an error analysis shedding light into the characteristics that make a justification unreliable.",

author = "Saketh Kotamraju and Eduardo Blanco",

note = "Funding Information: We gratefully acknowledge the support of NVIDIA Corporation with the donation of the Titan Xp GPU used for this research. The results presented in this paper were also obtained using the Chameleon testbed supported by the National Science Foundation (Keahey et al., 2020). Publisher Copyright: {\textcopyright} 2021 Association for Computational Linguistics.; 2021 Findings of the Association for Computational Linguistics, Findings of ACL: EMNLP 2021 ; Conference date: 07-11-2021 Through 11-11-2021",

year = "2021",

language = "English (US)",

series = "Findings of the Association for Computational Linguistics, Findings of ACL: EMNLP 2021",

publisher = "Association for Computational Linguistics (ACL)",

pages = "4206--4216",

editor = "Marie-Francine Moens and Xuanjing Huang and Lucia Specia and Yih, {Scott Wen-Tau}",

booktitle = "Findings of the Association for Computational Linguistics, Findings of ACL",

}

TY - GEN

T1 - Written Justifications are Key to Aggregate Crowdsourced Forecasts

AU - Kotamraju, Saketh

AU - Blanco, Eduardo

N1 - Funding Information: We gratefully acknowledge the support of NVIDIA Corporation with the donation of the Titan Xp GPU used for this research. The results presented in this paper were also obtained using the Chameleon testbed supported by the National Science Foundation (Keahey et al., 2020). Publisher Copyright: © 2021 Association for Computational Linguistics.

PY - 2021

Y1 - 2021

N2 - This paper demonstrates that aggregating crowdsourced forecasts benefits from modeling the written justifications provided by forecasters. Our experiments show that the majority and weighted vote baselines are competitive, and that the written justifications are beneficial to call a question throughout its life except in the last quarter. We also conduct an error analysis shedding light into the characteristics that make a justification unreliable.

AB - This paper demonstrates that aggregating crowdsourced forecasts benefits from modeling the written justifications provided by forecasters. Our experiments show that the majority and weighted vote baselines are competitive, and that the written justifications are beneficial to call a question throughout its life except in the last quarter. We also conduct an error analysis shedding light into the characteristics that make a justification unreliable.

UR - http://www.scopus.com/inward/record.url?scp=85129165554&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=85129165554&partnerID=8YFLogxK

M3 - Conference contribution

AN - SCOPUS:85129165554

T3 - Findings of the Association for Computational Linguistics, Findings of ACL: EMNLP 2021

SP - 4206

EP - 4216

BT - Findings of the Association for Computational Linguistics, Findings of ACL

A2 - Moens, Marie-Francine

A2 - Huang, Xuanjing

A2 - Specia, Lucia

A2 - Yih, Scott Wen-Tau

PB - Association for Computational Linguistics (ACL)

T2 - 2021 Findings of the Association for Computational Linguistics, Findings of ACL: EMNLP 2021

Y2 - 7 November 2021 through 11 November 2021

ER -

Written Justifications are Key to Aggregate Crowdsourced Forecasts

Abstract

Publication series

Conference

ASJC Scopus subject areas

Other files and links

Fingerprint

Cite this