Towards Improving Selective Prediction Ability of NLP Systems

Neeraj Varshney, Swaroop Mishra, Chitta Baral

Research output: Chapter in Book/Report/Conference proceedingConference contribution

7 Scopus citations

Abstract

It’s better to say “I can’t answer” than to answer incorrectly. This selective prediction ability is crucial for NLP systems to be reliably deployed in real-world applications. Prior work has shown that existing selective prediction techniques fail to perform well, especially in the out-of-domain setting. In this work, we propose a method that improves probability estimates of models by calibrating them using prediction confidence and difficulty score of instances. Using these two signals, we first annotate held-out instances and then train a calibrator to predict the likelihood of correctness of the model’s prediction. We instantiate our method with Natural Language Inference (NLI) and Duplicate Detection (DD) tasks and evaluate it in both In-Domain (IID) and Out-of-Domain (OOD) settings. In (IID, OOD) settings, we show that the representations learned by our calibrator result in an improvement of (15.81%, 5.64%) and (6.19%, 13.9%) over MaxProb –a selective prediction baseline– on NLI and DD tasks respectively.

Original languageEnglish (US)
Title of host publicationACL 2022 - 7th Workshop on Representation Learning for NLP, RepL4NLP 2022 - Proceedings of the Workshop
PublisherAssociation for Computational Linguistics (ACL)
Pages221-226
Number of pages6
ISBN (Electronic)9781955917483
StatePublished - 2022
Event7th Workshop on Representation Learning for NLP, RepL4NLP 2022 at ACL 2022 - Dublin, Ireland
Duration: May 26 2022 → …

Publication series

NameProceedings of the Annual Meeting of the Association for Computational Linguistics
ISSN (Print)0736-587X

Conference

Conference7th Workshop on Representation Learning for NLP, RepL4NLP 2022 at ACL 2022
Country/TerritoryIreland
CityDublin
Period5/26/22 → …

ASJC Scopus subject areas

  • Computer Science Applications
  • Linguistics and Language
  • Language and Linguistics

Fingerprint

Dive into the research topics of 'Towards Improving Selective Prediction Ability of NLP Systems'. Together they form a unique fingerprint.

Cite this