TY - GEN
T1 - PREDICTING THE GENERALIZATION GAP IN DEEP MODELS USING ANCHORING
AU - Narayanaswamy, Vivek
AU - Anirudh, Rushil
AU - Kim, Irene
AU - Mubarka, Yamen
AU - Spanias, Andreas
AU - Thiagarajan, Jayaraman J.
N1 - Funding Information:
This work was supported in part by the ASU SenSIP Center, Arizona State University. This work was performed under the auspices of the U.S. Department of Energy by the Lawrence Livermore National Laboratory under Contract No. DE-AC52-07NA27344, Lawrence Livermore National Security, LLC.
Publisher Copyright:
© 2022 IEEE
PY - 2022
Y1 - 2022
N2 - We address the problem of predicting the generalization gap of deep neural networks under large, natural, and synthetic distribution shifts between source and target domains. This is crucial in understanding how models behave in uncontrollable 'in-the-wild' scenarios, but existing techniques fail when target domain becomes very different from the source. Accurately capturing the relationship and distance between the source and target domains is critical for a reliable post-hoc estimation of generalization. In this paper, we propose a novel strategy for directly predicting accuracy on unseen target data with the help of anchoring and pre-text encoding in predictive models. Anchoring has been shown previously to perform effectively in characterizing domain shifts, which we exploit for predicting the generalization gap. Our experiments on the PACS dataset along with synthetic ablations indicate that our approach produces well calibrated accuracy estimates outperforming existing baselines.
AB - We address the problem of predicting the generalization gap of deep neural networks under large, natural, and synthetic distribution shifts between source and target domains. This is crucial in understanding how models behave in uncontrollable 'in-the-wild' scenarios, but existing techniques fail when target domain becomes very different from the source. Accurately capturing the relationship and distance between the source and target domains is critical for a reliable post-hoc estimation of generalization. In this paper, we propose a novel strategy for directly predicting accuracy on unseen target data with the help of anchoring and pre-text encoding in predictive models. Anchoring has been shown previously to perform effectively in characterizing domain shifts, which we exploit for predicting the generalization gap. Our experiments on the PACS dataset along with synthetic ablations indicate that our approach produces well calibrated accuracy estimates outperforming existing baselines.
KW - Calibrated models
KW - Deep Neural Networks
KW - Predicting generalization
KW - Uncertainty Estimation
UR - http://www.scopus.com/inward/record.url?scp=85131227923&partnerID=8YFLogxK
UR - http://www.scopus.com/inward/citedby.url?scp=85131227923&partnerID=8YFLogxK
U2 - 10.1109/ICASSP43922.2022.9747136
DO - 10.1109/ICASSP43922.2022.9747136
M3 - Conference contribution
AN - SCOPUS:85131227923
T3 - ICASSP, IEEE International Conference on Acoustics, Speech and Signal Processing - Proceedings
SP - 4393
EP - 4397
BT - 2022 IEEE International Conference on Acoustics, Speech, and Signal Processing, ICASSP 2022 - Proceedings
PB - Institute of Electrical and Electronics Engineers Inc.
T2 - 47th IEEE International Conference on Acoustics, Speech, and Signal Processing, ICASSP 2022
Y2 - 23 May 2022 through 27 May 2022
ER -