TY - GEN
T1 - Generative multimodal models of nonverbal synchrony in close relationships
AU - Grafsgaard, Joseph
AU - Duran, Nicholas
AU - Randall, Ashley
AU - Tao, Chun
AU - D'Mello, Sidney
N1 - Publisher Copyright:
© 2018 IEEE.
PY - 2018/6/5
Y1 - 2018/6/5
N2 - Positive interpersonal relationships require shared understanding along with a sense of rapport. A key facet of rapport is mirroring and convergence of facial expression and body language, known as nonverbal synchrony. We examined nonverbal synchrony in a study of 29 heterosexual romantic couples, in which audio, video, and bracelet accelerometer were recorded during three conversations. We extracted facial expression, body movement, and acoustic-prosodic features to train neural network models that predicted the nonverbal behaviors of one partner from those of the other. Recurrent models (LSTMs) outperformed feed-forward neural networks and other chance baselines. The models learned behaviors encompassing facial responses, speech-related facial movements, and head movement. However, they did not capture fleeting or periodic behaviors, such as nodding, head turning, and hand gestures. Notably, a preliminary analysis of clinical measures showed greater association with our model outputs than correlation of raw signals. We discuss potential uses of these generative models as a research tool to complement current analytical methods along with real-world applications (e.g., as a tool in therapy).
AB - Positive interpersonal relationships require shared understanding along with a sense of rapport. A key facet of rapport is mirroring and convergence of facial expression and body language, known as nonverbal synchrony. We examined nonverbal synchrony in a study of 29 heterosexual romantic couples, in which audio, video, and bracelet accelerometer were recorded during three conversations. We extracted facial expression, body movement, and acoustic-prosodic features to train neural network models that predicted the nonverbal behaviors of one partner from those of the other. Recurrent models (LSTMs) outperformed feed-forward neural networks and other chance baselines. The models learned behaviors encompassing facial responses, speech-related facial movements, and head movement. However, they did not capture fleeting or periodic behaviors, such as nodding, head turning, and hand gestures. Notably, a preliminary analysis of clinical measures showed greater association with our model outputs than correlation of raw signals. We discuss potential uses of these generative models as a research tool to complement current analytical methods along with real-world applications (e.g., as a tool in therapy).
KW - Close relationships
KW - Couples therapy
KW - Facial expression
KW - LSTM
KW - Neural networks
KW - Nonverbal synchrony
UR - http://www.scopus.com/inward/record.url?scp=85049389284&partnerID=8YFLogxK
UR - http://www.scopus.com/inward/citedby.url?scp=85049389284&partnerID=8YFLogxK
U2 - 10.1109/FG.2018.00037
DO - 10.1109/FG.2018.00037
M3 - Conference contribution
AN - SCOPUS:85049389284
T3 - Proceedings - 13th IEEE International Conference on Automatic Face and Gesture Recognition, FG 2018
SP - 195
EP - 202
BT - Proceedings - 13th IEEE International Conference on Automatic Face and Gesture Recognition, FG 2018
PB - Institute of Electrical and Electronics Engineers Inc.
T2 - 13th IEEE International Conference on Automatic Face and Gesture Recognition, FG 2018
Y2 - 15 May 2018 through 19 May 2018
ER -