Prosody-based Automatic Detection of Annoyance and Frustration in Human-Computer Dialog

Jeremy Ang, Elizabeth Shriberg1, and Andreas Stolcke2
(Professor Nelson H. Morgan)
(DARPA) ROAR N66001-99-D-8504, DARPA Communicator Project at ICSI and University of Washington, (NASA) NCC 2-1256, and (NSF) IRI-9619921

We investigate the use of prosody for the detection of frustration and annoyance in natural human-computer dialog. In addition to prosodic features, we examine the contribution of language model information and speaking "style." Results show that a prosodic model can predict whether an utterance is neutral versus "annoyed or frustrated" with an accuracy on par with that of human interlabeler agreement. Accuracy increases when discriminating only "frustrated" from other utterances, and when using only those utterances on which labelers originally agreed. Furthermore, prosodic model accuracy degrades only slightly when using recognized versus true words. Language model features, even if based on true words, are relatively poor predictors of frustration. Finally, we find that hyperarticulation is not a good predictor of emotion; the two phenomena often occur independently.

1Staff, ICSI, SRI International
2Staff, ICSI, SRI International

More information ( or

Send mail to the author : (

Edit this abstract