Production and perception of Brazilian Portuguese intonation of assertions, questions, and exclamations: a multimodal analysis

This presentation discusses the results of a multimodal analysis of the production and perception of Brazilian Portuguese intonation of statements, questions and exclamations.

The aim of this paper was to analyze how the auditory and the visual channels are integrated in the perceptual recognition of four types of utterances, namely, assertion, echo question, wh-question and wh-exclamation. Ten speakers (five males and five females) from Rio de Janeiro were recorded and filmed. This study is divided in two stages. First, the production of the utterances is described both acoustically – in relation to the fundamental frequency (F0), intensity and duration parameters – and visually, in terms of Action Units (momentary muscular movements of the face).

The analysis indicates the acoustic and visual cues distinguishing these pragmatic meanings. F0 brings the clearest acoustic cues to discriminate the four intonational contours, followed by intensity. Regarding facial gestures, head movements and blinking were observed for assertion, head and eyebrow movements for wh-questions, and head, eyebrow and lip movements for the production of wh-exclamations. The echo question, in turn, showed more variation of facial movements. In the next stage, four audiovisual perceptual tests were applied with one hundred and ninety-one listeners, presenting stimuli in three modalities: audio only (A), video only (V) and audio combined with video (AV). The results of the audiovisual tests showed not only that these utterance types are recognized in audiovisual (AV), auditory (A) and visual (V) modalities, but also that listeners benefit from the bimodal condition.

It was concluded that the visual channel is integrated in the perception of the intonation of assertions, question and exclamations, and it has a special role in contributing to robustness of speech perception in contexts with either ambiguous information or degraded audio.

Luma da Silva Miranda received a Ph.D. in Portuguese language from Federal University of Rio de Janeiro (UFRJ). She currently works as a Brazilian Portuguese Lector at Eötvös Loránd University, in Budapest, Hungary. Her field of research is in acoustic phonetics, especially regarding Brazilian Portuguese prosody, including speech perception and production as well as multimodal intonation, and acquisition of L2 sounds.