Multimodal character viewpoint in quoted dialogue sequences

Kashmiri Stec, Mike Huiskes, Martijn Wieling, Gisela Redeker

Full Text: PDF   Paper Package: Stec2017_1.0 tar.gz zip PID: 11022/


We investigate the multimodal production of character viewpoint in spoken American English narratives by performing complementary qualitative and quantitative analyses of two quoted dialogues, focusing on the storyteller’s use of character viewpoint gestures, character intona- tion, character facial expression, spatial orientation and gaze. A micro-analysis revealed that the extent of multimodal articulation depends on (i) the quoted speaker, with different multimodal articulatory patterns found for quotes by the speaker’s past self vs. a third-person character, and (ii) the position of the quoted utterance within the quoted dialogue, with mid-dialogue utter- ances garnering less co-articulation than initial or nal utterances within the quoted dialogue. We further investigated these observations using a quantitative approach, which was based on generalized additive modeling (GAM). The GAM analysis revealed different multimodal patterns for each quoted character, as indicated by the number of co-produced multimodal articulators. These patterns were found to hold regardless of the quote’s position within the narrative. We discuss these ndings with respect to previous work on multimodal quotation.