Emotion-Oriented Systems E-Book

Catherine Pelachaud

• Herausgeber: John Wiley & Sons
• Kategorie: Wissenschaft und neue Technologien
• Sprache: Englisch

Seitenzahl: 589

Veröffentlichungsjahr: 2013

Beschreibung

The Affective Computing domain, term coined by Rosalind Picard in 1997, gathers several scientific areas such as computer science, cognitive science, psychology, design and art. The humane-machine interaction systems are no longer solely fast and efficient. They aim to offer to users affective experiences: user's affective state is detected and considered within the interaction; the system displays affective state; it can reason about their implication to achieve a task or resolve a problem. In this book, we have chosen to cover various domains of research in emotion-oriented systems. Our aim is also to highlight the importance to base the computational model on theoretical foundations and on natural data.

Leseprobe

Table of Contents

Preface

PART 1: Foundations

Chapter 1: Contemporary Theories and Concepts in the Psychology of Emotions

1.1. Introduction

1.2. Emergence of a scientific approach to emotions

1.3. Basic emotions theories

1.4. Bi-dimensional theories of emotion

1.5. Appraisal theories of emotions

1.6. Conclusion

1.7. Glossary

1.8. Bibliography

Chapter 2: Emotion and the Brain

2.1. Introduction

2.2. The major role of affective neuroscience in understanding emotions

2.3. The historical and conceptual legacy of early conceptions of emotions and the brain

2.4. Initial neuro-anatomical emotion theories

2.5. Structures in the brain and their functions in emotional processes

2.6. The prefrontal cortex

2.7. The anterior cingulate cortex

2.8. The role of the insula in disgust

2.9. Temporal dynamic of brain processes in emotional genesis

2.10. Functional connectivity

2.11. Conclusion

2.12. Bibliography

PART 2: Non-verbal behavior

Chapter 3: Emotional Corpora: from Acquisition to Modeling

3.1. Introduction

3.2. Building corpora: “acted”, “induced” and real-life emotions

3.3. Current emotional corpora

3.4. Coding schemes

3.5. Complex emotions in spontaneous data

3.6. Applications for corpora

3.7. Conclusion

3.8. Bibliography

Chapter 4: Visual Emotion Recognition: Status and Key Issues

4.1. Introduction

4.2. What is a facial expression?

4.3. Overview of facial expression recognition methods

4.4. Spontaneous facial expressions

4.5. Expression intensity

4.6. Dynamic analysis

4.7. Multimodality

4.8. Conclusion

4.9. Bibliography

Chapter 5: Recognition of Acoustic Emotion

5.1. Introduction

5.2. Principles of automatic emotion-recognition systems

5.3. Acoustic descriptors

5.4. Automatic emotion classification

5.5. Performance and assessment

5.6. Conclusion

5.7. Bibliography

Chapter 6: Modeling Facial Expressions of Emotions

6.1. Expressive conversational agents

6.2. Expressions and their emotional states

6.3. Computational models for facial expressions of emotions

6.4. Conclusion

6.5. Acknowledgements

6.6. Bibliography

Chapter 7: Emotion Perception and Recognition

7.1. Introduction

7.2. Perception in vocal communication of emotion

7.3. Experimental paradigms and emotion-oriented automatic systems

7.4. Conclusion

7.5. Bibliography

PART 3: Functions

Chapter 8: The Role of Emotions in Human−Machine Interaction

8.1. Introduction

8.2. Interactive information and assistance systems

8.3. Video games

8.4. Intelligent tutoring systems (ITS)

8.5. Discussion and research perspectives

8.6. Bibliography

Chapter 9: Music and Emotions

9.1. The growing importance of music in society

9.2. Recognizing emotions and structural characteristics in music

9.3. Rules for modeling musical expression of emotions

9.4. Towards a continuous measure of emotional reactions to music

9.5. Multimodality in musical experience

9.6. Multimodal emotional synthesis in a musical context

9.7. The social active listening paradigm: the collective aspect of emotion

9.8. Conclusion and perspectives

9.9. Bibliography

Chapter 10: Literary Feelings in Interactive Fiction

10.1. Introduction: emotions and feelings

10.2. French novels and the representation of feelings

10.3. Madame Bovary: plot and scenes

10.4. Interactive fiction and emotional planning

10.5. Linguistic interaction and emotions

10.6. Emma Bovary’s virtuality

10.7. Conclusion

10.8. Bibliography

Chapter 11: The Design of Emotions: How the Digital is Making Us More Emotional

11.1. Representing, interpreting and evoking emotions

11.2. Emotion, mimicry and technical devices

11.3. Devices as an alternate source of emotion: photography

11.4. Art and computers: formal beginnings

11.5. The human behind the mechanics and the mechanics behind the human

11.6. Mirror interaction as an emotional vehicle

11.7. Trompe l’œil versus explicit expression

11.8. Three-dimensional universes: an empathetic experience

11.9. Empathy and identifying emotions

11.10. Making human−machine interaction and dialog effective

11.11. Conclusion: “revenge of the emotions”

11.11. Bibliography

List of Authors

Index

First published 2012 in Great Britain and the United States by ISTE Ltd and John Wiley & Sons, Inc.

Apart from any fair dealing for the purposes of research or private study, or criticism or review, as permitted under the Copyright, Designs and Patents Act 1988, this publication may only be reproduced, stored or transmitted, in any form or by any means, with the prior permission in writing of the publishers, or in the case of reprographic reproduction in accordance with the terms and licenses issued by the CLA. Enquiries concerning reproduction outside these terms should be sent to the publishers at the undermentioned address:

ISTE Ltd27-37 St George’s RoadLondon SW19 4EUUKJohn Wiley & Sons, Inc.111 River StreetHoboken, NJ 07030USAwww.iste.co.ukwww.wiley.com

© ISTE Ltd 2012

The rights of Catherine Pelachaud to be identified as the author of this work have been asserted by her in accordance with the Copyright, Designs and Patents Act 1988.

Library of Congress Cataloging-in-Publication Data

Emotion-oriented systems / edited by Catherine Pelachaud.

p. cm.

Includes bibliographical references and index.

ISBN 978-1-84821-258-9 (hardback)

1. User-centered system design. 2. Human-computer interaction. 3. Interactive computer systems. 4. Virtual computer systems. 5. Emotions and cognition. I. Pelachaud, Catherine.

QA76.9.H85E485 2012

004.2'1--dc23

2011034774

British Library Cataloguing-in-Publication Data

A CIP record for this book is available from the British Library

ISBN 978-1-84821-258-9

Preface

This book is composed of a series of studies focusing on emotions, and in particular on human−machine interaction systems. Interaction systems are no longer simply efficient, precise and fast information units. They aim to provide the end-user with an emotional experience (the iPhone being a good example). They need to consider and even act on end-users’ emotions (such as “serious games”, which are designed to create an optimum learning environment) and to display emotions (through Embodied Conversational Agents). This new generation of systems can detect the user’s emotional state, can identify trigger factors, act upon these emotions or even express an emotion.

Emotions are a complex phenomenon. They involve the evaluation of an event according to several dimensions including physiological, physical and cognitive changes. Emotions have been studied for a number of years through which several theoretical models have been explored, although there is no overall consensus on their various elements. Emotional theories also differ with regard to the (understanding of the) relation between evaluating an event and producing a physical expression and even the concept of emotional expression as a regulator or a purely social expression. However, the link between the cognitive and the physical aspects of emotions is also highlighted by a number of theories.

Emotions have been involved in the world of technology for a number of years. Affective computing (so-called by Rosalind Picard in 1997) addresses the computational modeling of emotions in human−machine interaction while also focusing on perception, cognition and generation. It aims to provide a machine with emotional intelligence, i.e. to provide it with the capability to monitor the end-user’s emotions by detecting and expressing them. In this way, recent years have seen the development of emotion-recognition technology based on analyzing visual, acoustic, physiological and even textual signals. The user’s emotional state is detected and considered during their interaction with the machine which then displays these emotions. The machine can therefore reason about the implications of carrying out a task, the task itself, solving a problem, making a decision, etc.

Emotional computational models require a theoretical basis in emotional models. This allows us to determine the choice of variables to be studied, to formalize complex phenomena and capture their intensity. Indeed, these computational models are becoming increasingly effective; they can test theoretical models, even predict specific results and validate theoretical models. This double exchange between theoretical and computational models is therefore of mutual benefit.

Emotion-oriented systems, by their very nature, are a multidisciplinary area of study. This book does not attempt to provide an exhaustive overview of the topic, but rather to examine a series of areas, ranging from psychology to technology and art, which explore the subject. The aim of this book is to provide the necessary foundations for anyone wishing to study this area of research. Each chapter is dedicated to a different area of study in the domain, which can serve as a pointer to further study. We have chosen to cover several areas to underline the diversity of emotional studies in emotion-oriented systems as well as demonstrate the importance of accounting for emotions, theoretical models and natural data when building these systems.

This book is composed of four parts, each comprising several chapters. Part 1, “Foundations”, is made up of two chapters focusing on theoretical models and affective neuroscience. Part 2, “Non-verbal behavior”, composed of five chapters, will examine the annotation, analysis and synthesis of emotional acoustic and visual behavior. Finally, Part 3, “Applications”, is composed of four chapters examining the role of emotions in human−machine interaction in the arts: in music, literary narration and, lastly, visual arts.

Chapter 1, “Contemporary Theories and Concepts in Emotional Psychology”, written by Géraldine Coppin and David Sander, offers an historical overview of various trends in emotional psychology. These trends are placed within their scientific context, indicating their significance in current research. This chapter also presents different contemporary theoretical models and their concepts. Three theories are further explained: those of basic emotions and bi-dimensional appraisal theories, for each of which their premises and characteristics are detailed. This is also accompanied by a critical analysis, demonstrating their individual limitations. These models, referred to throughout this book, are followed by a glossary of key concepts.

Chapter 2, “Emotion and the Brain”, by Andy Christen and Didier Grandjean, examines the relationship between the cerebral mechanisms underpinning the emotional process. It also examines the relatively new area of affective neuroscience. Following an historical overview of pioneering works, the chapter focuses on the brain’s structure and functions in emotional processes. This is illustrated looking at various parts of the brain (e.g. the amygdala or the prefrontal cortex) active in specific emotions, their role in the cognitive process, and the development of memories or even emotionally-linked physical behavior. These studies are based on a large number of neuroscientific techniques.

The following chapter, “Emotional Corpus: From Acquisition to Modeling”, by Laurence Devillers and Jean-Claude Martin, introduces Part 2 of the book. This chapter underlines the importance of using corpora to construct models based on observed data. It explains the different stages involved in creating and analyzing a corpus. In this way, it also highlights the difference between simulated, induced and experienced emotional data. The chapter then offers a summary of existing corpora. The annotation of corpora involves the creation of a precise coding system for several levels of abstraction and temporality. The importance of validating annotations and providing several measures for this purpose is also presented. The chapter concludes by presenting several applications originating from corpora annotations.

In Chapter 4, “Visual Emotion Recognition: Status and Key Issues”, Alice Caplier presents the latest practices in the field by analyzing work on emotion recognition analysis. The chapter begins by defining facial expressions and introducing Facial Action Coding System (FACS). Analyzing facial expressions requires the ability to detect a face in an image, which therefore means that we need to be able to locate and extract the facial characteristics. As with Chapter 3, there is a distinction between simulated and natural data, which raises the problem of classifying the expressions that need to be recognized. These expressions no longer correspond to the finite set of so-called six basic emotions. Furthermore, facial expressions are not static images − they are in fact dynamic and their temporal evolution is vital for recognizing the emotions they convey. Finally, the chapter concludes by examining multi-modal emotion recognition.

While Chapter 4 examines facial expressions, Chapter 5, “Acoustic Emotion Recognition” by Chloé Clavel and Gaël Richard, examines acoustic indicators in emotion. It details a range of high-level voice descriptors, such as prosody and voice quality, as well as low-level spectral and cepstral descriptors, which are defined and accompanied by their mathematical representations. Automatic classification of emotions using acoustic descriptors is carried out using learning algorithms. It is essentially achieved using supervised methods, i.e. the data (in this case acoustic) are sent to the learning program at the same time as the emotional labels that we want the machine to recognize. The chapter concludes by detailing the factors used to evaluate and compare acoustic recognition models.

The preceding chapters examine emotion analysis and recognition; the next will deal with synthesis.

Chapter 6 “Modeling Emotional Facial Expressions”, written by Sylwia Julia Hyniewska, Radoslaw Niewiadomski and Catherine Pelachaud, raises the question of how to generate emotional behavior in virtual agents. The chapter begins by presenting the three main approaches currently employed and explains how they differ in dealing with emotional facial expressions, followed by a synopsis of facial expression computational models. This overview is structured according to the emotional representation models on which these models are based. Facial expression models describing blends of emotions are also explored. As previously highlighted in Chapter 4, emotions are not expressed by static images but by dynamic sequences of expressions. A computational model of multimodal expression sequences concludes the chapter.

While Chapters 4 and 5 focus on automatic emotion recognition, Chapter 7, “Emotional Perception and Recognition” by Ioana Vasilescu, examines human perceptions of emotion. This chapter examines experimental perception studies used to determine objective characteristic emotional indicators and to understand how human perceptions can be exploited to identify emotions according to set categories. Chapter 7 focuses primarily on the voice, examining acoustic and prosodic indicators. Lexical and dialogical indicators are also studied in relation to voice quality. The results of perception studies can serve as a basis for automatic classification models.

Part 3 consists of four chapters, dedicated to human−machine interaction, music, interactive narration and visual art, respectively.

Chapter 8, “The Role of Emotions in Human−Machine Interaction”, by Magalie Ochs and Valérie Maffiolo, examines the notion of emotional intelligence within the framework of various types of applications involving human−machine interaction. The chapter analyzes how these applications can assess the emotions of the user and the machine. These applications indicate that emotions do not have the same function. Accounting for user emotions in interactive systems allows the system to have a better understanding of the user in order to better adapt to his or her needs. An area in which emotions play a particularly important role is that of “serious games” and video games. The emotions expressed by characters allow the game to feel more life-like and believable. In addition, accounting for the player’s emotions can increase their engagement in the game by stimulating and/or controlling specific emotions.

Chapter 9, “Music and Emotions” by Donald Glowinski and Antonio Camurri, addresses the theme of communicating emotions through music in a way that allows both the composer and listener to feel them. The influence of music on listeners has also been studied. The methods used to determine listeners’ and interpreters’ emotional states frequently rely on questionnaires. Emotions are represented according to diverse emotional theories (presented in Chapter 1) and it should be noted that in recent years the study of multimodality in musical experiences has made head way. Studies have shown that emotions are not only communicated by sounds, but also by listeners’ multimodal behavior, creating a multisensory experience. The authors present a technical platform that allows analysis of musicians’ multimodal behavior. The chapter concludes by presenting musical installations in which participants can interact with musical subject matter, creating a new social listening paradigm.

Another view of emotions is presented in Chapter 10, “Literary Feelings: the Psychology of Characters in Interactive Fiction”, by Marc Cavazza and David Pizzi. The authors focus not on communicational emotions, but the emotions experienced during cultural experiences, particularly during interactive narration. They use the word “feeling” rather than “emotion”, arguing for the necessity of narrative linguistic analysis to understand the nature of feelings. A study of the Madame Bovary narrative using drafts by Gustave Flaubert examines the ontology of feelings felt by the narrative’s characters. The second half of the chapter demonstrates an interactive narrative model in which the virtual characters’ feelings guide their actions, which, in turn, modifies their feelings. This is illustrated by the example of an interactive narration system based on Madame Bovary.

The last chapter of this volume, Chapter 11 “Designing Emotions and how Digitalization Awaits Us” by Annie Gentès, poses the problem of representing emotions with regard to the emotional potential of digital media. To better understand this hypothesis, the author focuses on media such as painting, photography and video, highlighting how interactive devices question how our own identity relates with the machine and our emotions when we interact with it. The creators of these devices rely on two premises: the subject of the work itself and the spectator’s dynamic interaction with it. Interacting in virtual worlds therefore offers an emotional experience based on actions, objects and virtual characters and our personal actions in this virtual world. Interactive works seek a form of empathy from their audience.

I would like to thank Radoslaw Niewiadomski, in particular, for helping me finalize this edition of this book, and Sylwia Hyniewska for her comments.

Catherine PELACHAUD

October 2011

PART 1

Foundations

Chapter 1

Contemporary Theories and Concepts in the Psychology of Emotions1

Is emotion more than a chapter heading?

Bentley, 1928

Everyone knows what an emotion is, until asked to give a definition. Then, it seems, no one knows.

Fehr and Russell, 1984

1.1. Introduction

Following the decisive change of the 1980s, which was marked by a veritable explosion in the scientific study of emotions, emotion is now considered to be a determining explanatory factor in human behavior. In this context, it seems important for any area of research into the functioning of psychological processes and their modeling, as is the case in informatics, to consider current theories and concepts in the psychology of emotions [SAN 09a, SAN 09b]. The central role of emotion in the cognitive system is illustrated by the fact that emotion occupies a “privileged status” in the human brain [DAV 04]; the majority of psychological mechanisms are either necessary for emotion, as such, or are influenced by emotion or involved in regulating emotions. However, as the above quotations suggest, conceptual precision of emotion remains a problem [ALV 02], so much so that there are almost as many definitions of emotion as there are theories and/or researchers working on this topic [KLE 81, STR 96]. In this introductory chapter, we will present the main contemporary theories and concepts of emotion, touching on the notions established towards the end of the 19th Century in which modern approaches are rooted. For each major current theory, we will present its premise, its main assumptions and its characteristics as well as the criticisms that have been aimed at it. As we will see, there is huge variation in the contribution to these theories towards discovering the functional architecture of emotional mechanisms which consists of characterizing the functional sub-systems responsible for the different stages of processing needed to produce emotions and specifying the organization and interaction of these sub-systems [FEL 05, KOR 11, SAN 02].

1.2. Emergence of a scientific approach to emotions

1.2.1. The emotional sequence: James-Lange versus Cannon-Bard

The first “scientific” emotional theory was proposed by William James in 1884 [JAM 84] and Carl Lange in 1885 [LAN 85], who each separately proposed a revolutionary “peripheralist” approach to emotion. This first theoretical conception was the focus of great controversy concerning the mechanisms responsible for triggering emotional feeling (the “sequence problem” [CAN 77]). According to James and Lange, what had previously been considered as the consequence of an emotion, they now considered to be the cause. According to them, the elicitation of a particular emotion is determined by the perception of a specific peripheral arousal pattern, i.e. we experience fear because we are trembling. It should be underlined that, according to James, this definition of emotion is only applicable to emotions that accompany a “specific physical reaction” and not all emotions, resulting in misinterpretations of this theory in later years. In 1885, Lange [LAN 85], summarized the problem of their theory in the following way:

“If I begin to tremble because I am threatened by a loaded pistol, does first a physical process occur within me, does terror arise, is that what causes my trembling, palpitation of the heart, and confusion of thought; or are these bodily phenomena produced directly by the terrifying cause, so that the emotion consists exclusively of the functional disturbances in my body?”