Review
Neural systems for recognizing emotion

https://doi.org/10.1016/S0959-4388(02)00301-XGet rights and content

Abstract

Recognition of emotion draws on a distributed set of structures that include the occipitotemporal neocortex, amygdala, orbitofrontal cortex and right frontoparietal cortices. Recognition of fear may draw especially on the amygdala and the detection of disgust may rely on the insula and basal ganglia. Two important mechanisms for recognition of emotions are the construction of a simulation of the observed emotion in the perceiver, and the modulation of sensory cortices via top-down influences.

Introduction

Emotional signals, either visual or auditory, can be considered as aspects of both an emotional response and social communication. There are multiple schemes for categorizing emotions (Table 1). This review focuses on so-called basic emotions that can be most reliably recognized from facial expressions: happiness, surprise, fear, anger, disgust, and sadness. Most brain structures that participate in the recognition of basic emotions involve both perceptual processing — identifying the geometric configuration of facial features in order to discriminate among different stimuli on the basis of their appearance — and recognition of the emotional meaning of a stimulus — knowing that a certain expression signals fear. Recognition relies on disparate strategies. For instance, recognition of fear from a facial expression may occur by linking the perceptual properties of the facial stimulus to various knowledge-based processes. These include the knowledge components of the concept of fear, the lexical label ‘fear’, the perception of the emotional fear response (or a central representation thereof) that the stimulus triggers in the subject, or knowledge about the motor representations required to produce the expression shown in the stimulus [1••].

Section snippets

Recognition of emotion from facial expressions

A large number of different structures participate in recognizing the emotion shown in a face: the occipitotemporal cortices, amygdala, orbitofrontal cortex, basal ganglia, and right parietal cortices, among others. These structures are engaged in multiple processes and at various points in time, making it difficult to assign a single function to a structure (Fig. 1).

Visual cortices

Regions of the occipital and posterior temporal visual cortices play a critical role in perceptual processing of socially and emotionally relevant visual stimuli. Single-unit studies in monkeys, intracranial field potential studies in neurosurgical human patients 2., 3., 4. and functional imaging studies, have all provided evidence that cortical areas in the lateral parts of the inferior occipital gyrus, fusiform gyrus, and superior temporal gyrus are disproportionately important in face

The amygdala

The amygdala participates in the recognition of emotional signals via at least two classes of input mechanisms: a subcortical route via the superior colliculus and the pulvinar thalamus, and a cortical route via the visual neocortex. Structures in the subcortical route are activated both when normal subjects are shown subliminal facial expressions of fear [16], and when subjects with blindsight, due to striate cortex damage, discriminate emotional facial expressions 17., 18•.. Subliminally

Orbitofrontal cortex

Damage to the orbitofrontal cortex, especially on the right, can result in impaired recognition of emotions from the face and the voice [50]. These findings are consistent with the activation found in right orbitofrontal cortex when comparing presentations of fearful and neutral faces [51••]. In contrast to the amygdala's activation in response to passive viewing of emotional faces or gender judgements, prefrontal regions may be activated when subjects are engaged in a cognitive task requiring

Somatosensory related cortices and the basal ganglia

Following a large number of lesion studies that demonstrated a role for right frontoparietal cortices in emotion recognition, an investigation of the association between lesion location and facial emotion recognition provided evidence that somatosensory related cortices in the right hemisphere played a critical role [58•]. The study revealed a consistent pattern for all emotions: lesions in the right ventral primary and secondary somatosensory areas, and, to a lesser extent in the insula and

Summary of recognizing emotion from faces

Upon presentation of an emotionally meaningful stimulus, one might envision a first, feed-forward sweep of information processing. This would proceed along occipital and temporal neocortices and extract perceptual information from faces and, after ∼100 ms in humans, would coarsely categorize the stimulus as expressing an emotion or not, on the basis of the structural properties of the image. Amygdala and orbitofrontal cortices could participate in the processes of emotion recognition from the

Recognition of emotion from prosody

Studies with computer-generated stimuli have provided evidence supporting two theories of emotion recognition from the patterns of stress and intonation during speech. First, data corroborate the idea that one or two simple auditory cues can be used by subjects to provide much of the information about the emotion in a prosodic stimulus, such as Fo range and amplitude variation. Second, findings also verify that the total number of cues available that can influence emotion judgements is large

Recognizing emotion in other sensory modalities

Our understanding of the neural correlates of emotion recognition in sensory modalities other than vision and audition is rudimentary, in large part because none of the studies have directly investigated recognition. Single-cell responses in animals have been reported to emotionally salient olfactory and gustatory stimuli especially in amygdala and orbitofrontal cortex (see [88] for review). Olfactory 85•., 89., 90. and gustatory [91] stimuli have been found to activate the human amygdala when

Conclusions and future directions

Some open questions regarding emotion recognition are indicated in Box 1. Although the recognition of emotion from facial expressions has been extensively investigated, little is known about the neural structures participating in recognition of emotion from other visual cues, such as body posture, from auditory cues, or from the integration of cues from multiple sensory modalities. Likewise, next to nothing is known about the neural basis of recognizing ‘social’ emotions (Table 1). Explorations

Update

In humans, the emotional reactions that can be triggered by stimuli also play a role in complex aesthetic judgements. Building on lesion studies that have shown dissociations between identifying melodies or recognizing emotion from music [96], a recent functional imaging study found that highly emotional music, which resulted in ‘shivers down the spine’ in the listener, activated a set of paralimbic structures including the ventral striatum, amygdala, and orbitofrontal cortex [97]. Just how to

Acknowledgements

The author is supported, in part, by grants from the National Institute of Mental Health, National Institute of Neurological Disorders and Stroke, the EJLB Foundation and the Klingenstein Fund.

References and recommended reading

Papers of particular interest, published within the annual period of review, have been highlighted as:

  • • of special interest

  • •• of outstanding interest

References (99)

  • P. Vuilleumier et al.

    Effects of attention and emotion on face processing in the human brain. An event-related fMRI study

    Neuron

    (2001)
  • B. Wild et al.

    Are emotions contagious? Evoked emotions while viewing emotionally expressive faces: quality, quantity, time course and gender differences

    Psychiatry Res

    (2001)
  • J.S. Morris et al.

    Saying it with feeling: neural responses to emotional vocalizations

    Neuropsychologia

    (1999)
  • T.W. Buchanan et al.

    Recognition of emotional prosody and verbal components of spoken language: an fMRI study

    Cognit Brain Res

    (2000)
  • P. Rama et al.

    Working memory of identification of emotional vocal expressions: an fMRI study

    Neuroimage

    (2001)
  • M.D. Pell

    Recognition of prosody following unilateral brain lesion: influence of functional and structural attributes of prosodic contours

    Neuropsychologia

    (1998)
  • E.D. Ross et al.

    Lateralization of affective prosody in brain and the callosal integration of hemispheric language functions

    Brain Lang

    (1997)
  • R. Adolphs et al.

    Intact recognition of emotional prosody following amygdala damage

    Neuropsychologia

    (1999)
  • I. Aharon et al.

    Beautiful faces have variable reward value: fMRI and behavioral evidence

    Neuron

    (2001)
  • R. Adolphs

    Recognizing emotion from facial expressions: psychological and neurological mechanisms

    Behav Cognit Neurosci Rev

    (2002)
  • T. Allison et al.

    Electrophysiological studies of human face perception. I. Potentials generated in occipitotemporal cortex by face and non-face stimuli

    Cereb Cortex

    (1999)
  • G. McCarthy et al.

    Electrophysiological studies of human face perception. II. Response properties of face-specific potentials generated in occipitotemporal cortex

    Cereb Cortex

    (1999)
  • A. Puce et al.

    Electrophysiological studies of human face perception. III. Effects of top-down processing on face-specific potentials

    Cereb Cortex

    (1999)
  • Haxby JV, Hoffman EA, Gobbini MI: Human neural systems for face recognition and social communication. Biol Psychiatry...
  • N. Kanwisher

    Domain specificity in face perception

    Nat Neurosci

    (2000)
  • N. Kanwisher et al.

    The fusiform face area: a module in human extrastriate cortex specialized for face perception

    J Neurosci

    (1997)
  • M.J. Tarr et al.

    FFA: a flexible fusiform area for subordinate-level visual processing automatized by expertise

    Nat Neurosci

    (2000)
  • J.V. Haxby et al.

    Distributed and overlapping representation of faces and objects in ventral temporal cortex

    Science

    (2001)
  • E.A. Hoffman et al.

    Distinct representations of eye gaze and identity in the distributed human neural system for face perception

    Nat Neurosci

    (2000)
  • D. Pizzagalli et al.

    Rapid emotional face processing in the human right and left brain hemispheres: an ERP study

    Neuroreport

    (1999)
  • E. Halgren et al.

    Cognitive response profile of the human fusiform face area as determined by MEG

    Cereb Cortex

    (2000)
  • Y. Sugase et al.

    Global and fine information coded by single neurons in the temporal visual cortex

    Nature

    (1999)
  • J.S. Morris et al.

    A subcortical pathway to the right amygdala mediating ‘unseen’ fear

    Proc Natl Acad Sci USA

    (1999)
  • B. de Gelder et al.

    Non-conscious recognition of affect in the absence of striate cortex

    Neuroreport

    (1999)
  • J.S. Morris et al.

    Differential extrageniculostriate and amygdala responses to presentation of emotional faces in a cortically blind field

    Brain

    (2001)
  • P.J. Whalen et al.

    Masked presentations of emotional facial expressions modulate amygdala activity without explicit knowledge

    J Neurosci

    (1998)
  • L. Liu et al.

    Single trial analysis of neurophysiological correlates of the recognition of complex objects and facial expressions of emotion

    Brain Topogr

    (1999)
  • R. Adolphs et al.

    Fear and the human amygdala

    J Neurosci

    (1995)
  • A.J. Calder et al.

    Facial emotion recognition after bilateral amygdala damage: differentially severe impairment of fear

    Cognit Neuropsychol

    (1996)
  • A.K. Anderson et al.

    Expression without recognition: contributions of the human amygdala to emotional communication

    Psychol Sci

    (2000)
  • R. Sprengelmeyer et al.

    Knowing no fear

    Proc R Soc London Ser B

    (1999)
  • A.J. Calder et al.

    Neuropsychology of fear and loathing

    Nat Rev Neurosci

    (2001)
  • R. Adolphs

    The human amygdala and emotion

    Neuroscientist

    (1999)
  • H. Schmolck et al.

    Impaired perception of facial emotions following bilateral damage to the anterior temporal lobe

    Neuropsychology

    (2001)
  • Adolphs R, Tranel D: Emotion recognition and the human amygdala. In The Amygdala. A Functional Analysis. Edited by...
  • P.J. Whalen

    Fear, vigilance, and ambiguity: initial neuroimaging studies of the human amygdala

    Curr Dir Psychol Sci

    (1999)
  • A.K. Anderson et al.

    Contribution of the anteromedial temporal lobes to the evaluation of facial emotion

    Neuropsychology

    (2000)
  • R. Adolphs et al.

    Emotion recognition from faces and prosody following temporal lobectomy

    Neuropsychology

    (2001)
  • J.S. Morris et al.

    A differential neural response in the human amygdala to fearful and happy facial expressions

    Nature

    (1996)
  • Cited by (1516)

    View all citing articles on Scopus
    View full text