Elsevier

Cognitive Brain Research

Volume 17, Issue 2, 15 July 2003, Pages 406-418
Cognitive Brain Research

Research report
Brain networks for analyzing eye gaze

https://doi.org/10.1016/S0926-6410(03)00143-5Get rights and content

Abstract

The eyes convey a wealth of information in social interactions. This information is analyzed by multiple brain networks, which we identified using functional magnetic resonance imaging (MRI). Subjects attempted to detect a particular directional cue provided either by gaze changes on an image of a face or by an arrow presented alone or by an arrow superimposed on the face. Another control condition was included in which the eyes moved without providing meaningful directional information. Activation of the superior temporal sulcus accompanied extracting directional information from gaze relative to directional information from an arrow and relative to eye motion without relevant directional information. Such selectivity for gaze processing was not observed in face-responsive fusiform regions. Brain activations were also investigated while subjects viewed the same face but attempted to detect when the eyes gazed directly at them. Most notably, amygdala activation was greater during periods when direct gaze never occurred than during periods when direct gaze occurred on 40% of the trials. In summary, our results suggest that increases in neural processing in the amygdala facilitate the analysis of gaze cues when a person is actively monitoring for emotional gaze events, whereas increases in neural processing in the superior temporal sulcus support the analysis of gaze cues that provide socially meaningful spatial information.

Introduction

The eyes move not only in the service of visual perception but also to support communication by indicating direction of attention, intention, or emotion [7]. Infants stare longer at the eyes than other facial features [39] and they spontaneously follow someone else’s gaze as early as 10 weeks of age [28]. Likewise, adults tend to automatically shift their attention in the direction of another person’s gaze [20], and when this happens, the outcome is that both people are attending to the same thing. This phenomenon of joint attention has been shown to facilitate the development of language and social cognition in children, and to particularly facilitate theory of mind skills—the understanding of another person’s mental state [5], [38], [41]. In addition, developmental delays in gaze-following have been shown to predict a later diagnosis of autism [4], so the infant’s behavioral response to gaze cues has become an important developmental marker.

Direct eye contact—when two people gaze directly at one another—is also an important aspect of gaze behavior [35]. Perceiving the direct gaze of another person, unlike averted gaze, indicates that the direction of attention is focused on the viewer. Perceiving eye contact directs and fixes attention on the observed face [23] and in visual search paradigms, it is detected faster than averted gaze [52]. Eye contact has been shown to increase physiological response in social interactions [43], and the amount and quality of eye contact are considered important indicators of social and emotional functioning [35]. Poor eye contact is a specific diagnostic feature of autism [3] and a key component of the negative symptom syndrome in schizophrenia [2]. Thus, investigating neural mechanisms of gaze may provide key insights for understanding neurobiological factors that mediate social development, social interactions, and, ultimately, how dysfunctions in these mechanisms might be related to symptoms observed in disorders such as autism and schizophrenia.

There is mounting evidence to suggest that specific regions of the temporal lobe, such as the fusiform gyrus, superior temporal sulcus (STS), and the amygdala, are involved in gaze processing [1], [26], [32], [33], [46], [47], [54]. Gaze is usually perceived in the context of a face, and faces are known to activate both the fusiform gyrus and the STS [1], [32], [46]. However, the fusiform gyrus responds more to whole faces and the STS responds more to facial features, particularly the eyes. Face identity judgments produce relatively more fusiform gyrus activity whereas gaze direction judgments of the same visual stimuli produce relatively more STS activity [26], [27]. Furthermore, lesions in the region of the fusiform gyrus can produce prosopagnosia, the inability to recognize familiar faces [16], [19], [31], [57]. Deficits in gaze direction discrimination are generally not found after fusiform damage but are found after STS damage [11].

Despite this evidence implicating the STS region in gaze perception, the exact nature of its contribution remains unresolved. Given that the STS responds to various kinds of biological motion, Haxby and colleagues [26] proposed that STS activity to gaze reflects processing of eyes as one of several movable facial features that is useful in social communication; in contrast, fusiform activity reflects processing of invariant facial features that are most useful for discerning personal identity. STS activation to eye and mouth movements [47] is consistent with this hypothesis, as is STS activation to passive viewing of averted and direct gaze [54] given that movable facial features have implied motion even when viewed as static pictures [37].

Another hypothesis about the STS, derived from neuronal recordings of STS activity in monkeys, emphasizes that this region processes cues about the direction of attention of others [44], [45]. STS cells show varying activity to pictures of different head orientations and gaze directions but show maximum firing when head and gaze are oriented in the same direction. Perrett and colleagues interpreted these data as evidence that the cells respond to the direction of attention of the observed individual [44], [45]. This idea has been especially influential because it relates to joint attention and the associated deficits in autism.

However, people automatically shift their own attention in response to the directional information in gaze [20], so it is difficult to separate perception of direction of attention of another from perception of the directional information inherent in that stimulus. Thus an alternative hypothesis is that the STS responds to any type of directional cue. This notion is supported by research showing activity in the STS and adjacent areas to directional attention cues that are not biological [34]. In addition, lesions in the posterior portion of the STS, e.g. the temporal parietal junction (TPJ), can compromise spatial attention skills [40]. Furthermore, during passive viewing of averted gaze [23], activity in the STS region correlates with activity in the intraparietal sulcus (IPS)—a brain area that has been consistently implicated in neural networks of spatial attention [14].

To obtain information about specific visual analyses taking place in STS, we designed an experiment to determine whether differential STS activity would be elicited by repetitive eye motion vs. eye motion providing relevant directional information vs. directional information from a nonfacial source. Similar STS responses to all types of eye motion would implicate a basic visual motion function, whereas preferential response to cues to direction of attention would implicate analyses more closely tied to the attentional relevance of the stimuli.

The amygdala is also centrally involved in gaze processing. Patients with bilateral amygdala damage experience difficulty identifying gaze direction [56]. Amygdala activation measured with positron emission tomography (PET) has been reported to passive viewing of both direct and averted gaze [54] and to active detection of eye contact (bilateral amygdala) and averted gaze (left amygdala) [33]. These results illustrate that the amygdala is involved in monitoring gaze and suggest that the right amygdala is instrumental in the perception of direct gaze. However, it is still unclear from these studies whether the aymgdala is responding to the presence of direct gaze or the process of monitoring for its appearance. Prior fMRI gaze studies were not able to contribute to these ideas because the region of the amygdala was not scanned [27], [47].

We investigated brain activations associated with gaze processing in two experiments. Although emotional facial expression was not of primary interest, we used happy and angry expressions in different blocks in both experiments. This design feature allowed us to minimize between-subject variability in spontaneous judgments of facial expression that tends to occur with neutral faces [18], and also allowed us to investigate effects of emotional expression on gaze processing.

In the first experiment we simulated the use of gaze as a cue to direction of attention. We directly tested whether regions such as STS would exhibit differential activation to gaze cues indicating direction of attention compared to nongaze cues providing directional information or to eye motion not providing directional information. In the primary condition, subjects viewed a face while the eyes of the face shifted in a fashion that implied eye motion, as if the individual was looking sequentially at different spatial locations. This Gaze task required that subjects discriminate whether or not the eyes gazed at a particular target location. Whereas prior fMRI experiments typically included only left and right gaze, we included ten different gaze positions such that fine-grained discriminations were required, thus approximating a more demanding and ecologically valid perceptual analysis of gaze cues. In control tasks an arrow, isolated or superimposed on a face, provided directional information instead of the eyes, or the eyes moved without providing relevant directional information (see Fig. 1 for examples of stimuli).

In a second experiment, we sought to determine whether amygdala activation was specifically associated with viewing direct gaze or with the act of monitoring gaze. Subjects identified direct gaze among direct and averted gaze trials, and trials were blocked so that direct gaze occurred on either 40% or 0% of the trials.

Section snippets

Tasks

In the Gaze task, a face remained continuously on the screen and the eyes of the face looked to a certain spatial location for 300 ms and looked back to the viewer for 900 ms. The subject pressed one button when the eyes shifted to a target location and another button when the eyes shifted towards any nontarget location. All gaze cues were based on a clock template such that the target location assigned on each run corresponded to one quadrant of the clock (i.e. 1 o’clock and 2 o’clock for

Experiment 1

A comparison between the Gaze task and all three control conditions combined gave an overview of neural regions responsive to gaze cues that indicate the direction of another person’s attention. This combined contrast controls for visual processing of faces per se, for non-meaningful eye motion, and for generic cognitive demands of extracting directional information from a stimulus. The neural network for gaze processing isolated by this analysis included three key regions: the posterior

Discussion

Patterns of activation revealed with fMRI in these two experiments showed that distinct contributions arise from brain networks in four regions thought to be involved in gaze processing. Face and gaze perception is accomplished via contributions from the STS, amygdala, fusiform gyrus, and PFC. Several insights into how gaze information is extracted can be gained by considering the distinct ways in which facial input is analyzed in these regions.

Acknowledgements

This research was supported by an NIMH Predoctoral Fellowship F31-MH12982 (CIH). The authors would like to thank Rick Zinbarg, William Revelle, and Marcia Grabowecky for helpful suggestions on experimental design, Katherine Byrne for assistance with stimuli production, and Joseph Coulson for assistance with manuscript preparation.

References (57)

  • Y.H. Kim et al.

    The large-scale neural network for spatial attention displays multifunctional overlap but differential asymmetry

    Neuroimage

    (1999)
  • R.T. Knight

    Decreased response to novel stimuli after prefrontal lesions in man

    Electroencephalogr. Clin. Neurophysiol.

    (1984)
  • K. Nichols et al.

    Eye Gaze and GSR

    J. Exp. Soc. Psychol.

    (1971)
  • B. Wicker et al.

    Brain regions involved in the perception of gaze: a PET study

    Neuroimage

    (1998)
  • N.C. Andreasen

    Negative symptoms in schizophrenia. Definition and reliability

    Arch. Gen. Psychiatry

    (1982)
  • Diagnostic and Statistical Manual of Mental Disorders

    (1994)
  • D. Baldwin

    Understanding the link between joint attention and language

  • F. Barcelo et al.

    Prefrontal modulation of visual processing in humans

    Nat. Neurosci.

    (2000)
  • S. Baron-Cohen

    Mindblindness: An Essay On Autism and Theory of Mind

    (1995)
  • E. Bonda et al.

    Specific involvement of human parietal systems and the amygdala in the perception of biological motion

    J. Neurosci.

    (1996)
  • L. Brothers

    The social brain: a project for integrating primate behaviour and neurophysiology in a new domain

    Concep. Neurosci.

    (1990)
  • V.P. Clark et al.

    Selective attention to face identity and color studied with fMRI

    Hum. Brain Mapping

    (1997)
  • D.L. Collins et al.

    Automatic 3D inter-subject registration of MR volumetric data in standardized Talairach space

    J. Comput. Assisted Tomogr.

    (1994)
  • M. Corbetta

    Frontoparietal cortical networks for directing attention and the eye to visual locations: identical, independent, or overlapping neural systems?

    Proc. Natl. Acad. Sci.

    (1998)
  • J. Decety et al.

    Brain activity during observation of actions. Influence of action content and subject’s strategy

    Brain

    (1997)
  • P. Ekman et al.

    Pictures of Facial Affect

    (1976)
  • C.K. Friesen et al.

    The eyes have it! Reflexive orienting is triggered by nonpredictive gaze

    Psychonom. Bull. Rev.

    (1998)
  • L.M. Frodi et al.

    Fathers’ and mothers’ responses to infants’ smiles and cries

    Infant Behav. Dev.

    (1978)
  • Cited by (186)

    • Body, Brain, Behavior: Three Views and a Conversation

      2022, Body, Brain, Behavior: Three Views and a Conversation
    • Comparative anatomy of the macaque and the human frontal oculomotor domain

      2021, Neuroscience and Biobehavioral Reviews
      Citation Excerpt :

      First, Ikkai and Curtis (2008) have reported that attentional shifts with or without eye movements (overt or covert) activate a cortical sector involving the superior and the inferior precentral sulcus and extending also in the caudal part of the inferior frontal sulcus. Furthermore, Bristow et al. (2007) and Hooker et al. (2003) observed activation at the inferior frontal junction or in the inferior frontal gyrus for gaze perception and attentional shifts guided by changes in gaze direction (see also Grosbras et al., 2005; Nummenmaa and Calder, 2009). Finally, Ishai et al. (2005) and Tsao et al. (2008) found that face perception activates cortical sectors in the caudal part of the right inferior frontal sulcus.

    View all citing articles on Scopus
    1

    Current address: Department of Psychology and Helen Wills Neuroscience Institute, University of California, Berkeley, 4143 Tolman Hall, Berkeley, CA 94720-5050, USA.

    View full text