Skip to main content

Thank you for visiting nature.com. You are using a browser version with limited support for CSS. To obtain the best experience, we recommend you use a more up to date browser (or turn off compatibility mode in Internet Explorer). In the meantime, to ensure continued support, we are displaying the site without styles and JavaScript.

  • Review Article
  • Published:

Bridging the gap between theories of sensory cue integration and the physiology of multisensory neurons

Key Points

  • Integration of multiple sensory cues is crucial for adaptive behaviour, but understanding its neural basis has been hampered by a divergence in the literature into two largely non-overlapping conceptual approaches. One approach seeks to predict and quantify cue integration behaviour in a rigorous psychophysical framework, whereas the other has focused on the empirical principles that govern multisensory integration by neurons, including its anatomical and developmental origins.

  • Tasked with reconciling these camps are the computational theorists who strive to create models of multisensory processing that are both biologically realistic and capable of explaining psychophysical performance. Progress has been made on this front, an example of which is the theory of probabilistic population coding. However, several obstacles remain, such as accounting for key differences between model predictions and neurophysiological data and a lack of neuronal recordings from behaving animals performing psychophysical cue integration tasks.

  • A recent set of studies on the integration of visual and vestibular cues for self-motion perception draws from both sets of approaches and has attempted to narrow the conceptual gap between them. This work uses macaque monkeys trained to report their heading direction in a virtual-reality apparatus, while researchers monitor single-unit activity in a multisensory region of the extrastriate visual cortex (the dorsal medial superior temporal area (MSTd)).

  • The emerging picture is one in which computations performed by multisensory neurons in area MSTd can be linked to psychophysical performance and may also help to explain the empirical principles that have driven the field for two decades. Divisive normalization, a common property of neuronal circuits, may be one of the key computations that give rise to the desired behaviour.

  • Further work will be needed to develop more complete models that connect the neurobiological details of multisensory integration with the probabilistic computations underlying cue integration performance and perception in general.

Abstract

The richness of perceptual experience, as well as its usefulness for guiding behaviour, depends on the synthesis of information across multiple senses. Recent decades have witnessed a surge in our understanding of how the brain combines sensory cues. Much of this research has been guided by one of two distinct approaches: one is driven primarily by neurophysiological observations, and the other is guided by principles of mathematical psychology and psychophysics. Conflicting results and interpretations have contributed to a conceptual gap between psychophysical and physiological accounts of cue integration, but recent studies of visual–vestibular cue integration have narrowed this gap considerably.

This is a preview of subscription content, access via your institution

Access options

Rent or buy this article

Prices vary by article type

from$1.95

to$39.95

Prices may be subject to local taxes which are calculated during checkout

Figure 1: Schematic of a generic cue integration cue-conflict psychophysical task.
Figure 2: A probabilistic population code framework accounts for optimal cue integration by summation of unisensory population activity.
Figure 3: Combined psychophysical and neurophysiological studies of visual–vestibular cue integration in the macaque.
Figure 4: The normalization model of multisensory integration.

Similar content being viewed by others

References

  1. Faisal, A. A., Selen, L. P. & Wolpert, D. M. Noise in the nervous system. Nature Rev. Neurosci. 9, 292–303 (2008).

    CAS  Google Scholar 

  2. Gepshtein, S. Two psychologies of perception and the prospect of their synthesis. Philos. Psychol. 23, 217–281 (2010).

    Google Scholar 

  3. Knill, D. C. & Pouget, A. The Bayesian brain: the role of uncertainty in neural coding and computation. Trends Neurosci. 27, 712–719 (2004). A concise review that provides a good introduction to the idea of sensory uncertainty and the Bayesian perspective on behaviour and neural coding, including the incorporation of priors and studies of motor control.

    CAS  PubMed  Google Scholar 

  4. Einstein, A. Über das Relativitätsprinzip und die aus demselben gezogenen Folgerungen. Jahrbuch Radioaktivität Elektronik 4, 411–462 (1907).

    Google Scholar 

  5. Angelaki, D. E., Shaikh, A. G., Green, A. M. & Dickman, J. D. Neurons compute internal models of the physical laws of motion. Nature 430, 560–564 (2004).

    CAS  PubMed  Google Scholar 

  6. Merfeld, D., Zupan, L. & Peterka, R. Humans use internal models to estimate gravity and linear acceleration. Nature 398, 615–618 (1999).

    CAS  PubMed  Google Scholar 

  7. Landy, M. S., Banks, M. S. & Knill, D. C. in Sensory Cue Integration (eds Trommershäuser, J., Kording, K. P. & Landy, M. S.) 5–29 (Oxford Univ. Press, 2011).

    Google Scholar 

  8. Angelaki, D. E., Gu, Y. & DeAngelis, G. C. Multisensory integration: psychophysics, neurophysiology, and computation. Curr. Opin. Neurobiol. 19, 452–458 (2009).

    CAS  PubMed  PubMed Central  Google Scholar 

  9. Kajikawa, Y., Falchier, A., Musacchia, G., Lakatos, P. & Schroeder, C. E. in The Neural Bases of Multisensory Processes (eds Murray, M. & Wallace, M.) (CRC Press, 2012).

    Google Scholar 

  10. Lakatos, P., Chen, C. M., O'Connell, M. N., Mills, A. & Schroeder, C. E. Neuronal oscillations and multisensory interaction in primary auditory cortex. Neuron 53, 279–292 (2007).

    CAS  PubMed  PubMed Central  Google Scholar 

  11. Kayser, C., Petkov, C. I., Remedios, R. & Logothetis, N. K. in The Neural Bases of Multisensory Processes (eds Murray, M. & Wallace, M.) (CRC Press, 2012).

    Google Scholar 

  12. Kayser, C. & Logothetis, N. K. Do early sensory cortices integrate cross-modal information? Brain Struct. Funct. 212, 121–132 (2007).

    PubMed  Google Scholar 

  13. Ghazanfar, A. A. & Schroeder, C. E. Is neocortex essentially multisensory? Trends Cogn. Sci. 10, 278–285 (2006).

    PubMed  Google Scholar 

  14. Driver, J. & Noesselt, T. Multisensory interplay reveals crossmodal influences on 'sensory-specific' brain regions, neural responses, and judgments. Neuron 57, 11–23 (2008).

    CAS  PubMed  PubMed Central  Google Scholar 

  15. Geisler, W. S. Contributions of ideal observer theory to vision research. Vision Res. 51, 771–781 (2011).

    PubMed  Google Scholar 

  16. Jacobs, R. A. Optimal integration of texture and motion cues to depth. Vision Res. 39, 3621–3629 (1999).

    CAS  PubMed  Google Scholar 

  17. Landy, M. S., Maloney, L. T., Johnston, E. B. & Young, M. Measurement and modeling of depth cue combination: in defense of weak fusion. Vision Res. 35, 389–412 (1995). Focusing on the array of visual cues available for the perception of depth, this paper develops several key ideas underlying contemporary ideal observer models of cue integration and also introduces a psychophysical procedure that has become a standard method for testing such models.

    CAS  PubMed  Google Scholar 

  18. Maloney, L. T. & Landy, M. S. in Proceedings of the SPIE (ed. Pearlman, W. A.) 1154–1163 (SPIE, 1989).

    Google Scholar 

  19. Clark, J. J. & Yuille, A. L. Data Fusion For Sensory Information Processing Systems (Kluwer, 1990).

    Google Scholar 

  20. Alais, D. & Burr, D. The ventriloquist effect results from near-optimal bimodal integration. Curr. Biol. 14, 257–262 (2004).

    CAS  PubMed  Google Scholar 

  21. Battaglia, P. W., Jacobs, R. A. & Aslin, R. N. Bayesian integration of visual and auditory signals for spatial localization. J. Opt. Soc. Am. A Opt. Image Sci. Vis. 20, 1391–1397 (2003).

    PubMed  Google Scholar 

  22. Ernst, M. O. & Banks, M. S. Humans integrate visual and haptic information in a statistically optimal fashion. Nature 415, 429–433 (2002). This study provides one of the earliest and clearest psychophysical demonstrations of optimal cue integration across separate sensory modalities. The authors showed that human subjects integrate vision and touch to estimate the width of a grasped object by taking into account the relative reliability of the cues and combining them to improve their performance. Importantly, cue reliability was varied randomly from trial to trial, suggesting that the brain may not need to explicitly learn or represent the uncertainty of the cues to accomplish the task.

    CAS  PubMed  Google Scholar 

  23. Sober, S. J. & Sabes, P. N. Flexible strategies for sensory integration during motor planning. Nature Neurosci. 8, 490–497 (2005).

    CAS  PubMed  Google Scholar 

  24. van Beers, R. J., Sittig, A. C. & Gon, J. J. Integration of proprioceptive and visual position-information: an experimentally supported model. J. Neurophysiol. 81, 1355–1364 (1999).

    CAS  PubMed  Google Scholar 

  25. Körding, K. P. & Wolpert, D. M. Bayesian decision theory in sensorimotor control. Trends Cogn. Sci. 10, 319–326 (2006).

    PubMed  Google Scholar 

  26. Gu, Y., Angelaki, D. E. & DeAngelis, G. C. Neural correlates of multisensory cue integration in macaque MSTd. Nature Neurosci. 11, 1201–1210 (2008). Using a visual–vestibular heading discrimination task, this study showed that monkeys, like humans, are capable of combining sensory cues to improve perceptual performance. The authors also characterized a population of neurons in extrastriate visual cortex (specifically, in area MSTd) that could underlie the behaviour.

    CAS  PubMed  Google Scholar 

  27. de Winkel, K. N., Weesie, J., Werkhoven, P. J. & Groen, E. L. Integration of visual and inertial cues in perceived heading of self-motion. J. Vis. 10, 1 (2010).

    PubMed  Google Scholar 

  28. Edwards, M., O'Mahony, S., Ibbotson, M. R. & Kohlhagen, S. Vestibular stimulation affects optic-flow sensitivity. Perception 39, 1303–1310 (2010).

    PubMed  Google Scholar 

  29. Butler, J. S., Smith, S. T., Campos, J. L. & Bulthoff, H. H. Bayesian integration of visual and vestibular signals for heading. J. Vis. 10, 23 (2010).

    PubMed  Google Scholar 

  30. Fetsch, C. R., Turner, A. H., DeAngelis, G. C. & Angelaki, D. E. Dynamic reweighting of visual and vestibular cues during self-motion perception. J. Neurosci. 29, 15601–15612 (2009).

    CAS  PubMed  PubMed Central  Google Scholar 

  31. Cochran, W. G. Problems arising in the analysis of a series of similar experiments. Appl. Stat. 4, 102–118 (1937).

    Google Scholar 

  32. Yuille, A. L. & Bülthoff, H. H. in Perception as Bayesian Inference (eds Knill, D. C. & Richards, W.) 123–162 (Cambridge Univ. Press, 1996).

    Google Scholar 

  33. Young, M. J., Landy, M. S. & Maloney, L. T. A perturbation analysis of depth perception from combinations of texture and motion cues. Vision Res. 33, 2685–2696 (1993).

    CAS  PubMed  Google Scholar 

  34. Knill, D. C. & Saunders, J. A. Do humans optimally integrate stereo and texture information for judgments of surface slant? Vision Res. 43, 2539–2558 (2003).

    PubMed  Google Scholar 

  35. Hillis, J. M., Watt, S. J., Landy, M. S. & Banks, M. S. Slant from texture and disparity cues: optimal cue combination. J. Vis. 4, 967–992 (2004).

    PubMed  Google Scholar 

  36. Clemens, I. A., De Vrijer, M., Selen, L. P., Van Gisbergen, J. A. & Medendorp, W. P. Multisensory processing in spatial orientation: an inverse probabilistic approach. J. Neurosci. 31, 5365–5377 (2011).

    CAS  PubMed  PubMed Central  Google Scholar 

  37. Ma, W., Beck, J. M. & Pouget, A. in Sensory Cue Integration (eds Trommershäuser, J., Kording, K. P. & Landy, M. S.) 393–405 (Oxford Univ. Press, 2011).

    Google Scholar 

  38. Oruc, I., Maloney, L. T. & Landy, M. S. Weighted linear cue combination with possibly correlated error. Vision Res. 43, 2451–2468 (2003).

    PubMed  Google Scholar 

  39. Rosas, P., Wagemans, J., Ernst, M. O. & Wichmann, F. A. Texture and haptic cues in slant discrimination: reliability-based cue weighting without statistically optimal cue combination. J. Opt. Soc. Am. A Opt. Image Sci. Vis. 22, 801–809 (2005).

    PubMed  Google Scholar 

  40. Rosas, P., Wichmann, F. A. & Wagemans, J. Texture and object motion in slant discrimination: failure of reliability-based weighting of cues may be evidence for strong fusion. J. Vis. 7, 3 (2007).

    PubMed  Google Scholar 

  41. Körding, K. P. et al. Causal inference in multisensory perception. PLoS ONE 2, e943 (2007).

    PubMed  PubMed Central  Google Scholar 

  42. Knill, D. C. Robust cue integration: a Bayesian model and evidence from cue-conflict studies with stereoscopic and figure cues to slant. J. Vis. 7, 5 (2007).

    PubMed  Google Scholar 

  43. Ernst, M. O. & Di Luca, M. in Sensory Cue Integration (eds Trommershäuser, J., Landy, M. S. & Kording, K. P.) 224–250 (Oxford Univ. Press, 2011).

    Google Scholar 

  44. Zaidel, A., Turner, A. H. & Angelaki, D. E. Multisensory calibration is independent of cue reliability. J. Neurosci. 31, 13949–13962 (2011).

    CAS  PubMed  PubMed Central  Google Scholar 

  45. Raposo, D., Sheppard, J. P., Schrater, P. R. & Churchland, A. K. Multisensory decision-making in rats and humans. J. Neurosci. 32, 3726–3735 (2012).

    CAS  PubMed  PubMed Central  Google Scholar 

  46. Stein, B. E. & Meredith, M. A. The Merging of the Senses (MIT Press, 1993).

    Google Scholar 

  47. Meredith, M. & Stein, B. Visual, auditory, and somatosensory convergence on cells in superior colliculus results in multisensory integration. J. Neurophysiol. 56, 640–662 (1986). This paper is among the first to demonstrate the impressive capacity of SC neurons to combine visual, tactile and auditory cues, yielding multisensory responses that were often considerably enhanced (and sometimes suppressed) relative to unisensory responses. These early observations laid the foundation for the well-known empirical 'principles' of multisensory integration, including the spatial and temporal principle and inverse effectiveness.

    CAS  PubMed  Google Scholar 

  48. Sparks, D. L. Translation of sensory signals into commands for control of saccadic eye movements: role of primate superior colliculus. Physiol. Rev. 66, 118–171 (1986).

    CAS  PubMed  Google Scholar 

  49. Stein, B. E. Development of the superior colliculus. Annu. Rev. Neurosci. 7, 95–125 (1984).

    CAS  PubMed  Google Scholar 

  50. Wurtz, R. H. & Albano, J. E. Visual-motor function of the primate superior colliculus. Annu. Rev. Neurosci. 3, 189–226 (1980).

    CAS  PubMed  Google Scholar 

  51. Stein, B. E. & Stanford, T. R. Multisensory integration: current issues from the perspective of the single neuron. Nature Rev. Neurosci. 9, 255–266 (2008).

    CAS  Google Scholar 

  52. James, T. W. & Stevenson, R. A. in The Neural Bases of Multisensory Processes (eds Murray, M. & Wallace, M.) (CRC Press, 2012).

    Google Scholar 

  53. Beauchamp, M. S., Lee, K. E., Argall, B. D. & Martin, A. Integration of auditory and visual information about objects in superior temporal sulcus. Neuron 41, 809–823 (2004).

    CAS  PubMed  Google Scholar 

  54. Sarko, D. K. et al. in The Neural Bases of Multisensory Processes (eds Murray, M. M. & Wallace, M. T.) (CRC Press, 2012).

    Google Scholar 

  55. Stanford, T. R., Quessy, S. & Stein, B. E. Evaluating the operations underlying multisensory integration in the cat superior colliculus. J. Neurosci. 25, 6499–6508 (2005). This study explores multisensory interactions in the SC in a more systematic manner than previous work by varying the intensity and timing of visual and auditory stimuli. The results suggest that most SC neurons combine their inputs additively and that the oft-cited phenomenon of 'superadditivity' may only occur for very weak stimuli.

    CAS  PubMed  PubMed Central  Google Scholar 

  56. Jiang, W., Jiang, H. & Stein, B. E. Two corticotectal areas facilitate multisensory orientation behavior. J. Cogn. Neurosci. 14, 1240–1255 (2002).

    PubMed  Google Scholar 

  57. Stein, B. E., Huneycutt, W. S. & Meredith, M. A. Neurons and behavior: the same rules of multisensory integration apply. Brain Res. 448, 355–358 (1988).

    CAS  PubMed  Google Scholar 

  58. Wilkinson, L. K., Meredith, M. A. & Stein, B. E. The role of anterior ectosylvian cortex in cross-modality orientation and approach behavior. Exp. Brain Res. 112, 1–10 (1996).

    CAS  PubMed  Google Scholar 

  59. Frens, M. A., Van Opstal, A. J. & Van der Willigen, R. F. Spatial and temporal factors determine auditory–visual interactions in human saccadic eye movements. Percept. Psychophys. 57, 802–816 (1995).

    CAS  PubMed  Google Scholar 

  60. Wallace, M. T., Meredith, M. A. & Stein, B. E. Converging influences from visual, auditory, and somatosensory cortices onto output neurons of the superior colliculus. J. Neurophysiol. 69, 1797–1809 (1993).

    CAS  PubMed  Google Scholar 

  61. Jiang, W., Wallace, M. T., Jiang, H., Vaughan, J. W. & Stein, B. E. Two cortical areas mediate multisensory integration in superior colliculus neurons. J. Neurophysiol. 85, 506–522 (2001).

    CAS  PubMed  Google Scholar 

  62. Wallace, M. T. & Stein, B. E. Development of multisensory neurons and multisensory integration in cat superior colliculus. J. Neurosci. 17, 2429–2444 (1997).

    CAS  PubMed  Google Scholar 

  63. Wallace, M. T. & Stein, B. E. Onset of cross-modal synthesis in the neonatal superior colliculus is gated by the development of cortical influences. J. Neurophysiol. 83, 3578–3582 (2000).

    CAS  PubMed  Google Scholar 

  64. Wallace, M. T. & Stein, B. E. Sensory and multisensory responses in the newborn monkey superior colliculus. J. Neurosci. 21, 8886–8894 (2001).

    CAS  PubMed  Google Scholar 

  65. Perrault, T. J. Jr, Vaughan, J. W., Stein, B. E. & Wallace, M. T. Neuron-specific response characteristics predict the magnitude of multisensory integration. J. Neurophysiol. 90, 4022–4026 (2003).

    PubMed  Google Scholar 

  66. Alvarado, J. C., Vaughan, J. W., Stanford, T. R. & Stein, B. E. Multisensory versus unisensory integration: contrasting modes in the superior colliculus. J. Neurophysiol. 97, 3193–3205 (2007).

    PubMed  Google Scholar 

  67. Perrault, T. J. Jr, Vaughan, J. W., Stein, B. E. & Wallace, M. T. Superior colliculus neurons use distinct operational modes in the integration of multisensory stimuli. J. Neurophysiol. 93, 2575–2586 (2005).

    PubMed  Google Scholar 

  68. Cuppini, C., Ursino, M., Magosso, E., Rowland, B. A. & Stein, B. E. An emergent model of multisensory integration in superior colliculus neurons. Front. Integr. Neurosci. 4, 6 (2010).

    PubMed  PubMed Central  Google Scholar 

  69. Patton, P. E. & Anastasio, T. J. Modeling cross-modal enhancement and modality-specific suppression in multisensory neurons. Neural Comput. 15, 783–810 (2003).

    PubMed  Google Scholar 

  70. Alvarado, J. C., Rowland, B. A., Stanford, T. R. & Stein, B. E. A neural network model of multisensory integration also accounts for unisensory integration in superior colliculus. Brain Res. 1242, 13–23 (2008).

    CAS  PubMed  PubMed Central  Google Scholar 

  71. Rowland, B. A., Stein, B. E. & Stanford, T. R. in Sensory Cue Integration (eds Trommershäuser, J., Kording, K. P. & Landy, M. S.) 333–344 (Oxford Univ. Press, 2011).

    Google Scholar 

  72. Green, D. M. & Swets, J. A. Signal Detection Theory and Psychophysics (Wiley, 1966).

    Google Scholar 

  73. Kim, B. & Basso, M. A. Saccade target selection in the superior colliculus: a signal detection theory approach. J. Neurosci. 28, 2991–3007 (2008).

    CAS  PubMed  PubMed Central  Google Scholar 

  74. Kim, B. & Basso, M. A. A probabilistic strategy for understanding action selection. J. Neurosci. 30, 2340–2355 (2010).

    CAS  PubMed  PubMed Central  Google Scholar 

  75. Ma, W. J., Beck, J. M., Latham, P. E. & Pouget, A. Bayesian inference with probabilistic population codes. Nature Neurosci. 9, 1432–1438 (2006). This theoretical study outlines a simple and flexible strategy for performing optimal Bayesian inference with populations of neurons, using multisensory cue integration as a primary example. By exploiting a mathematical property of neuronal noise, the authors show that simple summation of population activity in sensory areas can be sufficient to implement Bayes-optimal cue integration.

    CAS  PubMed  Google Scholar 

  76. Beck, J. M. et al. Probabilistic population codes for Bayesian decision making. Neuron 60, 1142–1152 (2008).

    CAS  PubMed  PubMed Central  Google Scholar 

  77. Jazayeri, M. & Movshon, J. A. Optimal representation of sensory information by neural populations. Nature Neurosci. 9, 690–696 (2006).

    CAS  PubMed  Google Scholar 

  78. Ma, W. J. & Pouget, A. Linking neurons to behavior in multisensory perception: a computational review. Brain Res. 1242, 4–12 (2008).

    CAS  PubMed  Google Scholar 

  79. Populin, L. C. & Yin, T. C. Bimodal interactions in the superior colliculus of the behaving cat. J. Neurosci. 22, 2826–2834 (2002).

    CAS  PubMed  Google Scholar 

  80. Stanford, T. R. & Stein, B. E. Superadditivity in multisensory integration: putting the computation in context. Neuroreport 18, 787–792 (2007).

    PubMed  Google Scholar 

  81. Perrault, T. J. Jr, Stein, B. E. & Rowland, B. A. Non-stationarity in multisensory neurons in the superior colliculus. Front. Psychol. 2, 144 (2011).

    PubMed  PubMed Central  Google Scholar 

  82. Gibson, J. J. The Perception of the Visual World (Houghton-Mifflin, 1950).

    Google Scholar 

  83. Warren, W. H. Jr, Morris, M. W. & Kalish, M. Perception of translational heading from optical flow. J. Exp. Psychol. Hum. Percept. Perform. 14, 646–660 (1988).

    PubMed  Google Scholar 

  84. Benson, A. J., Spencer, M. B. & Stott, J. R. Thresholds for the detection of the direction of whole-body, linear movement in the horizontal plane. Aviat. Space Environ. Med. 57, 1088–1096 (1986).

    CAS  PubMed  Google Scholar 

  85. Fernandez, C. & Goldberg, J. M. Physiology of peripheral neurons innervating otolith organs of the squirrel monkey. II. Directional selectivity and force-response relations. J. Neurophysiol. 39, 985–995 (1976).

    CAS  PubMed  Google Scholar 

  86. Fernandez, C. & Goldberg, J. M. Physiology of peripheral neurons innervating otolith organs of the squirrel monkey. I. Response to static tilts and to long-duration centrifugal force. J. Neurophysiol. 39, 970–984 (1976).

    CAS  PubMed  Google Scholar 

  87. Guedry, F. E. in Handbook of Sensory Physiology, The Vestibular System (ed. Kornhuber, H. H. ) (Springer, 1974).

    Google Scholar 

  88. Chen, A., DeAngelis, G. C. & Angelaki, D. E. Representation of vestibular and visual cues to self-motion in ventral intraparietal cortex. J. Neurosci. 31, 12036–12052 (2011).

    CAS  PubMed  PubMed Central  Google Scholar 

  89. Gu, Y., Watkins, P. V., Angelaki, D. E. & DeAngelis, G. C. Visual and nonvisual contributions to three-dimensional heading selectivity in the medial superior temporal area. J. Neurosci. 26, 73–85 (2006).

    CAS  PubMed  PubMed Central  Google Scholar 

  90. Chen, A., DeAngelis, G. C. & Angelaki, D. E. Convergence of vestibular and visual self-motion signals in an area of the posterior sylvian fissure. J. Neurosci. 31, 11617–11627 (2011).

    CAS  PubMed  PubMed Central  Google Scholar 

  91. Duffy, C. J. MST neurons respond to optic flow and translational movement. J. Neurophysiol. 80, 1816–1827 (1998).

    CAS  PubMed  Google Scholar 

  92. Britten, K. H. & van Wezel, R. J. Electrical microstimulation of cortical area MST biases heading perception in monkeys. Nature Neurosci. 1, 59–63 (1998).

    CAS  PubMed  Google Scholar 

  93. Gu, Y., DeAngelis, G. C. & Angelaki, D. E. Causal links between dorsal medial superior temporal area neurons and multisensory heading perception. J. Neurosci. 32, 2299–2313 (2012).

    CAS  PubMed  PubMed Central  Google Scholar 

  94. Morgan, M. L., DeAngelis, G. C. & Angelaki, D. E. Multisensory integration in macaque visual cortex depends on cue reliability. Neuron 59, 662–673 (2008). The authors recorded neuronal responses in area MSTd to a wide array of stimulus combinations, and found a simple mathematical rule by which multisensory neurons combine their inputs. Neurons appeared to take a weighted sum of their inputs, with weights that depend on cue reliability — a surprising finding that conflicted with theoretical predictions and was not well understood until later.

    CAS  PubMed  PubMed Central  Google Scholar 

  95. Holmes, N. P. & Spence, C. Multisensory integration: space, time and superadditivity. Curr. Biol. 15, R762–R764 (2005).

    CAS  PubMed  PubMed Central  Google Scholar 

  96. Parker, A. J. & Newsome, W. T. Sense and the single neuron: probing the physiology of perception. Annu. Rev. Neurosci. 21, 227–277 (1998).

    CAS  PubMed  Google Scholar 

  97. Bradley, A., Skottun, B. C., Ohzawa, I., Sclar, G. & Freeman, R. D. Visual orientation and spatial frequency discrimination: a comparison of single neurons and behavior. J. Neurophysiol. 57, 755–772 (1987).

    CAS  PubMed  Google Scholar 

  98. Green, D. M. & Swets, J. A. Signal Detection Theory and Psychophysics (Wiley, 1966).

    Google Scholar 

  99. Britten, K. H., Shadlen, M. N., Newsome, W. T. & Movshon, J. A. The analysis of visual motion: a comparison of neuronal and psychophysical performance. J. Neurosci. 12, 4745–4765 (1992).

    CAS  PubMed  Google Scholar 

  100. Fetsch, C. R., Pouget, A., DeAngelis, G. C. & Angelaki, D. E. Neural correlates of reliability-based cue weighting during multisensory integration. Nature Neurosci. 15, 146–154 (2011). In this study, the authors recorded from area MSTd while monkeys performed a cue-conflict version of the visual–vestibular heading task. Behaviourally, the animals were able to re-weight the cues on a trial-by-trial basis as reliability varied, and neuronal activity accounted for the behavioural results. The authors also derived a mathematically optimal combination rule for this task and used it to help explain deviations from optimality at the level of behaviour and neural responses.

    PubMed  Google Scholar 

  101. Paradiso, M. A. A theory for the use of visual orientation information which exploits the columnar structure of striate cortex. Biol. Cybern. 58, 35–49 (1988).

    CAS  PubMed  Google Scholar 

  102. Seung, H. S. & Sompolinsky, H. Simple models for reading neuronal population codes. Proc. Natl Acad. Sci. USA 90, 10749–10753 (1993).

    CAS  PubMed  Google Scholar 

  103. Carandini, M. & Heeger, D. J. Normalization as a canonical neural computation. Nature Rev. Neurosci. 13, 51–62 (2011). This review summarizes the large and diverse literature on the divisive normalization, including candidate biophysical mechanisms as well as its benefits for the efficiency of neural coding.

    Google Scholar 

  104. Louie, K., Grattan, L. E. & Glimcher, P. W. Reward value-based gain control: divisive normalization in parietal cortex. J. Neurosci. 31, 10627–10639 (2011).

    CAS  PubMed  PubMed Central  Google Scholar 

  105. Ohshiro, T., Angelaki, D. E. & DeAngelis, G. C. A normalization model of multisensory integration. Nature Neurosci. 14, 775–782 (2011). This paper presents a computational model that relies on a widespread neural computation called divisive normalization. Normalization of responses at the level of multisensory integration helps to explain several key empirical findings: the reliability-dependent combination rule in area MSTd as well as the ubiquitous empirical principles that were initially described in classic studies of the SC.

    CAS  PubMed  Google Scholar 

  106. Meredith, M. A. & Stein, B. E. Spatial factors determine the activity of multisensory neurons in cat superior colliculus. Brain Res. 365, 350–354 (1986).

    CAS  PubMed  Google Scholar 

  107. Carandini, M., Heeger, D. & Movshon, J. Linearity and normalization in simple cells of the macaque primary visual cortex. J. Neurosci. 17, 8621–8644 (1997).

    CAS  PubMed  Google Scholar 

  108. Carandini, M. From circuits to behavior: a bridge too far? Nature Neurosci. 15, 507–509 (2012).

    CAS  PubMed  Google Scholar 

  109. Beauchamp, M. S., Pasalar, S. & Ro, T. Neural substrates of reliability-weighted visual-tactile multisensory integration. Front. Syst. Neurosci. 4, 25 (2010).

    PubMed  PubMed Central  Google Scholar 

  110. Helbig, H. B. et al. The neural mechanisms of reliability weighted integration of shape information from vision and touch. Neuroimage 60, 1063–1072 (2012).

    PubMed  Google Scholar 

  111. Takahashi, K. et al. Multimodal coding of three-dimensional rotation and translation in area MSTd: comparison of visual and vestibular selectivity. J. Neurosci. 27, 9742–9756 (2007).

    CAS  PubMed  PubMed Central  Google Scholar 

  112. Gu, Y., DeAngelis, G. C. & Angelaki, D. E. A functional link between area MSTd and heading perception based on vestibular signals. Nature Neurosci. 10, 1038–1047 (2007).

    CAS  PubMed  Google Scholar 

  113. Chen, A., DeAngelis, G. C. & Angelaki, D. E. A comparison of vestibular spatiotemporal tuning in macaque parietoinsular vestibular cortex, ventral intraparietal area, and medial superior temporal area. J. Neurosci. 31, 3082–3094 (2011).

    CAS  PubMed  PubMed Central  Google Scholar 

  114. Shams, L., Kamitani, Y. & Shimojo, S. Illusions. What you see is what you hear. Nature 408, 788 (2000).

    CAS  PubMed  Google Scholar 

  115. Shimojo, S. & Shams, L. Sensory modalities are not separate modalities: plasticity and interactions. Curr. Opin. Neurobiol. 11, 505–509 (2001).

    CAS  PubMed  Google Scholar 

  116. Sekuler, R., Sekuler, A. B. & Lau, R. Sound alters visual motion perception. Nature 385, 308 (1997).

    CAS  PubMed  Google Scholar 

  117. Avillac, M., Denève, S., Olivier, E., Pouget, A. & Duhamel, J.-R. Reference frames for representing visual and tactile locations in parietal cortex. Nature Neurosci. 8, 941–949 (2005).

    CAS  PubMed  Google Scholar 

  118. Wallace, M. T. et al. Unifying multisensory signals across time and space. Exp. Brain Res. 158, 252–258 (2004).

    CAS  PubMed  Google Scholar 

  119. Fetsch, C. R., DeAngelis, G. C. & Angelaki, D. E. Visual–vestibular cue integration for heading perception: applications of optimal cue integration theory. Eur. J. Neurosci. 31, 1721–1729 (2010).

    PubMed  PubMed Central  Google Scholar 

Download references

Acknowledgements

We acknowledge the prodigious contributions of Y. Gu (Institute of Neuroscience, Shanghai, China) to the research discussed in this Review. We also thank M. Morgan, T. Ohshiro and many other current and former laboratory members, collaborators and technicians who made the work possible. Research was supported by US National Institutes of Health grants R01-EY016178 (to G.C.D.) and R01-EY019087 (to D.E.A.).

Author information

Authors and Affiliations

Authors

Corresponding author

Correspondence to Dora E. Angelaki.

Ethics declarations

Competing interests

The authors declare no competing financial interests.

Related links

PowerPoint slides

Glossary

Normative

A general term referring to an idea, statement or model that describes how something ought to be: that is, relating to an ideal or standard of correctness.

Cue

Any signal or piece of information bearing on the state of some property of the environment. Examples include binocular disparity in the visual system, interaural time or level differences in audition and proprioceptive signals (for example, from muscle spindles) conveying the position of the arm in space.

Ideal observer

A theoretical construct used to quantify optimal performance in a given task, where optimality is defined according to a pre-defined mathematical function (for example, minimizing a cost function or maximizing a utility function). The term 'ideal' does not imply perfect (error-free) performance, which is generally impossible given the uncertainty associated with all sensory data.

Reliability

Although the term reliability can mean different things in different fields, here we use it as a synonym for the precision of a measurement, which is defined mathematically as its inverse variance.

Bayesian probability theory

The branch of statistics and probability theory in which probability is interpreted as the 'degree of belief' that an event will occur (or that a hypothesis is true) rather than the relative frequency with which it has occurred. It is chiefly associated with the process of updating a prior belief about a hypothesis in light of new data, but the essence of Bayesian theory is this way of thinking about probability itself, which permits the estimation of a statistical parameter (or property of the environment) from experimental observations (or sensory information).

Accuracy

Accuracy refers to how close the measurement is to the true value: that is, how unbiased it is.

Poisson-like variability

Neurons respond differently to repeated presentations of the same stimulus, and this variability often resembles a family of probability distributions that includes the Poisson distribution (hence the term 'Poisson-like'). A prominent feature of this family is that the variance of neuronal responses (that is, the variance of the number of action potentials across repeated stimulus presentations) is proportional to the mean response.

Heading

An organism's instantaneous direction of translational movement.

Motion coherence

A property of random-dot motion stimuli — used in visual psychophysics and neurophysiology — that is often varied to control stimulus strength and therefore task difficulty. Motion coherence is the percentage of dots moving in the prescribed direction (the 'signal') while the remaining dots are re-plotted randomly on every video frame (the 'noise').

Divisive normalization

A neural computation in which the would-be response of an individual neuron (that is, its excitatory drive) is divided by the summed activity of a pool of neurons before generating an output.

Rights and permissions

Reprints and permissions

About this article

Cite this article

Fetsch, C., DeAngelis, G. & Angelaki, D. Bridging the gap between theories of sensory cue integration and the physiology of multisensory neurons. Nat Rev Neurosci 14, 429–442 (2013). https://doi.org/10.1038/nrn3503

Download citation

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1038/nrn3503

This article is cited by

Search

Quick links

Nature Briefing

Sign up for the Nature Briefing newsletter — what matters in science, free to your inbox daily.

Get the most important science stories of the day, free in your inbox. Sign up for Nature Briefing