Abstract
Environmental stimuli and objects, including rewards, are often processed sequentially in the brain. Recent work suggests that the phasic dopamine reward prediction-error response follows a similar sequential pattern. An initial brief, unselective and highly sensitive increase in activity unspecifically detects a wide range of environmental stimuli, then quickly evolves into the main response component, which reflects subjective reward value and utility. This temporal evolution allows the dopamine reward prediction-error signal to optimally combine speed and accuracy.
This is a preview of subscription content, access via your institution
Access options
Subscribe to this journal
Receive 12 print issues and online access
$189.00 per year
only $15.75 per issue
Buy this article
- Purchase on Springer Link
- Instant access to full article PDF
Prices may be subject to local taxes which are calculated during checkout
Similar content being viewed by others
References
Schultz, W. Multiple dopamine functions at different time courses. Ann. Rev. Neurosci. 30, 259–288 (2007).
Ljungberg, T., Apicella, P. & Schultz, W. Responses of monkey dopamine neurons during learning of behavioral reactions. J. Neurophysiol. 67, 145–163 (1992).
Schultz, W., Dayan, P. & Montague, R. R. A neural substrate of prediction and reward. Science 275, 1593–1599 (1997).
Schultz, W. Predictive reward signal of dopamine neurons. J. Neurophysiol. 80, 1–27 (1998).
Waelti, P., Dickinson, A. & Schultz, W. Dopamine responses comply with basic assumptions of formal learning theory. Nature 412, 43–48 (2001).
Tobler, P. N., Dickinson, A. & Schultz, W. Coding of predicted reward omission by dopamine neurons in a conditioned inhibition paradigm. J. Neurosci. 23, 10402–10410 (2003).
Bayer, H. M. & Glimcher, P. W. Midbrain dopamine neurons encode a quantitative reward prediction error signal. Neuron 47, 129–141 (2005).
Pan, W.-X., Schmidt, R., Wickens, J. R. & Hyland, B. I. Dopamine cells respond to predicted events during classical conditioning: evidence for eligibility traces in the reward-learning network. J. Neurosci. 25, 6235–6242 (2005).
Lak, A., Stauffer, W. R. & Schultz, W. Dopamine prediction error responses integrate subjective value from different reward dimensions. Proc. Natl Acad. Sci. USA 111, 2343–2348 (2014).
Fiorillo, C. D., Tobler, P. N. & Schultz, W. Discrete coding of reward probability and uncertainty by dopamine neurons. Science 299, 1898–1902 (2003).
Schultz, W., Ruffieux, A. & Aebischer, P. The activity of pars compacta neurons of the monkey substantia nigra in relation to motor activation. Exp. Brain Res. 51, 377–387 (1983).
Schultz, W. Responses of midbrain dopamine neurons to behavioral trigger stimuli in the monkey. J. Neurophysiol. 56, 1439–1462 (1986).
DeLong, M. R., Crutcher, M. D. & Georgopoulos, A. P. Relations between movement and single cell discharge in the substantia nigra of the behaving monkey. J. Neurosci. 3, 1599–1606 (1983).
Romo, R. & Schultz, W. Dopamine neurons of the monkey midbrain: contingencies of responses to active touch during self-initiated arm movements. J. Neurophysiol. 63, 592–606 (1990).
Chiodo, L. A., Antelman, S. M., Caggiula, A. R. & Lineberry, C. G. Sensory stimuli alter the discharge rate of dopamine (DA) neurons: evidence for two functional types of DA cells in the substantia nigra. Brain Res. 189, 544–549 (1980).
Steinfels, G. F., Heym, J., Strecker, R. E. & Jacobs, B. L. Behavioral correlates of dopaminergic unit activity in freely moving cats. Brain Res. 258, 217–228 (1983).
Schultz, W. & Romo, R. Responses of nigrostriatal dopamine neurons to high intensity somatosensory stimulation in the anesthetized monkey. J. Neurophysiol. 57, 201–217 (1987).
Schultz, W. & Romo, R. Dopamine neurons of the monkey midbrain: contingencies of responses to stimuli eliciting immediate behavioral reactions. J. Neurophysiol. 63, 607–624 (1990).
Mirenowicz, J. & Schultz, W. Preferential activation of midbrain dopamine neurons by appetitive rather than aversive stimuli. Nature 379, 449–451 (1996).
Horvitz, J. C., Stewart, T. & Jacobs, B. L. Burst activity of ventral tegmental dopamine neurons is elicited by sensory stimuli in the awake cat. Brain Res. 759, 251–258 (1997).
Guarraci, F. A. & Kapp, B. S. An electrophysiological characterization of ventral tegmental area dopaminergic neurons during differential pavlovian fear conditioning in the awake rabbit. Behav. Brain Res. 99, 169–179 (1999).
Joshua, M., Adler, A., Mitelman, R., Vaadia, E. & Bergman, H. Midbrain dopaminergic neurons and striatal cholinergic interneurons encode the difference between reward and aversive events at different epochs of probabilistic classical conditioning trials. J. Neurosci. 28, 11673–11684 (2008).
Brischoux, F., Chakraborty, S., Brierley, D. I. & Ungless, M. A. Phasic excitation of dopamine neurons in ventral VTA by noxious stimuli. Proc. Natl Acad. Sci. USA 106, 4894–4899 (2009).
Matsumoto, M. & Hikosaka, O. Two types of dopamine neuron distinctively convey positive and negative motivational signals. Nature 459, 837–841 (2009).
Fiorillo, C. D., Song, M. R. & Yun, S. R. Multiphasic temporal dynamics in responses of midbrain dopamine neurons to appetitive and aversive stimuli. J. Neurosci. 33, 4710–4725 (2013).
Fiorillo, C. D., Yun, S. R. & Song, M. R. Diversity and homogeneity in responses of midbrain dopamine neurons. J. Neurosci. 33, 4693–4709 (2013).
Fiorillo, C. D. Two dimensions of value: dopamine neurons represent reward but not aversiveness. Science 341, 546–549 (2013).
Thorpe, S. J., Rolls, E. T. & Maddison, S. The orbitofrontal cortex: neuronal activity in the behaving monkey. Exp. Brain Res. 49, 93–115 (1983).
Ravel, S., Legallet, E. & Apicella, P. Responses of tonically active neurons in the monkey striatum discriminate between motivationally opposing stimuli. J. Neurosci. 23, 8489–8497 (2003).
Roitman, M. F., Wheeler, R. A. & Carelli, R. M. Nucleus accumbens neurons are innately tuned for rewarding and aversive taste stimuli, encode their predictors, and are linked to motor output. Neuron 45, 587–597 (2005).
Paton, J. J., Belova, M. A., Morrison, S. E. & Salzman, C. D. The primate amygdala represents the positive and negative value of visual stimuli during learning. Nature 439, 865–870 (2006).
Amemori, K.-I. & Graybiel, A. M. Localized microstimulation of primate pregenual cingulate cortex induces negative decision-making. Nat. Neurosci. 15, 776–785 (2012).
Nomoto, K., Schultz, W., Watanabe, T. & Sakagami, M. Temporally extended dopamine responses to perceptually demanding reward-predictive stimuli. J. Neurosci. 30, 10692–10702 (2010).
Kobayashi, S. & Schultz, W. Reward contexts extend dopamine signals to unrewarded stimuli. Curr. Biol. 24, 56–62 (2014).
Stauffer, W. R., Lak, A. & Schultz, W. Dopamine reward prediction error responses reflect marginal utility. Curr. Biol. 24, 2491–2500 (2014).
Bushnell, M. C., Goldberg, M. E. & Robinson, D. L. Behavioral enhancement of visual responses in monkey cerebral cortex. I. Modulation in posterior parietal cortex related to selective visual attention. J. Neurophysiol. 46, 755–772 (1981).
Treue, S. & Maunsell, J. H. R. Attentional modulation of visual motion processing in cortical areas MT and MST. Nature 382, 539–541 (1996).
Womelsdorf, T., Anton-Erxleben, K., Pieper, F. & Treue, S. Dynamic shifts of visual receptive fields in cortical area MT by spatial attention. Nat. Neurosci. 9, 1156–1160 (2006).
Nardo, D., Santangelo, V. & Macaluso, E. Stimulus-driven orienting of visuo-spatial attention in complex dynamic environments. Neuron 69, 1015–1028 (2011).
Annic, A., Bocquillon, P., Bourriez, J.-L., Derambure, P. & Dujardin, K. Effects of stimulus-driven and goal-directed attention on prepulse inhibition of the cortical responses to an auditory pulse. Clin. Neurophysiol. 125, 1576–1588 (2014).
Thompson, K. G., Hanes, D. P., Bichot, N. P. & Schall, J. D. Perceptual and motor processing stages identified in the activity of macaque frontal eye field neurons during visual search. J. Neurophysiol. 76, 4040–4055 (1996).
Ipata, A. E., Gee, A. L., Bisley, J. W. & Goldberg, M. E. Neurons in the lateral intraparietal area create a priority map by the combination of disparate signals. Exp. Brain Res. 192, 479–488 (2009).
Ipata, A. E., Gee, A. L. & Goldberg, M. E. Feature attention evokes task-specific pattern selectivity in V4 neurons. Proc. Natl Acad. Sci. USA 109, 16778–16785 (2012).
Pooresmaeili, A., Poort, J. & Roelfsema, P. R. Simultaneous selection by object-based attention in visual and frontal cortex. Proc. Natl Acad. Sci. USA 111, 6467–6472 (2014).
Shadlen, M. N. & Newsome, W. T. Neural basis of a perceptual decision in the parietal cortex (Area LIP) of the rhesus monkey. J. Neurophysiol. 86, 1916–1936 (2001).
Roitman, J. D. & Shadlen, M. N. Response of neurons in the lateral intraparietal area during a combined visual discrimination reaction time task. J. Neurosci. 22, 9475–9489 (2002).
Ringach, D. L., Hawken, M. J. & Shapley, R. Dynamics of orientation tuning in macaque primary visual cortex. Nature 387, 281–284 (1997).
Sugase, Y., Yamane, S., Ueno, S. & Kawano, K. Global and fine information coded by single neurons in the temporal visual cortex. Nature 400, 869–873 (1999).
Bredfeldt, C. E. & Ringach, D. L. Dynamics of spatial frequency tuning in macaque V1. J. Neurosci. 22, 1976–1984 (2002).
Hedgé, J. & Van Essen, D. C. Temporal dynamics of shape analysis in macaque visual area V2. J. Neurophysiol. 92, 3030–3042 (2004).
Roelfsema, P. R., Tolboom, M. & Khayat, P. S. Different processing phases for features, figures, and selective attention in the primary visual cortex. Neuron 56, 785–792 (2007).
Hedgé, J. Time course of visual perception: Coarse-to-fine processing and beyond. Prog. Neurobiol. 84, 405–439 (2008).
Lak, A., Arabzadeh, E., Harris, J. A. & Diamond, M. E. Correlated physiological and perceptual effects of noise in a tactile stimulus. Proc. Natl Acad. Sci. USA 107, 7981–7986 (2010).
Hung, C. P., Kreiman, G., Poggio, T. & DiCarlo, J. J. Fast readout of object identity from macaque inferior temporal cortex. Science 310, 863–866 (2005).
Ambroggi, F., Ishikawa, A., Fields, H. L. & Nicola, S. M. Basolateral amygdala neurons facilitate reward-seeking behavior by exciting nucleus accumbens neurons. Neuron 59, 648–661 (2008).
Peck, C. J., Lau, B. & Salzman, C. D. The primate amygdala combines information about space and value. Nat. Neurosci. 16, 340–348 (2013).
Mogami, T. & Tanaka, K. Reward association affects neuronal responses to visual stimuli in macaque TE and perirhinal cortices. J. Neurosci. 26, 6761–6770 (2006).
Stanisor, L., van der Togt, C., Pennartz, C. M. A. & Roelfsema, P. R. A unified selection signal for attention and reward in primary visual cortex. Proc. Natl Acad. Sci. USA 110, 9136–9141 (2013).
Morris, G., Arkadir, D., Nevet, A., Vaadia, E. & Bergman, H. Coincident but distinct messages of midbrain dopamine and striatal tonically active neurons. Neuron 43, 133–143 (2004).
Day, J. J., Roitman, M. F. & Wightman, R. M. & Carelli, R. M. Associative learning mediates dynamic shifts in dopamine signaling in the nucleus accumbens. Nat. Neurosci. 10, 1020–1028 (2007).
Kobayashi, S. & Schultz, W. Influence of reward delays on responses of dopamine neurons. J. Neurosci. 28, 7837–7846 (2008).
Fiorillo, C. D., Newsome, W. T. & Schultz, W. The temporal precision of reward prediction in dopamine neurons. Nat. Neurosci. 11, 966–973 (2008).
Budygin, E. A. et al. Aversive stimulus differentially triggers subsecond dopamine release in reward regions. Neuroscience 201, 331–337 (2012).
Redgrave, P., Prescott, T. J. & Gurney, K. Is the short-latency dopamine response too short to signal reward? Trends Neurosci. 22, 146–151 (1999).
Redgrave, P. & Gurney, K. The short-latency dopamine signal: a role in discovering novel actions? Nat. Rev. Neurosci. 7, 967–975 (2006).
Schultz, W., Apicella, P. & Ljungberg, T. Responses of monkey dopamine neurons to reward and conditioned stimuli during successive steps of learning a delayed response task. J. Neurosci. 13, 900–913 (1993).
Mitchell, D. S. & Gormezano, I. Effects of water deprivation on classical appetitive conditioning of the rabbit's jaw movement response. Learn. Motivat. 1, 199–206 (1970).
Mackintosh, N. J. The Psychology of Animal Learning (Academic Press, 1974).
Ljungberg, T., Apicella, P. & Schultz, W. Responses of monkey midbrain dopamine neurons during delayed alternation performance. Brain Res. 586, 337–341 (1991).
Pearce, J. M. & Hall, G. A model for Pavlovian conditioning: variations in the effectiveness of conditioned but not of unconditioned stimuli. Psychol. Rev. 87, 532–552 (1980).
Chelazzi, L. et al. Altering spatial priority maps via reward-based learning. J. Neurosci. 34, 8594–8604 (2014).
Darwin, C. On the Origin of Species by Natural Selection, or the Preservation of Favoured Races in the Struggle for Life (John Murray, 1859).
Montague, P. R., Dayan, P. & Sejnowski, T. J. A framework for mesencephalic dopamine systems based on predictive Hebbian learning. J. Neurosci. 16, 1936–1947 (1996).
Tobler, P. N., Fiorillo, C. D. & Schultz, W. Adaptive coding of reward value by dopamine neurons. Science 307, 1642–1645 (2005).
Enomoto, K. et al. Dopamine neurons learn to encode the long-term value of multiple future rewards. Proc. Natl Acad. Sci. USA 108, 15462–15467 (2011).
Richards, J. B., Mitchell, S. H., de Wit, H. & Seiden, L. S. Determination of discount functions in rats with an adjusting-amount procedure. J. Exp. Anal. Behav. 67, 353–366 (1997).
Day, J. J., Jones, J. L., Wightman, R. M. & Carelli, R. M. Phasic nucleus accumbens dopamine release encodes effort- and delay-related costs. Biol. Psychiat. 68, 306–309 (2010).
Pascal, B. Pensées (1658–1662) (Hackett, 2004).
Bernoulli, D. Specimen theoriae novae de mensura sortis. Comentarii Academiae Scientiarum Imperialis Petropolitanae 5, 175–192 (1738); English translation available in Exposition of a new theory on the measurement of risk. Econometrica 22, 23–36 (1954).
Bentham, J. An Introduction to the Principle of Morals and Legislations (Blackwell, 1948).
von Neumann, J. & Morgenstern, O. The Theory of Games and Economic Behavior (Princeton Univ. Press, 1944).
Kahneman, D. & Tversky, A. Prospect theory: an analysis of decision under risk. Econometrica 47, 263–291 (1979).
Luce, R. D. Individual Choice Behavior: A Theoretical Analysis (Wiley, 1959).
Kagel, J. H., Battalio, R. C. & Green, L. Economic Choice Theory: An Experimental Analysis of Animal Behavior (Cambridge Univ. Press, 1995).
Mas-Colell, A., Whinston, M. & Green, J. Microeconomic Theory (Oxford Univ. Press, 1995).
Savage, L. J. The Foundations of Statistics (Wiley, 1954).
Debreu, G. Cardinal utility for even-chance mixtures of pairs of sure prospects. Rev. Econ. Stud. 26, 174–177 (1959).
Rothschild, M. & Stiglitz, J. E. Increasing risk: I. A definition. J. Econ. Theory 2, 225–243 (1970).
Caraco, T., Martindale, S. & Whitham, T. S. An empirical demonstration of risk-sensitive foraging preferences. Anim. Behav. 28, 820–830 (1980).
Machina, M. J. Choice under uncertainty: problems solved and unsolved. J. Econ. Perspect. 1, 121–154 (1987).
McCoy, A. N. & Platt, M. L. Risk-sensitive neurons in macaque posterior cingulate cortex. Nat. Neurosci. 8, 1220–1227 (2005).
O'Neill, M. & Schultz, W. Coding of reward risk distinct from reward value by orbitofrontal neurons. Neuron 68, 789–800 (2010).
Prelec, D. & Loewenstein, G. Decision making over time and under uncertainty: a common approach. Management Sci. 37, 770–786 (1991).
Weber, B. J. & Chapman, G. B. Playing for peanuts: why is risk seeking more common for low-stakes gambles? Organ. Behav. Hum. Decis. Process. 97, 31–46 (2005).
Fehr-Duda, H., Bruhin, A., Epper, T. & Schubert, R. Rationality on the rise: why relative risk aversion increases with stake size. J. Risk Uncertain. 40, 147–180 (2010).
Kreps, D. M. A Course in Microeconomic Theory (Pearson Education, 1990).
Friedman, M. & Savage, L. J. The utility analysis of choices involving risk. J. Polit. Econ. 56, 279–304 (1948).
Markowitz, H. The utility of wealth. J. Polit. Econ. 6, 151–158 (1952).
Jones, S. R., Garris, P. A. & Wightman, R. M. Different effects of cocaine and nomifensine on dopamine uptake in the caudate-putamen and nucleus accumbens. J. Pharmacol. Exp. Ther. 274, 396–403 (1995).
Calabresi, P. et al. Dopamine and cAMP-regulated phosphoprotein 32 kDa controls both striatal long-term depression and long-term potentiation, opposing forms of synaptic plasticity. J. Neurosci. 20, 8443–8451 (2000).
Gurden, H., Takita, M. & Jay, T. M. Essential role of D1 but not D2 receptors in the NMDA receptor-dependent long-term potentiation at hippocampal-prefrontal cortex synapses in vivo. J. Neurosci. 20, RC106 (2000).
Reynolds, J. N. J., Hyland, B. I. & Wickens, J. R. A cellular mechanism of reward-related learning. Nature 413, 67–70 (2001).
Pawlak, V. & Kerr, J. N. D. Dopamine receptor activation is required for corticostriatal spike-timing-dependent plasticity. J. Neurosci. 28, 2435–2446 (2008).
Shen, W., Flajolet, M., Greengard, P. & Surmeier, D. J. Dichotomous dopaminergic control of striatal synaptic plasticity. Science 321, 848–851 (2008).
Zhang, J.-C., Lau, P.-M. & Bi, G.-Q. Gain in sensitivity and loss in temporal contrast of STDP by dopaminergic modulation at hippocampal synapses. Proc. Natl Acad. Sci. USA 106, 1328–1333 (2009).
Yagishita, S. et al. A critical time window for dopamine actions on the structural plasticity of dendritic spines. Science 345, 1616–1620 (2014).
Tsai, H.-C. et al. Phasic firing in dopaminergic neurons is sufficient for behavioral conditioning. Science 324, 1080–1084 (2009).
Witten, I. B. et al. Recombinase-driver rat lines: tools, techniques, and optogenetic application to dopamine-mediated reinforcement. Neuron 72, 721–733 (2011).
Adamantidis, A. R. et al. Optogenetic interrogation of dopaminergic modulation of the multiple phases of reward-seeking behavior. J. Neurosci. 31, 10829–10835 (2011).
Kim, K. M. et al. Optogenetic mimicry of the transient activation of dopamine neurons by natural reward is sufficient for operant reinforcement. PLoS ONE 7, e33612 (2012).
Steinberg, E. E. et al. A causal link between prediction errors, dopamine neurons and learning. Nat. Neurosci. 16, 966–973 (2013).
Ilango, A. et al. Similar roles of substantia nigra and ventral tegmental dopamine neurons in reward and aversion. J. Neurosci. 34, 817–822 (2014).
Kravitz, A. V., Tye, L. D. & Kreitzer, A. C. Distinct roles for direct and indirect pathway striatal neurons in reinforcement. Nat. Neurosci. 15, 816–818 (2012).
Schultz, W. Neuronal reward and decision signals: from theories to data. Physiol. Rev. 95, 853–951 (2015).
Morris, G., Nevet, A., Arkadir, D., Vaadia, E. & Bergman, H. Midbrain dopamine neurons encode decisions for future action. Nat. Neurosci. 9, 1057–1063 (2006).
Padoa-Schioppa, C. & Assad, J. A. Neurons in the orbitofrontal cortex encode economic value. Nature 441, 223–226 (2006).
Cai, X., Kim, S. & Lee, D. Heterogeneous coding of temporally discounted values in the dorsal and ventral striatum during intertemporal choice. Neuron 69, 170–182 (2011).
Kennerley, S. W., Behrens, T. E. J. & Wallis, J. D. Double dissociation of value computations in orbitofrontal and anterior cingulate neurons. Nat. Neurosci. 14, 1581–1589 (2011).
So, N.-Y. & Stuphorn, V. Supplementary eye field encodes reward prediction error. J. Neurosci. 32, 2950–2963 (2012).
Samejima, K., Ueda, Y., Doya, K. & Kimura, M. Representation of action-specific reward values in the striatum. Science 310, 1337–1340 (2005).
Lau, B. & Glimcher, P. W. Value representations in the primate striatum during matching behavior. Neuron 58, 451–463 (2008).
Ito, M. & Doya, K. Validation of decision-making models and analysis of decision variables in the rat basal ganglia. J. Neurosci. 29, 9861–9874 (2009).
Kim, H., Sul, J. H., Huh, N., Lee, D. & Jung, M. W. Role of striatum in updating values of chosen actions. J. Neurosci. 29, 14701–14712 (2009).
Seo, M., Lee, E. & Averbeck, B. B. Action selection and action value in frontal-striatal circuits. Neuron 74, 947–960 (2012).
Brown, J. R. & Arbuthnott, G. W. The electrophysiology of dopamine (D2) receptors: a study of the actions of dopamine on corticostriatal transmission. Neuroscience 10, 349–355 (1983).
Toan, D. L. & Schultz, W. Responses of rat pallidum cells to cortex stimulation and effects of altered dopaminergic activity. Neuroscience 15, 683–694 (1985).
Mink, J. W. The basal ganglia: focused selection and inhibition of competing motor programs. Prog. Neurobiol. 50, 381–425 (1996).
Zweifel, L. S. et al. Disruption of NMDAR-dependent burst firing by dopamine neurons provides selective assessment of phasic dopamine-dependent behavior. Proc. Natl Acad. Sci. USA 106, 7281–7288 (2009).
Aosaki, T., Graybiel, A. M. & Kimura, M. Effect of the nigrostriatal dopamine system on acquired neural responses in the striatum of behaving monkeys. Science 265, 412–415 (1994).
Hernández-López, S., Bargas, J., Surmeier, D. J., Reyes, A. & Galarraga, E. D1 receptor activation enhances evoked discharge in neostriatal medium spiny neurons by modulating an L-type Ca2+ conductance. J. Neurosci. 17, 3334–3342 (1997).
Hernández-López, S. et al. D2 dopamine receptors in striatal medium spiny neurons reduce L-type Ca2+ currents and excitability via a novel PLCβ1-IP3-calcineurin-signaling cascade. J. Neurosci. 20, 8987–8995 (2000).
Tai, L.-H. & Lee, A. M., Benavidez, N., Bonci, A. & Wilbrecht, L. Transient stimulation of distinct subpopulations of striatal neurons mimics changes in action value. Nat. Neurosci. 15, 1281–1289 (2012).
Badrinarayan, A. et al. Aversive stimuli differentially modulate real-time dopamine transmission dynamics within the nucleus accumbens core and shell. J. Neurosci. 32, 15779–15790 (2012).
Lerner, T. et al. Intact-brain analyses reveal distinct information carried by SNc dopamine subcircuits. Cell 162, 635–647 (2015).
Solomon, R. L. & Corbit, J. D. An opponent-process theory of motivation. Psychol. Rev. 81, 119–145 (1974).
Oleson, E. B., Gentry, R. N., Chioma, V. C. & Cheer, J. F. Subsecond dopamine release in the nucleus accumbens predicts conditioned punishment and its successful avoidance. J. Neurosci. 32, 14804–14808 (2012).
Lammel, S. et al. Input-specific control of reward and aversion in the ventral tegmental area. Nature 491, 212–217 (2012).
Stopper, C. M., Tse, M. T. L., Montes, D. R., Wiedman, C. R. & Floresco, S. B. Overriding phasic dopamine signals redirects action selection during risk/reward decision making. Neuron 84, 177–189 (2014).
Christoph, G. R., Leonzio, R. J. & Wilcox, K. S. Stimulation of the lateral habenula inhibits dopamine-containing neurons in the substantia nigra and ventral tegmental area of the rat. J. Neurosci. 6, 613–619 (1986).
Ji, H. & Shepard, P. D. Lateral habenula stimulation inhibits rat midbrain dopamine neurons through a GABAA receptor-mediated mechanism. J. Neurosci. 27, 6923–6930 (2007).
Matsumoto, M. & Hikosaka, O. Lateral habenula as a source of negative reward signals in dopamine neurons. Nature 447, 1111–1115 (2007).
Kapur, S. Psychosis as a state of aberrant salience: a framework linking biology, phenomenology, and pharmacology in schizophrenia. Am. J. Psychiatry 160, 13–23 (2003).
Robinson, T. E. & Berridge, K. C. The neural basis for drug craving: an incentive-sensitization theory of addiction. Brain Res. Rev. 18, 247–291 (1993).
Saunders, B. T. & Robinson, T. E. The role of dopamine in the accumbens core in the expression of Pavlovian-conditioned responses. Eur. J. Neurosci. 36, 2521–2532 (2012).
Rao, R. P. N. & Ballard, D. H. Predictive coding in the visual cortex: a functional interpretation of some extra-classical receptive-field effects. Nat. Neurosci. 2, 79–87 (1999).
Roeper, J. Dissecting the diversity of midbrain dopamine neurons. Trends Neurosci. 36, 336–342 (2013).
Fuster, J. M. Unit activity of prefrontal cortex during delayed-response performance: neuronal correlates of transient memory. J. Neurophysiol. 36, 61–78 (1973).
Apicella, P., Scarnati, E., Ljungberg, T. & Schultz, W. Neuronal activity in monkey striatum related to the expectation of predictable environmental events. J. Neurophysiol. 68, 945–960 (1992).
Hollerman, J. R., Tremblay, L. & Schultz, W. Influence of reward expectation on behavior-related neuronal activity in primate striatum. J. Neurophysiol. 80, 947–963 (1998).
Matsumoto, M. & Takada, M. Distinct representations of cognitive and motivational signals in midbrain dopamine neurons. Neuron 79, 1011–1024 (2013).
Satoh, T., Nakai, S., Sato, T. & Kimura, M. Correlated coding of motivation and outcome of decision by dopamine neurons. J. Neurosci. 23, 9913–9923 (2003).
Bromberg-Martin, E. S. & Hikosaka, O. Lateral habenula neurons signal errors in the prediction of reward information. Nature Neurosci. 14, 1209–1216 (2011).
de Lafuente, O. & Romo, R. Dopamine neurons code subjective sensory experience and uncertainty of perceptual decisions. Proc. Natl Acad. Sci. USA 49, 19767–19771 (2011).
Cohen, J. Y., Haesler, S., Vong, L., Lowell, B. B. & Uchida, N. Neuron-type-specific signals for reward and punishment in the ventral tegmental area. Nature 482, 85–88 (2012).
Stuber, G. D., Wightman, R. M. & Carelli, R. M. Extinction of cocaine self-administration reveals functionally and temporally distinct dopaminergic signals in the nucleus accumbens. Neuron 46, 661–669 (2005).
Howe, M. W., Tierney, P. L., Sandberg, S. G., Phillips, P. E. M. & Graybiel, A. M. Prolonged dopamine signalling in striatum signals proximity and value of distant rewards. Nature 500, 575–579 (2013).
Chesselet, M. F. Presynaptic regulation of neurotransmitter release in the brain: facts and hypothesis. Neuroscience 12, 347–375 (1984).
Sugam, J. A., Day, J. J., Wightman, R. M. & Carelli, R. M. Phasic nucleus accumbens dopamine encodes risk-based decision-making behavior. Biol. Psychiat. 71, 199–205 (2012).
Threlfell, S. et al. Striatal dopamine release is triggered by synchronized activity in cholinergic interneurons. Neuron 75, 58–64 (2012).
Chuhma, N., Mingote, S., Moore, H. & Rayport, S. Dopamine neurons control striatal cholinergic neurons via regionally heterogeneous dopamine and glutamate signaling. Neuron 81, 901–912 (2014).
Brimblecombe, K. R. & Cragg, S. J. Substance P weights striatal dopamine transmission differently within the striosome-matrix axis. J. Neurosci. 35, 9017–9023 (2015).
Robbins, T. W. & Arnsten, A. F. T. The neuropsychopharmacology of fronto-executive function: monoaminergic modulation. Ann. Rev. Neurosci. 32, 267–287 (2009).
Young, A. M. J., Joseph, M. H. & Gray, J. A. Increased dopamine release in vivo in nucleus accumbens and caudate nucleus of the rat during drinking: a microdialysis study. Neuroscience 48, 871–876 (1992).
Grace, A. A. Phasic versus tonic dopamine release and the modulation of dopamine system responsivity: a hypothesis for the etiology of schizophrenia. Neuroscience 41, 1–24 (1991).
Datla, K. P., Ahier, R. G., Young, A. M. J., Gray, J. A. & Joseph, M. H. Conditioned appetitive stimulus increases extracellular dopamine in the nucleus accumbens of the rat. Eur. J. Neurosci. 16, 1987–1993 (2002).
Cheng, J. J., de Bruin, J. P. C. & Feenstra, M. G. P. Dopamine efflux in nucleus accumbens shell and core in response to appetitive classical conditioning. Eur. J. Neurosci. 18, 1306–1314 (2003).
Young, A. M. J. Increased extracellular dopamine in nucleus accumbens in response to unconditioned and conditioned aversive stimuli: studies using 1 min microdialysis in rats. J. Neurosci. Meth. 138, 57–63 (2004).
Anzalone, A. et al. Dual control of dopamine synthesis and release by presynaptic and postsynaptic dopamine D2 receptors. J. Neurosci. 32, 9023–9034 (2012).
Grace, A. A., Floresco, S. B., Goto, Y. & Lodge, D. J. Regulation of firing of dopaminergic neurons and control of goal-directed behaviors. Trends Neurosci. 30, 220–227 (2007).
Acknowledgements
The author thanks A. Dickinson, P. Bossaerts, C. R. Plott and C. Harris for discussions about animal learning theory and experimental economics; his collaborators on the cited studies for their ingenuity, work and patience; and three anonymous referees for comments. The author is also indebted to K. Nomoto, M. Sakagami and C. D. Fiorillo, whose recent experiments encouraged the ideas proposed in this article. The author acknowledges grant support from the Wellcome Trust (Principal Research Fellowship, Programme and Project Grants: 058365, 093270 and 095495), the European Research Council (ERC Advanced Grant 293549) and the US National Institutes of Health Caltech Conte Center (P50MH094258).
Author information
Authors and Affiliations
Corresponding author
Ethics declarations
Competing interests
The author declares no competing financial interests.
Supplementary information
Supplementary information S1 (table)
Multi-component neuronal responses (PDF 539 kb)
Supplementary information S2 (box)
Aversive dopamine activations? (PDF 620 kb)
Glossary
- Behavioural pseudoconditioning
-
A situation in which the context (environment) is paired, through Pavlovian conditioning, to a reinforcer that is present in this environment. Any stimulus occurring in this context thus reflects the same association, without being explicitly paired with the reinforcer. Pseudoconditioning endows an unpaired stimulus with motivational value.
- Context conditioning
-
An association between a specific stimulus (for example, a reward or punisher) and a context (for example, an environment, including all stimuli except the specific explicit stimulus).
- Down states
-
Neuronal membrane states that are defined by hyperpolarized membrane potentials and very little firing.
- Economic utility
-
A mathematical, usually nonlinear function that derives the internal subjective reward value u from the objective value x. Utility is the fundamental variable that decision-makers maximize in rational economic choices between differently valued options.
- Hebbian learning
-
A cellular mechanism of learning, proposed by Donald Hebb, according to which the connection between a presynaptic and a postsynaptic cell is strengthened if the presynaptic cell is successful in activating a postsynaptic cell.
- Motivational salience
-
The ability of a stimulus to elicit attention due to its positive (reward) or negative (punishment) motivational value. Motivational salience is common to reward and punishment.
- Novelty salience
-
The ability of a stimulus to elicit attention due to its novelty.
- Physical salience
-
The ability of a stimulus to elicit attention by standing out, due to its physical intensity or conspicuousness.
- Rescorla–Wagner model
-
The prime error-driven reinforcement model for Pavlovian conditioning, in which the prediction error (reward or punishment outcome minus current prediction) is multiplied by a learning factor and added to the current prediction to result in an updated prediction.
- Surprise salience
-
The ability of a stimulus to elicit attention due to its unexpectedness.
- Temporal difference reinforcement models
-
A family of non-trial-based reinforcement learning models in which the difference between the expected and actual values of a particular state (prediction error) in a sequence of behaviours is used as a teaching signal to facilitate the acquisition of associative rules or policies to direct future behaviour. Temporal difference learning extends Rescorla–Wagner-type reinforcement models to real time and higher-order reinforcers.
- Up states
-
Neuronal membrane states that are defined by relatively depolarized membrane potentials and lots of action potential firing.
- Visual search task
-
An experimental paradigm in which subjects are asked to detect a 'target' item (for example, a red dot) among an array of distractor items (for example, many green dots).
- Voltammetry
-
An electrochemical measurement of oxidation-reduction currents across a range of imposed voltages, used in neuroscience for assessing concentrations of specific molecules, such as dopamine.
Rights and permissions
About this article
Cite this article
Schultz, W. Dopamine reward prediction-error signalling: a two-component response. Nat Rev Neurosci 17, 183–195 (2016). https://doi.org/10.1038/nrn.2015.26
Published:
Issue Date:
DOI: https://doi.org/10.1038/nrn.2015.26
This article is cited by
-
NSF DARE—transforming modeling in neurorehabilitation: a patient-in-the-loop framework
Journal of NeuroEngineering and Rehabilitation (2024)
-
Dopamine projections to the basolateral amygdala drive the encoding of identity-specific reward memories
Nature Neuroscience (2024)
-
State and rate-of-change encoding in parallel mesoaccumbal dopamine pathways
Nature Neuroscience (2024)
-
Exploring the steps of learning: computational modeling of initiatory-actions among individuals with attention-deficit/hyperactivity disorder
Translational Psychiatry (2024)
-
How does apathy impact exploration-exploitation decision-making in older patients with neurocognitive disorders?
npj Aging (2023)
