Mesolimbic Dopamine Dynamically Tracks, and Is Causally Linked to, Discrete Aspects of Value-Based Decision Making

doi:10.1016/j.biopsych.2014.10.024

Biological Psychiatry

Volume 77, Issue 10, 15 May 2015, Pages 903-911

https://doi.org/10.1016/j.biopsych.2014.10.024 Get rights and content

Abstract

Background

To make appropriate choices, organisms must weigh the costs and benefits of potential valuable outcomes, a process known to involve the nucleus accumbens (NAc) and its dopaminergic input. However, it is currently unknown if dopamine dynamically tracks alterations in expected reward value online as behavioral preferences change and if so, if it is causally linked to specific components of value such as reward magnitude and/or delay to reinforcement.

Methods

Electrochemical methods were used to measure subsecond NAc dopamine release during a delay discounting task where magnitude was fixed but delay varied across blocks (n = 7 rats). Next, to assess whether this dopamine signaling was causally related to specific components of choice behavior, we employed selective optogenetic stimulation of dopamine terminals in the NAc using a modified delay discounting task in which both delay and magnitude varied independently (n = 23 rats).

Results

Cues predictive of available choices evoked dopamine release that scaled with the rat’s preferred choices and dynamically shifted as delay to reinforcement for the large reward increased. In the second experiment, dopamine signaling was causally related to features of decision making, as optogenetically enhanced dopamine release within the NAc during predictive cue presentation was sufficient to alter subsequent value-related choices. Importantly, this dopamine-mediated shift in choice was limited to delay-based, but not magnitude-based, decisions.

Conclusions

These findings indicate that NAc dopamine dynamically tracks delay discounting and establishes a causal role for this signaling in a subset of value-based associative strategies.

Section snippets

Behavior

Detailed methods are described in Supplement 1. Briefly, male Sprague Dawley rats (n = 7; Harlan, Indianapolis, Indiana) were trained on a delay discounting task comprised of three trial types. On forced-choice delay trials (Figure 1A, left), a cue light was illuminated for 5 seconds followed by extension of two levers. A single press on the associated lever positioned below that cue light resulted in a large reward (three sucrose pellets) delivered after a period of delay. During forced-choice

Dopamine Signaling Tracks Associative Cue Value Related to Subjective Choice Preferences

During the delay discounting task used with FSCV, rats discriminated the different reward options during both forced- and free-choice trials. On forced-choice trials, rats showed accurate responding (89.6% correct), which was modulated by delay costs (Figure S1A–D in Supplement 1). During free-choice trials, rats’ initial preference for the large-reward lever decreased as delays for that outcome increased across blocks (F_2,7 = 21.68, p < .001) (Figure 1B). In the no-delay block, rats strongly

Discussion

The present findings reveal a role for dopaminergic input to the NAc in mediating discrete aspects of value-based decision making. Using FSCV in a delay discounting task where reward magnitudes were constant but delays to reinforcement shifted, dopamine release during predictive cues scaled between differently valued options that reflected information about the rats’ preferred responses in forced-choice trials tracked the value of the preferred choices and was dynamically modulated by delay

Acknowledgments and Disclosures

This research was supported by DA034021 to RMC, DA030307 to JAS, and DA035322 to MPS.

We are grateful for outstanding technical support from Xuefei Wang.

The authors report no biomedical financial interests or potential conflicts of interest.

References (30)

J.J. Clark et al.
Pavlovian valuation systems in learning and decision making
Curr Opin Neurobiol
(2012)
J.J. Day et al.
Phasic nucleus accumbens dopamine release encodes effort- and delay-related costs
Biol Psychiatry
(2010)
J.L. Jones et al.
Basolateral amygdala modulates terminal dopamine release in the nucleus accumbens and conditioned responding
Biol Psychiatry
(2010)
J.A. Sugam et al.
Phasic nucleus accumbens dopamine encodes risk-based decision-making behavior
Biol Psychiatry
(2012)
C.D. Fiorillo
Transient activation of midbrain dopamine neurons by reward risk
Neuroscience
(2011)
C.M. Stopper et al.
Overriding phasic dopamine signals redirects action selection during risk/reward decision making
Neuron
(2014)
I.B. Witten et al.
Recombinase-driver rat lines: Tools, techniques, and optogenetic application to dopamine-mediated reinforcement
Neuron
(2011)
J.A. Sugam et al.
Nucleus accumbens neurons track behavioral preferences and reward outcomes during risky decision making
Biol Psychiatry
(2014)
H.L. Fields et al.
Ventral tegmental area neurons in learned appetitive behavior and positive reinforcement
Annu Rev Neurosci
(2007)
J.J. Day et al.
Associative learning mediates dynamic shifts in dopamine signaling in the nucleus accumbens
Nat Neurosci
(2007)

W.X. Pan et al.

Dopamine cells respond to predicted events during classical conditioning: Evidence for eligibility traces in the reward-learning network

J Neurosci

(2005)

P. Waelti et al.

Dopamine responses comply with basic assumptions of formal learning theory

Nature

(2001)

W. Schultz et al.

A neural substrate of prediction and reward

Science

(1997)

M.R. Roesch et al.

Dopamine neurons encode the better option in rats deciding between differently delayed or sized rewards

Nat Neurosci

(2007)

C.D. Fiorillo et al.

Discrete coding of reward probability and uncertainty by dopamine neurons

Science

(2003)

Cited by (82)

The effects of early life stress on impulsivity
2022, Neuroscience and Biobehavioral Reviews
Elevated impulsivity is a symptom shared by various psychiatric disorders such as substance use disorder, bipolar disorder, and attention-deficit/hyperactivity disorder. However, impulsivity is not a unitary construct and impulsive behaviors fall into two subcategories: impulsive action and impulsive choice. Impulsive choice refers to the tendency to prefer immediate, small rewards over delayed, large rewards, whereas impulsive action involves difficulty inhibiting rash, premature, or mistimed behaviors. These behaviors are mediated by the mesocorticolimbic dopamine (DA) system, which consists of projections from the ventral tegmental area to the nucleus accumbens and prefrontal cortex. Early life stress (ELS) alters both impulsive choice and impulsive action in rodents. ELS also changes DA receptor expression, transmission, and activity within the mesocorticolimbic system. This review integrates the dopamine, impulsivity, and ELS literature to provide evidence that ELS alters impulsivity via inducing changes in the mesocorticolimbic DA system. Understanding how ELS affects brain circuits associated with impulsivity can help advance treatments aimed towards reducing impulsivity symptoms in a variety of psychiatric disorders.
Dopamine release in the nucleus accumbens core signals perceived saliency
2021, Current Biology
Citation Excerpt :
Biological evidence for dopamine neurons encoding an RPE signal was first demonstrated by Schultz and colleagues,1 showing increases in dopamine neuron firing rates when an unexpected reward is encountered, a learning-dependent shift in firing to cues that predict reward delivery, and a decrease in firing rates when expected rewards are withheld. Similar outcomes have been observed across species in ventral tegmental area (VTA) cell bodies3,12–15 and in dopamine release in the nucleus accumbens (NAc).4,16–21 As a result, the RPE model of the role of dopamine in learning has been dominant in the field for the last 20 years.
A large body of work has aimed to define the precise information encoded by dopaminergic projections innervating the nucleus accumbens (NAc). Prevailing models are based on reward prediction error (RPE) theory, in which dopamine updates associations between rewards and predictive cues by encoding perceived errors between predictions and outcomes. However, RPE cannot describe multiple phenomena to which dopamine is inextricably linked, such as behavior driven by aversive and neutral stimuli. We combined a series of behavioral tasks with direct, subsecond dopamine monitoring in the NAc of mice, machine learning, computational modeling, and optogenetic manipulations to describe behavior and related dopamine release patterns across multiple contingencies reinforced by differentially valenced outcomes. We show that dopamine release only conforms to RPE predictions in a subset of learning scenarios but fits valence-independent perceived saliency encoding across conditions. Here, we provide an extended, comprehensive framework for accumbal dopamine release in behavioral control.
The rostromedial tegmental (RMTg) “brake” on dopamine and behavior: A decade of progress but also much unfinished work
2021, Neuropharmacology
Between 2005 and 2009, several research groups identified a strikingly dense inhibitory input to midbrain dopamine neurons arising from a previously uncharted region posterior to the ventral tegmental area (VTA). This region is now denoted as either the rostromedial tegmental nucleus (RMTg) or the “tail of the VTA” (tVTA), and is recognized to express distinct genetic markers, encode negative “prediction errors” (inverse to dopamine neurons), and play critical roles in behavioral inhibition and punishment learning. RMTg neurons are also influenced by many categories of abused drugs, and may drive some aversive responses to such drugs, particularly cocaine and alcohol. However, despite much progress, many important questions remain about RMTg molecular/genetic properties, diversity of projection targets, and applications to addiction, depression, and other neuropsychiatric disorders.
This article is part of the special Issue on ‘Neurocircuitry Modulating Drug and Alcohol Abuse'.
Trial-by-trial dynamics of reward prediction error-associated signals during extinction learning and renewal
2021, Progress in Neurobiology
Reward prediction errors (RPEs) have been suggested to drive associative learning processes, but their precise temporal dynamics at the single-neuron level remain elusive. Here, we studied the neural correlates of RPEs, focusing on their trial-by-trial dynamics during an operant extinction learning paradigm. Within a single behavioral session, pigeons went through acquisition, extinction and renewal - the context-dependent response recovery after extinction. We recorded single units from the avian prefrontal cortex analogue, the nidopallium caudolaterale (NCL) and found that the omission of reward during extinction led to a peak of population activity that moved backwards in time as trials progressed. The chronological order of these signal changes during the progress of learning was indicative of temporal shifts of RPE signals that started during reward omission and then moved backwards to the presentation of the conditioned stimulus. Switches from operant choices to avoidance behavior (and vice versa) coincided with changes in population activity during the animals’ decision-making. On the single unit level, we found more diverse patterns where some neurons’ activity correlated with RPE signals whereas others correlated with the absolute value during the outcome period. Finally, we demonstrated that mere sensory contextual changes during the renewal test were sufficient to elicit signals likely associated with RPEs. Thus, RPEs are truly expectancy-driven since they can be elicited by changes in reward expectation, without an actual change in the quality or quantity of reward.
Cue-Evoked Dopamine Neuron Activity Helps Maintain but Does Not Encode Expected Value
2019, Cell Reports
Cue-evoked midbrain dopamine (DA) neuron activity reflects expected value, but its influence on reward assessment is unclear. In mice performing a trial-based operant task, we test if bidirectional manipulations of cue or operant-associated DA neuron activity drive learning as a result of under- or overexpectation of reward value. We target optogenetic manipulations to different components of forced trials, when only one lever is presented, and assess lever biases on choice trials in the absence of photomanipulation. Although lever biases are demonstrated to be flexible and sensitive to changes in expected value, augmentation of cue or operant-associated DA signaling does not significantly alter choice behavior, and blunting DA signaling during any component of the forced trials reduces choice trial responses on the associated lever. These data suggest cue-evoked DA helps maintain cue-value associations but does not encode expected value as to set the benchmark against which received reward is judged.
Choice impulsivity after repeated social stress is associated with increased perineuronal nets in the medial prefrontal cortex
2024, Scientific Reports

View all citing articles on Scopus

¹: Authors MPS and JAS contributed equally to this work.

View full text

Archival ReportMesolimbic Dopamine Dynamically Tracks, and Is Causally Linked to, Discrete Aspects of Value-Based Decision Making

Abstract

Background

Methods

Results

Conclusions

Section snippets

Behavior

Dopamine Signaling Tracks Associative Cue Value Related to Subjective Choice Preferences

Discussion

Acknowledgments and Disclosures

Curr Opin Neurobiol

Biol Psychiatry

Biol Psychiatry

Biol Psychiatry

Neuroscience

Neuron

Neuron

Biol Psychiatry

Ventral tegmental area neurons in learned appetitive behavior and positive reinforcement

Annu Rev Neurosci

Associative learning mediates dynamic shifts in dopamine signaling in the nucleus accumbens

Nat Neurosci

Dopamine cells respond to predicted events during classical conditioning: Evidence for eligibility traces in the reward-learning network

J Neurosci

Dopamine responses comply with basic assumptions of formal learning theory

Nature

A neural substrate of prediction and reward

Science

Dopamine neurons encode the better option in rats deciding between differently delayed or sized rewards

Nat Neurosci

Discrete coding of reward probability and uncertainty by dopamine neurons

Science

Archival Report
Mesolimbic Dopamine Dynamically Tracks, and Is Causally Linked to, Discrete Aspects of Value-Based Decision Making