Skip to main content

Thank you for visiting nature.com. You are using a browser version with limited support for CSS. To obtain the best experience, we recommend you use a more up to date browser (or turn off compatibility mode in Internet Explorer). In the meantime, to ensure continued support, we are displaying the site without styles and JavaScript.

  • Article
  • Published:

Learning the value of information in an uncertain world

Nature Neuroscience volume 10, pages 1214–1221 (2007)Cite this article

Abstract

Our decisions are guided by outcomes that are associated with decisions made in the past. However, the amount of influence each past outcome has on our next decision remains unclear. To ensure optimal decision-making, the weight given to decision outcomes should reflect their salience in predicting future outcomes, and this salience should be modulated by the volatility of the reward environment. We show that human subjects assess volatility in an optimal manner and adjust decision-making accordingly. This optimal estimate of volatility is reflected in the fMRI signal in the anterior cingulate cortex (ACC) when each trial outcome is observed. When a new piece of information is witnessed, activity levels reflect its salience for predicting future outcomes. Furthermore, variations in this ACC signal across the population predict variations in subject learning rates. Our results provide a formal account of how we weigh our different experiences in guiding our future actions.

This is a preview of subscription content, access via your institution

Access options

Buy this article

  • Purchase on Springer Link
  • Instant access to full article PDF
Buy now

Prices may be subject to local taxes which are calculated during checkout

Figure 1: Probability-tracking task.
Figure 2: Behavior of Bayesian learner and human subjects.
Figure 3: Experiment II, cingulate activity reflecting estimated volatility.
Figure 4: Region-of-interest analysis and potential confounding factors.
Figure 5: Estimated volatility and variance on r.
Figure 6: VTA correlate of reward prediction.

Similar content being viewed by others

References

  1. Ernst, M.O. & Banks, M.S. Humans integrate visual and haptic information in a statistically optimal fashion. Nature 415, 429–433 (2002).

    Article  CAS  PubMed  Google Scholar 

  2. Kording, K.P. & Wolpert, D.M. Bayesian integration in sensorimotor learning. Nature 427, 244–247 (2004).

    Article  PubMed  Google Scholar 

  3. Kahneman, D. & Tversky, A. Choices, Values and Frames (Cambridge University Press, Cambridge, 2000).

    Book  Google Scholar 

  4. Montague, P.R., Dayan, P., Person, C. & Sejnowski, T.J. Bee foraging in uncertain environments using predictive hebbian learning. Nature 377, 725–728 (1995).

    Article  CAS  PubMed  Google Scholar 

  5. Samejima, K., Ueda, Y., Doya, K. & Kimura, M. Representation of action-specific reward values in the striatum. Science 310, 1337–1340 (2005).

    Article  CAS  PubMed  Google Scholar 

  6. Daw, N.D., O'Doherty, J.P., Dayan, P., Seymour, B. & Dolan, R.J. Cortical substrates for exploratory decisions in humans. Nature 441, 876–879 (2006).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  7. Bayer, H.M. & Glimcher, P.W. Midbrain dopamine neurons encode a quantitative reward prediction error signal. Neuron 47, 129–141 (2005).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  8. Rescorla, R.A. & Wagner, A.R. in Classical Conditioning II: Current Research and Theory (eds. Black, A.H. & Prokasy, W.F.) 64–99 (Appleton-Century Crofts, New York, 1972).

    Google Scholar 

  9. Sutton, R.S. & Barto, A.G. Reinforcement Learning: an Introduction (MIT Press, Cambridge, Massachusetts, 1998).

    Google Scholar 

  10. Dayan, P., Kakade, S. & Montague, P.R. Learning and selective attention. Nat. Neurosci. 3 Suppl, 1218–1223 (2000).

    Article  CAS  PubMed  Google Scholar 

  11. Doya, K. Metalearning and neuromodulation. Neural Netw. 15, 495–506 (2002).

    Article  PubMed  Google Scholar 

  12. Pearce, J.M. & Hall, G. A model for Pavlovian learning: variations in the effectiveness of conditioned, but not of unconditioned, stimuli. Psychol. Rev. 87, 532–552 (1980).

    Article  CAS  PubMed  Google Scholar 

  13. Dickinson, A. & Mackintosh, N.J. Classical conditioning in animals. Annu. Rev. Psychol. 29, 587–612 (1978).

    Article  CAS  PubMed  Google Scholar 

  14. Cox, R.T. Probability, frequency and reasonable expectaion. Am. J. Phys. 14, 1–13 (1946).

    Article  Google Scholar 

  15. Kakade, S. & Dayan, P. Acquisition and extinction in autoshaping. Psychol. Rev. 109, 533–544 (2002).

    Article  PubMed  Google Scholar 

  16. Courville, A.C., Daw, N.D. & Touretzky, D.S. Bayesian theories of conditioning in a changing world. Trends Cogn. Sci. 10, 294–300 (2006).

    Article  PubMed  Google Scholar 

  17. Yu, A.J. & Dayan, P. Uncertainty, neuromodulation and attention. Neuron 46, 681–692 (2005).

    Article  CAS  PubMed  Google Scholar 

  18. Sugrue, L.P., Corrado, G.S. & Newsome, W.T. Matching behavior and the representation of value in the parietal cortex. Science 304, 1782–1787 (2004).

    Article  CAS  PubMed  Google Scholar 

  19. Kennerley, S.W., Walton, M.E., Behrens, T.E., Buckley, M.J. & Rushworth, M.F. Optimal decision making and the anterior cingulate cortex. Nat. Neurosci. 9, 940–947 (2006).

    Article  CAS  PubMed  Google Scholar 

  20. Gallistel, C.R., Mark, T.A., King, A.P. & Latham, P.E. The rat approximates an ideal detector of changes in rates of reward: implications for the law of effect. J. Exp. Psychol. Anim. Behav. Process. 27, 354–372 (2001).

    Article  CAS  PubMed  Google Scholar 

  21. Procyk, E., Tanaka, Y.L. & Joseph, J.P. Anterior cingulate activity during routine and nonroutine sequential behaviors in macaques. Nat. Neurosci. 3, 502–508 (2000).

    Article  CAS  PubMed  Google Scholar 

  22. Walton, M.E., Devlin, J.T. & Rushworth, M.F. Interactions between decision making and performance monitoring within prefrontal cortex. Nat. Neurosci. 7, 1259–1265 (2004).

    Article  CAS  PubMed  Google Scholar 

  23. Niki, H. & Watanabe, M. Prefrontal and cingulate unit activity during timing behavior in the monkey. Brain Res. 171, 213–224 (1979).

    Article  CAS  PubMed  Google Scholar 

  24. Ullsperger, M. & von Cramon, D.Y. Error monitoring using external feedback: specific roles of the habenular complex, the reward system and the cingulate motor area revealed by functional magnetic resonance imaging. J. Neurosci. 23, 4308–4314 (2003).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  25. Brown, J.W. & Braver, T.S. Learned predictions of error likelihood in the anterior cingulate cortex. Science 307, 1118–1121 (2005).

    Article  CAS  PubMed  Google Scholar 

  26. Ito, S., Stuphorn, V., Brown, J.W. & Schall, J.D. Performance monitoring by the anterior cingulate cortex during saccade countermanding. Science 302, 120–122 (2003).

    Article  CAS  PubMed  Google Scholar 

  27. Matsumoto, K., Suzuki, W. & Tanaka, K. Neuronal correlates of goal-based motor selection in the prefrontal cortex. Science 301, 229–232 (2003).

    Article  CAS  PubMed  Google Scholar 

  28. Smith, S.M. et al. Advances in functional and structural MR image analysis and implementation as FSL. Neuroimage 23 Suppl 1, S208–S219 (2004).

    Article  PubMed  Google Scholar 

  29. Koechlin, E., Ody, C. & Kouneiher, F. The architecture of cognitive control in the human prefrontal cortex. Science 302, 1181–1185 (2003).

    Article  CAS  PubMed  Google Scholar 

  30. Strick, P.L., Dum, R.P. & Picard, N. Motor areas on the medial wall of the hemisphere. Novartis Found Symp. 218, 64–75; discussion 75–80, 104–8 (1998).

    CAS  PubMed  Google Scholar 

  31. Van Hoesen, G.W., Morecraft, R.J. & Vogt, B.A. in Neurobiology of Cingulate Cortex and Limbic Thalamus (eds. Vogt, B.A. & Gabriel, M.) (Birkhauser, Boston, 1993).

    Google Scholar 

  32. McCoy, A.N., Crowley, J.C., Haghighian, G., Dean, H.L. & Platt, M.L. Saccade reward signals in posterior cingulate cortex. Neuron 40, 1031–1040 (2003).

    Article  CAS  PubMed  Google Scholar 

  33. McCoy, A.N. & Platt, M.L. Risk-sensitive neurons in macaque posterior cingulate cortex. Nat. Neurosci. 8, 1220–1227 (2005).

    Article  CAS  PubMed  Google Scholar 

  34. Fiorillo, C.D., Tobler, P.N. & Schultz, W. Discrete coding of reward probability and uncertainty by dopamine neurons. Science 299, 1898–1902 (2003).

    Article  CAS  PubMed  Google Scholar 

  35. Preuschoff, K., Bossaerts, P. & Quartz, S.R. Neural differentiation of expected reward and risk in human subcortical structures. Neuron 51, 381–390 (2006).

    Article  CAS  PubMed  Google Scholar 

  36. Aston-Jones, G. & Cohen, J.D. An integrative theory of locus coeruleus-norepinephrine function: adaptive gain and optimal performance. Annu. Rev. Neurosci. 28, 403–450 (2005).

    Article  CAS  PubMed  Google Scholar 

  37. Engle, R.F. Autoregressive conditional Heteroscedasticity with estimates of the variance of UK inflation. Econometrica 50, 987–1008 (1982).

    Article  Google Scholar 

  38. Waelti, P., Dickinson, A. & Schultz, W. Dopamine responses comply with basic assumptions of formal learning theory. Nature 412, 43–48 (2001).

    Article  CAS  PubMed  Google Scholar 

  39. O'Doherty, J. et al. Dissociable roles of ventral and dorsal striatum in instrumental conditioning. Science 304, 452–454 (2004).

    Article  CAS  PubMed  Google Scholar 

  40. Haruno, M. et al. A neural correlate of reward-based behavioral learning in caudate nucleus: a functional magnetic resonance imaging study of a stochastic decision task. J. Neurosci. 24, 1660–1665 (2004).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  41. Tanaka, S.C. et al. Prediction of immediate and future rewards differentially recruits cortico-basal ganglia loops. Nat. Neurosci. 7, 887–893 (2004).

    Article  CAS  PubMed  Google Scholar 

  42. Kunishio, K. & Haber, S.N. Primate cingulostriatal projection: limbic striatal versus sensorimotor striatal input. J. Comp. Neurol. 350, 337–356 (1994).

    Article  CAS  PubMed  Google Scholar 

  43. Amiez, C., Joseph, J.P. & Procyk, E. Reward encoding in the monkey anterior cingulate cortex. Cereb. Cortex 16, 1040–1055 (2006).

    Article  CAS  PubMed  Google Scholar 

  44. Yoshida, W. & Ishii, S. Resolution of uncertainty in prefrontal cortex. Neuron 50, 781–789 (2006).

    Article  CAS  PubMed  Google Scholar 

  45. Fitzgerald, K.D. et al. Error-related hyperactivity of the anterior cingulate cortex in obsessive-compulsive disorder. Biol. Psychiatry 57, 287–294 (2005).

    Article  PubMed  Google Scholar 

  46. Critchley, H.D., Mathias, C.J. & Dolan, R.J. Neural activity in the human brain relating to uncertainty and arousal during anticipation. Neuron 29, 537–545 (2001).

    Article  CAS  PubMed  Google Scholar 

  47. Botvinick, M.M., Cohen, J.D. & Carter, C.S. Conflict monitoring and anterior cingulate cortex: an update. Trends Cogn. Sci. 8, 539–546 (2004).

    Article  PubMed  Google Scholar 

  48. Rushworth, M.F., Buckley, M.J., Behrens, T.E., Walton, M.E. & Bannerman, D.M. Functional organization of the medial frontal cortex. Curr. Opin. Neurobiol. 17, 220–227 (2007).

    Article  CAS  PubMed  Google Scholar 

  49. Hampton, A.N., Bossaerts, P. & O'Doherty, J.P. The role of the ventromedial prefrontal cortex in abstract state-based inference during decision making in humans. J. Neurosci. 26, 8360–8367 (2006).

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  50. Preuschoff, K. & Bossaerts, P. Adding prediction risk to the theory of reward learning. Ann. N Y Acad. Sci. 1104, 135–146 (2007).

    Article  PubMed  Google Scholar 

Download references

Acknowledgements

The authors would like to thank K. Watkins for advice with the study and the manuscript. This work was supported by the UK Medical Research Council (T.B.), the Engineering and Physical Sciences Research Council (M.W.W.), the Wellcome trust (M.E.W.) and the Royal Society (M.F.S.R.).

Author information

Authors and Affiliations

  1. FMRIB Centre, University of Oxford, John Radcliffe Hospital, Oxford, OX3 9DU, UK

    Timothy E J Behrens, Mark W Woolrich & Matthew F S Rushworth

  2. Department of Experimental Psychology, University of Oxford, South Parks Road, Oxford, OX1 3UD, UK

    Timothy E J Behrens, Mark E Walton & Matthew F S Rushworth

Authors
  1. Timothy E J Behrens
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Mark W Woolrich
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Mark E Walton
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Matthew F S Rushworth
    View author publications

    You can also search for this author in PubMed Google Scholar

Contributions

All four authors were involved in generating the hypothesis, designing the experiment and writing the manuscript. Where specific roles can be assigned: T.E.J.B. and M.W.W. built the model. T.E.J.B. acquired and analyzed the data. M.E.W. supplied the necessary incisive wit. M.F.S.R. supervised the project.

Corresponding author

Correspondence to Timothy E J Behrens.

Ethics declarations

Competing interests

The authors declare no competing financial interests.

Supplementary information

Supplementary Text and Figures

Supplementary Figures 1 and 2, Table 1, Supplementary Information (PDF 1986 kb)

Rights and permissions

Reprints and permissions

About this article

Cite this article

Behrens, T., Woolrich, M., Walton, M. et al. Learning the value of information in an uncertain world. Nat Neurosci 10, 1214–1221 (2007). https://doi.org/10.1038/nn1954

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1038/nn1954

This article is cited by

Search

Advanced search

Quick links

Nature Briefing

Sign up for the Nature Briefing newsletter — what matters in science, free to your inbox daily.

Get the most important science stories of the day, free in your inbox. Sign up for Nature Briefing