Explanations for color phenomena are often sought in the retina, lateral geniculate nucleus, and V1, yet it is becoming increasingly clear that a complete account will take us further along the visual-processing pathway. Working out which areas are involved is not trivial. Responses to S-cone activation are often assumed to indicate that an area or neuron is involved in color perception. However, work tracing S-cone signals into extrastriate cortex has challenged this assumption: S-cone responses have been found in brain regions, such as the middle temporal (MT) motion area, not thought to play a major role in color perception. Here, we review the processing of S-cone signals across cortex and present original data on S-cone responses measured with fMRI in alert macaque, focusing on one area in which S-cone signals seem likely to contribute to color (V4/posterior inferior temporal cortex) and on one area in which S signals are unlikely to play a role in color (MT). We advance a hypothesis that the S-cone signals in color-computing areas are required to achieve a balanced neural representation of perceptual color space, whereas those in noncolor-areas provide a cue to illumination (not luminance) and confer sensitivity to the chromatic contrast generated by natural daylight (shadows, illuminated by ambient sky, surrounded by direct sunlight). This sensitivity would facilitate the extraction of shape-from-shadow signals to benefit global scene analysis and motion perception.