The auditory scene is dynamic, changing from 1 min to the next as sound sources enter and leave our space. How does the brain resolve the problem of maintaining neural representations of the distinct yet changing sound sources? We used an auditory streaming paradigm to test the dynamics of multiple sound source representation, when switching between integrated and segregated sound streams. The mismatch negativity (MMN) component of event-related potentials was used as index of change detection to observe stimulus-driven modulation of the ongoing sound organization. Probe tones were presented randomly within ambiguously organized sound sequences to reveal whether the neurophysiological representation of the sounds was integrated (no MMN) or segregated (MMN). The pattern of results demonstrated context-dependent responses to a single tone that was modulated in dynamic fashion as the auditory environment rapidly changed from integrated to segregated sounds. This suggests a rapid form of auditory plasticity in which the longer-term sound context influences the current state of neural activity when it is ambiguous. These results demonstrate stimulus-driven modulation of neural activity that accommodates to the dynamically changing acoustic environment.