Skip to main content
SpringerLink
Log in

Audio–tactile superiority over visuo–tactile and audio–visual combinations in the temporal resolution of synchrony perception

  • Research Article
  • Published:
Experimental Brain Research Aims and scope Submit manuscript

Abstract

To see whether there is a difference in temporal resolution of synchrony perception between audio–visual (AV), visuo–tactile (VT), and audio–tactile (AT) combinations, we compared synchrony–asynchrony discrimination thresholds of human participants. Visual and auditory stimuli were, respectively, a luminance-modulated Gaussian blob and an amplitude-modulated white noise. Tactile stimuli were mechanical vibrations presented to the index finger. All the stimuli were temporally modulated by either single pulses or repetitive-pulse trains. The results show that the temporal resolution of synchrony perception was similar for AV and VT (e.g., ~4 Hz for repetitive-pulse stimuli), but significantly higher for AT (~10 Hz). Apart from having a higher temporal resolution, however, AT synchrony perception was similar to AV synchrony perception in that participants could select matching features through attention, and a change in the matching-feature attribute had little effect on temporal resolution. The AT superiority in temporal resolution was indicated not only by synchrony–asynchrony discrimination but also by simultaneity judgments. Temporal order judgments were less affected by modality combination than the other two tasks.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5

Similar content being viewed by others

Notes

  1. In this paper, we focus on the basic characteristics of multisensory temporal synchrony perception for simple, non-speech stimuli. The AV temporal window for complex stimuli is known to be wider than for simple stimuli (e.g., Dixon and Spitz 1980). Several recent studies also demonstrate that multimodal temporal synchrony perception for speech stimuli is somewhat special (Vatakis and Spence 2006a, b, 2007, 2008).

  2. Another problem with SJ arises when one attempts to measure the temporal acuity limit in terms of the upper temporal frequency for correct synchrony perception using repetitive stimuli. For the AV combination, and possibly for other modality combinations (Yau et al. 2009), when the stimulus temporal frequency is higher than the limit of synchrony perception, while lower than the limit of perception of visual flickers and auditory flutters, observers often report apparent synchrony even if the paired stimuli were different in temporal frequency. Further in AV, the visual flicker rate is apparently captured by the auditory flutter (auditory driving, see Recanzone 2003; Shipley 1964). These phenomena suggest the mechanism underlying SJ is complex, which makes the interpretation of the SJ data harder. In contrast, synchrony–asynchrony discrimination tasks can be used similarly for non-repetitive and repetitive stimuli because auditory driving is observed only at high frequencies where synchrony–asynchrony discrimination is impossible. This effect is negligible as far as observers’ performance is evaluated in terms of the threshold of discrimination.

  3. Since the numbers of synchronous and asynchronous trials were fixed, it was theoretically possible for the participants to use feedback to guess the next stimulus more accurately by chance. We consider, however, that the effect of this factor was negligible, since the obtained psychometric functions always dropped to the chance level for all the participants, including the authors.

  4. It is suggested that the tactile system can sense much smaller inter-skin time differences than suggested by our TT data when touch source localization is tested using skin locations properly selected for this task (Von Békésy 1959).

  5. Use of one-tailed tests is justified for testing whether the higher resolution for AT than for AV was preserved in the A2T1 and A1T2 conditions but a two-tailed t test does not indicate a significant difference in either case. Unfortunately, presumably because of the task complexity, A1T2 and A2T1 showed a large individual difference. In particular, the performance of one naïve participant was very low. For the other six participants, the threshold was consistently higher for A1T2 and A2T1 than for AV, and the differences were statistically significant even by two-tailed paired t tests (P = 0.0102, P = 0.0104, respectively).

  6. Additional data were also collected for the TT condition with hand crossed. See Discussion and footnote 10.

  7. 1.746 = 1.178/0.674, where 1.178 is the x value at which the normal distribution function is one half of the peak (corresponding to HWHH of SJ), while 0.674 is that at which the cumulative normal distribution function is 75% (corresponding to JND of TOJ).

  8. If mid-level temporal resolution is indeed found to be different across modalities, what kind of mechanism is responsible for the difference? Given that the mid-level signals are computed from low-level signals, although the temporal resolution significantly drops, the mid-level signals might still inherit a difference in temporal resolution from peripheral signals. Thus, vision is still worse than the other modalities even in terms of mid-level temporal resolution. Alternatively, the modality difference might come from a difference in the attentional load (Lavie and Tsal 1994) required to perform similar operations in different modalities.

  9. A reviewer suggested that, because an event involving touch is likely to occur around the body, the superiority of AT over AV in temporal resolution might reflect the fact that asynchronies between A and T are usually smaller than those between A and V in a natural environment. Our sensory system might have developed the cross-modal difference in temporal resolution through adaption to the natural environment (e.g., Fujisaki et al. 2004; Hanson et al. 2008; Harrar and Harris 2008; Navarra et al. 2007; Vroomen et al. 2004). However, this idea cannot explain why VT does not have a high temporal resolution. The reviewer also suggested that the elevation of temporal acuity by spatial misalignment of stimulus pair (Spence et al. 2003) might be particularly strong for AT, but this is not supported by the finding that spatial coincidence has no effects on the temporal acuity of AT for sighted participants (Occelli et al. 2008; Zampini et al. 2005a).

  10. A more striking difference between TOJ and SJ can be demonstrated with TT judgments with hand crossed (Fig. 4e). With our system and participants, we confirmed that TOJ was heavily disrupted by crossing the hands (Shore et al. 2002; Yamamoto and Kitazawa 2001), while SJ was unaffected (Axelrod et al. 1968). HWHH values are 18.2 and 19.8 ms in Figs. 4d and 5e, respectively.

References

  • Aghdaee SM, Cavanagh P (2007) Temporal limits of long-range phase discrimination across the visual field. Vis Res 47(16):2156–2163

    Article  PubMed  Google Scholar 

  • Amano K, Johnston A, Nishida S (2007) Two mechanisms underlying the effect of angle of motion direction change on colour-motion asynchrony. Vis Res 47(5):687–705

    Article  PubMed  Google Scholar 

  • Arnold DH (2005) Perceptual pairing of colour and motion. Vis Res 45(24):3015–3026

    Article  PubMed  Google Scholar 

  • Arrighi R, Alais D, Burr D (2006) Perceptual synchrony of audiovisual streams for natural and artificial motion sequences. J Vis 6(3):260–268

    Article  PubMed  Google Scholar 

  • Axelrod S, Thompson LW, Cohen LD (1968) Effects of senescence on the temporal resolution of somesthetic stimuli presented to one hand or both. J Gerontol 23(2):191–195

    PubMed  CAS  Google Scholar 

  • Caetano G, Jousmaki V (2006) Evidence of vibrotactile input to human auditory cortex. Neuroimage 29(1):15–28

    Article  PubMed  Google Scholar 

  • Dixon NF, Spitz L (1980) The detection of auditory visual desynchrony. Perception 9(6):719–721

    Article  PubMed  CAS  Google Scholar 

  • Duncan J, Martens S, Ward R (1997) Restricted attentional capacity within but not between sensory modalities. Nature 387(6635):808–810

    Article  PubMed  CAS  Google Scholar 

  • Foxe JJ, Wylie GR, Martinez A, Schroeder CE, Javitt DC, Guilfoyle D et al (2002) Auditory-somatosensory multisensory processing in auditory association cortex: an fmri study. J Neurophysiol 88(1):540–543

    PubMed  Google Scholar 

  • Fujisaki W, Nishida S (2005) Temporal frequency characteristics of synchrony–asynchrony discrimination of audio–visual signals. Exp Brain Res 166(3–4):455–464

    Article  PubMed  Google Scholar 

  • Fujisaki W, Nishida S (2007) Feature-based processing of audio–visual synchrony perception revealed by random pulse trains. Vis Res 47(8):1075–1093

    Article  PubMed  Google Scholar 

  • Fujisaki W, Nishida S (2008a) Temporal limits of within- and cross-modal cross-attribute bindings. Paper presented at the 9th international multisensory research forum, Hamburg, Germany

  • Fujisaki W, Nishida S (2008b) Top-down feature-based selection of matching features for audio–visual synchrony discrimination. Neurosci Lett 433(3):225–230

    Article  PubMed  CAS  Google Scholar 

  • Fujisaki W, Shimojo S, Kashino M, Nishida S (2004) Recalibration of audiovisual simultaneity. Nat Neurosci 7(7):773–778

    Article  PubMed  CAS  Google Scholar 

  • Fujisaki W, Koene A, Arnold D, Johnston A, Nishida S (2006) Visual search for a target changing in synchrony with an auditory signal. Proc Biol Sci 273(1588):865–874

    Article  PubMed  Google Scholar 

  • Hanson JV, Heron J, Whitaker D (2008) Recalibration of perceived time across sensory modalities. Exp Brain Res 185(2):347–352

    Article  PubMed  Google Scholar 

  • Harrar V, Harris LR (2008) The effect of exposure to asynchronous audio, visual, and tactile stimulus combinations on the perception of simultaneity. Exp Brain Res 186(4):517–524

    Article  PubMed  Google Scholar 

  • Hirsh IJ, Sherrick C E Jr (1961) Perceived order in different sense modalities. J Exp Psychol 62:423–432

    Article  PubMed  CAS  Google Scholar 

  • Holcombe AO, Cavanagh P (2001) Early binding of feature pairs for visual perception. Nat Neurosci 4(2):127–128

    Article  PubMed  CAS  Google Scholar 

  • Hoshiyama M, Kakigi R, Tamura Y (2004) Temporal discrimination threshold on various parts of the body. Muscle Nerve 29(2):243–247

    Article  PubMed  Google Scholar 

  • Kandil FI, Fahle M (2001) Purely temporal figure-ground segregation. Eur J Neurosci 13(10):2004–2008

    Article  PubMed  CAS  Google Scholar 

  • Kayser C, Petkov CI, Augath M, Logothetis NK (2005) Integration of touch and sound in auditory cortex. Neuron 48(2):373–384

    Article  PubMed  CAS  Google Scholar 

  • Kelly DH (1979) Motion and vision. Ii. Stabilized spatio-temporal threshold surface. J Opt Soc Am 69(10):1340–1349

    Article  PubMed  CAS  Google Scholar 

  • Lavie N, Tsal Y (1994) Perceptual load as a major determinant of the locus of selection in visual attention. Percept Psychophys 56(2):183–197

    PubMed  CAS  Google Scholar 

  • Lee BB, Martin PR, Valberg A, Kremers J (1993) Physiological mechanisms underlying psychophysical sensitivity to combined luminance and chromatic modulation. J Opt Soc Am A 10(6):1403–1412

    Article  PubMed  CAS  Google Scholar 

  • Levanen S, Jousmaki V, Hari R (1998) Vibration-induced auditory-cortex activation in a congenitally deaf adult. Curr Biol 8(15):869–872

    Article  PubMed  CAS  Google Scholar 

  • Lewkowicz DJ (1996) Perception of auditory-visual temporal synchrony in human infants. J Exp Psychol Hum Percept Perform 22(5):1094–1106

    Article  PubMed  CAS  Google Scholar 

  • Lu ZL, Sperling G (2001) Three-systems theory of human visual motion perception: Review and update. J Opt Soc Am A Opt Image Sci Vis 18(9):2331–2370

    Article  PubMed  CAS  Google Scholar 

  • Lu T, Liang L, Wang X (2001) Temporal and rate representations of time-varying signals in the auditory cortex of awake primates. Nat Neurosci 4(11):1131–1138

    Article  PubMed  CAS  Google Scholar 

  • Martens W, Woszczyk W (2004) Perceived synchrony in a bimodal display: optimal intermodal delay for coordinated auditory and haptic reproduction. Paper presented at the ICAD 04-tenth meeting of the international conference on auditory display, Sydney, Australia

  • Moore B (2003) An introduction to the psychology of hearing, 5th edn. Academic Press, San Diego

    Google Scholar 

  • Navarra J, Soto-Faraco S, Spence C (2007) Adaptation to audiotactile asynchrony. Neurosci Lett 413(1):72–76

    Article  PubMed  CAS  Google Scholar 

  • Nishida S, Johnston A (2002) Marker correspondence, not processing latency, determines temporal binding of visual attributes. Curr Biol 12(5):359–368

    Article  PubMed  CAS  Google Scholar 

  • Occelli V, Spence C, Zampini M (2008) Audiotactile temporal order judgments in sighted and blind individuals. Neuropsychologia 46(11):2845–2850

    Article  PubMed  Google Scholar 

  • Pöppel E (1988) Mindworks: time and conscious experience. New York, Harcourt Brace Jovanovich

    Google Scholar 

  • Recanzone GH (2003) Auditory influences on visual temporal rate perception. J Neurophysiol 89(2):1078–1093

    Article  PubMed  Google Scholar 

  • Schneider KA, Bavelier D (2003) Components of visual prior entry. Cogn Psychol 47(4):333–366

    Article  PubMed  Google Scholar 

  • Schroeder CE, Lindsley RW, Specht C, Marcovici A, Smiley JF, Javitt DC (2001) Somatosensory input to auditory association cortex in the macaque monkey. J Neurophysiol 85(3):1322–1327

    PubMed  CAS  Google Scholar 

  • Shapiro KL, Arnell KM, Raymond JE (1997) The attentional blink. Trends Cogn Sci 1:291–296

    Article  Google Scholar 

  • Shipley T (1964) Auditory flutter-driving of visual flicker. Science 145:1328–1330

    Article  PubMed  CAS  Google Scholar 

  • Shore DI, Spry E, Spence C (2002) Confusing the mind by crossing the hands. Brain Res Cogn Brain Res 14(1):153–163

    Article  PubMed  Google Scholar 

  • Soto-Faraco S, Deco G (2009) Multisensory contributions to the perception of vibrotactile events. Behav Brain Res 196(2):145–154

    Article  PubMed  Google Scholar 

  • Spence C, Baddeley R, Zampini M, James R, Shore DI (2003) Multisensory temporal order judgments: When two locations are better than one. Percept Psychophys 65(2):318–328

    PubMed  Google Scholar 

  • Van de Burg E, Olivers CN, Bronkhorst AW, Theeuves J (2008) Pip and pop: nonspatial auditory signals improve spatial visual search. J Exp Psychol Hum Percept Perform 34(5):1053–1065

    Article  PubMed  Google Scholar 

  • van de Par S, Kohlrausch A (2004) Visual and auditory object selection based on temporal correlations between auditory and visual cues. Paper presented at the 18th int. congress on acoustics, Kyoto, Japan, 4–9 April 2004

  • van de Par S, Kohlrausch A, Juola F (2002) Some methodological aspects for measuring asynchrony detection in audio–visual stimuli. Paper presented at the Forum Acousticum Sevilla, Sevilla, Spain

  • van Eijk RL, Kohlrausch A, Juola JF, van de Par S (2008) Audiovisual synchrony and temporal order judgments: effects of experimental method and stimulus type. Percept Psychophys 70(6):955–968

    Article  PubMed  Google Scholar 

  • Vatakis A, Spence C (2006a) Audiovisual synchrony perception for music, speech, and object actions. Brain Res 1111(1):134–142

    Article  PubMed  CAS  Google Scholar 

  • Vatakis A, Spence C (2006b) Audiovisual synchrony perception for speech and music assessed using a temporal order judgment task. Neurosci Lett 393(1):40–44

    Article  PubMed  CAS  Google Scholar 

  • Vatakis A, Spence C (2006c) Temporal order judgments for audiovisual targets embedded in unimodal and bimodal distractor streams. Neurosci Lett 408(1):5–9

    Article  PubMed  CAS  Google Scholar 

  • Vatakis A, Spence C (2007) How ‘special’ is the human face? Evidence from an audiovisual temporal order judgment task. Neuroreport 18(17):1807–1811

    Article  PubMed  Google Scholar 

  • Vatakis A, Spence C (2008) Investigating the effects of inversion on configural processing with an audiovisual temporal-order judgment task. Perception 37(1):143–160

    Article  PubMed  Google Scholar 

  • Vatakis A, Navarra J, Soto-Faraco S, Spence C (2008) Audiovisual temporal adaptation of speech: temporal order versus simultaneity judgments. Exp Brain Res 185(3):521–529

    Article  PubMed  Google Scholar 

  • Verstraten FA, Cavanagh P, Labianca AT (2000) Limits of attentive tracking reveal temporal properties of attention. Vision Res 40(26):3651–3664

    Article  Google Scholar 

  • Von Békésy G (1959) Similarities between hearing and skin sensations. Psychol Rev 66(1):1–22

    Article  Google Scholar 

  • Von Békésy G (1963) Interaction of paired sensory stimuli and conduction in peripheral nerves. J Appl Physiol 18:1276–1284

    Google Scholar 

  • Vroomen J, Keetels M, de Gelder B, Bertelson P (2004) Recalibration of temporal order perception by exposure to audio–visual asynchrony. Brain Res Cogn Brain Res 22(1):32–35

    Article  PubMed  Google Scholar 

  • Yamamoto S, Kitazawa S (2001) Reversal of subjective temporal order due to arm crossing. Nat Neurosci 4(7):759–765

    Article  PubMed  CAS  Google Scholar 

  • Yau JM, Olenczak JB, Dammann JF, Bensmaia SJ (2009) Temporal frequency channels are linked across audition and touch. Curr Biol 19(7):561–566

    Article  PubMed  CAS  Google Scholar 

  • Zampini M, Shore DI, Spence C (2003) Audiovisual temporal order judgments. Exp Brain Res 152(2):198–210

    Article  PubMed  Google Scholar 

  • Zampini M, Brown T, Shore DI, Maravita A, Roder B, Spence C (2005a) Audiotactile temporal order judgments. Acta Psychol (Amst) 118(3):277–291

    Article  Google Scholar 

  • Zampini M, Guest S, Shore DI, Spence C (2005b) Audio–visual simultaneity judgments. Percept Psychophys 67(3):531–544

    PubMed  Google Scholar 

Download references

Acknowledgment

Most of this work was carried out when WF was a JSPS Research Fellow at NTT Communication Science Laboratories.

Author information

Authors and Affiliations

  1. National Institute of Advanced Industrial Science and Technology (AIST), Tsukuba Central 6, 1-1-1, Higashi, Tsukuba, Ibaraki, 305-8566, Japan

    Waka Fujisaki

  2. NTT Communication Science Laboratories, NTT Corporation, 3-1, Morinosato-Wakamiya, Atsugi-shi, Kanagawa, 243-0198, Japan

    Shin’ya Nishida

Authors
  1. Waka Fujisaki
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Shin’ya Nishida
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Waka Fujisaki.

Rights and permissions

Reprints and permissions

About this article

Cite this article

Fujisaki, W., Nishida, S. Audio–tactile superiority over visuo–tactile and audio–visual combinations in the temporal resolution of synchrony perception. Exp Brain Res 198, 245–259 (2009). https://doi.org/10.1007/s00221-009-1870-x

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s00221-009-1870-x

Keywords

Search

Navigation