Audio-visual integration of offset signals(Summaries of Awarded Presentation at the 28th Annual Meeting)

(1)

The Japanese Psychonomic Society

NII-Electronic Library Service

The JapanesePsychonomic Society

TheJbpanesefournatofPsychonomicScinnce

201O,VoL29,No,1,83-84

Summary

ofAwarded

Presentation6-116

Audio-visual

integration

of

offset

signalsi}

Fuminori

ONo*

and

Katsumi

WATANABE*･

*2･ *3

71}re

Lihiversit),

of

7bizyo',

fS7"'2,

and

AIST'"

Accumulated

evidence

has

shown that the perception of a visual object

is

altered

by

an auditory stimulus. This study investigatedhow the"offset" _timing

of an auditory stimulus would

influencethe _perceived _positionof a moving visual object, A visual _targetmoved smoothly ina

horizontal

direction

and disappeared at an unpredictable position.

Concurrently

with the visual

target,a _pure-tone sound was continuously _presented. The onset of the sQund was temporally

aligned with thatof the visua] target

but

theoffset timing varied.

The

results showed thatthe

perceived

final

posjtion

changed systematically with the temporal _gap between the visual and

sound offsets. This findingsuggests that an offset of an auditory signal may signal changes of a visual object.

Key words: audio-visual integration,motion _perception,mislocation

Information carried by differentsensory channels

is

effortlessly and automatically

integrated,

Numer-ous studies

have

also shown that subjective visual experience can

be

dramatically

altered

by

auditory

signals

(e.g.

Shams,

Kamitani,

Thompson,

&

Shi-mojo, 2000). For example, Scheier,Nijwahan,

Shi-mojo,

(1999)

have

shown thatvisua] temporal resolu-tioncan be either improved or degraded by sounds,

depending on the temporal relationship, They

showed thatwhen two

lights

were turned on with a small temporal delay,the accuracy of temporal erder

judgment

was

better

when one sound preceded, and another followed,the visual stimulL Incontrast, the

subjects' perforrnance

became

worse when two sounds were insertedbetween thetwe visual stimuli.

This findingdemonstrated that auditory signa]s can

positively

interact

with the timing of visual stimuli,

Whereas most of the_previous studies have examined

stimulus "onsets",

less

attention

has

been

devoted

to stimulus

"offsets".

We therefore asked the _question of whether an

offset of auditory signal woulcl alter visual

percep-tion.

To

address thisquestion,we

investigated

how

the offset timing of an auditory stimulus would

infl-uence the perceived offset _position of a smoothly

moving visual object.

Methods

Participants. Fourteen _paid volunteers

partici-pated.

All

participants reperted

having

normal

hear-ing and normal, or corrccted-to-normal, vision.

Apparatus and

Stimuli.

The _participants were

seated

in

front

ofa rnonitor

inaquiet

dark

room, and

were wearing headphones. The monitor background

was split

into

two gray areas which were

different-iatedby the screen luminance: the upper area was

brighter

than the

lewer

area

(Figure

1).

A

white

fixationcross was presented at 7degrees under the

center of screen. The target was a black disk

(1

degree

in

diameter),

which was

O,5

degrees

above the

boundary lineof the background, The auditory

*

b

Research

Center

for

Advanced

Science

and

Technology,

The

University

of Tokyo, 4-6-1,

Komaba, Meguro-ku, Tokyo 153-8904

The study was supported by the Hokuriku

Innovatien Cluster for Health Science,

Japan

Figure

Society for the Promotion of Science,

Japan

Science

and Technolegy Agency, and MEXT,

1,

Schematic

representation of the

stimulus. The arrow indicatesthe directionof

the target's motien.

The JapanesePsychonomic Society,Allrights reserved. NII-Electronic

(2)

The Japanese Psychonomic Society

NII-Electronic Library Service

TheJapanesePsychonomic Society

84 t

stimulus was a continuous pure tone with a

fre-quency of

1,OOO

Hz.

Procedure. The participantinitiatedeach trialby

pressing the space

bar

of a

keyboard.

The

initial

positionof thetargetwas 15 degrees away from thc

midpoint of

the

disp]ay.

After

a

delay

of

500ms

from

theappearance of thetarget,

it

moved smoothly

totheright and

disappeared

at one of threerandom

positions:either

-O.3,

O,

or

O.3 degrees

frorn

the

display center. Concurrentty with thevisual target,

a pure-tone sound was presented.

The

onset

timing

of the sound was temporally aligned with the onset

of the

targct

motion. The

temporal

delay between

the targetoffset and the sound offset was either

-120,

-80,

-40,

_O,_40,_80,_or ₁₂₀ _rns. _After₂₀₀ _ms

from thetargetoffset,a mouse cursor was _prcsentcd

at the eenter of the display. The

_participants

were

required

to

indicate

the

offset positionas accurately as they could

by

using the mouse cursor.

Each

participantperformed 21O trials.

Results

Figure 2shows the mean mislocation of the target

for

each time

difference

between

the sound offset

and the visuai offset. A _positiveva]ue mcans that thc offset _position of the moving target was

per-ceived as after the actual offset _position.

Analysis

with a one-way ANOVA revealed a significant main

effect

[F(6,

78)=-

10.42,

_P<O.OOI].

Planned

compari-sons showed that whcn thesound offset was

-120

ms the _perccived _tcrmination _positionof the visual

targetwas earlier than

it

was at a sound offset of O

ms

[t(13)=2,99,

P<O.Ol]. Furthermore, when the sound termination was 120 ms after thc target offsct,

the perceived target

location

was more advanced

than

it

was atOms

[t(13)=2.54,P<O.Ol].

Discussion

Our results clearly showed that the offset of an

auditory stimulus

influenced

the perceived position

at which a moving visual object disappeared.

Spe-cifically, when

the

offset

timing

of

the

concurrent

sound was earli6r than that of thc visual target,the

offset positionof the targetwas perceived as

before

{i.e.,

earlier than) the actual offset position. In

con-I'he

Japanese

Journal

of Psychenomic

Science

Voi.29,No. 1

02 oo.15 g :Ol : !. O.05 e :o

2 8-eos

!'

-Ol

:t

-O

IS

-D?

-no

-eo

"o D 4o eo 1?o

Tlmedseund ±eTnlnetienlelativetoterge ±±erminetion(ms}

Figure2. The mean mislocation for each

temporal

delay.

Vertical

bars

indicate

the

standard error.

trast,when theoffset tiniingof theconcurrent sound

was laterthan that of the visual target,the offset

positionof the targetwas _perceived as ahead of the

actual location. These

findings

suggest that the offset of an auditory signal and thatof a visual signal

can also

be

integrated,

A phenomenon relevant to the_present study

is

the

temporal ventriloquism effect, in which the per-ceived timing of a visua] stimulus

is

associated with

that of the sound to which the visual stimulus is

bound

{Morein-Zamir,

Soto-Farace,

&

Kingstone,

2003; Vroomen

&

de

Gelder,

2004). The _prescnt

findjng

can

be

attributed totheoffset timing of the

visual target also

being

advanced or

delayed,

de-pending on theoffset timing of the concurrent sound.

W'hether

this

integration

occurs

in

other modali-tics

(e,g.

vision and touch), and whcthcr the effects

depend

on the spatial proximity of the stimulus,

warrant

further

investigations.

References

Morein-Zamir, S. Soto-Faraco, S,,& Kingstone, A.

(2003).

iXuditory capture of vision: Examining

poralventriloquism. CognitiveBrain Research, 17,

154-163.

Scheier,C.R. Nijwahan, R,,& Shimojo, S.

(1999).

Sound

alters x･'i'suaitemporal resolution.

tiveOPhthalmology & VisualScience,40,4169.

Shams,

L.

Kamitani, Y.

Thompsen,

S. &

Shimojo,

S. (2000).

What you see iswhat _you hear. ?Vature,

408, 788.

Vroomcn,

J.,

&

de Gclder,B.

_(2004}.

Temporal

triloquism:

Sound

modu]ates the flash-lageffect.