Exploring multimodal integration in two functions of perception(Spatio-temporal integration of multimodal sensations,Symposium 2 at the 27th Annual Meeting)

NII-Electronic Library Service

7Zhe

_.lapanese

"]umatofRsychonomicScience

2009,Vol.28,No.1,13e-L34

Lecture

Exploringmultimodal

integration

in

of

_perception

two

functions

Kenzo

SAKURAI*

7bhohu

Galeuin

Lbiiversit),*

Focusing

on the topics of event perception and self-motion perception,

in

this paper,

I

introduce eur recent research on the

integration

of visual

information

with auditory and

vcstibu-lar

information.

We

have been investigating the limitsof audiofvisual integration

by

modifying

conventional streamlbounce

disp]ays

in

spatial and ternporaldomains, We found thata sound has

a markedly _greater organizing

infiuence

on visual _{perception than} was _{previously thought,}

influencing the resolution of visual motion sequences over a wide range of spatiotemporal

manipulations,

Regarding

_the

integration

of visual and vestibular

inforrnation

in _perceived

se]f-motion, theresults ofour experiments, inwhich we manipulated thecongruency

between

vestibu-lar

and visual

(eptic

fiew}inputs,suggest

that

the

multlmodal

integration

is

an cither-or _process when the discrepancy

between

visual and vestibu]ar information islarge,but the integrationisa

weighted combjnation of both inputs when thatdifferenceissmaH,

Key

words: stream/bounce effect, multimodal perception, rnultisensory integration,audio/xrisual

interactions

Introduction:

Two

major

functions

of

perception

Human _{perception provides} at

least

two

functions.

One

is

to

identify,

locate

and track objects or events

in

the environment based on the retinal tmages and

other receptors

<sound,

touch,

sme]L etc.).

Another

is

to perceive self-positjon or sel,f-motion

from

retinal optical fiow,vestibular stimulation, auditory signals,

etc.

Focusing

on thesetwo

iunctions

of perception

I

wi]1 introduce our recent investigations on multisen-sory integration in this article.

First,

Iwill report some of our research on theaudio/visual

integration.

Second,

I

will

describe

investigations

on the

integra-tion of visual

'information

with vestibu]ar

informa-tion.

Spatiotemporal

limits

of audio!visual

integration

in

the

stream/bounce effect

Our interestand understanding about

audio/vis-* _{Department of Psychology,}

_Toheku

_Gakuin

Un'iverslty,

2-1-1

Tenjinzawa,

Izumi-ku,

Sendai

981-3193

Copyright2009.

ual integration has increa$ed significantly over the

Last

decade.

Prior

tothissurge

ln

interest,

a classlcal

view of audio/visua] integrati'onwas that vision

dominates

confiicttng auditory

information.

A

typi-cal exarnple isthe ventriloquist cffect inwhich

ob-servers _perceive the ventriloquist's voice as ifit

comes

frorn

the puppet's mouth.

A

contrary rnodern

view of audiofvisua] integration isthat there are

instances

where auditory

information

influences

vi-sion. A seminal

finding

was Sekuler, Sekuler and

Lau's

₍₁₉₉₇₎

report of thc so-called `stream/bounce'

effect, a transient-induced shift

in

_perceptual bias

when resolving an ambiguous motion djsplay. Ina

typicaldisplay,two identicaltargcts

(usua]ly

dots or

squares), movc toward one another from opposite

sides of adisplay at a constant speed, superjmpose at

the center

(it

isreferred to as "the

point of

coinci-dence''),and continue _past one another _tothe other

Qbject's starting _point. This visual sequencc is equally consistent with Lhe two objects "streaming"

past one another with their

individual

motions un-changed or "bouncing"

off of one another where the

targets reverse thei,r motion after superimposirig

(Figure

1).Streaming isthe

dominant

_perception

in

The JapanescPsychonomic Secicty,Allrights reservecl. NII-Electronic

Streaming

Percept

Figure

K.

SAKuRAi:

Exploring

rnultimodal

integration

in

two

functions

of perception

or

1.

Two

possible

percepts

a typical stream/bounce display.

Beuncing

Percept

generated in

visual only

displays,

though bouncing is

occasion-ally reported

CBertenthal,

Banton,

&

Bradbury,

l993;

Sekuler & Sekuler, 1999), Interestingly,however,

Sekuler

et a], showed that a

brief

auditory tone

presented at or near the point of coincidence alters

thisbiasfrom _{predomi/nant]y} streaming to

predomi-nantly

bouncing.

This stream!bounce effect have

been

replicated and examined

in

detail

by

several

subsequent

investigators

(e.g.

Fujisaki,

Shimejo,

Kashino,

&

Nishida,

2004)

though allof them

have

employed ambiguous visual motion sequences,

lead-ing tothe assumption that auditory stimulation has

little

influence,

sufficient only to

bias

the resolution

of ambiguous visual displays,

Inorder to more thoroughly assess _the organizing strength of an auditory stimu]us on a visual

display,

we have been investigating thespatiotemporal limits of audio/visual

integration

by

manipulating the con-ventional streamfbounce display inspatial and

tern-poral domains.

Spatial

domain

In

the

spatial

dornain,

we

began

with a

conven-tional,ambiguous, streamlbounce display and pro-gressively offset the rnotion trajectoriesof

the

mov-ing

targets either vertically

in

a 2-D display or in

depth ina 3-D disp]ay

(Figures

2 and 3). We found

that

the

bias

toward$

bouncing,

induced

by

an

audi-tor}rtransient,

_persists

despite

the trajectoryoffsets reducing the_probability of a motion reversal

(Grove

&

Sakurai,

2009).

Offset conditions were cornbined with two

audi-t.oryconditions

(tone

or no tone at the point of

coin-cidence)

in

the presence or absence of a central

oc-cluder.

In

conditions with no sound, streaming was

reported on a clear majority of tria]sregardless of spatial offset. When a transienttone was _presented,

reported motion reversa]s

dominated

and persisted

forincreasing

2-D

vertical offsets up to 17,9min arc

a)

b)

Stream

Bounce

131

Figure

2.

Schematic

illustration

of observers'

view and possible perceived trajectoriesof

the targetsina 2-D display. Ina) no occluder

is present. Observers could perceive the

targets

to

either stream past

{left

panel)or to reverse their trajectory after ceincidence

(right

panel}. In b) possible streaming and

bouncing percepts when the targets coincjde

behind

a central occ]uder.

Dashed

lines

in

the

left

panel of a) and b)a]So represent the

objective path of the targets.

a)

b)Figure3,

Oblique view illustrating _possible

perceived trajectoriesof the targets

in

a

3-D

display.

In

a) no occluder

is

present.

vers could _perceive the targets to either

stream _past

(]eft

panel) or to reverse their

trajectory

after coincidence

(right

panel).

Motion

reversal after coincidence would

involve

targetsswitching depth planes after

the point of coincidence. In b) _possible streaming

(left

_panel)

and bouncing

{right

panel) percepts when the targets coincide

behind

a central occluder. Dashed linesin

the leftpane] of a) and b)also represent the

NII-Electronic Library Service

132

TheJapanese

Journal

of Psychonomic Science Vol.28, No. 1

and for3-D trajectory offsets up to

25.6

min arc. The

persistence of the

bounce

promoting effect of an audltory tone at the_point of coincidence, indisplays

rendered unambiguous and more consistent with

streaming, shows that the organizing strcngth of audttery stimulation on visual _perception isstronger

than previous]y thought, Ternporal domain

To

explore the tempora] properties of audiovisual

interaction,we employed a novel motion sequence, a

multiple streamfbounce

display,

consisting of two

disc-pairstracing orthogona] oblique

(

±45 deg)

tra-jectorics

at equal speeds

(13.44

deg/sec),

and

coincid-ing

at acentraL

fixation

paint

CKawachL

Grove,

Saku-rai, & Gyoba, 2008)

{Fig.

4). We discovered that in

the

absence of any

transient

stimulL when a]1

four

discssimultancously coincided at the center of the

display,

perceivcd

bouncing

dominates.

Ilowever,

when one ofthe

disc-pairs

lags

behind

theother such

that

the disc-pairscoincide at

different

tirnes,

vi.$ual only sequences are resolved as streaming events.

Bouncing

of

both

disc-pairs

is

recovered,

however,

if

a transienttone is_presented simultaneously with the

first

coincidence event. Therefore,

in

the temporal

･domain,

_we

_found

_{that one auditory tone}

crossmo-dallyaffects mulLiple visual events. Our experiments are systematically exploring this

_phenomenon.

Inour firstexperiment with _thi$

display,

w･e

ma-nipulated the temporal offsets

between

the

coinci-dences of the

disc-pairs

(O-250ms)

by staggering

metion onset

between

the _pairs,

_A

_brief

tone,

syn-chronou$ with the

first

coincidence, was _presented

on half the trials. Observers

judged

whether all

disc-pairsappeared tobounce or not. The tone

pro-moted bounce _pcrcepts lnboth disc-pairsin spite of

increasing

offsets up to250 ms, again

demonstrating

a robust effect of auditory stimulation on visual

perception.

However,

it

is

possible that perceived bouncing in the firstdisc-pair per se promoted

bouncing insecond. A subsequent experiment with

three disc-pairsruled out this_possibilityby eliciting vision-only re]iable bouncing in the firstmotion

se-quence

{simuLtaneous

coincidence of

four

discs}

with-out atransient sound

{Kawachi

ct al., 2008). Results

Figure4.

The

inu]tjple stream!bounce

play,

The

left

sequence shows the O ms

delay

condition

in

which all the objects

simultaneously begin movjng toward the

center, simultaneously coincide and then

move away

from

one another.

The

right

illustrates

the

delay

conditions inwhich one

of two-disc _pairs moves firstand then the

other _pairmoves such thatthe coincidence of

the two

discpairs

isno

]onger

simultaneous.

showed

that

transient

free

bouncing

in

the

first

coin-cidence didnot promote bouncing inthc second. We

conclude thatone auditory tone alters the_perception

of multipLe visua] events.

Pereeived

direetion

ef self-motion

by

visual/vestibular

integration

To

investigate

the

integration

of visual and

vesti-bular

information

in

perceived self-motion, we

meas-ured the apparent

direction

of self-motion when the directionsof visuaL and vestibuLar information are

inconsistent,

extending our previous research

{Saku-raL

Kikuchi,

KikuchL

&

Misawa,

2002;

2003).

Ob-servers were seated on an osci]lating motor-drlve'n

swing at various orientations to the direction of

swing motion, while they viewed

expandingfcon-tracting optic fiow consistent with

lorwardlback-ward selFmotion via a head mounted

display,

Inall

condiLions, the optic

fiow

xiv'asphase

Iocked

to

the

actual motion of the swing.

Observers

_performed a

rod-pointing tasktoreport the perceived directienof self-motion.

Whe" observers' bodies were orlented ]essthan 90

degrees

away

from

the

direction

of motion specified

by

optic flow,$ome observers _perceived mid-way

K.

SAKuRAI:

Exploring multimodal

integration

in

two

functions

of

_perception

133

i"/

Figure5. The vestibular produced pattern

(RDP)

video camera arrow and the Gray arrow RDPCCD

tt

stimuiation expandinglcontracting

by

lmage mounted

indicates

synchronize indicates contraction of the RDP.

motor-driven swing for

and synchronized

optical

fiow

stirnu!i

capturing the randomdot on a board with a CCD

on a swing, White

observer's rightward motion

'

d

expansion of the

_RDP.

Ieftward motion and

vestibular

peroeption

Figure6. Illustration of the integration of

visual and vestibu]ar information for srnall

(<90

degrees)

differences

between

visual and

vestibular

inputs.

When

observers

ience

rightward vestibular stirnulation

panding optic flow, ebservers _perceive mid-way diagonaL self-motion direction,

diagonal se!f-motion, consistent with a weighted

cornbination of vjsual and vestibular

input$.

For

oricntations more than 90 degrees away

from

the

direction

of optic

flow,

observers reported the

yeridi-cal

djrection

of their

body

rnotion.

That

isthey

reported the

direction

of motion as signa]ed by the vestibular organs, discounting thevtsual input. The

results suggest that the multirnodal integration in

the _perception of se]f-motion isan either-or _process

when the

direction

difference

between

visual and

vestibular information

ls

large,

but

the

integration

is

a combination of

both

inputs

when that

difference

is

smalL

Question

about multimodal

integration

Here I address a _questioR whether there isany

common rule ofmultimodal

integration

in

two

func-tionsof perception. Ernst and Banks

(2002)

showed

the

multimodal

integration

between

visual and

hap-tic

information

isa weight ¢d cornbination of both

inputs. Their claim can be _generalized toany other

combinatien of sensory

inputs

as

far

as

it

is

the

object orevent _perception.

On

the other hand, forits generalization,

there

are not enough

data

of

psycho-physical measurements on the multimodal

integra-tion

in

self-motion

_percepti,on,

Our

data shows the possibilitythat the finding by

Ernst

and Banks

(2002)

can be generalized tothe mu]timodal

integra-tionof visual and vestibular inputs.

Conclusions

The bias towards

bouncing,

induced

by

an

audi-tory transient,persistsas theprobability ofa motion

reversal was reduced by

introductng

a spatial offset either vertically ina 2-D display or

in

depth

in

a

3-D

djsplay.

A

single auditory tone crossmodally affects

multiple visual evcnts with various temporal otfsets

between the two coincidences. Thesc rcsu]ts

demon-strate that auditory

information

has a $tronger

or-ganizing

influence

on visual _perception than

previ-ously thought and contradicts the old view that vi-sion dominates theother senses.

The

apparent _perceived

direction

of se]f-motion

is

crossmodally

integrated

from

visual and vestibular

senses when theinputs are moderately disparate,but

is an either-or _process, with the vestibular sense

dominating,

when the

inputs

are widely different.

On

two major

functions

oi perccption, a question

is

left

whether a common rule of the multimodal

inte-gration exists.

Acknowledgements

I thank all rny collaborators, Phi]ip M.

Grove,

Saka-NII-Electronic Library Service

134

The

Japanese

Journal

of

Psychonomic

Science

Vol.28,

No. I

moto,

Jiro

Gyoba,

and YOiti Suzuki. This research

was supported

by

a

Grant-in-Aid

of

MEXT

for

Spe-cially

Promoted

Research

(no,

19001004}.

References

Bcrtenthal, B

.I.

Banton, T,,& Bradbury, A.

_(1993}.

Directional bias in the perception of _translating

pattems,

Percoption,

22,

193-207.

Ernst,

M.

O.,

&

Banks,

M,

S.

_(2O02).

Humans

integrate

visual and hapticinformation ina statistically

timal

fashjen.

Nliture,

415, 429-433,

FujisakL

W,,

Shimojo.

S,,

Kashino,

M.

&

Nishida,

S.

(2004).

Recalibration of audiovisual simultaneity.

IVlatureArL?uroscience,7,773-778.

Grove,

P.M., &

Sakurai,

K,

_(2007).

EquivaLent

eamlbounce perception in cyclopean and

nance

defined

displays.

[Abstract]

lburaal

of

sion, 7(9):302,

302a,

http:1/journalofvision.org/7/

9f303L

doi:10,116717.9.302.

Grove,

P.

M.

&

Sakurai,

K.

(2009).

Auditory

induced

bounce

perception persist$as the probability of a

motion reversal

is

reduced.

Perception

advance

online

_pubLication,

doj:10,1068fp5860.

KawachL Y,,Grove, P,M., SakuraL K.,& Gyoba,

J.

(2008),

Crossmodal

effects of a single auditor}r

tone

on multiple visual events. PtarcqPtion,

ment), 27,

Sakurai,

K.

KjkuchL A.,

Kikuchi,

R,,

&

Misawa,

Y.

(2002).

Perceived

direction

of self-motion

from

ua] and vestibular sensory integration.PV,oceedings

of

theSecond Asian Conference on Vision,102,

Sakurai,

K.

KikuchL

A.

Kikuchi,

R.

&

Misawa,

Y.

(2003),

Perceived direction and distance of

mQtion frornvisual and vestibular sensory

tion.

Perception,

32CSupplemenO,

71.

Sekuler,

R.,

&

Sekuler,

A.B.

(1999).

Collisions

tween moving visual targets:what controls

native ways of seeing an ambiguous display?

cePtion,

28,

415-32.

Sekuler,

R.,

Sekuler,

A.

B.,

&

I.au,

R,

C1997).

Sound

alters visual motion _perception.Ndture,385, 308.