The effects of stimulus strength on the timing of audiovisual multisensory facilitation(Spatio-temporal integration of multimodal sensations,Symposium 2 at the 27th Annual Meeting)

7'he

ldpanese

,Journal

oj'Rs,vchonomic

Science

2oog,

VoL

28,

Ne.

].,

1.z3-129

Lecture

Theeffects

of

stimulus

strength

audiovisual

multisensory

on

the

timing

of

facilitation

Mark

E.

McCouRT*and

NOrth

Dakota

State

Lynnette

LEoNE*

[Jdaiversity*

Four

experiments

examined

tcmporal

_properties

of

audiovisual multisensory

integration.

Experiment

1

measured reaction

time

to

100

ms

auditory

(A)

and

visual

(V)

stimulL

and

te

audiovisua]

(AV)

combtnations with stimulus

onset

asynchronles

_(SOAs)

ranging

frorn

-100

to

+200

ms.

Significant

violations

of

Miller's

inequality

(signjfying

neural coacttvation)

occurred

only

fer

simultaneous

presentation

(AV

SOA=-Oms).

Experiments

2

and

3

were

identieal

to

Experiment

1

with

the

following

exceptions.

In

Experiment

2

auditory

stimulus

intensity

was

adjusted

to

clamp

performance

in

auditory-only

trials

at a

d'=2,

while visua] stimulus contrast was

clearly

suprathreshold

(d'>4).

The

results

indicated

that

neural

coactivation

occurred

over

an

expanded

range

of

AV

SOAs

from

-60

to

O

ms.

In

Experiment

3

visual stimulus contrast was adjusted

to

c]amp

performance

in

visual-only

trials

at

a

d'=2,

while

auditory

stimulus

intensity

was

clearly

suprathreshold

(d'>4).

Neural

coactivation

in

thjs

case

also

occurred

over

an

expanded

range

of

AV

SOAs,

frorn

O

to

+60

rns.

In

Experiment

4

the

intensity

of

both

A

and

V

stimuli

was adjusted

to

clamp

performance

in

unisensory

trials

at

a

d'=2.

As

in

Experirnent

1,

neural coactivation occurred only

for

simultaneous

AV

_presentation

(AV

SOA.='O

ms).

These

results

have

jmplications

for

early

multisensory

processing,

the

rele

of

attentjon,

and

the

general-ity

of

the

inverse

effectiveness rule.

Key

words: multisensory

facilitatlon,coactivation,inverse

effectiveness, rcaction

time,

d'

Multisensory

integration

(MI)

refers

to

the

process

by

which convergence of

information

from

two

or

more

individual

sensory

systems

onto

particular

neurens resu]ts

in

a neuronal response

that

is

quali-tatively

and

_{quantitatively}

different

than

the

re-sponses

of

these

neurons

to

individual

unisensory

signais

(Calvert,

2001

).

According

to

Meredith

<2002)

di,fferent

types

of responses

can

result

from

neuronal convergence,

Depending

on stimulus

_properties

such

as

spatial

and

temporal

coincidence,

the

out-come

ot

multisensory

convergence

can

be

either re-sponse

facilitation

or

suppression.

Stimuli

that

are

closely

aligned

in

space

and

time

lead

to

response

*

_{Cerrespondence}

Center

for

Visual

Neuroscience,

Department

of

Psychology,

NDSU

Department

2765,

PO

Box

6050,

College

of

Science

and

Mathematics.

North

Dakota

State

University,

Fargo,

ND

58108-6050,

U.S.A.

TEL:

(701)

231-8625,

FAX]

(701)

231-8426

E-mail:

[email protected]

facilitation

while

those

that

are

temporal]y

and

spa-tially

disparate

may result

in

response suppression

(Meredith

&

Stein,

l986;

Meredith,

Nemitz,

&

Stein,

1987;

Meredith

&

Al]man,

2008).

Response

facj],ita-tion

to

multisensory

convergence

is

especially

robust

when

the

unisensory

stimuli

themselves

produce

weak

(near

thresho]d)

responses.

This

_phenomenon,

conspicuous

i,n

the

responses of multisensory neu-rons of

the

superior colliculus,

has

been

termed

the

"inverse

effectiveness

rule"

(Meredith

&

Stein,

1986).

Perhaps

the

rnost

basic

example of

facilitative

MI

is

the

modulati,on

of

reaction

time

(RT)

to

_pairings

of

sensory

stirnu]i

presented

over

multiple

sensory

channels,

where

RTs

are

faster

than

those

predicted

by

the

optimal statistical combination of responses

to

the

unisensory signals,

This

enhancement

in

the

speed

of

_processing

has

been

termed

the

"redundant

signals effect", or

RSE

(MMer,

1982).

Miller

(1982)

compared

two

different

modcls

that

cou!d

poten-tially

account

for

the

RSE:

1)

separate

activation

or

"race"

_mode]s;

_and

₂₎

_neural

_coactivation

models.

124

TheJapanese

Journal

of

Psychonomic

Science

VoL28,

No.

1

Race

models

assume

that

each red'undant signa]

is

processed

independently.

On

a

_given

trial

the

chan-nel

that

_processcs

the

stimulus most

quickly,

and

thus

initiates

the

overt response, "wins"

_the

race.

In

race models redundancy

_gains

result

from

statistical

probability

summation and no neural

interaction

be-tween

the

activated

sensory

channels

is

requiredi,

In

contrast, neural coactivation models

_posit

that

signals

from

different

channels

interact

in

some

fash-ion

to

facilitate

the

response.

Coactivation

by

the

separate

signals

builds

until a response criterion

has

becn

satisfied.

This

can occur

before

the

criterion

would

be

reached

by

activation

from

either

individ-ual

signal,

Miller

C1982)

developed

a

mathematical

inequalaty,

which,

according

to

race

rnodels,

_places

a

ceiling

on

the

_probability

of obtaining responses

to

redundant

stimu".

This

equation specifies

that

the

_probabiLity

of obtaining

the

fastest

responses

to

redundant

sig-nals must

be

less

than

or equal

to

the

_prebability

of

obtaining

the

fastest

responses

to

_individual

stimuli.

T,

he

inequality

states,

for

a

_given

pair

ef

stimulL

Le.

auditory

(A)

and visual

(V),

at a

_given

response

Ia-tency

<t),

that:

P<RT<tlA

and

V)S[P(RT<tlA>+

P(R

T<

t

1

V>]-

_[P(RT

<

t

1

A)"

P(RT<

ti

V)]

where

P(RT

<tlA

and

VD

is

the

_probability

of obtaining an

RT

Iess

than

time

(t>

in

response

to

the

auditory

and

visual stimulus.

This

_probability

must

be

less

than

or equal

to

the

sum of

the

_{probabilities}

of

obtaining

reaction

times

less

than

time

(t)

in

re$ponse

to

the

individual

stimulL

P(RT<tlA)+P(RT<tlV),

taking

into

account

their

joint

probability,

P(RT<tiA)*

P

(RT<t[V).

Violations

of

the

inequalit}r

indicate

that

some

neural

interaction

beyond

mere

probability

surnmation

i,s

occurring,

which

implies

neural coacti-vation.

Both

channels contribute

to

the

satisfaction of

the

response criterlon

se

responses

to

redundant

signals will

be

faster

than

those

_predicted

by

statisti-cal

facilitation,

Both

coactivation

and

the

inverse

effectiveness

rule

can

be

evaluated

by

investigating

the

temporal

properties

of

the

RSE

under

conditions

in

which

the

speed of response

to

either

the

auditory

or

visual

signal

is

manipulated.

Psychophysically,

higher

in-tensity

stimuli are

generatty

processed

rnere rapidly

than

low

intensity

stimuli, as

indexed

using

both

reaction

tirme

and

electrophysio]ogical

measures,

For

example,

increasing

the

eontrast of a visual stimulus, or

the

loudness

of an auditory stimulus,

leads

to

decreases

in

reaction

time,

a

_phenomenon

known

as

Pteron's

law

(Pjeron,

1952;

Mansfield,

l973;

JaSkowski,

l985).

The

_present

work

investigates

whether

the

reZative

_processing

speed

of

twe

sensory

channels

determines

the

optimal

SOA

for

neural co-activation.

If

the

processing

speed

in

one sensory channel

is

increased

relative

to

another

(in

response

to

a

stronger

stimulus,

for

example),

we

_predicted

that

the

optirnal

SOA

for

neural coactivation would shift such

that

the

weaker

(slower)

stimulus musL

be

presented

earller

than

the

stronger

{faster)

stimulus,

This

_predietion

is

based

on

the

suggestion

that

the

RSE

is

the

result

of

integration

of

signals

at

the

neuronal

leve],

We

hypothesized

that

information

coming

from

the

separate

sensory

channels

would

need

to

arrive at

those

brain

regions

performing

multisensory

integration

at roughly

the

same

time

in

order

for

facilitation

to

occur.

Therefore,

the

stimu-lus

that

is

processed

more

s]owly

would

need

more

time

to

reach

thcse

multisensoTy

areas,

hence

it

would need

to

be

_presented

earlier.

We

conducted an experiment with

four

conditions

designed

to

address

the

foLlowing

objectives:

1)

To

assess

the

range of stimulus onset asynchrony

(SOA)

over

which

neuronal

coactivation

occurs

when

both

unisensory stimuli

are

relatively

weak;

2>

To

evalu-ate

whether

the

range of

SOAs

_producing

neuronal

coactivation

depends

upon

the

relative

speed

of

proc-essing unisensory signals;

3)

To

evaluate

whether

neuronal coactivation

depends

upon uniscnsory stimu]us

intensity,

as

predicted

by

the

inversc

ef-fectiveness

rule; and

4)

To

examine

the

role of

at-tention

in

multisensory

_processing.

Condition

1:

We

asked

the

question:

What

is

the

range of audiovisual

SOA

over which neural coacti-vation exists when

both

unisensery stimuli are

rela-tively

weak

(objective

1).

Subjects

were

n=7

(4

male;

mean age=30

_years)

al] of whom

had

normal

(or

corrected

to

normal) viston

and

nerma]

hearing.

The

workdescribed

here

M.

E.

McCovRT

and

L.LEoNE:

Audiovisual

multisensory

faci]itat,ion

125

was carried out

in

accordance wtth

The

Code

of

Ethics

of

the

World

Medical

Association

_{Declaration

of

HelsinkO

for

experiments

involving

humans.

Prior

to

their

particjpation

in

the

study aU

partici-pants

provided

written

informed

consent.

Visual

stimuli

were

circular

Gabor

patches

of

low

(l

cfd) spatial

frequency

and variable contrast with a

Gaussian

standard

deviation==

1

degree,

centered

at

2.25

degrees

eccentricity

from

fixation

in

the

upper

left

quadrant

of

the

viewing

area.

Stimuli

were

presented

on

a

CRT

with

mean

luminance=-=-60.89

cdfm2 and refresb rate of

1OO

Hz.

Individual

subjects

contrast

settings

were

between

1-4%

contrast

for

all

conditions.

Visual

stimulus

duration

was

100

rns.

The

auditory stimulus was a

1kHz

_pure

tone

of

variable

]oudness

(range=31.l-49.0

dB),

presented

via

a

speaker

approximately

co-located

with

the

vis-ual

stimuli.

Auditory

stimulus

duration

was

100

ms,

The

experimental

paradigm

was

single-interval

sig-nal

detection.

Participant's

task

was

to

respond via

button

press

as

_quickly

and

accurately

as

_possible

to

the

detection

of any stimulus.

Subjects

_performed

34

blecks

of

_tria]s.

Blocks

centained

9

unisensory

vis-ual

stimuli,

9

unisensory

auditory stimuli,

9

catch

trials,

and

3

each

of

audiovisual multisensory combi-nations at

SOAs

ranging

from

-

1OO

ms

to

200

ms

(in

20

ms

increments)

for

a

total

of

75

trials

per

block,

Negative

values

indicated

that

the

auditory

stimulus occurred

_prior

to

the

visual stimulus.

Catch

trials

were

trials

on

which

no

stimu]us

was

_presented.

In

order

to

ensure criterion response

(d'='L2)

was

main-tained,d'

was

ealculated

for

both

auditory

and

visual

unisensory

conditions

following

the

completion

of

6

trials

per

condition

and

unisensory stlmulus

inten-sity

levels

were

adjusted.

Additionally,

cumulative

d'

was

calculated

across

SOA

for

each

subject

and

for

the

_group.

Reaction

time

(RT}

was recorded

to

the

nearest mi]lisecond and outliers

(RT<

100

rns or

>1OOO

ms)

were

removed.

Independent

samples

t-tests

com-pared

RT

in

each

AV

stimulus

condition

with

RT

in

each unisensory

stirnu,Ius

condition.

Data

were

sub-jected

to

bootstrapping

analysis

(Fe$ter

&

Bjschof,

1991)

of

1000

iteration$

where each

iteTation

resam-pled

the

data

at each

SOA

{with

replacement).

This

process

yielded

a

distribution

of

cumulative

prob-ability

density

function

(CDF)

curves whose

stan-dard

deviation

was

used

to

compute

95%

confidence

intervals

for

subsequent

calculations

of

Miller's

in-equality.

For

condition

1,

sensitivity

increases

were

com-puted

for

multisensDry

conditions.

Significant

in-creases

were

considered

to

be

those

that

exceeded

simple

quadratic

summation

or

the

expected

in-crease

in

sensitivity

to

the

_presentation

of

two

sig-nals

as

compared

to

sensitivity

to

the

presentation

oi

one

signal

(Campbell

&

Green,

1965;

Legge,

1984).

Response

enhancement

versus

response

szipPression:

Despite

having

been

rnatched

for

detectability,

mean

RT

to

the

unisensory

auditory

stimulus

(421.8

ms)

was

significantly

faster

than

to

the

unisensory

visual

stimulus

{432.8

ms),

t(3151)=-=･--2.45,P=--O.Ol.

There-fore,

in

order

to

assess

whether

significant

RT

facili-tation

occurred

in

multisensory conditions,

RT

to

AV

stimuli

at

all

SOAs

was

compared with

RT

in

the

auditory unisensory stimulus condition.

A

series of

independent-sarnples

t-tests

compared mean

RTs

in

the

AV

stirnulus

conditions

to

RT

to

the

unisensory

auditory stimulus.

Mean

RT

in

AV

conditions was significantly

faster

than

mean

RT

to

the

fastest

uni-sensory

stimuLus

(A)

over

a

range

of

SOAs(-

80

ms

to

40

ms)

indicating

rnultisensory response

facilita-tion

at

those

SOAs.

In

order

to

assess

whether significant

RT

suppres-sion occurred

in

mu]tisensory conditions,

RT

to

AV

stimuli

at

alL

SOAs

was

compared

with

RT

in

the

visual unisensory

stimu]us

condition.

Independent-sarnples

t-tests

cornparing

RT

to

AV

stimu]us

combi-nations

with

RT

_to

_the

unisensory

visual

stimulus

showed

that

mean

RT

to

AV

combinations was sig-nificantly

slower

than

mean

RT

to

the

slower

unisen-sory

stimulus

(V)

over

a

range

of

SOA

(140

rns

to

200

ms)

indicating

multisensory response suppression.

Multisenso7oJ

integration:

Exarninatlon

of

the

mean values of

the

bootstrapped

Mil]er's

inequalities

as

a

function

of

RT

and

AV

SOA

demonstrated

that

neu-ral coactivation, as

indicated

by

significant

126

The

Japanese

Journal

of

Psychonomic

Science

Vol.28,

No,

1

an

SOA

ef

O

rns

<simultanelty).

The

d'

value

pre-dicted

based

on

the

optimal

statistical

combination

of

unisensory

_perforrnance

according

to

_quadratic

summatien

was

d'=-r2.76.

Comparison

of

this

pre-dicted

va}ue

with observed sensitivities

to

AV

stim-uli

indicated

no

significant

gains

in

multisensory

sensitivity

for

any

SOA.

Condition

2;

This

condition

addressed

the

effect

of

visual

stimulus

intensity

on

the

audiovisual

RSE,

We

evaluated

the

experimenta!

_question:

Does

an

increa$e

in

visual

stimutus

intensity

lead

to

changes

in

the

time-course

of

the

RSE?

The

_predietion

was

that

by

increasing

visual

stimulus

intensity

(and

thereby

decreasing

RT

to

unisensory visual stirnuli),

the

optimal

SOA

for

neural

coactivation

would

shift

such

that

the

auditorv

stimulus

would

have

to

occur

'

before

the

visual stimu]us

(objective

2),

The

same subjects who

_particapated

in

condition

1

also

participated

in

condition

2,

The

experimental

paradigm

replicated condition

1

with

the

following

exceptions.

Visual

stimulus contrast was

increased

to

intenslties

that

produced

criterion

response of

d'>4

for

each

_participant.

In

order

to

ensure

crUe-rion

response,

d'

was

calculated

for

unisensory

audi-tory

and visual stimuli every

6

trials

throughout

the

experiment

and

stimulus

intensity

level

was

ad-justed.

Response

enhancement

vexsus

response

suPPression:

As

expeeted,

relative

to

condition

1,

mean

RT

to

the

high

contrast

visual

stimulus

{333,4ms}

decreased

by

99

ms,

t{3679)

r=29.9,

P<O.OO1.

In

order

to

assess whether significant

RT

facilitation

occurred

in

multi-sensory conditions,

RT

to

AV

stjmuli' at al]

SOAs

was compared with

RT

in

the

visual unisensory stimulus

condition.

The

results

of

independent-samples

t-tests

cornparing

RT

to

AV

stimulus

combinations

with

RT

to

the

unisensory

visual

stimulus

for

the

combined

group

data

showed

that

mean

RT

to

AV

combina-tions

was significantly

faster

than

mean

RT

to

the

fastest

unisensory stimulus

(V)

over a range of

SOA

(OOms

to

40

ms)

indicating

rnu]tisensory response

facilitation

at

those

SOAs.

In

order

te

assess whether significant

RT

suppres-sion occurred

in

mu]tisensory conditions,

RT

_to

AV

stimuli at a]1

SOAs

was

compared

with

RT

in

the

auditory

unisensory stimulus condition.

Indepen-dent-samples

t-tests

comparing

RT

to

AV

stimulus

combinations with

RT

to

the

auditory stimulus

(451.4

rns)

demonstrated

that

mean

RT

to

AV

combi-nations was

faster

than

mean

RT

_to

_the

auditory

stimulus

over

the

entire range of

SOA

indicating

multisensory

response suppression

did

not occur

in

this

condition,

A42tltisenso7],

integralion:

Analysis

of

_group

data

indicated

that

neura],

coactivation

(as

demonstrated

by

significant

violations of

Miller's

Inequaljty)

oc-curred

across

the

range

of

AV

SOAs

from

O

ms

to

60

ms.

These

results

were

opposite

to

the

predicted

direction.

The

results

are

also

jnconsistent

with

the

inverse

effectiveness

rule,

since

increases

in

stimulus

intensity

led

to

increases

in

ncura] coactivation

(de-fined

by

the

wtdening range of

SOAs

over

which

violations

occurred).

Condition

3:

Here

we

addressed

the

effect

of

audi-tory

stimulus

intensity

on

the

audiovisual

RSE,

Spe-cifical]y, we

investigated

the

e'ffect of an

increase

in

auditor), stimulus

intensity

on

the

time-course

ef

the

RSE

(objective

2).

The

prediction

was

that

b}r

in-creasing auditory stimulus

intensity

(and

thereby

decreasing

RT

to

unisensory auditory stimuli),

the

SOA

for

neural

coactivation

would shift such

that

the

visual stimulus would

have

to

occur

before

the

auditory

stimulus.

The

experimental

paradigm

repli-cated condi.tion

1

with

the

following

exceptions:

In-tensity

oi

the

auditory

stimulus

was

increased

to

intensities

that

_produced

response of

d'

>4

_Untensity

range=49.0-･

44.5

dB>.

Response

Enhancement

ve7:sus response suPPression:

As

expected,

relative

to

condition

1,

mean

RT

to

a

higher

intensity

audiitory stimulus

(307,Ims)

de-creased

by

115

ms,t(3091>=

-27,1,

P<O.Ol.

In

erder

to

assess whether significant

RT

facilitation

oc-curred

in

multisensory

conditions,

independent-samples

Vtests

compared

RT

to

AV

stimuli

at

all

M.

E.

McCouRT

and

LLEoNE:

Audiovisua]

rnultisensory

facilitation

127

condition.

Mean

RT

to

AV

combinatiolls was

sig-nificantlv

faster

than

mean

RT

to

the

fastest

unisen-sory

stimulus

_(A)

at

-60

ms,

-20

ms

and

simultane-ous

indicatlng

multisensory response

facilitation

at

those

SOAs.

Independent-sarnples

t-tests

also a$sessed whether

significant

RT

suppression

occurred

in

multisensory

conditions.

RT

to

AV

stimuli

at

all

SOAs

was

com-pared

with

RT

in

the

visual

unisensory

condition.

Mean

RT

to

AV

combinations

was

faster

than

mean

RT

to

the

visual stirnulus

(446.9

rns) over

tlte

entire range

of

SOA

tested

indicating

rnultisensory

re-sponse

suppression

did

not

occur

in

thj,s

condition.

Mutcisensor:y

integration:

Analysis

of

_group

data

indicated

that

neural coactivation

(as

demonstrated

by

violations of

Miller's

Inequality)

occurred across

the

range

of

AV

SOAs

from

-60

ms

to

O

ms.

Again,

the

shift

in

AV

SOA

for

neural eoactivation

that

occurred was opposite

to

the

hypothesized

direction,

These

results are

also

inconsistent

with

the

inverse

effectjveness rule, since

increases

in

stimulus

inten-sity

ied

to

increases

in

neural

coactivation

(defined

by

the

widening range of

SOAs

over which

viola-tions

occurred).

An

important

question

to

consider

in

light

of

the

results of conditions

2

and

3

is

why

does

the

range of

optimal

SOAs

for

neural

coactivation

shift

toward

SOAs

at

which

the

stronger

stimulus

comes

earlier

(rather

than

later,

as

hypothesized)?

A

_pessible

ex-planation

for

these

results

could

be

that

the

stronger

stimulus

is

acting

as

an

exogenous

cue

which

facili-tates

_processing

of

the

weaker stimulus.

A

closer

Iook

at

the

roie

of

attention

was

needed.

Condition

4:

This

condition

further

considered

the

effect of

stimulus

strength on

the

audiovisual

RSE

in

order

to

identify

the

role of attention

in

multisensory

faciLitation

(objective

4).

We

asked whether an

in-crease

in

both

visual stimulus contrast and auditory stimulus

intensity

would

lead

to

changes

in

the

time-course of

the

RSE.

If

attentional contributions can explain

the

results

of

conditions

2

and

3,

then

the

prediction

was

_that

increasing

strength

of

both

uni-sensory

stimu]i

would

increase

the

range

of

SOAs

at

which

neural

coactivation

occurred

to

include

all

SOAs

at

which

neural

coactivation

was

found

in

both

conditions

2

and

3.

The

experimental

_paradigm

replicated condition

1

with

the

foLlowing

exception:

Both

the

visual

and

the

auditery

st!mulus

were

in-creased

in

intensity

to

values

that

produced

response

accuracies

of

d'

>4,

ResPonse

enhancement

versus

response

szaPPression:

As

expected,

relative

to

condition

1,

mean

RT

to

higher

intensity

auditory stirnulus

decreased

by

135

ms,

t<2783)

=:

30.8,

P<O.Ol.

Alse

_{as expected, relative}

to

condition

1,

mean

RT

to

higher

intensity

visual

stimu]us

decreased

by

l16

ms,

t(2770)=28.8,P<

O.Ol.

Mean

reaction

tirne

te

the

visual stimulus

(316

ms) was significantly slower

than

mean reaction

time

to

the

auditory

stimu]us

(286ms),

t(2402)=8,74,

P<

O.Ol.

Thereiore,

for

this

condition,

in

order

to

assess

whether significant

RT

facilitation

occurred

in

rnulti-sensory conditions,

RT

to

AV

stimuLi

at

a]1

SOAs

was

compared with

RT

in

the

unisensory auditory

condi-tion.

Independent-samples

t-tests

comparing

RT

to

AV

stimulus

combinations with

RT

tQ

the

fastest

unisensory

<A)

stimulus showed mean

RT

to

AV

combinations

was

significantly

faster

than

mean

RT

to

the

auditory stimulus at

SOAs

-80

rns,

-40

ms,

-20

_{ms, and}

_O

_ms

_indicating

_{multisensory response}

facilitation

at

those

SOAs.

In

order

to

assess whether significant

RT

suppres-sion

occurred

in

multisensory

conditions,

RT

to

AV

stimuLi

at

all

SOAs

was

compared

wjth

RT

in

the

unisensory

visual

condition.

The

results oi

_the

inde-pendent-samples

t-tests

comparing

RT

to

AV

stimu-lus

combinations

with

RT

to

the

unisensory visual stimulus

indicatecl

that

mean

RT

to

AV

combina-tions

was

equal

to

or

faster

than

mean

RT

Lo

the

slower unisensory stimulus

(V)

over

the

entire range

of

SOA

indicating

multisensory response

suppres-sion

did

not occur

in

this

condition.

Multisensory

integration:

Ana]ysis

of

_group

data

of

the

mean value of

the

bootstrapped

MMer's

inequal-ity

as

a

function

of

RT

and

AV

SOA,

indicated

that

optimal

AV

SOA

for

neurai

coactivation,

as

indicated

128

_The

_Japanese

_journal

of

Psychonomic

Science

VoL

28,

No.

1

occurred

only

at

SOA=O

($imultaneous

presenta-tion).

Discussion

The

results

of

conclition

1

indicate

that

for

rela-tively

weak unisen$ory

stimuli,

physical

simultane-ity

of

stimuli

is

critical

for

facilitative

multisensory

integration

with

respect

to

RT.

Although

mean

RT

to

multisensory stimuli was

significantly

faster

than

to

the

fastest

unisensory

stirnulus

over

a

range

of

SOAs,

this

facilitation

did

not exceed

the

increase

predicted

by

statistical

faciljtation

exccpt

when

the

A

and

V

stimuli were

prcsented

simultaneously.

This

result

supports

the

temporat

rage

of

multisen-sory

integration

which

suggests

that

_greater

facilita-tion

occurs

when stimuli

occur

at approximately

_the

same

time

(Stein

&

Meredith,

1993).

When

the

strength of

the

visua] stimulus

is

increased,

as

in

condition

2,

and visual

RT

is

significantly

decrcased

relative

to

auditory

RT,

the

range

of

optimal

SOAs

for

facilitative

multisensory

integration

(as

evi-denced

by

increases

in

RT

that

exceed

the

increase

predjcted

by

statistical

facilitatiQn)

undergoes

two

changes:

it

cxpands significant]y

to

encompass

a

larger

range

of

SOA,

and shifts

toward

SOAs

for

which

the

visual stimulus occurs

first.

Condition

3

shows

that

when

the

strength

of

the

auditory

stimu-lus

is

increased

and auditory

RT

is

significantly

faster,

the

rangc of optimal

SOAs

for

facilitative

mu}tisensory

integration

al$o

undergoes

two

changes:

it

expands

significantly

to

encompass a

larger

range

of

SOA,

and

shifts

toward

SOAs

for

which

the

auditory

stimulus

occurs

first.

For

combi-natiens of relatively

higher

strength unisensory stimuli,

as

in

condition

4,

simultaneity

is

also

essen-tial

for

facilitative

multisensory

integration

with

re-spect

to

RT,

There

is

a surprising

lack

of eiiidence

for

multisensory

facilitation

with

respec"o

sensitiv-ity

(d').

The

results of condition

1

when compared with aLI other conditions showed

that

multisensory suppres-sion occurred only

in

this

condition and only

for

cornbinations of weaker stimuli at very

long

SOAs,

One

explanation

for

this

could

be

that

the

modula-tory

effects

of multisensory neurons may

be

_greater

when

stimuli

have

weaker

effects

on

unisensory

sys-tems,

It

may

be

the

case

that

when

the

incoming

signals

from

unisensory systerns are

weaker,

the

sup-pressive

effects

of

multisensory

neurona]

conver-gence

are more apparent,

Interestingly,

the

mean

RT

to

the

unjsensory

audi-tory

stimulus

in

condition

2

was significantiy

longer

in

latency

than

in

condition

1,

despite

the

fact

that

intensity

was

adjusted

to

clamp

sensitivity

to

a

d'=

2

in

both

cases.

Additionally,

the

mean

RT

to

the

unisensory visual stimuLus

in

condition

3

was

sig-njficantly

longer

in

latency

than

in

condition

1,

de-spite

the

fact

that

intensity

was

adjusted

to

clamp

sensitivity

to

a

d'

J-

2

in

both

of

these

cases

as

well.

Given

that

the

extent

of

multisensory

facilitation

in

conditions

2

and

3

is

greater

than

in

condition

1,

one

suggestion

is

that

the

enhanced

salience

of

the

high

intensity

unisensory stimulus may

have

exerted a suppressive

effect

on responses

_to

weaker

unisen-sory stimuli.

One

interpretation

of

these

results

considers

that

relative

_timing

of

the

individual

sensory

systems may not

be

responsible

for

facilitation.

Instead,

the

rnultisensory

stimuli

in

conditions

2

and

3

(a

combi-nation of a weaker stimulus with a rnore

intense

stirnu]us) represent an

intermediate

level

of

totat

stimulus

intensity

between

combinations

of weak

stimu]i and sLrong stimu!i.

As

mentioRed

previously,

numerous studies seem

to

support a

linear

increase

in

facilitation

with

decreases

in

stimulus

intensities.

Many

of

these

studies,

however

(e.g.

Callan

et

aL,

2001:

Frassinetti

et

aL

2002;

Diederich

&

Colonius,

2004:

Serino

et

aL

2007)

may

not

have

presented

a

sufficient range of stimulus

intensities

to

idicntify

where maximal

facilitation

is

likely

to

occur under

different

stimulus

conditions.

Even

studies

that

do

present

a range of stimulus

intensities

and claim

their

results

"generalLy''

follow

the

rule

(Lakotas

et

al.

2007)

show

the

greatest

facilitation

at

interrnedi-ate

levels

of stimulus

intensity.

Ross

et

aL

(2006)

propose

that

perhaps

there

ts

an

intermediate

"zene"

of

maximal

facilitation

between

stimulus extrernes.

They

suggest

that

higher

leve]

cognitive

_processes

such as speech

_perception

may

stimu-M,

E,

McCouRT

and

L.

LEoNE:

AudlovlsuaI

multisensory

facilitation

129

lus

intensity

because

they

require

a

minimum amount of stimulus

input

in

order

for

recognition

to

occur and

that

these

intermediate

levels

of

stirnulus

intensity

encompass

this

minimum requirement.

Our

results support

the

idea

of maximal multisen-sory

faci]itation

at

intermediate,levels

of

stimulus

intensity,

The

three

types

of combinations of

unisen-sory stimuli

(weakfweak,

weakfstrong, and strongf

strong)

represent

the

three

levels

of mu]tisensory stimulus

intensity,

These

results

indicate

that

the

greatest

]evel

of

facilitation

occurs

at

intermediate

levels

of stimulus

intensity.

Additionally,

our results

suggest

that

this

maximal zone of

facilitation

in

responses

includes

earlier

signal

detection

processes,

Referenees

Callan,

E.,

Callan,

A.,

Kroos,

C.,

&

Vatikiotis-Bateson,

E,

(2001).

Multimodal

contribution

to

speech

ception revealed

by

independent

component

lysis:

A

single

sweep

EEG

case

study.

CQgnitive

Brain

Research,

10,

349-353.

Calvert

G.

{2001),

Crossmoda!

processing

in

the

man

brain:

Tnsights

from

functional

neuroimaging

studies,

Cerebral

Cortex,

11,

111O-1123.

Camp6ell,

F.

&

Green,

D.

(1965).

Monocular

vs.

ocular visua] acuity,

Nbeture,

208,

191-192.

Diederich,

A.,

&

Colonius,

H.

_(2004),

Birnodal

and

trimodal

multisensary enhancement:

Effects

of

mulus

onset

and

intensity

on

reaction

time.

tion

and

Rsychophpusics,

66,

1388-1404.

Frassinetti,

F.

Bo]ognini,

N.

&

Ladavas,

E,

_<2002),

Enhancement

of vlsual

_perception

by

crossmodal

visuo-auditory

interaction.

Experimentat

Btzzin

Re-search,

147,

332-343.

JaSkowski,

P,

_(1985).

The

effect

of

visual adaptation

on simple motor reaction

time:

Pt,

1.

Studia

tagica,

27,

191-201,

Lakotas,

P,,

Chen,

C.

O"ConnelL

M.,

Mills,

A.

&

eder,

C.

(2007).

Neurona]

oscillations

and

sory

interaction

in

_primary

auditory cortex.

ron,

53,

279-292.

Legge,

G.

_{1984).

Binocular

contrast

summation

II.

Quadratic

summation,

Vision

Research,

24,

394.Mansfield,

R.

_J.

W,

(1973).

Latency

function

in

hurnan

vision,

Vision

Research,

13,

2219--2234.

Meredith,

A.

_(2002).

On

the

neuronal

basi$

for

sensory convergence:

A

brief

overview.

Cognitive

Brain

Research,

14,

1-40,

Meredith,

A.,

&

Allman

B.

{2008).

Subthreshold

essing

in

cat auditory

certex.

AlaurorePort

(in

_press),

Meredith,

A.,

Nemitz,

_J.

&

Stein,

B.

_(1987).

nants

of

multisensory

integration

in

superior

liculus

neurons.

I.

Temporai

factors.

foumal

of

Alleu-roscience,

7,

3215-3229.

Meredith,

A.

&

Stein,

B,

(1986).

Visuai

auditory

and

sornatosensory convergence on cells

in

superior

colliculus results

in

multisensory

integration.

Iburnat

of

Nburoscience,

56,

640-662.

Miller,

J.

(1982).

Divided

attention:

Evidence

for

tivation

with redundant

signats.

CQgnitive

cLg]J,

14,

247-279.

Pi6ron,

H.

_(1952).

The

sensations:

71heirjunctions,

cesses

and

mechanisms.

London:

Frederick

Muller

Ltd,Ross,

L,,

Saint-Amour,

D.

Leavitt,

V.

Javitt,

D.

&

Foxe,

J.

(2007).

Do

you

see what

I"m

saying?

Exploring

visual

enhancernent

of

speech

hension

in

noisy environments.

Cerebral

Coriex,

17,

1147-1153.

Serino,

A.

Farne,

A,,

Rinaldesi,

M.,

Haggard,

P.

&

Ladavas,

E.

{2007).

Can

vision

of

the

body

rate

impaired

somatosensory

function?

chologia,

45,

1101-1107,

Stein,

B.,

&

Meredith,

A,

_(1993).

Ihe

merging

of

the

senses.

Cambridge:

MIT

Press.

Author

Disclosure

Role

of

funding

source:

This

publication

was made

_po$sible

by

following

grants:

NIH

P20

RR020151

(MEM)

and

EPS-O132289

(MEM),

The

National

Center

for

Research

Resources

(NCRR)

is

a component of

the

National

Institutes

of

Health

(NIH),

EPSCOR

(EPS)

is

a

divisi,on

of

the

National

Science

Founclation.

The

contents

of

this

report are solely

the

responsibility

of

the

authors

and

do

not necessari]y reflect

the

official views of

the

NIH,

NCRR,

or

NSF.

These

agencies

had

no

role

in

study

design;

in

the

co]lection,

analysis

and

interpre-tation

of

data;

in

the

writing of

the

report; and

in

the

decision

to

submit

the

_paper

for

_publication.

Acknowledgments

The

authors

thank

Huanzhong

(Dan)

Gu

for

pro-grarnming

the

experiments and

Dr,

Wolfgang

Teder-Sa}ejarvi

for

technical

advice.

Commcrcial

relatjonships: