DOI: http://doi.org/10.14947/psychono.35.14
Functional MRI studies of multisensory integration
underlying self-motion perception
Mark W. Greenlee
Institute for Experimental Psychology, University of Regensburg
Since the advent of functional magnetic resonance imaging cognitive science has experienced a turn towards neuroscience. Models of perceptual and cognitive functions can now be tested against patterns of human brain activ-ity in anatomically well-defined regions of interest. Structural and functional connectivactiv-ity analyses can inform us about how different brain regions are interconnected and interact in perceptual and cognitive tasks, as well as during resting states. In this study I review the results of a series of experiments that aimed to reveal the visual-vestibular sensory processing underlying self-motion perception. We (Frank, Baumann, Mattingley, & Greenlee, 2014; Frank, Wirth, & Greenlee, 2016) localized regions in the posterior insula using fMRI with visual and vestibular stimuli. The results suggest that two areas in this part of the brain are involved in self-motion perception: the parieto-insular ves-tibular cortex (PIVC) for the processing of vesves-tibular information and posterior insular cortex (PIC) for the integra-tion of visual and vestibular informaintegra-tion. The results suggest that these two regions play different roles in the inte-gration of visual and vestibular cues related to self-motion perception.
Keywords: vestibular cortex, caloric vestibular stimulation, functional MRI, self-motion perception Introduction
We experience our environment via a continuous exchange between our different sense modalities in order to optimally plan and execute behavioral responses to the source of these sensory signals (Ghazanfar & Schroeder, 2006). Frequently, signals from the environment can be weak, noisy and ambigu-ous (e.g., an approaching vehicle in the fog). Multisensory in-tegration of such weak sensory signals enhances our ability to integrate the available information by adjusting the weights as-signed to each sense according to the ongoing variance in the signal (Ernst & Banks, 2002). By these means, we can draw valid conclusions about the nature of the objects that led to these sensory signals. Multisensory integration guides our be-havior to optimally interact with stationary and moving ob-jects in our environment, even in circumstances when we our-selves are in motion (Berthoz & Viaud-Delmon, 1999). Such complex processing of sensory information from the different sense modalities requires efficient integration to be able to re-spond quickly to the demands placed on us by our environ-ment (Mergner, Maurer, & Peterka, 2003). Indeed, impairenviron-ment
in any one of these sense systems can lead to maladaptation and thus suboptimal behavior (van der Kooij, Jacob, Koop-man, & Van der Helm, 2001).
Self-motion perception involves the integration of sensory signals arising from the visual, vestibular, somatosensory and motor systems (Bremmer, 2011; Britten 2008; Greenlee, 2000; Greenlee et al., 2016; Lappe, Bremmer, & van den Berg, 1999). Image motion on the retina is, by definition, ambiguous with respect to the source of this motion. To separate retinal image motion evoked by self motion from image motion arising from movements of objects in the immediate environment, the brain needs to compare signals arising from head/body movements from signals related to objects that move indepen-dently of the observer. This problem is related to the well known efference copy model (von der Holst & Mittelstaedt, 1950; Sperry, 1950), where a copy of the motor control signal is compared to the arising sensory signals. If the motion-evoked sensory signals can be explained by the motor action, the signals are cancelled and do not reach conscious percep-tion. Such efference copy signals would be an important mechanism underlying perceptual stability during head and eye movements (Greenlee & Kimmig, in press; Morrone, 2014).
Head motion is an important source of self-motion percep-tion. Accelerations and/or rotations of the head lead to chang-Copyright 2016. The Japanese Psychonomic Society. All rights reserved. Corresponding address: Institute for Experimental Psychology,
University of Regensburg, D-93050 Regensburg, Germany. E-mail: [email protected]
es in endolymph flow in the otoliths and semicircular canals of the vestibular organ (Bárány, 1907; Lopez, Blanke, & Mast, 2012). The cortical processing of these vestibular signals un-derlying self motion perception, as well as the integration of vestibular signals with those coming from visual and somato-sensory pathways, has been studied in primates and humans (for reviews, Angelaki, Gu, & DeAngelis, 2011; Hitier, Bes-nard, & Smith, 2014; Lopez & Blanke, 2011). However, the in-vestigation of the neural mechanisms underlying self-motion perception is constrained by the fact that neurophysiological recordings usually require that the animal or human partici-pant remain stationary. Natural vestibular stimulation de-mands, on the other hand, movement or acceleration of the head. In primates this can be achieved by placing the animal on a motion platform together with the single- or multi-unit recording equipment. Such laboratory setups have led to a wealth of information about the cortical processing of vestibu-lar signals in, for example, the parieto-insuvestibu-lar vestibuvestibu-lar cortex (PIVC) and dorsal visual area MSTd (Angelaki et al., 2011; Guldin & Grüsser, 2008) in occipito-temporal cortex. In these studies, multisensory neurons have been found that exhibit re-sponse selectivity for the relative direction of visual and vestib-ular inputs. In contrast, studies using functional MRI in hu-mans rely on artificial stimulation of the vestibular system either with galvanic vestibular nerve stimulation or caloric vestibular stimulation (Deutschländer et al., 2002; Lopez and Blanke, 2011; Lobel, Klein, Bihan, Leroy-Willing, & Berthoz, 1998; Stephan et al., 2005). To advance on our knowledge of cortical processing during caloric vestibular stimulation, we have recently introduced a fMRI-compatible caloric vestibular stimulation (CVS) device that can induce the illusory sense of self motion during functional MRI measurements (Frank & Greenlee, 2014).
Caloric vestibular stimulation during functional MRI
Using repeated, bithermal caloric vestibular stimulation, we investigated the responses of human vestibular cortex (Frank & Greenlee, 2014) and compared these responses to those evoked by visual motion and combination of visual and vestib-ular stimulation (Frank et al., 2014; Frank et al., 2016). In these experiments, participants were positioned in the supine position with eyes open in the MRI-scanner (3-Tesla Siemens Allegra). We presented them with visual stimuli, either in the form of a field of randomly moving dots or a single fixation
cross. Participants fixated the fixation cross presented at the screen center, or we asked them to close their eyes, while ca-loric vestibular stimulation (CVS) was performed. CVS was applied by a MRI-compatible, micro-pump system, where hot (48°C), cold (5°C or 20°C), or neutral (37°C) water flowed through left and right ear pods leading to differential caloric vestibular stimulation conditions (Figure 1c, see: Frank & Greenlee, 2014). Periods of bithermal stimulation (i.e., hot on one side–cold on the other) were always followed by periods of neutral stimulation (warm̶37°C̶on both sides). During each trial, participants indicated the presence or absence of self-motion sensations and, if present, the main direction of self motion. The fMRI BOLD response was contrasted be-tween conditions of caloric and neutral stimulation.
To study the extent to which vestibular cortex can be influ-enced by visual motion, we combined caloric vestibular stimu-lation with visual motion cues. Participants viewed limited-lifetime randomly moving dots during episodes of caloric vestibular stimulation. Participants indicated whether visual and vestibular (self) motion cues were in same or opposite di-rections (Frank et al., 2014). To determine the location of visu-ally responsive regions in the posterior insular cortex in indi-vidual participants, we first conducted independent localizer scans. Visual responses in the visual-vestibular area posterior insular cortex (PIC) were localized with purely (100％) coher-ent motion vs. static dots (Figure 1a; see Frank et al., 2016). With these localizer runs we could define visual motion areas like MT, MST, STS and PIC. An example of one participant is presented in Figure 1a. Next we determined responses to ca-loric vestibular stimulation, while the participants were scanned. In this localizer experiment, the participants were asked to close their eyes and to concentrate on any sensations of self motion. The results of these localizer runs are shown in Figure 1b. Area PIVC shows robust responses to caloric stimu-lation. Several other cortical regions also exhibit activations during vestibular stimulation, including area PIC, which is lo-cated posterior to PIVC (Figure 1a). The mean time course of vestibular responses in PIC is presented in Figure 1d. We found significant activations with a peak response approx. 30 s after CVS onset. The delay of the maximum response corre-sponds well with our expectation related to the time required for the caloric stimulus to affect endolymph flow in the hori-zontal canal (Barnes, 1995). The response amplitudes for self-motion sensations to the left or right during CVS did not dif-fer significantly. Moreover, significant activations were also
present in PIC during combined visual and vestibular stimula-tion, although the mean response did not seem to depend on the relative direction of experienced self motion (Frank et al., 2014).
Based on these results it appears that area PIC, in addition to the well established area PIVC, is part of the cortical vestib-ular network and plays a role in the integration of visual and vestibular motion cues for the perception of self motion. The responsiveness of area PIC to visual motion differs from the response of area PIVC to visual motion. PIVC has been shown to respond either weakly to visual motion (Chen et al., 2010)
or its response can even be suppressed by visual motion (Brandt et al., 1998; Deutschländer et al., 2002; Dieterich et al., 1998; Kleinschmidt et al., 2002).
The results described above suggest that area PIC integrates visual and vestibular signals for self-motion perception, whereas PIVC is involved in the processing of vestibular sig-nals (Chen et al., 2010). Similar to area MST in temporal cor-tex (Gu, Angelaki, & DeAngelis, 2008), PIC (or the primate analog area VPS̶ventral parietal sulcus) could be involved in the differential analysis of sensory cues related to self and ob-ject movements (Chen, DeAngelis, & Angelaki, 2011). The re-Figure 1. Schematic illustration of the method of bithermal caloric vestibular stimulation (CVS) used to explore the
vestibu-lar responses in the posterior insuvestibu-lar cortex area (PIC). a) Visual motion localizer runs showing significant activations in PIC in the posterior end of the lateral sulcus (LS). In addition to PIC, other motion-sensitive regions in visual and parietal cortex respond well to visual motion stimuli. b) Results for one participant during caloric vestibular stimulation. The partic-ipant closed their eyes and was instructed to pay attention to any sense of self motion. c) Schematic illustration of the meth-od of caloric vestibular stimulation: hot (red) and cold (blue) water was pumped from outside of the scanner room into ear pods, which were inserted into the ear canals of the participant. The glass ear pods were mounted on ear-protection muffs that covered the outer ear and pinna. The water circulated within a close-loop system and was transported back outside the scanner room to a draining barrel (grey). By positioning four valves we could direct hot water either to the left or right ear, while cold water was directed to the opposite ear. In the baseline conditions all valves were opened and hot and cold waters mixed to provide a neutral (near body temperature) stimulation. Under this bithermal stimulation most participants report-ed the sense of self motion, in which the participant’s head was perceived as slowly rotating to the side with hot water. On baseline trials most participants reported no sense of self motion. d) Average time–course (n＝9 participants) of activation of PIC after 30 s of caloric vestibular stimulation. The curves signify activations according to reported directions of the self-motion sensation (light green: self self-motion to the left; dark green: self self-motion to the right). After Frank & Greenlee, 2014 and Frank et al., 2016 (with permission). For a color version of this figure, see online article.
Figure 1. Schematic illustration of the method of bithermal caloric vestibular stimulation (CVS) used to explore the vestibu-lar responses in the posterior insuvestibu-lar cortex area (PIC). a) Visual motion localizer runs showing significant activations in PIC in the posterior end of the lateral sulcus (LS). In addition to PIC, other motion-sensitive regions in visual and parietal cortex respond well to visual motion stimuli. b) Results for one participant during caloric vestibular stimulation. The partic-ipant closed their eyes and was instructed to pay attention to any sense of self motion. c) Schematic illustration of the meth-od of caloric vestibular stimulation: hot (red) and cold (blue) water was pumped from outside of the scanner room into ear pods, which were inserted into the ear canals of the participant. The glass ear pods were mounted on ear-protection muffs that covered the outer ear and pinna. The water circulated within a close-loop system and was transported back outside the scanner room to a draining barrel (grey). By positioning four valves we could direct hot water either to the left or right ear, while cold water was directed to the opposite ear. In the baseline conditions all valves were opened and hot and cold waters mixed to provide a neutral (near body temperature) stimulation. Under this bithermal stimulation most participants report-ed the sense of self motion, in which the participant’s head was perceived as slowly rotating to the side with hot water. On baseline trials most participants reported no sense of self motion. d) Average time–course (n＝9 participants) of activation of PIC after 30 s of caloric vestibular stimulation. The curves signify activations according to reported directions of the self-motion sensation (light green: self self-motion to the left; dark green: self self-motion to the right). After Frank & Greenlee, 2014 and Frank et al., 2016 (with permission).
sults of our fMRI experiments indicate that PIC can be driven by visual motion and vestibular inputs. Interestingly, PIVC shows, if anything, a tendency for a negative response during visual motion stimulation. Recent imaging studies in humans found activation in the posterior Sylvian fissure during self-motion sensations (i.e., vecton) induced by visual self-motion cues (see Cardin & Smith 2010; Huang, Chen, & Sereno, 2015; Ue-saki & Ashida, 2015). The reported location of this activation coincides well with the location of PIC (Figure 1a), suggesting that these earlier studies probably found activations in PIC and not PIVC.
Conclusions
The reviewed findings suggest that the human vestibular cortex is located in the posterior part of the Sylvian fissure (i.e., in posterior insular and retroinsular cortex). Our results indicate that the parieto-insular vestibular cortex (PIVC) and the posterior insular cortex (PIC) represent two areas that are activated by caloric vestibular stimulation. Galvanic vestibular stimulation also activates these same areas (Billington & Smith 2015; Smith, Wall, & Thilo, 2012). Together these fMRI results point to a network of areas in the Sylvian fissure that responds to vestibular cues related to self motion. On-going research in our laboratory explores the structural connectivity of these re-gions to each other and to the rest of the brain using diffusion-weighted MRI and probabilistic fiber tracking (Wirth, Frank, Beer, & Greenlee, submitted for publication). The results of these studies should inform us on the widespread connectivity of the vestibular cortex with many regions of the brain. As such, these brain circuits underlie our sense of self motion and help us to interpret signals arising from self and object mo-tion. The study of these Sylvian areas will enhance our under-standing of the physiological processing of self-motion cues. It should also promote our insight into pathologies of vestibular pathways as well as conflicts between vestibular, visual and motor signals can lead to vertigo, dizziness and reduced pos-tural stability.
References
Angelaki, D. E., Gu, Y., & DeAngelis, G. C. (2011). Visual and vestibular cue integration for heading perception in ex-trastriate visual cortex. The Journal of Physiology, 589, 825– 833. Advance online publication doi.org/10.1113/jphysiol. 2010.194720
Bárány, R. (1906). Über die vom Ohrlabyrinth ausgelöste Ge-genrollung der Augen bei Normalhörenden, Ohrenkranken
und Taubstummen. European Archives of
Oto-Rhino-Laryn-gology, 68, 1–30. Advance online publication doi.
org/10.1007/BF01834666
Barnes, G. (1995). Adaptation in the oculomotor response to caloric irrigation and the merits of bithermal stimulation.
British Journal of Audiology, 29, 95–106. Advance online
publication doi.org/10.3109/03005369509086586
Berthoz, A., & Viaud-Delmon, I. (1999). Multisensory integra-tion in spatial orientaintegra-tion. Current Opinion in Neurobiology,
9, 708–712. Advance online publication doi.org/10.1016/
S0959-4388(99)00041-0
Billington, J., & Smith, A. T. (2015). Neural mechanisms for discounting head-roll-induced retinal motion. The Journal
of Neuroscience, 35, 4851–4856. Advance online publication
doi.org/10.1523/JNEUROSCI.3640-14.2015
Brandt, T., Bartenstein, P., Janek, A., & Dieterich, M. (1998). Reciprocal inhibitory visual-vestibular interaction. Visual motion stimulation deactivates the parieto-insular vestibu-lar cortex. Brain: a Journal of Neurology, 121, 1749–1758. Bremmer, F. (2011). Multisensory space: from eye-movements
to self-motion. The Journal of Physiology, 589, 815–823. Ad-vance online publication doi.org/10.1113/jphysiol.2010. 195537
Britten, K. H., Shadlen, M. N., Newsome, W. T., & Movshon, J. A. (1992). The analysis of visual motion: A comparison of neuronal and psychophysical performance. Journal of
Neu-roscience, 12, 4745–4765.
Cardin, V., & Smith, A. T. (2010). Sensitivity of human visual and vestibular cortical regions to egomotion-compatible vi-sual stimulation. Cerebral Cortex, 20, 1964–1973. Advance online publication doi.org/10.1093/cercor/bhp268
Chen, A., DeAngelis, G. C., & Angelaki, D. E. (2010). Macaque parieto-insular vestibular cortex: Responses to self-motion and optic flow. Journal of Neuroscience, 30, 3022–3042. Ad-vance online publication doi.org/10.1523/JNEUROSCI. 4029-09.2010
Chen, A., DeAngelis, G. C., & Angelaki, D. E. (2011). Repre-sentation of vestibular and visual cues to self-motion in ventral intraparietal cortex. Journal of Neuroscience, 31, 12036–12052. Advance online publication doi.org/10.1523/ JNEUROSCI.0395–11.2011
Deutschländer, A., Bense, S., Stephan, T., Schwaiger, M., Brandt, T., & Dieterich, M. (2002). Sensory system interac-tions during simultaneous vestibular and visual stimulation in PET. Human Brain Mapping, 16, 92–103. Advance online publication doi.org/10.1002/hbm.10030
Dieterich, M., Bucher, S. F., Seelos, K. C., & Brandt, T. (1998). Horizontal or vertical optokinetic stimulation activates vi-sual motion-sensitive, ocular motor and vestibular cortex areas with right hemispheric dominance. An fMRI study.
Brain: a Journal of Neurology, 121, 1479–1495.
Ernst, M. O., & Banks, M. S. (2002). Humans integrate visual and haptic information in a statistically optimal fashion.
Nature, 415, 429–433. Advance online publication doi.
org/10.1038/415429a
caloric stimulation device for the investigation of human vestibular cortex. Journal of Neuroscience Methods, 235, 208–218. Advance online publication doi.org/10.1016/j. jneumeth.2014.07.008
Frank, S. M., Baumann, O., Mattingley, J. B., & Greenlee, M. W. (2014). Vestibular and visual responses in human poste-rior insular cortex. Journal of Neurophysiology, 112, 2481– 2491. Advance online publication doi.org/10.1152/ jn.00078.2014
Frank, S. M., Wirth, A. M., & Greenlee, M. W. (2016). Visual-vestibular processing in the human Sylvian fissure. Journal
of Neurophysiology, jn.00009.2016. Advance online
publica-tion doi.org/10.1152/jn.00009.2016
Ghazanfar, A., & Schroeder, C. (2006). Is neocortex essentially multisensory? Trends in Cognitive Sciences, 10, 278–285. Advance online publication doi.org/10.1016/j.tics.2006.04. 008
Greenlee, M. W. (2000). Human cortical areas underlying the perception of optic flow: Brain imaging studies.
Internation-al Review of Neurobiology, 44, 269–292. Advance online
publication doi.org/10.1016/S0074-7742(08)60746-1 Greenlee, M. W., Frank, S. M., Kaliuzhna, M., Blanke, O.,
Bremmer, F., Churan, J., ... & Smith, A. T. (2016). Multisen-sory integration in self motion perception. MultisenMultisen-sory
Re-search. Advance online publication doi.org/10.1163/2213
4808-00002527
Greenlee, M. W., & Kimmig, H. (2016). Visual perception dur-ing eye movements. In C. Klein, & U. Ettdur-inger (Eds.),
Ad-vances in eye movement research. Berlin: Springer Verlag.
Gu, Y., Angelaki, D. E., & DeAngelis, G. C. (2008). Neural cor-relates of multisensory cue integration in macaque MSTd.
Nature Neuroscience, 11, 1201–1210. Advance online
publi-cation http://doi.org/10.1038/nn.2191
Guldin, W. O., & Grüsser, O. J. (1998). Is there a vestibular cortex? Trends in Neurosciences, 21, 254–259.
Hitier, M., Besnard, S., & Smith, P. F. (2014). Vestibular path-ways involved in cognition. Frontiers in Integrative
Neurosci-ence, 8, 59. Advance online publication doi.org/10.3389/
fnint.2014.00059
Holst, von, E., & Mittelstaedt, H. (1950). Das Reafferenz-prinzip: Wechselwirkungen zwischen Zentralennervensys-tem und Peripherie. Die Naturwissenschaften, 20, 464–476. Huang, R.-S., Chen, C.-F., & Sereno, M. I. (2015). Neural
sub-strates underlying the passive observation and active con-trol of translational egomotion. Journal of Neuroscience, 35, 4258–4267. Advance online publication doi.org/10.1523/ JNEUROSCI.2647-14.2015
Kleinschmidt, A. (2002). Neural correlates of visual-motion perception as object- or self-motion. NeuroImage, 16, 873–
882. http://doi.org/10.1006/nimg.2002.1181
Kooij, H., Jacobs, R., Koopman, B., & Van der Helm, F. (2001). An adaptive model of sensory integration in a dynamic en-vironment applied to human stance control. Biological
Cy-bernetics, 84, 103–115. Advance online publication doi.
org/10.1007/s004220000196
Lappe, M., Bremmer, F., & van den Berg A. V. (1999). Percep-tion of self-moPercep-tion from visual flow. Trends in Cognitive
Sci-ences, 3, 329–336.
Lobel, E., Kleine, J. F., Bihan, D. L., Leroy-Willig, A., & Ber-thoz, A. (1998). Functional MRI of galvanic vestibular stim-ulation. Journal of Neurophysiology, 80, 2699–2709. Lopez, C., & Blanke, O. (2011). The thalamocortical vestibular
system in animals and humans. Brain Research Reviews, 67, 119–146. Advance online publication doi.org/10.1016/j. brainresrev.2010.12.002
Lopez, C., Blanke, O., & Mast, F. W. (2012). The human vestib-ular cortex revealed by coordinate-based activation likeli-hood estimation meta-analysis. Neuroscience, 1–21. Ad-vance online publication doi.org/10.1016/j.neuroscience. 2012.03.028
Mergner, T., Maurer, C., & Peterka, R. J. (2003). A multisenso-ry posture control model of human upright stance. Progress
in Brain Research, 142, 189–199. Advance online
publica-tion doi.org/10.1016/S0079-6123(03)42014-1
Morrone, M. C. (2014). Interactions between eye movements and vision: Perception during saccades. In J. S. Werner, & L. M. Chalupa (Eds.), The new visual neurosciences (pp. 947– 962). Cambridge: MIT Press.
Smith, A. T., Wall, M. B., & Thilo, K. V. (2012). Vestibular in-puts to human motion-sensitive visual cortex. Cerebral
Cor-tex, 22, 1068–1077. Advance online publication doi.
org/10.1093/cercor/bhr179
Sperry, R. W. (1950). Neural basis of the spontaneous optoki-netic response produced by visual inversion. Journal of
Comparative and Physiological Psychology, 43, 482–489.
Stephan, T., Deutschländer, A., Nolte, A., Schneider, E., Wies-mann, M., Brandt, T., & Dieterich, M. (2005). Functional MRI of galvanic vestibular stimulation with alternating cur-rents at different frequencies. NeuroImage, 26, 721–732. Ad-vance online publication doi.org/10.1016/j.neuroimage. 2005.02.049
Uesaki, M., & Ashida, H. (2015). Optic-flow selective cortical sensory regions associated with self-reported states of vec-tion. Frontiers in Psychology, 6, 775. Advance online publi-cation doi.org/10.3389/fpsyg.2015.00775
Wirth, A. M., Frank, S. M., Beer, A. L., & Greenlee, M. W. (2016). White matter connections in the vestibular and vi-sual-vestibular cortex. submitted for publication
