The direction of another’s gaze provides a strong cue to their intentions and future actions. The perception of gaze is a remarkably plastic process: adaptation to a particular direction of gaze over a matter of seconds or minutes can cause marked aftereffects in the perceived direction of others’ gaze. Computational modelling of data from be-havioural studies of gaze adaptation allows us to make inferences about the functional principles that govern the neural encoding of gaze direction. This in turn provides a foundation for testing computational theories of neuro-psychiatric conditions in which gaze processing is compromised, such as autism.
Keywords: gaze direction, visual adaptation, social attention, face perception, sensory coding Introduction
Gaze perception plays a critical role in human communica-tion and interaccommunica-tion (Argyle & Cook, 1976). Vision is an active process whereby the eyes tend to track objects of interest. Such movements serve to position the retinal image of the object of interest on the fovea, ensuring that it is processed with the highest acuity visual mechanisms. To an observer, the direc-tion of your gaze reveals where you are looking and hence what you are looking at. This might be an object of shared at-tention or it might be the observer him or herself. The direc-tion of your gaze is thus a strong social signal to your inten-tions and future acinten-tions (Baron-Cohen, 1995). As such, understanding the mechanisms by which another’s gaze is perceived and interpreted is an active area of interest in the burgeoning field of social neuroscience (Nummenmaa & Calder, 2009).
The phenomenon of visual adaptation demonstrates that the perception of gaze direction is a remarkably plastic process that can be affected by the recent history of stimulation (Jen-kins, Beaver, & Calder, 2006; Seyama & Nagayama, 2006). Ad-aptation is an overloaded term that refers to three inter-related elements: procedure, process and percept (Wade & Verstraten, 2005). The procedure of adaptation is exposure to a particular diet of sensory stimulation. In response to changes in
stimula-tion, our sensory systems change the way they process incom-ing information. These changes in sensory processincom-ing give rise to aftereffects in our perception.
Adaptation is well-established as a fundamental characteris-tic of low-level sensory processing but it is only relatively re-cently that adaptation to higher-level visual qualities associat-ed with faces and objects has been explorassociat-ed (Clifford & Rhodes, 2005). Adaptation to a series of faces displaying avert-ed gaze can cause markavert-ed changes in perception, such as whether or not gaze appears to be directed at the observer (Jenkins et al., 2006; Seyama & Nagayama, 2006). The deter-minants and phenomenology of adaptation are diagnostic as to the processes by which our brains represent the direction of the gaze of others. Here, I describe the phenomenon of adap-tation to the direction of another’s gaze and discuss its impli-cations for our understanding of gaze processing in the human brain.
Gaze Aftereffects
Gaze aftereffects following adaptation to a series of images of faces with gaze averted in a particular direction are general-ly ‘repulsive’ or ‘negative’. That is to say, the perceived direc-tion of a subsequent test stimulus is repelled away from the adapting direction compared to perception of that same test in an unadapted baseline condition. Robust gaze aftereffects are evident with various techniques of measurement. For example, using a forced-choice judgment of gaze as leftwards or right-wards, Seyama and Nagayama (2006) found that adaptation to gaze averted horizontally by 35° biased the perception of sub-Copyright 2018. The Japanese Psychonomic Society. All rights reserved. Correspondence address: School of Psychology, UNSW
Sydney, Sydney, NSW 2052, Australia. E-mail: colin.clifford@unsw.edu.au
sequently presented test stimuli throughout the range ±4° such that they were more likely to be reported as looking in the opposite direction to the adaptor. Similarly, using a forced-choice categorization of gaze as leftwards/direct/rightwards, Jenkins et al. (2006) found that adaptation to 25° averted gaze tended to cause test stimuli averted 5–10° to the same side as the adaptor to be reported as looking direct.
More recent studies have used a continuous rather than cat-egorical measure, where participants adjust the direction of an on-screen pointer to indicate perceived direction (Palmer & Clifford, 2017a, b; Palmer, Lawson, Shankar, Clifford, & Rees, 2018). Using a pointer has the advantage of allowing the ef-fects of adaptation to be measured metrically (e.g., how much the perceived direction of gaze shifts in degrees), and across the whole gamut of physically realizable gaze directions. These latter studies have consistently found that, for adaptors averted by 25°, the strongest aftereffects are observed for test stimuli averted by around 10° to the same side as the adaptor. The peak magnitude of these aftereffects is approximately 8°, which corresponds to roughly half the width of the
partici-pant’s head at the viewing distance of 50 cm used in these studies. Thus, adaptation caused gaze directed at the partici-pant’s ear to appear to be directed straight at them! This is il-lustrated in Figure 1.
Using Adaptation to Reveal the Structure of the Neural Channels Coding Gaze Direction
Channels are a well-established concept in psychophysics, referring to cell populations in the nervous system that are tuned along a given stimulus dimension. Adaptation can be modelled as a reduction in the responsiveness of each channel proportional to how strongly it is engaged by the adapting stimulus (Graham, 1989). The perceptual effects of adaptation can thus indicate how a stimulus property is represented across a system of such channels. Adaptation studies have pro-vided important new insights into the channel structure cod-ing gaze direction.
The earliest studies showed dissociable perceptual afteref-fects of adaptation to leftwards and rightwards averted gaze (Jenkins et al., 2006; Seyama & Nagayama, 2006) indicating Figure 1. Illustration of the magnitude of gaze aftereffects based on fitted data from 28 neurotypical adults (Palmer et al.,
2018). Leftmost column denotes the adapting condition. Subsequent columns represent the perceived direction of gaze of faces with eyes averted horizontally by ‐10, 0 and +10 degrees, respectively.
explicitly for gaze directed at the observer (Calder, Jenkins, Cassel, & Clifford, 2008; Palmer & Clifford, 2017a).
Calder et al. (2008) investigated the range of test gaze direc-tions that observers categorised as being directed at them, and observed opposite effects of adaptation on perceived gaze di-rection depending on whether adaptation was to a series of faces (i) all gazing directly at the observer or (ii) alternating between leftwards and rightwards gaze averted by 25°. Calder
et al. reasoned that, within each of these two adaptation
condi-tions, channels tuned to leftwards and rightwards gaze should be engaged to the same extent. Consequently, in a two-channel opponent system, the two adaptation conditions should have qualitatively the same effect on the range of test gaze direc-tions categorized as direct. However, adaptation to direct gaze was found to narrow the range of test gaze directions catego-rized as direct whereas adaptation to alternating leftwards and rightwards gaze broadened the range (see Lawson, Clifford, & Calder, 2009, 2011 for analogous findings regarding adapta-tion to body and head direcadapta-tion, respectively). Calder et al. reasoned that the most parsimonious account of this pattern of data was a system of three broadly tuned channels tuned to leftwards, rightwards and direct gaze, respectively. Under such a system, adaptation to direct gaze would engage primarily the direct channel, causing the range of subsequent test directions perceived as direct to narrow. Conversely, adaptation to alter-nating leftwards and rightwards gaze would preferentially en-gage the leftwards and rightwards channels, causing the range to broaden.
The use of a categorical measure of perceived gaze direction (e.g., left/direct/right) requires observers to adopt a decision criterion as to where precisely the boundaries between catego-ries lie. It is conceivable that adaptation might affect such sub-jective category boundaries in a systematic way, rather than the perceptual experience of gaze direction per se (Storrs, 2015). For example, if exposure to a series of directly gazing faces served as a repeated reference as to what constitutes di-rect gaze then it might both narrow the range of test faces
cat-Palmer and Clifford (2017a) revisited the question of what channel structure underlies the coding of horizontal gaze di-rection using a different response method. Participants were required to use a pointer to indicate perceived direction, avoiding the need for them to adopt subjective category boundaries in their responding. Although not finding clear evidence of an aftereffect of adaptation to direct gaze (see also Kloth & Schweinberger, 2010), Palmer and Clifford observed in their data a novel characteristic diagnostic of the existence of a direct channel. Specifically, they found that the magnitude of aftereffects to 25° averted gaze was tuned for test direction, with the maximum aftereffects evident for test stimuli averted 10–15° to the same side as the adaptor (see also Palmer & Clif-ford, 2017b; Palmer et al., 2018).
Using computational modelling to simulate hypothetical channel structures, Palmer and Clifford (2017a) demonstrated that tuning of aftereffect magnitude for test direction is char-acteristic of a system comprising a small number of broadly tuned mechanisms whose activity is subject to divisive nor-malization (Figure 2). Their simulations supported the intui-tive notion that the effects of adaptation on perception should be most evident when the test stimulus engages channels dif-ferentially affected by the adapting stimulus. This leads to dis-tinct predictions for the tuning of aftereffects in two-channel opponent and three-channel systems. For example, following adaptation to leftwards averted gaze, a system of only two op-ponent channels would produce the strongest aftereffects for direct test stimuli, as this is the direction for which the strong-ly adapted (leftwards) and relativestrong-ly unadapted (rightwards) channels are equally engaged. In a three-channel system, how-ever, the strongest aftereffects following adaptation to left-wards averted gaze would be evident for a test direction where leftwards and direct channels are equally engaged, i.e. moder-ately averted gaze to the same side as the adaptor, as observed empirically.
The findings of Palmer and Clifford (2017a) thus support the conclusions of Calder et al. (2008) that the coding of horizontal
gaze direction is inconsistent with the operation of a two-chan-nel opponent system but can be parsimoniously accounted for within a three-channel framework. Furthermore, Palmer and Clifford’s computational modelling highlights the role of divi-sive normalization in the coding of gaze direction. Dividivi-sive nor-malization serves as a form of gain control, ensuring that the relative activation across channels provides a code that is robust to variation in the absolute level of stimulation. Divisive nor-malization has been argued to be a canonical feature of nervous system function (Carandini & Heeger, 2012), though it is rela-tively unexplored in the context of higher-level, social vision. In general, the effect of adaptation on sensory coding is complicat-ed by its potential to act on both driving and suppressive mech-anisms (Solomon & Kohn, 2014). Here, the precise tuning pro-file of gaze aftereffect magnitude predicted by the model arises because signals from the adapted channel(s) feed into not only the driving mechanism (‘N’ in Figure 2) but also into the nor-malization signal (‘D’ in Figure 2).
Gaze Adaptation in People with Autism Spectrum Disorder
Autism spectrum disorder (ASD) is a heterogeneous devel-opmental condition whose characteristics include atypicalities in social communication and interaction (American Psychiat-ric Association, 2013), including gaze-based behaviours (Bar-on-Cohen, 1995). Reduced effects of adaptation to gaze direc-tion have been observed in ASD both in children (Pellicano, Rhodes, & Calder, 2013) and adults (Lawson, Aylward, Roiser, & Rees, 2017).
It has been suggested that altered divisive normalization may contribute to a wide array of the behavioural
consequenc-es in ASD (Rosenberg, Patterson, & Angelaki, 2015). Palmer et
al. (2018) sought evidence for atypical divisive normalization
in ASD in the context of gaze adaptation. They simulated the effect of varying the degree of divisive normalization (‘1-w’ in Figure 2) in the model of horizontal gaze coding proposed by Palmer and Clifford (2017a). On the basis of their computa-tional modelling, Palmer et al. (2018) predicted that a reduc-tion in the degree of divisive normalizareduc-tion should lead to a broader tuning profile of gaze aftereffect magnitude as a func-tion of test direcfunc-tion.
However, 27 adults with a diagnosis of ASD showed no dif-ference from matched neurotypical controls in either the over-all magnitude of their gaze aftereffects or the degree of divisive normalization inferred from fitting the model to their data. On the basis of a Bayesian statistical analysis, Palmer et al. (2018) concluded that their results provide strong support for there being no difference between ASD and control groups in how the effects of adaptation differ across test directions. Nor was there a significant correlation at an individual level be-tween the strength of adaptation or normalization and autistic features (ADOS and AQ scores).
The observation by Palmer et al. (2018) of strong gaze after-effects in adults with ASD appears at odds with previous find-ings of reduced effects of adaptation to gaze direction in ASD both in children (Pellicano et al., 2013) and adults (Lawson et
al., 2017). However, Palmer et al. note that participants in
their study indicated the perceived direction of gaze by setting the rotation of a pointer, while in the previous two studies par-ticipants made a categorical judgment as to whether the face was looking directly towards them. Palmer et al. conjecture that the reduced effects of adaptation in individuals with ASD Figure 2. Inside the Eye Direction Detector. Schematic representation of the functional architecture proposed by Palmer et
al. (2017a) to underlie the coding of horizontal gaze direction. Gaze direction is encoded by the pattern of activation across
three channels tuned to leftwards, direct, and rightwards, respectively. The outputs of these channels are then combined through a process of divisive normalization to generate a metric estimate of gaze direction.
gaze direction. Gaze direction can be encoded across three neural channels tuned broadly to leftwards, rightwards, and direct gaze (Calder et al., 2008). Furthermore, the encoding of gaze direction entails the divisive normalization of channel re-sponses (Palmer & Clifford, 2017a), supporting the idea that this is a canonical feature of sensory processing extending to social aspects of perception.
Recent data from participants with Autism Spectrum Disor-der (Palmer et al., 2018) do not support the notion of there be-ing a widespread difference in the normalization of sensory re-sponses in ASD. Instead, they demonstrate the underlying function of the gaze system in ASD in terms of channel struc-ture, adaptive coding and normalization, despite their being dif-ferences in how individuals with ASD respond to others’ gaze.
Acknowledgments
I am grateful to all of my collaborators on this work, princi-pally Dr Colin Palmer, Dr Rebecca Lawson and the late Dr Andy Calder. Thank you to A/Prof Yumiko Otsuka and Prof Katsumi Watanabe for hosting my visit to Japan to give this lecture. This work has been supported by Australian Research Council Discovery Projects DP120102589 and DP160102239.
References
American Psychiatric Association. (2013). Diagnostic and
sta-tistical manual of mental disorders: DSM-5 (5th ed.).
Arling-ton, VA: American Psychiatric Association.
Argyle, M., & Cook, M., (1976). Gaze and mutual gaze. Cam-bridge, UK: Cambridge University Press.
Baron-Cohen, S. (1995). Mindblindness: An essay on autism
and theory of mind. Cambridge: MIT Press.
Calder A. J., Beaver J. D., Winston, J. S., Dolan, R. J., Jenkins, R., Eger, E., & Henson, R. N. (2007). Separate coding of dif-ferent gaze directions in the superior temporal sulcus and inferior parietal lobule. Current Biology, 17, 20–25.
Calder, A. J., Jenkins, R., Cassel, A., & Clifford, C. W. G. (2008). Visual representation of eye gaze is coded by a non-opponent multichannel system. Journal of Experimental
Psychology: General, 137, 244–261.
Carandini, M., & Heeger, D. J. (2012). Normalization as a
ca-Kloth, N., & Schweinberger, S. R. (2010). Electrophysiological correlates of eye gaze adaptation. Journal of Vision, 10, 17. Lawson, R. P., Clifford, C. W. G., & Calder, A. J. (2009). About
turn: The visual representation of human body orientation revealed by adaptation. Psychological Science, 20, 363–371. Lawson, R. P., Clifford, C. W. G., & Calder, A. J. (2011). A real
head turner: Horizontal and vertical head directions are multichannel coded. Journal of Vision, 11: 17, 1–17. Lawson, R. P., Aylward, J., Roiser, J. P., & Rees, G. (2017).
Ad-aptation of social and non-social cues to direction in adults with autism spectrum disorder and neurotypical adults with autistic traits. Developmental Cognitive Neuroscience, 29, 108–116.
Nummenmaa, L., & Calder, A. J. (2009). Neural mechanisms of social attention. Trends in Cognitive Sciences, 13, 135–143. Palmer, C. J., & Clifford, C. W. G. (2017a). Functional
mecha-nisms encoding others’ direction of gaze in the human ner-vous system. Journal of Cognitive Neuroscience, 29, 1725– 1738.
Palmer, C. J., & Clifford, C. W. G. (2017b). The visual system encodes others’ direction of gaze in a first-person frame of reference. Cognition, 168, 256–266.
Palmer, C. J., Lawson, R. P., Shankar, S., Clifford, C. W. G., & Rees, G. (2018). Autistic adults show preserved normalisa-tion of sensory responses in gaze processing. Cortex, 103, 13–23.
Pellicano, E., Rhodes, G., & Calder, A. J. (2013). Reduced gaze aftereffects are related to difficulties categorising gaze direc-tion in children with autism. Neuropsychologia, 51, 1504– 1509.
Rosenberg, A., Patterson, J. S., & Angelaki, D. E. (2015). A computational perspective on autism. Proceedings of the
Na-tional Academy of Science of the United States of America, 112, 9158–9165.
Seyama, J., & Nagayama, R. S. (2006). Eye direction aftereffect.
Psychological Research, 70, 59–67.
Solomon, S. G., & Kohn, A. (2014). Moving sensory adapta-tion beyond suppressive effects in single neurons. Current
Biology, 24, R1012–1022.
Storrs, K. R. (2015). Are high-level aftereffects perceptual?
Frontiers in Psychology, 6, 157.
Wade, N. J., & Verstraten, F. A. J. (2005). Accommodating the past: A selective history of adaptation. In C. W. G. Clifford & G. Rhodes (Eds.), Fitting the mind to the world:
Adapta-tion and aftereffects in high-level vision (pp. 83–101).
