In the sci-fi user interface criticism blog [???] the guy complains that it’s really bad user interface design for a HUD in a flying machine to have things expand in place, because it triggers a human “startle response”, which stresses people out, makes them blink, and distracts them.
Presumably the underlying phenomenon here is that if something is expanding rapidly in place in your visual field, it’s usually an object flying rapidly at your face, which means it could hit you soon.
I was alarmed and concerned, because I’ve been playing a bit with ZUIs, and in ZUIs it’s really common for things to expand in place. I started thinking of workarounds: could we maybe zoom into a thing by first zooming in on a point to the right of the screen, then a point to the left of the screen, thus avoiding this response?
But does this visual "startle response" to zooming in exist at all? If so, will it cause problems? Or is the related “orienting response” more relevant? Here’s a cursory literature review.
The Wikipedia article on "startle response" says:
In animals, including humans, the startle response is a largely unconscious defensive response to sudden or threatening stimuli, such as sudden noise or sharp movement, and is associated with negative affect.1 Usually the onset of the startle response is a startle reflex reaction. The startle reflex is a brainstem reflectory reaction (reflex) that serves to protect vulnerable parts, such as the back of the neck (whole-body startle) and the eyes (eyeblink) and facilitates escape from sudden stimuli. It is found across the lifespan of many species.
Startle and Surprise on the Flight Deck, Rivera et al., 2014. This is a five-page paper on the difference between the “startle reflex” and the “surprise emotion”. It says that startle was “studied extensively in the 1970s and 1980s”, but refers primarily to the acoustic startle response, although it says, “The startle reflex can be elicited through auditory, visual, or tactile stimuli,” citing Carlsen et al. 2008, Davis et al. 1982, and Yeomans et al. 2002. With regard to effects we might care about in ZUIs, it says, “startle has been found to impair information processing on mundane tasks,” and “may induce brief disorientation and short-term psychomotor impairments which are likely to lead to task interruptions, or brief confusion,” and that “after startle”, “decision making can be significantly impaired, especially higher-order functions”.
It does mention ways the startle response can be evoked visually: sudden illumination of flight crews by lasers, for example.
The "Startle Reflex" article in the Encyclopedia of Autism Spectrum Disorders, by Sterling, defines it as an “automatic response to the sudden onset of an acoustic, visual, or tactile stimulus… that varies systematically with the individual’s emotional state… characterized by a fast series of muscle contractions around the head, neck, and shoulders;” this says very little about what kind of stimuli will cause it, other than “sudden”, or what kinds of cognitive effects it might have.
Human On-Line Response to Visual and Motor Target Expansion, Cockburn and Brock 2005 or so, says that expanding mouse targets in place as the mouse approaches them is “visually appealing” but also “improves acquisition performance”. Their main result is that it's easier to click on things when they expand when you move the mouse toward them, and that this is mostly due to the visual feedback, only slightly due to the larger click target. While this is basically not about startling users at all, but rather about how long it takes to click on things when they’re expanding (related to Fitts’ Law), it provides a sort of useful negative evidence — in their experiments, users were able to click on things faster when they expanded in place, while if expanding stuff in place was invoking a strong “startle reflex”, you’d expect to see their performance worsen. But maybe they didn’t expand fast enough or far enough, or maybe the performance improvement would have been even greater without the “startle response” confounding it.
The Wikipedia article on “fear-potentiated startle” mentions bright light as a stimulus that elicits “startle”, and says that the “startle” is stronger when the organism is afraid, or when the person is traumatized or not depressed, and mentions some symptoms of “startle”: blinking and faster pulse. However, apparently, usually the experiments use loud noises.
The Uncanny Valley in Games and Animation, by Tinwell, p.106, section “Lack of Visual Startle Reflex and Psychopathy,” says that we make surprised faces when “we experience a sight or sound that frightens or surprises us”, except for psychopaths. It’s talking about things like “seeing a shadow when we thought we were alone,” though, not seeing windows expand on a screen, and it doesn’t mention cognitive effects. The author is bringing this up because apparently people who act scared or surprised without showing fear “in the upper facial region” are creepy (“uncanny”) and that maybe this is a reason animations can be creepy.
Boo! Culture, Experience, and the Startle Reflex, by Simons, 1996, is a 288-page book about the “startle reflex”. It goes into quite a bit of philosophizing about the relationships between culture, biology, and psychology, saying things like, “Being startled changes… one's relationship with the entire experienced universe,” and “Pythagoras… showed that the soul is mathematical.”
The author is a collaborator of Ekman and Friesen who himself did original research on startle some years back, using pistol sounds.
It explains, “The subject of the book is a more general one: (1) How one's experience of being in the world is shaped by neurophysiology and how it is shaped by prior experience, beliefs, values, and social condition, and (2) how these different types of shaping influences might be consid- ered jointly in single explanatory formulations.”
It gives some more detail on what the startle response is, exactly (p. 9):
The usual human startle response is precisely the set of behaviors that would be maximally effective in minimizing the chance of a fatal encounter. The response includes both physical events and alterations in attention, thought, and mood. Instantaneously and without reflection, forward motion is checked. The raised foot is arrested or drawn back and the upper limbs are drawn in to safety. Visual attention is locked onto the eliciting stimulus, whatever had been in consciousness is obliterated, and the mind is filled with one thought: “Snake!” Startling stimuli other than snakes elicit similar immediate patterned rapid-avoidance behaviors and similar redirection of attention.
In the context of ZUIs, redirecting your attention to what you’re zooming into might be good, and that would be consonant with the positive results Cockburn and Brock got from inflating click targets.
He confirms the 14-ms number Sourakov (the butterfly dude) cites for human startle response time, but for tightening jaw muscles, not blinking.
He cites “Hoffman, 1984, p.275”, which turns out to be “Methodological factors in the behavioral analysis of startle,” as a paper that mentions a “startling sight.” His Table 1.2 on p.14 lists startling stimuli; the visual items include “dangerous or appalling sights” (“snakes and creepy-crawlies, other dangers, appalling sights”), “bright lights”, “sudden motion”, “great beauty”, and “cessation of a stimulus”. Later (p.21), he reports on visual stimuli that startled his informant Barbara Haldane:
I mentioned before my startle reaction to the sight of a large spider (or a picture or drawing of one); other large images of things with spindly legs, such as ants and other insects, produce a similar but less-violent startle reaction.
A real snake (but not a picture of one)…
As I told you, the unexpected sight of a human skull--of nearly life-sized or larger proportions, anyway--induces a shock in me ... especially if it has dark eye-sockets.
The sight of a larger-than-life human face looking directly at me (at the camera, in the case of a photo) has a similar shock value. I recall one instance of being quite unpleasantly startled by the face of Sophia Loren on a magazine cover, and more than once by an ad in a magazine featuring eyes (larger than life) looking directly at the viewer. (Both eyes usually have to be present — one big eye doesn't have as much effect.) [T]hese reactions are invol- untary and uncontrollable unless the sight in question is antici- pated, and even then I am apt to experience an internal moment of panic.
And his hyperstartler informant Gould startles easily with a variety of visual stimuli, including boys jumping out at her at a ranch, her children stepping out into the hall in front of her, a classroom of her students having turned their desks around while she wasn’t paying attention, and putting on the wrong kind of glasses.
The exaggerated startle reaction he reports of Gould, in which she screams for a while, falls to her knees, and throws whatever she’s holding, would certainly be a UX problem in a ZUI; it would likely break her phone. However, it seems to be mostly related to seeing unexpected things, not to a lower-level stimulus such as a thing expanding in her field of vision, regardless of whether it’s expected or not.
Emotion, attention, and the startle reflex, by Lang et al. in 1990, is a theoretical model of emotion in general, based on the “startle reflex”, which is an intellectual overreach if I ever heard of one.
Visual Pathways Involved in Fear Conditioning Measured with Fear-Potentiated Startle: Behavioral and Anatomic Studies, Shi and Davis 2001, startled rats with sounds or electric shocks but conditioned them to associate the “startle response” with visual stimuli. Then they injected different drugs into different parts of the rats’ brains to see which parts were involved in the association. This paper doesn’t use an innate visual startle response or human subjects, so it is of no interest.
The pupil as a measure of emotional arousal and autonomic activation, Bradley et al. 2013 had people look at pleasant or unpleasant pictures, measuring their emotional response to the pictures by their pupil diameter, skin conductance, and pulse. They also found that people’s pupils contracted even more when the pictures were bright than when they were unpleasant. This came up because they startled some people, but not others, with loud noises before showing the pictures. However, they didn’t report on the results of this part of the experiment in the paper!
Postural and eye-blink indices of the defensive startle reflex, by Hillman et al. 2003, reports on an experiment where they startled 24 volunteers with loud noises through headphones and measured how much they blinked and cringed, rocking first forwards and then backwards. Their results were that people who blinked more also cringed further backwards.
A Double Dissociation in the Affective Modulation of Startle in Humans: Effects of Unilateral Temporal Lobectomy, by Funayama et al., 2001, found differences in the “startle response” in people who had had one of their temporal lobes surgically removed. They were startling the subjects with loud noises as a way to measure the emotional effect of looking at pleasant or unpleasant pictures or being told they would be electrically shocked. Their results are somewhat involved and they draw a bunch of conclusions about the role of the amygdala in fear that I don’t care about.
Perceiving Threat In the Face of Safety: Excitation and Inhibition of Conditioned Fear in Human Visual Cortex, by Miskovic and Keil, 2013, startled 29 subjects with loud noises that the subjects could predict by the speed a light flickered at.
Greater general startle reflex is associated with greater anxiety levels: a correlational study on 111 young women, by Poli and Angrilli, 2015, startled young women with loud noises in both ears after questioning them about how anxious they felt, finding that more anxious women blinked more (but only with their left eyes) and hated the noises more. As background, it mentions that old people and men startle less.
Extraordinarily Quick Visual Startle Reflexes of Skipper Butterflies (Lepidoptera: Hesperiidae) are Among the Fastest Recorded in the Animal Kingdom, by Sourakov, 2009, is about visual startle reflexes in butterflies. I think butterfly eyes and visual cortices evolved separately from our own, so I doubt that conclusions about startle reflexes in butterflies are applicable to humans. It does, however, mention that loud noises can make people blink within 14 ms, puffs of air can make us blink within 30–50 ms, but these butterflies jump into the air in less than the researcher’s camera’s exposure time of 17 ms when visually startled by the flash; however, it claims that this is “twice as fast as the fastest startle reflex of humans”, so, I don’t even know.