Preface: Following the previous #AI series and #MarvinMinsky series. This post is co-written with Claude, which is named after Shannon Claude.

How Does Attention Work?

Alfred Yarbus (1914–1986) was a Soviet biophysicist who spent the 1950s and 1960s studying how the human eye moves. In 1965 he published Eye Movements and Vision in Russian; the English translation followed in 1967, and the book went on to become one of the most cited classics in eye-movement research.

When we look at something, it feels as though our gaze sweeps smoothly across the whole scene. It doesn't. The eyes move in jumps: they stop in one place, leap quickly to another, and stop again. A fixation is the eye pausing briefly in one spot; a saccade is the jump from one pause to the next.

A saccade is one of the fastest movements the human body can make, taking only a few dozen milliseconds. During the jump, the image on the retina is blurred, and the brain takes in almost nothing. Everything we actually "see" happens during the pauses. So when someone looks at a painting, what they're really doing is a string of pauses and jumps. To know what they looked at, you have to record those pauses one by one.

The problem is that eye movements are fast and tiny, and in the 1950s there were no high-speed cameras and no computers. Yarbus's solution was to attach the recording device to the eye itself. He used a tiny suction cap, pressed onto the subject's cornea and held in place by suction. A small hole in the center let the subject still see; a small angled mirror was mounted on the edge of the cap. The subject's head was held still, and a beam of light was aimed at the mirror and reflected onto photographic paper. When the eye turned, the mirror turned with it, and the spot of light moved across the paper. The path the spot traced was the path the eye had taken.

The first suction caps were made by a technician in his group, and they were so uncomfortable they were nearly unusable. Yarbus kept refining the design, through a long process of trial and error, until he had a range of caps that worked. It was for this device, and the research he did with it, that he received his doctorate in biology in 1964. Even after the improvements, wearing it was an ordeal. Yarbus recommended that no session last longer than twelve minutes, yet he also recorded that most subjects felt a painful drying of the cornea after just two or three. Today's eye trackers don't touch the eye at all; they use infrared light and cameras to record the position of the pupil from a distance. But the paths Yarbus drew with suction caps and photographic paper are still the most cited images in the field.

At the time, though, the recordings looked like a tangle of fine lines and ink dots. Where the dots clustered, the eye had paused; the fine lines between them were the jumps. Yarbus found that when people look at a face, their gaze circles between the eyes, nose, and mouth in a triangle, over and over. With a complex painting, too, the gaze keeps returning to a few places that carry the most information.

The painting he used most was Ilya Repin's They Did Not Expect Him: a man returning from exile walks through the door, and the family inside looks up at him. I wrote about this painting in detail in 1890s, Russia/ The Wanderers. In the Soviet Union, nearly everyone knew it, and his subjects knew what story it told.

He had a single subject look at the same painting seven times, three minutes each time, with a different task each time: look freely; estimate the family's material circumstances; give the ages of the people; surmise what the family had been doing before the visitor arrived; remember the clothes everyone was wearing; remember where the people and objects were in the room; estimate how long the visitor had been away. Same person, same eyes, same painting, and the recorded eye movements were completely different. Asked about money, the gaze settled on clothing, furniture, and the furnishings of the room; asked about ages, it went almost entirely to the faces; asked how long the visitor had been gone, it shuttled between the visitor's face and the family's, as if reading their expressions. What he found was that where the eyes land isn't decided by the painting. It's decided by what the viewer wants to know.

Budget: Kahneman, Attention and Effort, 1973

From the 1970s on, psychologists found several metaphors for attention, each tied to a set of experiments. The first metaphor is a budget. In 1973, Daniel Kahneman published Attention and Effort. Before him, Broadbent and Treisman had argued over "at which stage the filtering happens," as if attention were a gate and the only question was where to install it. Kahneman asked a different question: what if attention is a resource that can be allocated? His view was that the mental effort a person can invest at any moment has a limited total. That effort can be divided among several things at once, but the sum can't exceed the limit.

The capacity model holds that attention is a limited supply of mental effort that can be split among several things at once, as long as the total stays under the limit. The size of this budget isn't fixed. The more alert or keyed up a person is, the more effort they can draw on; when they're sleepy or tired, the less. How it gets divided follows rules too: some things automatically claim a share, like a loud bang or your own name; the rest goes to whatever you're trying to do at the moment. The harder a task, the more effort it takes; once it becomes practiced, the same task takes less.

Mental effort is invisible, so how do you measure it? Kahneman used the pupil. In 1966, he and Jackson Beatty published an experiment in Science: subjects heard a string of digits, one per second, paused for two seconds, then repeated them back at the same pace. Throughout, a camera photographed their pupils. With each digit heard, the pupil widened a little; at the end of the list, it reached its widest; once recall began, it narrowed a little with each digit spoken. The more digits there were to hold, the wider the pupil grew. The pupil worked like a gauge, showing second by second how much the mind was carrying. Later research found that once the digits went beyond the limit of memory, the pupil stopped widening: the budget was spent, and adding more to the task added nothing.

The biggest difference between this model and filter theory is that it allows several things to happen at once. The filter lets in only one stream at a time; the budget only requires that the total stay within bounds. An experienced driver can drive and talk at the same time, because the two together stay within budget. But once the road gets complicated, rain, a lane change, the car ahead braking hard, drivers often stop talking mid-sentence. Kahneman later used a similar example in Thinking, Fast and Slow: when passing a large truck on a narrow road, people in the car tend to fall quiet on their own. It isn't that they don't want to talk. The budget has run out.

Spotlight: Posner, "Orienting of Attention," 1980

In 1980, Michael Posner of the University of Oregon published "Orienting of Attention" in the Quarterly Journal of Experimental Psychology.

The experiment worked like this. Subjects sat in front of a screen and stared at a cross in the center, with a box on either side of it. On each trial, a cue appeared first, telling the subject which side the target was likely to appear on: sometimes an arrow at the center pointing left or right, sometimes a sudden brightening of one of the boxes. A moment later, the target appeared in one of the boxes, and subjects pressed a key the instant they saw it. The experiment recorded how many milliseconds the response took.

The key was that the cue wasn't always right. About 80 percent of the time, the target appeared on the side the cue pointed to; the rest of the time, it appeared on the other side. On some trials, the cue pointed to neither side, as a control. When the cue was right, subjects responded faster than in the control trials; when it was wrong, slower. The speed-up was the benefit, the slowdown the cost. The differences were only a few dozen milliseconds, but they were very consistent.

Covert attention is attention moving to another location while the eyes stay still. This showed that attention doesn't have to follow the eyes. Like a spotlight, it can move ahead to a spot in the field of view and light it up, so that whatever appears there gets processed faster. When the cue pointed the wrong way, the spotlight had to pull out of the wrong spot before moving to the right one, which is why responses were slower. Posner later broke this process into three steps: disengaging from the current location, moving, and engaging at the new one.

The two kinds of cue also worked differently. With the arrow at the center, subjects had to understand what it meant and then deliberately move their attention; when a box suddenly brightened, attention was pulled there without any thought. These correspond neatly to the two kinds of attention James described long before: active and passive.

Glue: Treisman and Gelade, "A Feature-Integration Theory of Attention," 1980

Also in 1980, Anne Treisman, then at the University of British Columbia in Canada, and Garry Gelade of Oxford published "A Feature-Integration Theory of Attention" in Cognitive Psychology.

The method they used is called visual search. A crowd of colored letters appears on the screen, and subjects have to decide as fast as they can whether a particular target is among them, pressing one key if it is and another if it isn't. The number of letters changes from trial to trial, from one to thirty.

In the first kind of search, the target can be identified by a single feature. The screen shows a crowd of brown Ts and green Xs, and the target is a blue letter, or the letter S. Blue is the only color of its kind; S is the only shape of its kind. Whether the screen held five letters or thirty, finding the target took almost the same time. It popped out on its own.

In the second kind, the target can be identified only by a combination of two features. The screen shows the same crowd of brown Ts and green Xs, and this time the target is a green T. Green is nothing unusual; green Xs are everywhere. T is nothing unusual either; brown Ts are everywhere. Only a letter that is both green and a T is the target. This time, the more letters there were, the slower the search, with the time rising almost in a straight line with the number of letters: each additional letter added a roughly fixed amount of time.

There was one more detail. When the target wasn't on the screen, subjects had to look at every letter before they could be sure; when it was, they found it on average halfway through. So when the target was absent, search time rose with the number of letters about twice as fast as when it was present. That's exactly the result you'd get from checking one letter at a time.

Their explanation was that the brain can process individual features like color and shape across the whole field of view at once, which is why a unique color or a unique shape pops out. But binding "green" to "T-shaped," recognizing "a green T," takes attention, which has to stick them together one at a time.

Feature integration theory holds that individual features can be processed all at once, but combining several features into a single whole object requires attention to bind them one at a time. If attention really is glue, then when there isn't enough glue, features can get stuck together wrong. In 1982, Treisman and Hilary Schmidt published a set of experiments in which several colored letters flashed briefly on the screen while subjects were kept busy with another task. Subjects often reported seeing combinations that weren't on the screen at all: shown a red X and a green O, for instance, they would say they'd seen a red O. They had seen both the colors and the shapes; they had just stuck them together wrong. This is called an illusory conjunction. Attention is the glue that binds scattered features into a whole object.

1999: Looking Without Seeing

In the 1970s, Ulric Neisser of Cornell University laid three separately filmed recordings on top of one another: a team in white shirts passing a ball, a team in black shirts passing a ball, and a woman with an umbrella walking through the frame. Once superimposed, all three were semi-transparent, like layers of ghosts overlapping in the same space. Subjects had to count the passes made by one of the teams. The woman with the umbrella was in the frame for about four seconds, and only about one in five people saw her. But the result had a loophole: the picture was semi-transparent to begin with, hard to make out, so missing her wasn't surprising.

In 1999, Daniel Simons and Christopher Chabris at Harvard set out to close that loophole. They shot a new video, real and opaque, and also made a semi-transparent version for comparison. The video ran 75 seconds: two teams of three, one in white shirts and one in black, moving around in front of a bank of elevator doors and passing an orange basketball. Subjects had to watch one of the teams and silently count how many passes it made. In the easy version of the task, they counted all passes; in the hard version, they kept separate counts of bounce passes and aerial passes.

Between 44 and 48 seconds into the video, something unexpected happened. In half the recordings, a tall woman holding an open umbrella walked across the frame from left to right; in the other half, a shorter woman in a full-body gorilla suit did the same. It lasted about five seconds. Afterward, the researchers first asked for the pass count, then asked a series of increasingly specific questions: Did you notice anything unusual? Did you see anyone besides the players? Only at the end did they ask directly: Did you see a gorilla walk across the screen?

In all, 228 subjects took part, most of them Harvard undergraduates. Overall, 46 percent failed to notice the person who had walked into the scene at all. The woman with the umbrella was easier to see than the gorilla: 65 percent noticed her, while only 44 percent noticed the gorilla. The opaque video made the intruder easier to see than the semi-transparent one, and the harder the task, the fewer people saw it.

The biggest difference came down to which team people were watching. The gorilla was black. Of those counting the black team's passes, 58 percent saw it; of those counting the white team's, only 27 percent. People counting the white team had been shutting out everything black, and along with the players in black, the black gorilla got shut out too.

The version that later spread widely was a separate one they filmed: the gorilla walks to the middle of the frame, stops, turns to face the camera, thumps its chest, and walks off, staying in view for about nine seconds in all. Even so, in that version about half the viewers missed it. When told there had been a gorilla, many didn't believe it until they watched the video again.

Inattentional blindness is the failure to see an obvious, unexpected thing right in front of you when your attention is focused on something else. The name was coined by the psychologists Arien Mack and Irvin Rock in the 1990s, and in 1998 they published a book with that title. The experiment shows that looking and seeing are not the same thing. Light entering the eye doesn't mean the thing has entered awareness. What isn't attended to is, as far as experience goes, as good as absent.

The study won the 2004 Ig Nobel Prize in Psychology. In 2010, Simons and Chabris turned it into a book, The Invisible Gorilla. More on this in the next post. 🧪

Timeline

• 1950s–1960s: Alfred Yarbus records eye movements with a suction-cap device

• 1964: Yarbus receives his doctorate in biology for his eye-movement research

• 1965: Yarbus publishes Eye Movements and Vision in Russian; the English translation follows in 1967

• 1966: Kahneman and Beatty publish their experiment on pupil size and memory load in Science

• 1970s: Neisser's superimposed "selective looking" videos; most viewers miss the woman with the umbrella

• 1973: Kahneman publishes Attention and Effort, setting out the capacity model

• 1980: Posner publishes "Orienting of Attention"

• 1980: Treisman and Gelade publish "A Feature-Integration Theory of Attention"

• 1982: Treisman and Schmidt publish their experiments on illusory conjunctions

• 1998: Mack and Rock publish Inattentional Blindness

• 1999: Simons and Chabris publish the gorilla experiment

• 2004: The gorilla experiment wins the Ig Nobel Prize in Psychology

• 2010: Simons and Chabris publish The Invisible Gorilla

• 2011: Kahneman publishes Thinking, Fast and Slow

Sources

• Tatler et al. — Yarbus, eye movements, and vision (i-Perception, 2010)

• Cabinet Magazine — Sasha Archibald on Yarbus

• Kahneman & Beatty — Pupil Diameter and Load on Memory (Science, 1966)

• Kahneman — Attention and Effort (Prentice-Hall, 1973)

• Posner — Orienting of Attention (1980)

• Treisman & Gelade — A Feature-Integration Theory of Attention (1980)

• Treisman & Schmidt — Illusory Conjunctions in the Perception of Objects (1982)

• Neisser & Becklen — Selective Looking: Attending to Visually Specified Events (1975)

• Simons & Chabris — Gorillas in Our Midst: Sustained Inattentional Blindness for Dynamic Events (1999)

• APA Monitor — Inattentional Blindness (2001)