Preface: Following the previous #AI series and #MarvinMinsky series. This post is co-written with Claude.
The "attention" in machine learning is a borrowed word. Before it was borrowed, psychologists had been studying it for more than a century. In 1890, William James of Harvard published The Principles of Psychology. The two-volume work took him twelve years to write. Psychology had only just split off from philosophy to become a discipline of its own; in Germany, Wilhelm Wundt had set up the world's first psychology laboratory only in 1879. James ran few experiments. His method was mainly introspection: observing how his own mind worked, then writing it down.
The book has an entire chapter on attention, and it opens with a passage that has been quoted ever since: "Everyone knows what attention is. It is the taking possession by the mind, in clear and vivid form, of one out of what seem several simultaneously possible objects or trains of thought. Focalization, concentration, of consciousness are of its essence. It implies withdrawal from some things in order to deal effectively with others."
The passage can be taken apart. First, the objects of attention aren't only things out in the world; they include trains of thought. You can attend to a sound outside the window, or to something you're thinking about. Second, "simultaneously possible." At any given moment, several things in front of your eyes, in your ears, and in your head are competing to be noticed. Attention doesn't happen in a vacuum. It happens in the middle of a competition.
Third, "clear and vivid." What gets attended to isn't just selected; it becomes sharper. Focus on one person talking in a noisy room, and their voice seems louder and clearer than everyone else's, even though the sound itself hasn't changed. Fourth, "withdrawal." To do one thing well, you have to pull back from others. That's the cost. The things you let go of don't disappear because you've stopped noticing them. You've just stopped processing them for now.
James also wrote about attention's opposite. Its real opposite, he said, is "the confused, dazed, scatter-brained state which in French is called distraction, and Zerstreutheit in German." The German word means, literally, "scattered." Attention gathers the mind together; distraction scatters it. In the same chapter he wrote something bolder: "My experience is what I agree to attend to. Only those items which I notice shape my mind." Far more happens in the world than we ever experience, and what sorts the one from the other is attention.
James also distinguished two kinds of attention. One is active: you decide to finish reading the page, so you have to keep dragging your thoughts back to it. The other is passive: a loud bang, a flash of light, a familiar name, and your attention is pulled toward it without your deciding anything. More than a century later, neuroscientists found two systems in the brain that correspond to these two kinds of attention. From the very beginning, attention was defined as a trade-off.
But for the next half century, the subject all but disappeared from psychology. Behaviorism, which rose in the early twentieth century, held that psychology should study only observable behavior; things like "attention" and "consciousness," which could only be described through introspection, didn't count as science. Attention didn't return to the laboratory until after the Second World War. The war left behind a set of very practical problems: radar operators had to watch several screens at once, and pilots had to pick out instructions from the control tower amid a din of radio chatter. How much information can a person actually handle at once? That question brought psychologists back to attention.
1953: The Cocktail Party
Real experimental research wouldn't begin for more than half a century, and its starting point was a scene everyone has lived through: at a noisy party, you can make out just what the person across from you is saying and shut out every other voice in the room. The cocktail party effect is the ability to focus attention on one voice in an environment where many voices are speaking at once.
In 1953, Colin Cherry, in Britain, brought this scene into the laboratory. He had subjects put on headphones and played two different spoken passages at the same time, one in the left ear and one in the right, and asked them to repeat aloud only one of them. Afterward, they remembered almost nothing of what the other ear had heard. When the passage in that ear switched partway through from English to German, they didn't notice; when it switched to a recording played backward, they only sensed that something was a little off. But if the voice in that ear switched from a man's to a woman's, they caught it.
The unattended ear wasn't hearing nothing. It heard the pitch of the voice, but not the meaning of the words.
(writing)