Preface: Before there were computers that learned from reward, there were cats escaping puzzle boxes, rats pressing levers, and a pigeon trained to steer a missile. This is the strange animal-lab origin story of the AI that beat the world at Go.
Following the previous #AI series and #MarvinMinksy series. This post is co-written with Claude.

The usual story of reinforcement learning begins in a computer lab. A program plays a game against itself millions of times, keeps what works, discards what doesn’t, and one day beats the best human player in the world. The vocabulary of that story — reward, punishment, trial and error, reinforcement — sounds like engineering language, invented for machines. In fact, every one of those words came out of animal laboratories, and most of them existed before the first electronic computer was built. The people who built AlphaGo did not invent the idea of a machine that learns from consequences. They inherited half a century of work from psychologists who spent fifty years watching cats, dogs, rats, and pigeons figure out how to get fed.

1896: Chickens in a Philosopher’s Basement

Edward Thorndike was a graduate student at Harvard who wanted to study how animals learn. Harvard would not give him a laboratory. So he kept chicks in his rented room, until his landlady objected. His teacher, William James, then the most famous psychologist in America, let him move the birds into the basement of his own house in Cambridge. James’s children sometimes came down to watch.

In that basement, Thorndike stood books on end to build simple mazes, and timed how long the chicks took to find their way out to food and company. The setup was crude, but the question behind it was not. Thorndike wanted to know whether animals reason their way to solutions, or whether something simpler was at work.

He was reacting against a popular kind of book at the time: collections of anecdotes about clever animals, the dog that opened a gate latch, the cat that rang a doorbell. Thorndike thought these books proved nothing. A story about a dog opening a gate tells you only that the dog opened the gate. It does not tell you how the dog learned to do it, or whether the dog understood what it was doing. He complained that the existing literature was less a psychology than a eulogy of animals. What he wanted was measurement. He was, it has to be said, another scientist who pushed his curiosity as far as it would go, the same way the best artists push theirs.

1898: The Cat in the Box

In 1897, Thorndike moved to Columbia on a fellowship and switched from chicks to cats. He built a device he called a puzzle box: a wooden crate about twenty inches long, with a door held shut by a latch. Inside the box, a loop of string, a lever, or a button would release the door. Outside, in plain sight, was food.

A hungry cat was placed in the box. At first it made no attempt to study the latch. It clawed at the bars, pushed its nose through the gaps, bit at the walls, and circled the box with no apparent method. Eventually, by accident, it pulled the string. The door opened. The cat got out and ate. Then Thorndike put it back in. (😅)

The second time, the cat got out a little faster. The third time, a little faster again. Over many repetitions, the useless movements gradually dropped away, and the cat went more and more directly to the string. Thorndike recorded the escape time for every trial and plotted the results on a graph. The curve sloped downward, but not in a sudden drop. A sudden drop would have meant the cat understood something all at once. The actual curve was gradual. This is the origin of the learning curve. The gradualness was the point. If the cat had been reasoning, there would have been a moment of insight, followed by fast escapes every time. Instead the data showed slow, incremental improvement. Thorndike concluded that the cat never “figured anything out.” Each time the successful action brought food, it was “stamped in” a little deeper, and the unsuccessful actions were slowly stamped out.

In 1898 he published these results as his doctoral dissertation, Animal Intelligence: An Experimental Study of the Associative Processes in Animals. It was the first dissertation in the history of psychology to use animals rather than humans as subjects. In 1911, he gave the principle a name. The Law of Effect: an action followed by a satisfying result becomes more likely to be repeated; an action followed by discomfort becomes less likely to be repeated.

That sentence is the conceptual core of reinforcement learning. A modern algorithm that raises the probability of a move because the move led to a win is doing the same thing Thorndike described his cat doing with the string. Late in his career, Thorndike revised his own law: he found that reward was a far more powerful teacher than punishment, and that punishment could not simply erase what reward had built.

1903: The Dog That Didn’t Have to Do Anything

Around the same time, in St. Petersburg, Ivan Pavlov was studying digestion in dogs. He noticed that the dogs began to salivate before the food arrived: at the sight of the lab assistant, at the sound of footsteps. He began pairing a neutral signal with food, and found that eventually the signal alone was enough to trigger salivation. He received the Nobel Prize in 1904, for his work on digestion rather than conditioning. The conditioning experiments became famous later.

Pavlov’s work and Thorndike’s are often taught together, but the difference between them matters for everything that follows. Pavlov’s dog does not have to do anything. The bell rings, the food comes, and the dog learns to predict the food. Thorndike’s cat has to act. The food appears only if the cat does the right thing. One is learning to predict. The other is learning to choose.

Classical conditioning: an animal learns that one event predicts another. Instrumental conditioning (later called “operant conditioning”): an animal learns that its own action produces a result. Reinforcement learning in machines needs both. A system has to predict which situations lead to good outcomes, and it has to choose the actions that lead it toward those situations. In 1900, these were a cat and a dog in two different countries. By the 1950s, they would become two separate problems in mathematics. The word “reinforcement” itself entered English through the 1927 English translation of Pavlov’s lectures.

1938: The Box That Ran Itself

B.F. Skinner entered Harvard as a graduate student in 1928. By his own account, he had tried to become a novelist before that, and failed. He took Thorndike’s puzzle box and changed one thing, which turned it into an instrument. In Thorndike’s box, the animal escaped and the trial was over. In Skinner’s box, the animal stayed inside. A rat pressed a lever, a food pellet dropped into a tray, and the rat could press again. The experiment could run for hours without anyone touching it. Skinner connected the lever to a pen that drew a continuous line on a rotating roll of paper, rising slightly with every press. He called it a cumulative recorder. The shape of the line showed the rate of behavior at a glance: steep when the rat pressed often, flat when it stopped.

He coined the term operant for behavior that acts on the environment to produce consequences. In 1938 he published The Behavior of Organisms. The device everyone came to call the “Skinner box” became the standard equipment of American psychology for the next forty years.

Skinner had rats press a lever for food, but he changed the rules for how many presses, and when, would earn food, and then watched how the rats’ behavior changed in response. There were four kinds of rules, and each has an everyday counterpart.

The first: food for every press. This was the simplest, and the rats learned it fastest. But they also gave up fastest: once the food stopped, the rats quickly noticed, and stopped pressing. It is like a vending machine. You put in a coin, something comes out. If one day you put in a coin and nothing comes out, you try once or twice and walk away. You don’t keep feeding it coins.

The second: food for every ten presses. The number was fixed. The rats would press in a rapid burst, take a short break after getting the food, then press in another rapid burst. It is like being paid by the piece: finish ten pieces, get paid. After getting paid, people take a breather, then start rushing again.

The third: food only after a minute has passed. This had nothing to do with how many times the rat pressed, only with time. After the last delivery of food, no amount of pressing did anything; once the minute was up, the next press counted. The rats quickly learned: right after being fed, they barely pressed at all, and as the minute approached, they pressed more and more often. It is like waiting for a package or for an exam. At first you’re in no hurry, but the closer the deadline gets, the more anxious you become, and the more often you check.

The fourth: random. Sometimes food came after three presses, sometimes after twenty, sometimes after seven, and the rat had no way to predict it. The result was that the rats pressed at a very high rate, steadily, without pausing, because any press might be the one. And even after the food stopped entirely, they kept pressing for a very, very long time before giving up. This is the slot machine. It is also scrolling on your phone: you never know whether the next refresh will turn up something interesting, so you keep scrolling.

Different rules of reward produce completely different learned behavior. The same rat, the same lever, the same food: change only the way the reward is given, and it becomes an entirely different pattern of behavior. And the most persistent responding came from the unpredictable rule: an animal rewarded after a random number of actions will keep pressing even after the rewards have stopped. Those of you who like casinos, take note.

1943: Pigeons in a Missile

In 1940, on a train to Chicago, Skinner watched a flock of birds flying in tight formation, rising and wheeling alongside the car. He began to wonder whether birds could be used to steer.

By then he was at the University of Minnesota, and had switched from rats to pigeons. He trained pigeons to peck at an image of a target on a screen, rewarding them with grain. A peck at the center of the screen meant the target was straight ahead. A peck off-center meant the target had drifted, and the direction of the drift could be translated into a steering correction. If the screen showed what the nose of a falling bomb was seeing, the pigeon could guide it.

After Pearl Harbor, the military took an interest in this strange research project. The first money came from an unexpected place: General Mills, the cereal company. They gave Skinner $5,000 and a room on the top floor of a flour mill in Minneapolis. In 1943, the National Defense Research Committee signed a $25,000 contract with General Mills, and the project was named Project Pigeon.

Skinner’s team built a nose cone divided into three compartments, each with one pigeon facing its own screen, so that if one bird made a mistake, the other two could outvote it. The pigeons performed reliably through noise, vibration, and the acceleration of a dive. The scientists evaluating the project could not get past the image of birds flying a bomb, and it was dropped in 1944. After the war, however, the Navy revived it, and canceled it again in 1953, by which time electronic guidance had become reliable enough. Skinner later described it as the history of a crackpot idea.

Project Pigeon is usually treated as a wartime curiosity. But it was also the first serious attempt to install a trained learning system inside a machine and hand it control. The learning system just happened to be a bird.

1948: The Superstitious Pigeon

In 1948, Skinner ran an experiment that exposed the central weakness of learning from reward. He put hungry pigeons in a box and dropped grain into the tray every fifteen seconds, no matter what the pigeons did. Nothing the pigeons did had any effect on the food.

Even though they didn’t need to learn anything to get fed, the pigeons picked up some new tricks anyway. One turned in counterclockwise circles between feedings. Another kept thrusting its head into an upper corner of the cage. Another bobbed its head up and down. Each pigeon had been doing something at the moment the food happened to drop, and that action got stamped in, exactly as the Law of Effect predicts. The pigeon then repeated it, which made it more likely to be doing the same thing the next time food dropped, which stamped it in deeper. Skinner titled the paper “‘Superstition’ in the Pigeon.”

The pigeons had not failed to learn. They had learned perfectly, from a signal that had nothing to do with their behavior. The reward told them that something they did was good, but not which thing. This became the credit assignment problem: when a reward arrives, which of the actions that came before it deserves the credit?

For the cat in the puzzle box, pulling the string and getting out were only a second or two apart. For a Go program, a single move and the end of the game can be two hundred moves apart. Skinner’s circling pigeons are the simplest demonstration of why this problem is so hard. Solving it would occupy everyone who built learning machines for the next forty years.

1948: Two Keys for the Schoolmaster

In the summer of 1948, at the National Physical Laboratory outside London, Alan Turing wrote a report titled “Intelligent Machinery.” In it he described something he called an unorganized machine: a network of simple, randomly connected components, with no built-in program, that could be shaped by training into something useful. He suggested that the cortex of an infant might be exactly this kind of machine. One version, which he called the P-type, had only two inputs from the outside world besides its senses: one representing pleasure, or reward, and one representing pain, or punishment. When the machine did something wrong, pain would cancel whatever it had just tried. When it did something right, pleasure would make the change permanent. Turing worked through examples of training it by hand, with pencil and paper. The director of the laboratory, Sir Charles Darwin, grandson of the naturalist, is reported to have dismissed the report as a schoolboy essay.

Turing never let go of the idea. In 1950, in “Computing Machinery and Intelligence,” he proposed that instead of trying to program an adult mind, one should build a child’s mind and then educate it, using rewards and punishments the way a teacher would. In a talk around 1951, he put it more concretely: the machine should have two keys for the schoolmaster to press, one representing pleasure and one representing pain. That is the entire architecture of reinforcement learning, proposed by the founder of computer science in the vocabulary of a nineteenth-century classroom. What Turing lacked was a way to make it actually work. There was no algorithm for deciding which internal connections to change, no machine fast enough to run millions of trials, and no answer to the problem Skinner’s pigeons had just demonstrated. Then, in 1951, a twenty-four-year-old Princeton mathematics graduate student named Marvin Minsky built a machine out of vacuum tubes. 🧪