Preface: By the late 1970s, computer science had forgotten that you can learn from rewards, except for one stubborn man. Following the previous #AI series and #MarvinMinsky series. This post is co-written with Claude.

The story of reinforcement learning is usually told starting from AlphaGo, or from the 2025 Turing Award: two pioneers, a lifetime of work, finally honored with computer science's highest prize. But when these two men started out, they weren't founding a field. They were hired to evaluate someone else's idea, one that most of their peers didn't take seriously. They spent years on a project nobody cared about, going back to read animal psychology from half a century earlier, and digging back up the idea Samuel had sealed inside his checkers program. Barto later recalled that the field was "not fashionable" at the time, and that they were "kind of in the wilderness."

Klopf: The Hedonistic Neuron

A. Harry Klopf, born in 1941, was a scientist at Wright-Patterson Air Force Base in Ohio, responsible for machine intelligence research in the avionics directorate. He had an idea that ran against the mainstream of his day.

In the 1960s and 70s, most people who studied adaptive systems believed that the goal of a learning system was to maintain balance. If the temperature is too high, bring it down; too low, bring it up; if the error is too big, correct it until the error is zero. This is called homeostasis. Neural network research of that era followed the same logic: give the machine a correct answer, and however wrong it is, correct it by that much, until it stops making mistakes.

Klopf thought this missed the point. Living things aren't after balance. They're after more: more food, more pleasure, less pain. He gave this state a name, "heterostasis," meaning pushing certain things as high as they will go rather than holding them in the middle. And he went a step further: it isn't just the whole animal that seeks pleasure. Every neuron in the brain is a tiny hedonist, too. It strengthens the connections that bring it a "pleasure" signal and weakens the ones that bring "pain."

This sounds a lot like Thorndike's Law of Effect, just shrunk down from a cat to a single cell. And it differed in one fundamental way from the error correction that was fashionable at the time. Error correction requires someone to tell the machine what the right answer is. Seeking pleasure only requires telling it whether the outcome was good or bad; working out what's right is up to the machine, by trial. Just as Thorndike, standing outside the puzzle box, never taught his cat which string to pull.

Klopf had no experimental evidence and no algorithm that could actually run. But he had one thing: he was inside the Air Force, and he knew how to get money for a curious idea.

1977: A Project That Only Delivered Reports

In 1977, the Air Force Office of Scientific Research began funding a project at the University of Massachusetts Amherst. Three computer science professors, Michael Arbib, William Kilmer, and Nico Spinelli, were its principal investigators. The project's mission was unusual. It wasn't to build a machine or solve a military problem, but to evaluate whether Klopf's theory had any scientific merit, comparing it against all previous research on learning and adaptive systems. The only deliverables were a report each year and a final report.

To do this, the project hired a postdoc: Andrew Barto.

Barto was born in 1948 and went to the University of Michigan as an undergraduate, where he started out studying shipbuilding, in naval architecture and engineering. Later he became interested in something else: using computers and mathematics to model the brain. In 1970 he graduated with distinction in mathematics, and in 1975 he earned a PhD in computer science at Michigan, with a dissertation on cellular automata, a mathematical model made of many simple cells that influence one another according to rules. He spent two years as an assistant professor at the State University of New York at Binghamton, then came to Amherst.

It was a strange job. The project asked for almost nothing beyond a report each year. Barto later described it as pure curiosity-driven basic research. They could spend nearly all their time on one thing: reading. Reading Klopf's ideas, reading all the literature connected to them, and then figuring out what, exactly, was new about them.

1978: A Psychology Student

Not long after Barto arrived, a young man joined the project: Richard Sutton.

Sutton was born around 1957 in Toledo, Ohio, and grew up near Chicago. At Stanford he studied not computer science but psychology, with a minor in computer science. He graduated in 1978 and had just been admitted to the computer science graduate program at Amherst. He became Barto's first doctoral student. A little later, Charles Anderson, a fresh computer science graduate from the University of Nebraska, joined as well.

A mathematician who had started out in shipbuilding and a student of psychology: the pairing happened to fit exactly what they had been asked to evaluate. Half of Klopf's idea came from animal psychology, and the other half pointed toward a machine that could compute. Sutton knew the work of Thorndike, Pavlov, and Skinner well and understood how animals learn; Barto knew how to turn an idea into mathematics.

The more they read, the more they noticed something odd. The kind of learning Klopf was describing, where no teacher supplies the correct answer and the learner is only told whether the outcome was good or bad and left to pursue long-term reward on its own, had long been common knowledge in psychology. As earlier posts in this series showed, it had also made its way into engineering and artificial intelligence: Minsky's SNARC, Samuel's checkers, Michie's matchboxes. But by the end of the 1970s, it had all but vanished from computer science. When people talked about "learning," they almost always meant something else: showing a machine the correct answers and having it learn to imitate them.

Klopf hadn't invented a new idea. He had simply, stubbornly, remembered it in an era when everyone else had forgotten.

Rereading Pavlov: Surprise Is the Teacher

The most important thing they found in psychology wasn't Thorndike. It was Pavlov.

As the first post explained, reinforcement learning needs two things: prediction and choice. Thorndike's cat stands for choice, Pavlov's dog for prediction. The earlier machines, SNARC and the matchboxes, learned choice: which move to make. But they all got stuck in the same place: the reward came too late.

Barto and Sutton realized that the answer might be hiding in the dog's half.

In the decades after Pavlov, psychologists ran a great many conditioning experiments and discovered a crucial phenomenon. Suppose an animal has already learned that when a bell rings, food follows. Now, along with the bell, a light comes on, and then the food arrives. After many repetitions, the light is shown on its own, and the animal barely reacts. It hasn't learned the connection between the light and the food.

Why? Because the food was already predicted by the bell. The light told the animal nothing new, and when the food arrived, nothing about it was a surprise.

In 1972, the psychologists Robert Rescorla and Allan Wagner captured phenomena like this in a concise model. Its core is a single sentence: how much an animal learns depends on how surprised it is. Prediction error is the gap between what actually happens and what was predicted. The bigger the gap, the more is learned; with no gap, nothing is learned at all. Surprise is the teacher. A reward that was completely expected teaches an animal nothing.

In 1981, Barto and Sutton proposed their own model of conditioning, pushing this idea a step further. Rescorla and Wagner's model looked only at the final outcome of each trial: did the food come or not? But in a real experiment, time is continuous. The moment the bell rings, the dog's prediction about the future has already changed; it doesn't wait for the food to arrive. What they wanted to describe was this kind of prediction, updated moment by moment.

Picture what happens during a single trial. The bell rings, and five seconds later, the food arrives. In Rescorla and Wagner's model, those five seconds are a single unit: each trial counts as one round, and at the end of the round, the model checks whether the food came and calculates the surprise once. What happens inside those five seconds doesn't matter to it.

But it matters to the dog. The moment the bell rings, the dog's anticipation rises and it starts to drool, and the closer the food gets, the more it drools. If the gap between the bell and the food is too long, the dog has a hard time connecting the two; if the food comes first and the bell afterward, the dog learns almost nothing. The order and the length of time are crucial to learning, and a model that only settles accounts at the end of each round can't see any of it.

There was an even stranger phenomenon, one Pavlov himself had discovered. First, the dog learns that food follows the bell. Then the training changes: first a light comes on, then the bell rings, but no food is given. After a few repetitions, when the light comes on by itself, the dog drools too. Yet the light had never once appeared together with food.

This is called second-order conditioning. Think of ordering takeout. At first, what makes you happy is the food itself. After you've ordered a few times, the knock at the door makes you happy, because the knock means the food is here. Later still, when your phone shows "Your driver is 500 meters away," that makes you happy too. But you can't eat a notification. It isn't even the knock. All it tells you is that the knock is coming soon. You're happy about good news that's on its way.

It's the same with the dog and the light. The light is the notification, the bell is the knock, and the food is the food. The light never appeared together with food; it only ever appeared together with the bell. But in the dog's mind, the bell was already a good thing, so the light that announced the bell became a good thing too.

This shows that an animal's learning signal comes not only from real rewards, but also from changes in prediction. When the bell rings after the light, the dog's expectations about the future suddenly improve, and that improvement is itself a kind of reward, enough to give the light meaning.

This is what Barto and Sutton's model set out to capture, in two parts. First, predictions aren't updated only at the end of a round; they are updated at every moment in time. Second, one prediction can learn from another: the light's value comes from the bell's value, and the bell's value comes from the food. Value is passed back like this, one step at a time.

This is exactly what the Bellman equation says: the value of a situation equals the reward right now, plus the value of the next situation. Only this time it wasn't written on paper by a mathematician. It was worked out by a dog in a laboratory.

This problem is the same as the one in Samuel's checkers program. Every time Samuel's program made a move, it updated its judgment of the position; every time Pavlov's dog heard a signal, it updated its expectation of food. One was a program an engineer wrote at IBM, the other was animal behavior a psychologist observed in a lab, and they were doing the same thing.

1983: The Actor and the Critic

In 1983, Barto, Sutton, and Anderson published a paper titled "Neuronlike Adaptive Elements That Can Solve Difficult Learning Control Problems." The problem they chose was the very one Michie's BOXES had tackled fifteen years earlier: balancing a pole on a cart.

The cart, the pole, and the track were all simulated on a computer, just as in BOXES. The rules were the same too: at every moment, the cart could only be pushed left or right with a fixed force, and the states of the cart and pole were chopped into cells. The hardest part was this: the only signal the machine ever received came at the moment the pole fell. Before that, whether it was pushing well or badly, nobody told it anything. And the fall of the pole was often sealed dozens of steps earlier. This is the credit assignment problem in its purest form.

Their solution was to split the machine in two.

One part was called the "actor." Its job was to make decisions: in each cell, push left or push right. This is the half that belongs to Thorndike's cat: learning to choose.

The other part was called the "critic." It made no decisions at all. It did only one thing: predict. For each cell, it had to learn to judge, with the cart and pole in that state, how far they were from falling. This is the half that belongs to Pavlov's dog: learning to predict.

The key was how the two worked together. At every step, the critic took a look: after that push, did things get better or worse? If its prediction for the new state was more optimistic than its prediction for the old one, that meant the actor had just pushed well, and the critic gave the actor a little "pleasure." If it was more pessimistic, it gave a little "pain."

This way, the actor didn't have to wait for the pole to fall. It got feedback from the critic with every step. And where did the critic's own predictions come from? It learned them the same way: it used the next moment's prediction to correct this moment's prediction. The moment the pole fell simply nailed a definite answer onto the end of the chain.

This was Samuel's "first impression and second thought," lifted out of checkers and turned into a separate component.

The result was that the machine made of an actor and a critic learned to balance the pole faster and more reliably than Michie's BOXES. The paper got few citations at first, but later became one of the most cited papers in the field. The "actor-critic" design is still one of the most widely used structures in reinforcement learning today.

In 1984, Sutton finished his doctoral dissertation, and its title named the very problem this series has been following: "Temporal Credit Assignment in Reinforcement Learning." In 1986, Anderson's dissertation combined this approach with multilayer neural networks, letting the machine learn on its own how to divide up the states instead of having a person chop them into cells in advance. It was one of the earliest forerunners of today's deep reinforcement learning.

1988: Temporal Difference Learning

By 1988, Sutton was working at GTE Laboratories in Massachusetts. That year, he published a paper titled "Learning to Predict by the Methods of Temporal Differences." It took the ideas of the past decade off the pole, out of the dog, and out of the checkers program, and wrote them down as a general rule that could be applied to any problem. Temporal difference learning is using the next moment's prediction to correct this moment's prediction.

Take an example everyone has lived through. Sutton and Barto later used a similar one in their textbook: driving home from work. At 6:00, you leave the office. You glance at the time and the weather and figure you'll be home in thirty minutes. At 6:05, you get in the car and notice it's started to rain. Traffic is worse in the rain, so you change your estimate to forty minutes. At 6:20, you get on the highway, and it's moving better than you expected. You change your estimate again, to thirty-five minutes. At 6:30, you exit the highway and get stuck behind a slow truck. At 6:43, you get home. The question is: how should that 6:00 estimate of thirty minutes be corrected?

One way is to wait. Wait until you get home, see that it actually took forty-three minutes, and then look back and tell yourself: next time I leave at six, I should guess longer. That's the matchbox method: wait until the whole game is over, then settle the accounts.

The other way is not to wait. At 6:05, when you notice the rain and change your estimate from thirty to forty, you already know at that moment that the 6:00 estimate was too optimistic. You don't have to wait until you get home; you can immediately use the new 6:05 estimate to correct the old 6:00 one. All the way home, every estimate gets corrected by the one that comes after it, and the moment you walk through the door simply nails a real number onto the last stretch.

The gap between two successive predictions is the "temporal difference." It's the same thing as the prediction error Rescorla and Wagner described, except that instead of comparing a prediction with the final outcome, you compare this moment's prediction with the next moment's.

This has three advantages.

First, you learn early. You don't have to wait for something to end; you learn a little with every step. And if something never ends, like a person's life, this is the only way.

Second, you learn precisely. A lot happens on the way home: the rain, the open highway, the truck. If you only look at the final forty-three minutes, you can't tell which stretch caused the problem. Temporal difference spreads the responsibility across each stretch: how much the estimate changed when the rain started, how much when you got stuck behind the truck. This is the answer to the credit assignment problem.

Third, you don't need a map. Bellman's equation also works out the value of one situation from the value of the next, but it requires knowing the entire map in advance: how long each road is, how likely it is to rain. Temporal difference doesn't. All it needs is for you to actually drive home.

Bellman defined value. Samuel was the first to approximate it from experience, in checkers. Rescorla and Wagner saw prediction error in animals. Barto and Sutton put them together in the same machine. Sutton's 1988 paper finally wove these threads into a single rule.

Ninety years had passed since Thorndike put his first cat in a puzzle box.

One Rule

Temporal difference learning, in 1988, was just a rule. It worked on the cart and pole, and on a few simple prediction problems. Whether it could accomplish something truly hard, like beating the best humans at a complex game, nobody yet knew.

Klopf's "hedonistic neuron" became a book in 1982. His theory itself was never borne out. But the "evaluation project" he had persuaded the Air Force to fund ended up, without meaning to, rescuing a field that was close to being forgotten. He died in the late 1990s and never saw what it became.

The year after Sutton's paper, a doctoral student at Cambridge turned in his dissertation, titled "Learning from Delayed Rewards." Three years after that, a researcher at IBM took Sutton's rule and had a program play itself at backgammon more than a million times. Some thirty years earlier, at IBM's lab in Poughkeepsie, Samuel's checkers program had played itself the very same way. 🧪

Sources:

• CACM — Barto, Rediscovering Reinforcement Learning

• Wikipedia — Harry Klopf

• Wikipedia — Andrew Barto

• Wikipedia — Richard S. Sutton

• Boston.com / AP — AI pioneers who channeled hedonistic machines

• WBUR — UMass Turing Prize

• Sutton & Barto 1987 — A Temporal-Difference Model of Classical Conditioning

• Barto, Sutton & Anderson 1983 — Neuronlike Adaptive Elements