Preface: Following the previous #AI series and #MarvinMinksy series. This post is co-written with Claude.

Three claims about Arthur Samuel get repeated more than any others: that he coined the term "machine learning" and gave it its classic definition, "the field of study that gives computers the ability to learn without being explicitly programmed"; that in 1962 his program beat a state checkers champion; and that this was the first time a machine had beaten a human expert at a game of skill.

These claims turn up again and again in textbooks and lecture slides. The first is only half right, the second is wrong, and the third needs a heavy discount. But underneath the inflated stories, what Samuel actually accomplished matters more than the legend: he was the first to get a machine to judge whether a move was good or bad from its own experience, without waiting for the game to end.

1956: Checkers on Television

On the morning of February 24, 1956, IBM broadcast a live checkers demonstration on television. One of the players was an IBM 701, the company's first large commercial stored-program computer, a machine that filled an entire room.

IBM's founder, Thomas Watson Sr., made a point of arranging for the company's shareholders to watch, and predicted that after the demonstration IBM's stock would rise fifteen points. Supposedly it did. A few months later, Watson died. Standing behind the machine was a soft-spoken engineer in his fifties. He wasn't a mathematician or a psychologist. His specialty was vacuum tubes.

1901: The Vacuum Tube Engineer

Arthur Samuel was born in 1901 in Emporia, Kansas. He studied at the College of Emporia while working on and off at General Electric in Schenectady, New York. He earned a master's degree from MIT in 1926, stayed on as an instructor for two years, and joined Bell Labs in 1928.

He spent nearly two decades at Bell Labs, working mainly on electron tubes, the vacuum tubes we met in the SNARC post. During the war, he helped develop a key component of radar: a switch that protected the receiver from being burned out in the instant the radar fired off its powerful pulse.

In 1946 he became a professor of electrical engineering at the University of Illinois and joined a project to build one of the early electronic computers. Money was tight, and Samuel came up with an idea: write a program that played checkers and have it challenge the world champion. If it won, there would be no better way to show everyone how powerful electronic computers could be. At the time, he thought it wouldn't be hard.

The computer wasn't finished in time, so the checkers program was never completed at Illinois. In 1949 Samuel moved to IBM's lab in Poughkeepsie, New York, to help the company move from electromechanical equipment to electronic computers. He brought the checkers idea with him. He later said that the problem of writing a program that seemed intelligent took up nearly all his spare time for as long as he was at IBM.

Why Checkers

In 1952, Samuel wrote his first checkers program on the IBM 701.

He chose checkers deliberately. Its rules are much simpler than chess, with only two kinds of pieces, ordinary pieces and kings, yet the number of possible positions is around five hundred billion billion, a 5 followed by twenty zeros. Simple rules meant the program wasn't too hard to write, and most of the machine's effort could go into "thinking." The huge number of positions meant there was no way to win by working through all of them. Bellman's curse of dimensionality applied here too. The program had to learn to judge whether the position in front of it was good, without being able to calculate all the way to the end of the game.

This was the same problem Hassabis ran into as a teenager writing his Othello program. Samuel's program also had two parts.

One part looked ahead. The program spread out every possible move, and every reply the opponent could make to each one, layer by layer like the branches of a tree. It looked a few moves ahead, then used minimax, assuming the opponent would always make the move that was best for them, to carry the scores back from the future to the present.

The other part was the evaluation function. After looking a few moves ahead, the game was still far from over, so the program had to give those future positions a score. Samuel's approach was to pick out a set of features, like how many pieces each side had left, how many kings, who controlled the center of the board, whose pieces were more exposed to capture, give each feature a weight, and add them up into a total score.

An evaluation function is a formula for scoring a position. It is made up of a number of features, each multiplied by a weight, and the sum is the position's score.

Up to this point, this was only a program that played checkers, not a program that learned. The weights were set by a person. If the person set them well, the program played well; if not, it kept playing badly. What Samuel wanted was for the program to adjust those weights itself.

Samuel had the program learn in two ways.

The first he called "rote learning." As it played, the program stored the positions it encountered along with the scores it had worked out for them. The next time it looked ahead and found a position it had seen before at the end of a branch, it didn't need to analyze it again; it could simply pull up the stored score. And that stored score had itself come from looking several moves ahead the last time. In effect, the program could see further than it appeared to. This kind of learning was very slow, but it kept improving, and it was especially useful in the opening and the endgame, where positions repeat most often.

The second kind of learning was more interesting, and it's what earned Samuel his place in the history of reinforcement learning. He called it "generalization learning." What it adjusted were the weights in the evaluation function.

The question was which way to adjust them. The most direct approach would be to wait until the game was over, then strengthen the judgments used in that game if it was won and weaken them if it was lost. That's the logic of machines like SNARC and the matchboxes: wait for the reward, then look back and hand out credit. But a game of checkers can run dozens of moves, and by the end there's no way to tell which of the earlier judgments went wrong.

Samuel's idea was: don't wait for the end of the game.

Before each move, the program gave the position in front of it a score. This was its "first impression," what the evaluation function could see at a glance. Then it looked a few moves ahead, spread out the branches, and carried the scores of those future positions back, arriving at a new score. This new score was its "considered judgment," because it took into account what might happen over the next few moves, and it was usually more accurate than the first impression.

If the two scores differed, that meant the evaluation function's first impression was off. So the program adjusted the weights to bring its next first impression closer to the considered judgment.

Here's an example. At first glance, the program thinks it has a slight edge in some position and gives it a score of +3. After looking a few moves ahead, it discovers that the opponent has a double jump coming that will take two of its pieces, and the score carried back is only −5. The gap is 8 points, which means the evaluation function is too optimistic about this kind of position. So the program checks which features produced the +3. Say "control of the center" was weighted too heavily and "how exposed my pieces are" too lightly. It turns the first down a little and the second up a little.

This way, the program learned with every move, without waiting for the game to be decided. Every lookahead was a small test, with the self that thought more deeply correcting the self that thought more shallowly.

This was Samuel's answer to the credit assignment problem. SNARC gave the credit to the last few steps before the reward, because all it could do was wait for the finish. Samuel's program didn't wait. At every step, it used the later judgment to correct the earlier one, and in this way the error was passed back, one step at a time.

The idea is strikingly close to Bellman's equation: the value of a position can be worked out from the positions that follow it. But there's a fundamental difference. Bellman needed to know all the rules of the world in advance and sit down to calculate every situation. Samuel didn't. He had the program estimate the value of each position, step by step, from the experience of actually playing. Thirty years later, Richard Sutton turned this idea into a formal algorithm called temporal difference learning. In their textbook, Sutton and Barto devote a whole section to Samuel's checkers program, treating it as the most important forerunner of the idea.

Two Selves

To let the program adjust its own weights, Samuel also designed a way for it to play against itself.

He had two versions of the program running at once, one called Alpha and one called Beta. Alpha kept adjusting its evaluation function during every game in the way just described, while Beta stayed fixed. The two played each other. If Alpha won, that meant its adjusted evaluation function was better, so Beta was replaced with Alpha's current version, and the games continued. If Alpha kept failing to win, that meant it had drifted in the wrong direction and needed a shake-up to start again from a different direction.

This was one of the first attempts to make a machine stronger by having it play itself. More than fifty years later, AlphaGo Zero used the same basic idea, on a scale vastly larger.

In 1959, Samuel published a paper titled "Some Studies in Machine Learning Using the Game of Checkers." In the abstract, he wrote that with only about eight to ten hours of machine playing time, the program could learn to play a better game of checkers than the person who wrote it. The paper helped popularize the term "machine learning."

But the definition most often quoted, "the field of study that gives computers the ability to learn without being explicitly programmed," can't be found in the paper. It's a later summary of Samuel's work that eventually got attributed to him.

1962: An Exaggerated Victory

By 1962, Samuel's program had been moved to the faster IBM 7094. In a public match, it played a Connecticut player named Robert Nealey, and won. IBM's promotional materials described Nealey as a former Connecticut checkers champion and one of the best players in the country. The news spread nationwide, and many people came away believing that checkers had been conquered by a machine.

It hadn't. At the time of the match, Nealey wasn't a state champion; he didn't win that title until four years later, in 1966. He had never played in top-level events like the U.S. championship, and his reputation came mostly from beating local players. And in that game, Nealey made an obvious blunder. In the rematch that followed, Nealey won one game and drew five. The program didn't win a single game.

In 1965 the program played a few games against the world checkers champion, Walter Hellman. Hellman won all four games played by mail, and the one game played across the board was a draw. Samuel himself acknowledged in 1967 that his program still couldn't beat the best human players.

But the impression that checkers had been solved had already taken hold. Researchers moved on to chess, and checkers was largely neglected for more than twenty years. It wasn't until 1994 that Chinook, a program from the University of Alberta in Canada, truly reached world-champion level.

The Ceiling

Why did Samuel's program stop where it did? He gave the answer himself in his 1967 review.

Generalization learning was much faster than rote learning and did surprisingly well in the middle game, but its progress soon leveled off. The reason was that all the program could adjust were the weights, while the features themselves, what counted as "control of the center," what counted as "exposed to capture," had all been chosen by a person. If the features a person chose left out something important, no amount of reweighting could make up for it. What the program could learn would never go beyond the angles a person had taught it to look from.

This was the same wall Hassabis's Othello program ran into. He had to write "bonus points for corners" in by hand, and the machine just calculated according to his intuition. It would take AlphaGo for a machine to start deciding for itself what to look at on the board.

Samuel retired from IBM in 1966 and went to Stanford as a lecturer, later becoming a research professor, and taught there until 1982. He kept improving his checkers program until later programs completely outclassed it. He also never stopped writing code. He contributed to Stanford's operating system and to Donald Knuth's TeX typesetting system, and at eighty-six he was still writing programs for multi-font printing, possibly the oldest working programmer in the world at the time. He died at Stanford Hospital in July 1990 from complications of Parkinson's disease, at the age of eighty-eight.

While Samuel was at IBM getting his program to play against itself, on the other side of the Atlantic, an Englishman who had broken German codes at Bletchley Park also wanted to teach a machine to play a game. He had no IBM 701; he didn't even have an electronic computer. What he used was some three hundred matchboxes. 🧪