Preface: A runaway teenager who taught himself logic hiding in a Detroit library, a novelist who never wrote his novel, and a grad student with a box of vacuum tubes. Between them, they invented the idea behind every neural network running today.
Following the previous #AI series and #MarvinMinksy series. This post is co-written with Claude.

Neural networks are usually thought of as something that belongs to computers: masses of numbers stored on chips, trained on graphics cards, written in code. Today’s versions can’t be seen or touched. But the first neural network that could learn was a physical object. It was the size of a grand piano, crammed with glowing glass tubes, and full of knobs, a chain, and clutches. The motor that drove it came out of a World War II bomber. When it learned, it did so by mechanically turning its knobs, a little at a time. In the last post, we saw Turing imagine, in 1948, a machine that a schoolmaster would educate by pressing keys. He had no way to build it. Three years later, a twenty-three-year-old math graduate student did.

1943: The Neuron as a Circuit

The idea of the neural network came from two people who studied the brain, and who could hardly have been more different.

Warren McCulloch was born in 1898. Trained as a doctor, he was a neurophysiologist and psychiatrist, and in the early 1940s he was studying the brain at the University of Illinois in Chicago. For years he had been turning over an idea: that the brain’s workings might be described in the language of logic. He had read Russell and Whitehead’s Principia Mathematica, which set out to show that all of mathematics could be built up, step by step, from the most basic rules of logic. McCulloch wanted to do the same thing for the brain.

Walter Pitts was born in 1923 to a working-class family in Detroit. His father wanted him to leave school and get a job, but Pitts kept slipping off to the public library, where he taught himself Greek, Latin, logic, and mathematics. His friends later told a story about him. When he was twelve, running from boys who were chasing him through the streets, he ducked into the library and hid there until it closed. Somewhere in the stacks he found Principia Mathematica. He stayed for three days, read the whole thing, and then wrote to Russell to point out what he thought were problems in the first half of the first volume. Russell supposedly wrote back and invited him to come study in England.

At fifteen, Pitts heard that Russell was coming to lecture at the University of Chicago, so he ran away from home and went to Chicago. He never saw his family again. He never enrolled, and he had nowhere to live; he drifted around the university, sat in on lectures, and got by on odd jobs. The logician Rudolf Carnap eventually encountered him. As the story goes, Pitts walked into Carnap’s office without introducing himself and simply handed him a copy of Carnap’s own book, covered in annotations. A real-life Good Will Hunting.

In early 1942, McCulloch was introduced to Pitts and was struck by the eighteen-year-old’s mind. He invited Pitts, who was still homeless, to move into his family’s home. By day McCulloch went to the lab; in the evenings the two sat together, one bringing his knowledge of nerve cells, the other his mastery of logic. In 1943 they published their joint paper in the Bulletin of Mathematical Biophysics, titled “A Logical Calculus of the Ideas Immanent in Nervous Activity.” Pitts was twenty that year.

Real nerve cells are enormously complex, and McCulloch and Pitts simplified them to the extreme. In their model, a neuron has only two states: it fires, or it doesn’t, with nothing in between. It has a number of inputs that receive signals from other neurons. Each neuron has a threshold: if the signals arriving at the same moment add up to the threshold, it fires once; if they fall short, it stays silent. There is also a kind of signal that acts as a veto: if it arrives, the neuron will not fire, no matter how many other signals come in.

Suppose a neuron has two inputs and a threshold of 2. It fires only when both inputs receive a signal at the same time. This is the logical “and”: the conclusion holds only if A and B are both true. Take the same two inputs and change the threshold to 1. Now the neuron fires if either input receives a signal. This is “or”: the conclusion holds if either A or B is true. Add the veto signal, and you get “not”: if a certain condition is present, the conclusion does not hold. “And,” “or,” and “not” are the three basic building blocks of logic, and any logical reasoning, however complex, can be built out of them. From this, McCulloch and Pitts proved that if enough of these neurons are connected in the right way, the network can express any logical proposition. The notation they used in the paper came straight from the Principia Mathematica that Pitts had read at twelve. They also noticed that if the connections in a network looped back into a circle, a signal could keep circulating around the loop even after the original stimulus had disappeared. A network could therefore “remember” something that had happened in the past.

The significance of the paper was that it described the brain and the machine in the same language. “Firing or not firing” was the same thing as “on or off” in an electrical circuit. If a neuron could be treated as a switch, then a group of connected neurons could, in principle, be built out of wires and switches; and conversely, a machine made of switches could, in principle, think the way a brain does. Two years later, when the mathematician John von Neumann drafted the design report for one of America’s first stored-program computers, he borrowed McCulloch and Pitts’s neuron notation to describe the computer’s components.

1949: The More You Use It, the Stronger It Gets

McCulloch and Pitts’s neurons could compute, but they could not learn. The missing piece came from a Canadian who had originally wanted to be a novelist.

Donald Hebb was born in 1904 in Chester, a small town in Nova Scotia, and was taught at home by his mother until he was eight. He graduated from Dalhousie University in 1925 with a degree in English, intending to write novels. The novel never got written. He spent a few years as a schoolteacher in Quebec, became interested in psychology, and enrolled at McGill University for a master’s degree. His path was almost identical to Skinner’s: both men first wanted to be writers, and only after failing turned to the study of behavior. In 1934, Hebb went to the University of Chicago to study under the psychologist Karl Lashley, then followed Lashley to Harvard, where he received his PhD in 1936.

Lashley was engaged in a stubborn pursuit at the time: finding the place in the brain where memories are stored. He trained rats to run mazes, then removed different parts of their cerebral cortex to see whether they would forget the route. He assumed that if he cut in the right place, the memory would disappear. Instead, no matter which part he removed, the rats still remembered the way. The more he removed, the worse they performed, but there was no single area whose removal made the memory vanish entirely. Memory seemed not to be stored in any one place.

Hebb later saw something similar in humans. From 1937 to 1939, he worked for the neurosurgeon Wilder Penfield at the Montreal Neurological Institute, giving psychological tests to patients who had undergone brain surgery. Some of them had had sizable portions of their frontal or temporal lobes removed, yet their scores on intelligence tests did not drop nearly as much as expected.

These two puzzles became the starting point for Hebb’s thinking over the next decade: if knowledge and memory are not stored in any one place, then where are they? A synapse is the point of connection between two nerve cells. A signal passing from one cell to another must cross a synapse, and each synapse transmits signals with a different strength, some strong, some weak.

From 1942 to 1947, Hebb studied chimpanzees at the Yerkes Laboratories of Primate Biology in Florida, where he began writing a book. In 1949 he published The Organization of Behavior. At its core was a hypothesis about the synapse. Hebb proposed that if cell A is near enough to cell B, and repeatedly takes part in making cell B fire, then some kind of growth or metabolic change takes place in one or both cells, so that A afterward becomes more effective at making B fire. Put more plainly: if two nerve cells are often activated one right after the other, the connection between them grows stronger and stronger.

In Hebb’s view, learning is not the brain acquiring something new, but existing connections changing in strength. Walk a path often enough, and it becomes a road. Building on this, Hebb proposed a larger concept, which he called the “cell assembly.” A group of cells that are often activated together will, as the connections among them grow stronger, gradually bind into a single unit. One assembly represents one thing: a face, a sound, a concept. A cell assembly is a group of nerve cells that have bound together into a single unit because they are frequently activated at the same time and their connections have strengthened. Together, they represent a thing or an idea.

This idea answered both of the puzzles left by Lashley and Penfield. The cells of an assembly are spread across a large area of the cortex, so if one part is removed, the remaining cells are still connected and can still represent the thing. Memory could not be found in a fixed location because it was never in one location; it was distributed across an entire network. And because the connections within an assembly are dense, activating just some of its cells is enough to light up the whole assembly. You see half of a familiar face, and you recognize the whole person. The most widely repeated summary today is “neurons that fire together, wire together.” The phrase is catchy, but Hebb never said it. It is a later distillation of his idea.

When Hebb proposed his theory, it was purely a hypothesis. He had no way of proving that such changes actually happened in the brain. It was not until 1966 that Terje Lømo, a researcher in Oslo, Norway, found while experimenting on the hippocampus of rabbits that after a set of neural pathways was stimulated repeatedly, the connections there really did become stronger, and the strengthening lasted for a long time. In 1973, he and Timothy Bliss formally published the finding, which was later named “long-term potentiation.” Twenty-four years had passed since Hebb’s book came out.

On the machine side, a few years after Hebb’s book was published, a group of researchers at IBM tried to simulate cell assemblies on a computer. The attempt was not very successful.

Hebb’s rule says that “connections used together grow stronger,” but it does not say which outcomes are worth remembering. If a rat takes a wrong turn in a maze, the connections behind that wrong turn were also used together, and by Hebb’s rule they too would grow stronger. For connections to strengthen only when the rat has “gotten it right,” something more is needed: a signal that tells the brain that what just happened was good. That signal is what Thorndike and Skinner called reward. Two years later, Minsky put both of these things into the same machine.

1951: A Letter to Harvard

Marvin Minsky had read McCulloch and Pitts's paper as an undergraduate at Harvard. By 1951 he was a mathematics graduate student at Princeton, but what fascinated him was still the brain. He wanted to do what McCulloch and Pitts had not: build a machine whose connections could change on their own, and teach it to do something.

At Princeton he met Dean Edmonds, a graduate student in physics. Edmonds had done electronics work during the war and was good with his hands. The two estimated that each artificial neuron would need about six vacuum tubes. Minsky wrote up the design in a letter to the Harvard psychologist George Miller, and Miller secured a few thousand dollars from the Office of Naval Research. That summer, the two went to Harvard and started building. They named the machine SNARC, for "Stochastic Neural Analog Reinforcement Calculator." The word "reinforcement" came straight from Skinner. Minsky talked with Skinner many times while designing it. What he wanted to build was a machine that could be reinforced.

SNARC was made up of about forty units. Each unit played the role of a synapse, with one input and one output. The forty units were wired together at random through a plugboard to form a network, with the output of one unit feeding into the input of another.

When a signal reached a unit, it did not necessarily get through. That was controlled by a knob on the unit. The knob was graduated from 0 to 1. Turned to 0.9, the signal got through almost every time; turned to 0.2, only two times out of ten. Turned to 0, it never got through. Any single time, there were only two outcomes: through, or not through. But over many tries, the effect of the knob became clear. This is what "stochastic" in the name means. Every step the machine took involved chance, and learning was a matter of slowly shifting which way the chance leaned.

Why rely on chance instead of making each unit a simple switch? Because a switch gives the same answer every time. Starting from the same point, the signal would always take the same path. If that path was wrong, it would stay wrong forever. With chance built in, the signal would every so often take a different path, and the machine would have the opportunity to stumble onto a better one.

SNARC: Six Vacuum Tubes

Each unit held about six vacuum tubes. A vacuum tube is a glass tube with the air pumped out. At its center is a metal filament called the cathode, which heats up when current runs through it, and electrons fly off its surface. Around the outside is a metal plate called the anode, connected to the positive terminal of the power supply. Electrons carry a negative charge, so they are drawn toward the positive plate, flying from the cathode to the anode, and current flows through the tube.

Between the cathode and the anode sits a fine metal mesh called the grid. The electrons have to pass through it to reach the anode. Put even a slight negative voltage on the grid, and it pushes the oncoming electrons back; the current weakens, or stops entirely. Remove that voltage, and the electrons can pass through the mesh again. The grid works like a valve on a water pipe. It needs only a tiny voltage itself, yet it decides whether a much larger current can get through. The tube is evacuated so that air molecules don't block the electrons, and so that the hot filament doesn't burn out on contact with oxygen. Later, the transistor replaced the vacuum tube. It is smaller, more durable, and needs no heating, but it does the same thing: it uses a faint signal to switch a stronger current on and off. A computer chip today contains billions of transistors, and every one of them is a descendant of the vacuum tubes of 1951.

In SNARC, the signal arriving from the previous unit was that slight voltage on the grid. If the unit "let it through" this time, the vacuum tube opened and released a stronger current. The current split two ways: one path went on to the next unit, carrying the signal forward; the other flowed into the unit's own capacitor. A capacitor is a component that can store electricity temporarily. It slowly leaks, and after a few seconds it is empty. Think of it as a bucket with a small hole in the bottom: when the signal passes through, the bucket is filled; afterward the water drains out, a little at a time.

SNARC: The Electromagnetic Clutch

The capacitor was connected to an electromagnetic clutch inside the unit. A clutch is a device that can join two parts together or separate them. While there was still charge in the capacitor, the electromagnet had enough strength to hold the clutch engaged; once the charge had leaked away, the clutch released. The other side of the clutch was connected to the knob. The entire machine had only one motor. It had been taken from the autopilot of a B-24 bomber. The war had just ended, and surplus military equipment like this was plentiful and cheap. The motor drove a chain that ran past each of the forty units in turn.

Most of the time, the motor sat still. When the lights on the front of the machine showed that the rat had reached the goal, the operator pressed a button. The button closed the motor's circuit, the motor started, and the chain moved. As it passed each unit, if that unit's clutch was engaged, the chain turned the knob up a little; if the clutch was released, the chain ran past without touching it. Once the chain had made its full circuit, the motor stopped. After that one pass, the only knobs that had been turned up were those on units where "a signal had just passed through and the capacitor still held a charge." None of the other units had moved at all.

Whichever unit the signal reached, the corresponding light on the front of the machine lit up. To an observer, it looked like a rat wandering around a maze. At first, all the knobs were set about the same, and the rat blundered around at random. Every so often, it happened to reach the goal. The operator pressed the button, the chain moved, and the units the rat had passed through last were turned up a little. The next time, a signal reaching those units would get through more easily, and the rat would be more likely to follow the same path.

The machine never knew which path was right. It did only one thing: at the moment a reward arrived, it strengthened the units that had just been used. But the path that led to the goal was used every time the rat reached the goal, so its knobs got turned up every time. Paths that never reached the goal never got the button, so their knobs stayed where they were. After dozens, then hundreds of runs, the knobs along the right path turned higher and higher, and the rat headed more and more directly for the exit. The person, meanwhile, only had to judge whether the rat had reached the goal and press the button. Which knob to turn, and by how much, was the machine's business.

Three years earlier, Turing had imagined in his report a machine that a schoolmaster would educate by pressing keys. In SNARC, that key became a physical object for the first time.

SNARC: Surprises and Shortcomings

SNARC's vacuum tubes broke down often. But even with a few of them dead, the machine kept working. In a randomly connected network, a few missing connections still left the signal room to route around them. It also gave Minsky and Edmonds a surprise. Because of a mistake in the circuit design, two or three rats could appear in the maze at once. Minsky later recalled that these rats influenced one another: when one found a good path, the others tended to follow it. They had not designed this. It was a property that grew out of the circuitry on its own.

When the reward arrived, only units that the signal had passed through in the last few seconds still held a charge. So the only part that got strengthened was the short stretch of path the rat had covered just before reaching the goal. In a maze, that was enough. But in a game of chess, the move that decides the outcome may come dozens of moves before the end, and by the time the result is known, that move's capacitor has long since drained. The machine would have no memory of it at all. This is exactly the problem at the end of the previous post: when the reward comes, which of the earlier decisions deserves the credit? SNARC's answer was "the most recent few." That answer is simple, but it only works in the simplest cases. If the rat took a completely pointless detour before reaching the goal, that detour got turned up too.

After 1954

In 1954, Minsky completed his doctoral dissertation on neural networks. Over the following decade and more, he grew increasingly skeptical of this approach, and eventually became one of its most famous critics. We covered that turn in "The Unsolvable XOR Gate." SNARC itself did not survive. Minsky later said that he had lent the machine to some students at Dartmouth, and after that it could not be found. All that can be seen today is a single unit, a small metal rack holding six vacuum tubes. In those same years when Minsky and Edmonds were turning knobs at Harvard, at the RAND Corporation in Santa Monica, California, a mathematician was working on the same question: when you make a series of decisions, what counts as getting them right? Minsky answered it with a machine. The California mathematician answered it with equations. 🧪