Preface: Rise of GPU and how NVIDIA came to dominate the industry, with a little bit of intro to backpropagation algorithm. Co-written with Gemini. 

If you are like me, someone who’s into arts and film and not at all tech, you probably didn’t hear about NVIDIA until maybe 2022, and you were probably like, who’s this Taiwanese called Jensen Huang that’s always wearing a leather jacket, and why the hell is the GPU industry so important and popping? If you were also like me, in that you’d like to procrastinate in doing everything, like all great artists do, like I’m writing and researching this instead of working on a screenplay, you probably didn’t spend time researching him despite your curiosity, so let me break it down for you. I’m learning this as I write. Gemini says, the popularity and eventual dominance of the Graphics Processing Unit (GPU) by NVIDIA resulted from a multi-decade transition from niche gaming hardware to the essential engine for artificial intelligence and scientific research. While no single person "invented" the concept, the technology evolved through several key milestones. In the 1970s-80s, arcade systems and early consoles used specialized graphics circuits to offload rendering tasks from the CPU. NEC μPD7220 (1982) was the first graphics display processor for personal computers integrated onto a single chip. 3Dlabs Glint Gamma (1997), A UK-based company, 3Dlabs, introduced the workstation-class "Glint Gamma," which they referred to as a "geometry processor unit" (GPU). Under CEO Jensen Huang, NVIDIA coined and popularized the acronym "GPU" (Graphics Processing Unit). They marketed the GeForce 256 as the "world's first GPU" because it was the first consumer-grade single-chip processor to handle complex transform and lighting (T&L) operations previously managed by the CPU. 

The GPU's rise was driven by its unique ability to perform parallel processing—the capacity to execute thousands of simple mathematical operations simultaneously. The gaming revolution in the 1990s created  demand for realistic 3D graphics in video games that required rapid manipulation of pixels, a task perfectly suited for parallel architecture. Beyond gaming, GPUs became essential for non-gaming tasks, such as Windows Vista’s graphical interface in 2006. GPUs were also found to be exceptionally efficient at the massive matrix and vector calculations required to train deep neural networks. This turned GPUs from "niche gaming parts" into essential hardware for every major data center.  Jensen Huang co-founded NVIDIA in 1993, the year of the rooster, the year I was born. NVIDIA is as old as I am, I see.  The company was conceived in 1993 at a Denny’s restaurant in San Jose, California. Jensen Huang, then an engineer at LSI Logic, met with Chris Malachowsky and Curtis Priem, both from Sun Microsystems, to discuss the future of computing. They believed the next wave of computing would be accelerated computing, specifically dedicated to solving problems that general-purpose processors (CPUs) couldn't handle efficiently. They chose video games as their first target market because gaming had a massive demand for high-performance graphics and offered a path to high-volume sales that could fund their research into more complex computational problems. 

The early years were fraught with technical and financial peril. Their first chip, the NV1 (released in 1995), used a mathematical technique called "quadrilateral primitives" instead of the "triangles" that were becoming the industry standard. When Microsoft announced that its DirectX software would standardize on triangles, the NV1 became obsolete almost overnight. By 1996, the company was weeks away from running out of money. Huang had to lay off roughly half the staff and put all the company's remaining resources into a "hail mary" project: the RIVA 128. The RIVA 128 was a success, selling one million units in its first four months and saving the company. This set the stage for their major breakthrough in 1999 with the launch of the GeForce 256, which NVIDIA marketed as the "world's first GPU". NVIDIA's eventual dominance over competitors like 3Dfx and ATI was driven by Jensen Huang's unique management and technical strategies. While the rest of the industry released new chips every 18 months, Huang pushed NVIDIA to release new products every six months, effectively outrunning the competition. In 2007, the company released CUDA, a software layer that allowed its gaming chips to be used for general-purpose mathematical research. This was a massive financial risk that didn't pay off for nearly a decade, but it eventually made NVIDIA the default hardware for the AI revolution.

The transition from 2007 to the present is the era where NVIDIA shifted from being a "gaming company" to becoming the backbone of the global economy. This transformation was driven by a long-term bet on software called CUDA, which perfectly positioned them for the AI explosion. Jensen Huang believed that "accelerated computing" would eventually be needed for every industry, not just games. His realization that AI would be the defining future of the company crystallized around 2012 with the success of AlexNet. The world changed in 2012 with AlexNet, a neural network that won the ImageNet computer vision competition by a massive margin. AlexNet was trained using two NVIDIA GTX 580 GPUs. This proved to the scientific world that NVIDIA’s parallel processing was thousands of times more efficient than CPUs for training neural networks. Suddenly, every AI researcher in the world needed NVIDIA hardware. 

They created specialized software (like cuDNN) for AI developers, ensuring that if you wanted to build AI, it worked best—and sometimes only—on NVIDIA. They began selling entire racks of servers (the DGX systems) rather than just individual cards. In 2020, NVIDIA acquired Mellanox for $7 billion. Mellanox specialized in high-speed data center networking, allowing NVIDIA to connect thousands of GPUs together into a single "giant GPU". NVIDIA became a household name when the world realized that the generative AI revolution (ChatGPT, Midjourney, etc.) was physically impossible without their chips. As of 2025, NVIDIA controls approximately 92% of the discrete GPU market and a similar share of the data center AI market. In 2024, NVIDIA’s market cap crossed $3 trillion, briefly making it the most valuable company in the world. While competitors like Intel and AMD viewed the GPU as a peripheral for video games, NVIDIA viewed it as a universal computer. By spending 15 years building the software (CUDA) and networking (Mellanox) to support that vision, they were the only ones ready when the "AI moment" finally arrived.

And that solved the Minsky’s second proposed problem in one of the two impossibles in his first book, Perceptrons, that is he proved that as problems become more complex, the number of neurons and the size of the "weights" (coefficients) needed can grow exponentially, becoming physically impossible for computers of that era to handle. With GPU, it is now possible to run those calculations. However, this doesn’t solve problem #1 yet, which was, let me remind you, even though they knew solving the XOR problem was impossible on a 2D plane, with hidden layers, it was possible to solve, yet there was no known mathematical way to "teach" the hidden layers to a machine. This was later also solved, by this algorithm called the Backpropagation. 

But first, let me explain what an algorithm is. In today’s context, when we talk about “algorithms” in tech, we are talking about an algorithm we can teach to the machine. Since the late 17th century (denoting the Arabic or decimal notation of numbers), it meant a variant (influenced by Greek arithmos ‘number’) of Middle English algorism, via Old French from medieval Latin algorismus . The Arabic source, al-Ḵwārizmī ‘the man of Ḵwārizm’ (now Khiva), was a name given to the 9th-century mathematician Abū Ja‘far Muhammad ibn Mūsa, author of widely translated works on algebra and arithmetic. But now, this word means, “a process or set of rules to be followed in calculations or other problem-solving operations, especially by a computer”, or “a data-tracking system in which an individual's internet search history and browsing habits are used to present them with similar or related material on social media or other platform”. It doesn’t just mean a process of calculations anymore, it means a process of calculations you can teach to a machine, not just humans. 

The first time someone was able to teach an “algorithm” to a machine was Turning, as mentioned in the previous post, and the movie The Imitation Game. However, it was the second breakthrough that taught machines how to “figure out” a formula by itself. It was in 1959, Arthur Samuel developed one of the world's first successful examples of a "learning" algorithm by programming a computer to play checkers. This program was a landmark in AI because it demonstrated that a machine could achieve master-level performance through experience rather than just following rigid, pre-coded instructions. Samuel's program utilized a success-based reward system. It functioned by evaluating the state of the board and adjusting its strategy based on the outcome of its moves. The algorithm had to decide how much "credit" or "blame" to give to specific moves and strategies for a win or loss. It assigned "weights" to different board features by correlating them with eventual success. If the existing strategy (the "ingredients") was inadequate, the algorithm could test products of preexisting terms to invent new ways to evaluate the game board. Most notably, the program could play against itself, essentially creating a training loop that allowed it to learn at a rate much faster than playing against humans alone. Samuel’s work laid the groundwork for many concepts we see in modern AI. It proved that machines do not always need to be explicitly "programmed" for every scenario; they can be given a framework to learn from data. 

In 1986, Backpropagation algorithm emerged. This algorithm provided the mathematical breakthrough needed to "teach" deep networks. The Backpropagation algorithm is the fundamental mathematical method used to train multi-layer neural networks by enabling them to learn from their errors. It effectively solved the "learning problem" that Minsky and Papert identified in 1969, which had previously limited AI to simple, single-layer tasks. The process involves a continuous loop of testing and adjustment. Data is fed into the network, passing through various "hidden layers" where each neuron applies a weight to the information before reaching an output. The network's output is compared against the correct answer to determine the "error" or "loss". The algorithm calculates how much each specific weight in the network contributed to that error. It "propagates" this error signal backward from the output layer through the hidden layers. Using calculus (specifically gradient descent), the algorithm tweaks every weight just enough to make the next prediction more accurate. This way, the algorithm can teach itself to be closer and closer to the expected result by adjusting the coefficients. This way, we have successfully given the machine instructions on how to figure out the math. 

Teaching algorithms reached a "Big Bang" moment when massive datasets and increased compute power aligned. Unlike traditional sequential computers, modern GPUs (Graphics Processing Units) allow algorithms like backpropagation to run on millions of "agents" simultaneously. A GPU achieves massive parallelism by utilizing a "throughput-oriented" architecture designed to perform thousands of simultaneous operations, which is the physical realization of what Minsky called parallel computation. While a standard CPU (Central Processing Unit) is like a single scholar solving complex problems one by one (serial), a GPU is like a stadium full of students solving millions of simple math problems at the same time. More on parallel computation in the next post. ☀️

References & Recommended Reading

1. The NVIDIA Origin Story: From Gaming to "Universal Computer"

• Hruska, J. (2014). "The History of the GPU: 1990s and Beyond." ExtremeTech.

  • Why read: A deep dive into the 3D graphics war. It covers the RIVA 128 (the "Hail Mary" project) and the GeForce 256—the chip that birthed the term "GPU."

• CNBC Tech Reports (2023). "How NVIDIA Grew From Gaming to AI Giant."

  • Why read: Perfect for non-techies. It documents Jensen Huang’s pivotal "bet the company" moments, from the near-bankruptcy in 1996 to the launch of CUDA in 2007.

• NVIDIA Press Archive (1999). "NVIDIA GeForce 256: The World’s First GPU."

  • Why read: Historical context on why "Transform & Lighting" (T&L) moving to a single chip was the first step toward the parallel processing power we use for AI today.

2. The 2012 "Big Bang": AlexNet and the Death of "Impossible"

• Krizhevsky, A., Sutskever, I., & Hinton, G. E. (2012). "ImageNet Classification with Deep Convolutional Neural Networks."

  • Why read: This is the most important paper in modern AI. It describes how two NVIDIA GTX 580 GPUs solved the "impossible" complexity Minsky warned about in Perceptrons.

• Viso Suite (2024). "AlexNet: Revolutionizing Deep Learning in Image Classification."

  • Why read: A clear breakdown of how AlexNet used ReLU (for speed) and GPUs (for scale) to crush the competition, proving that "brute force" math was the path forward.

3. Solving the "Math Problem": Backpropagation & Arthur Samuel

• Samuel, A. L. (1959). "Some Studies in Machine Learning Using the Game of Checkers." IBM Journal.

  • Why read: The primary source for the first "learning" machine. It explains the "weights" and "scoring polynomials" you mentioned—the ancestors of modern neural network parameters.

• Rumelhart, D. E., Hinton, G. E., & Williams, R. J. (1986). "Learning representations by back-propagating errors." Nature.

  • Why read: The math that saved AI. It describes how to "teach" hidden layers using calculus (gradient descent), finally solving Minsky’s XOR problem.

• Thinkstack AI (2025). "What is the XOR Problem? Geometric and Representational Intuition."

  • Why read: A modern, visual explanation of why Minsky thought AI was "stuck" in 2D and how hidden layers allow us to "fold" the math into 3D to solve it.

4. The 2026 Landscape: Vertical Integration & The "Giant GPU"

• Dell’Oro Group (Dec 2025). "Data Center Infrastructure in 2026: The Rise of Vera Rubin."

  • Why read: Outlines NVIDIA’s latest shift—from selling chips to selling "racks" of servers. It explains how the Mellanox acquisition turned thousands of small GPUs into one "giant GPU."

• TechPolicy Press (2025). "NVIDIA’s Strategic Blueprint: The Generative Industrial Revolution."

  • Why read: Discusses Jensen Huang’s 2026 vision of "Sovereign AI" and why nations, not just companies, are now buying GPUs as if they were oil or gold.

🧬 Key "Artist-Friendly" Definitions

• Parallel Computing (GPU): Like a stadium of 10,000 students each solving one simple addition problem at the same time.

• Serial Computing (CPU): Like one genius professor solving a complex equation line by line.

• Backpropagation: The "Review Session" where the machine looks at its wrong answer, sees which "neurons" caused the mistake, and nudges them to do better next time.

• CUDA: The "Translator" that allows scientists to speak to a gaming chip and tell it to do physics or biology instead of just drawing 3D monsters.