How do neural networks work?

Modern AI runs on something called a neural network. ChatGPT is one. So is the thing that finds faces in your photos, the thing that turns your voice into text, and most of the AI from the ten years before chatbots showed up.

“Neural network” sounds fancy, and a little biological. But it's really a few simple operations done over and over again, millions or billions of times. And at the heart of it is one operation in particular, called the dot product.

We're going to learn what a dot product is, and then build a neural network out of them, one brick at a time. No programming needed. Everything on this page is live: drag the dots, press the buttons, break things.

If you read How does AI work?, this goes one layer deeper. That post treated the model as a mystery equation full of knobs. This one opens the equation up.

1Distance

Before the dot product, let's start with something familiar: distance.

Here's a flat, 2D plane, like a sheet of graph paper. Every point on it has two coordinates: how far right it is (x) and how far up it is (y). Drag the points around.

We can measure the distance between any two points. You'd use a ruler; a computer uses a formula from Pythagoras. But you don't need the formula. What matters is the shape of the idea:

distance(A, B) → one number

Distance is a function: you hand it two points, and it hands you back a single number. A small number means the points are close. A big number means they're far apart. Hang on to that shape. Everything else on this page has it.

2Alignment

Distance isn't the only way to compare two points. Another is alignment: do two points lie in the same direction, as seen from some starting spot?

Let's make that concrete with baseball. Picture a field from above, with home plate at the bottom. When a batter hits the ball, it generally goes one of three ways: to the left field side, up the middle, or to the right field side.

We can put coordinates on the field just like on the graph paper. Home plate is [0, 0]. The right field foul line is the x direction and the left field foul line is the y direction. One unit is 100 feet, so a ball at [3, 0] landed right on the right field line, 300 feet out.

We can measure the distance between two batted balls exactly as before. But say we want to know whether two balls were hit to the same side of the field. Distance is bad at this. Look:

A and B landed close together, but A went to left and B went to right. C went the same way as A, just much deeper, so distance calls C the odd one out. Distance mixes up two different things: which way the ball went, and how far it went.

What we really want is the angle from home plate. Every ball leaves home at some angle: 0° is straight down the right field line, 90° is straight down the left field line, and 45° is dead center.

Now A and C, both hit to left, have nearly the same angle, and B is way off. We can wrap this up as a function, just like distance:

alignment(A, B) = the difference between their angles

Zero means the exact same direction. Bigger means less aligned. Drag the balls and compare the two functions: alignment ignores how far the ball went, which is exactly what we wanted.

3The dot product

The dot product is a measure of alignment too. It just calculates it a different way. Instead of measuring angles, you multiply the matching coordinates together and add up the results:

A = [x1, y1] B = [x2, y2] dot(A, B) = x1 × x2 + y1 × y2

That's the whole thing. Multiply, multiply, add. Here it is next to our other two functions:

distance(A, B)—

alignment(A, B)—

dot(A, B)—

Play with it and a few things jump out:

• Hit both balls in the same direction, and the dot product is big.
• Put one on each foul line, 90° apart, and it's exactly zero. (One side of each multiplication is zero there.)
• It runs the opposite way from alignment: for the dot product, bigger means more aligned.

The dot product also grows when the balls go farther, so it measures alignment and length together. That's usually fine, and in a neural network it's often useful: a long arrow is a loud signal. If you ever want pure alignment, divide by both lengths. That's called cosine similarity.

So why bother, when angles work fine? Because the dot product is just multiplying and adding, it works the same way in any number of dimensions. On a baseball field each ball has one angle. In 3D, a direction needs two angles, like latitude and longitude, and comparing them gets fiddly. For the dot product, a third dimension is just one more multiply and one more add:

dot(A, B) = x1 × x2 + y1 × y2 + z1 × z2

angle between—

dot(A, B)—

Neural networks routinely take dot products in thousands of dimensions. That's thousands of multiplies and adds, but still one number at the end, and no angles anywhere.

4Dot products answer questions

Here's where it gets interesting. We can arrange a few dot products to answer a question:

Was this ball hit to left, center, or right?

We start with the ball that was hit, as a single coordinate:

ball = [x, y]

Then we keep one arrow for each part of the field, pointing into the middle of that zone:

left = [1, 4] center = [3, 3] right = [4, 1]

We take the dot product of the ball with each arrow:

dot_left = dot(ball, left) dot_center = dot(ball, center) dot_right = dot(ball, right)

And then we pick the biggest one:

max(dot_left, dot_center, dot_right)

Whichever arrow the ball lines up with best tells us where it was hit. Drag the ball:

Three dot products and a max: we've built a classifier. The diagram on the right is the same calculation, drawn the way neural networks usually are. The inputs are on the left. Each line multiplies by the number written on it. Each circle adds up whatever flows into it. That circle-and-line picture is a neural network. A tiny one, but a real one.

Why [1, 4] for left and not something like [0, 5]? Because the dot product rewards length as well as direction. With left as [0, 5] and center as [5, 5], center's arrow would be longer and win every single time. Keeping the three arrows about the same length makes it a fair fight about direction.

5Learning the numbers

I picked those six numbers, 1, 4, 3, 3, 4 and 1, by hand. That works for baseball, where we understand the problem. For harder problems nobody can pick the numbers by hand. Instead, we learn them from examples.

Let's warm up with something familiar: the line of best fit. Here's how far some batted balls traveled, plotted against how many seconds they hung in the air. A line y = m × x + b has two adjustable numbers, m and b, which we call parameters. And one number, the loss, measures how bad the line is: here, the average gap between a dot and the line.

slope m

intercept b

Train

the line—

loss (average miss)—

Drag the sliders to find a good fit, or press Train and let the computer do it. Training is a loop: check the loss, nudge every parameter in whichever direction makes the loss smaller, and repeat. (How does AI work? walks through this slowly, if you want more.)

The same trick works on dot products. Below are 72 batted balls, each labeled left, center or right by the official scorer, who splits the field into three equal slices. Our classifier starts out with three random arrows. Their six numbers are its parameters. The loss is a number that's big when the classifier picks the wrong zone, and small when it confidently picks the right one.

Watch the arrows swing into place as the loss falls. Nobody told it where left field is. It worked that out from examples. It even beats my hand-picked arrows, which put the zone lines at about 27° and 63° instead of the scorer's 30° and 60°, and so get 7 of these 72 balls wrong.

How does it know which way to nudge? For each parameter, it asks: if this number went up a tiny bit, would the loss go up or down? Then it moves the number a small step the other way. It does that for every parameter at once, then repeats. Our classifier has six parameters. ChatGPT has hundreds of billions. It's the same question, asked of each one.

The method is called gradient descent. The clever trick that answers “which way?” for billions of parameters at once, without test-driving each one, is called backpropagation.

6The secret sauce: non-linearity

So far a neural network is a bunch of dot products. Can we make it smarter by stacking them, feeding the outputs of some dot products into more dot products?

As it stands, no. Say one dot product doubles its input, and a second one triples whatever it gets:

first: h = 2 × x second: out = 3 × h put together: out = 6 × x

The two dot products collapse into one. That always happens, no matter how many you stack or how wide you make them. Stacked dot products only ever add up to a straight-line relationship, which a single layer could have done on its own. A thousand layers would just be a very expensive single layer.

The fix is almost embarrassingly simple. Between layers, we squash each number with a function that isn't a straight line. The most popular one is called ReLU, and all it does is replace negative numbers with zero:

relu(x) = max(0, x)

That one bend is enough. Each dot product can now switch off for part of the input, and the pieces can no longer collapse into a single line. Here's a small network with one input, 8 dot products in the middle, and one output, trying to fit a wavy curve. Train it with the squash, then without:

Train Start over Squash with ReLU

training steps0

loss (average miss)—

Without the squash, the best it can do is a straight line, however long it trains. With it, the faint lines show each middle dot product's bent contribution, and together they add up to a curve. The squash is called a non-linearity, and it's the secret sauce. It's what makes stacking worth doing.

7Going deep

With a squash between them, stacking pays off. The outputs of one layer of dot products become the inputs to the next layer. The number of layers is the network's depth, and that's the “deep” in deep learning.

Every line in the picture is one multiplication by a learned number: one parameter. Each circle also gets one more learned number added in, called a bias. Make the network wider or deeper and watch the count climb:

Parameters pile up fast, because every circle connects to every circle in the next layer. One layer 10,000 wide feeding another 10,000 wide is 100 million parameters on its own. Stack a thousand of those and you're at 100 billion.

That's what people mean when they talk about a “100B model”: a neural network with 100 billion learned numbers inside its dot products. Some real sizes:

• Our line of best fit2
• Our baseball classifier6
• GPT-2 (2019)1.5 billion
• GPT-3 (2020)175 billion
• Llama 3.1, largest (2024)405 billion

8Any function at all

There's a famous result called the universal approximation theorem. In plain English: a neural network that's big enough, with enough training data, can learn to imitate any function.

Remember, a function is anything that takes something in and gives a result back. Draw any curve you like below, and watch a network learn it. The slider sets how many dot products sit in the middle. A few give a rough sketch; more give a closer fit.

dot products in the middle

Clear and draw your own

parameters—

training steps0

loss (average miss)—

The theorem doesn't promise that training is easy, or say how big “big enough” is. But it does say there's no shape a network can't learn, in principle. And an enormous number of useful things can be framed as a function:

• Photo labelingpixel colors in → “cat” or “dog” out
• Playing a gamewhat's on screen in → which button to press out
• Speech to textsound in → words out
• Chatthe conversation so far in → the next word out

That's why neural networks are everywhere in modern AI. Instead of writing rules for each problem, you collect examples of inputs and outputs, and let training find the function.

9Words as coordinates

So how does a pile of dot products become ChatGPT? The first step is to turn words into coordinates, just like batted balls.

Imagine a simple 3D space where each direction measures how much a word is some quality. Say x measures female, y measures male, and z measures old. Grandma is very female and very old, so she sits around [1, 0, 0.95]. A boy is around [0, 1, 0.1]. Every word becomes a point, and words with similar meanings land near each other. Pick a word to see its dot product with all the others:

The dot product finds related words for the same reason it found balls hit to the same side. It also shows the limits of three directions: queen and woman barely differ, and fossil scores well against grandpa because “old” is all they share. Real models use thousands of directions, and nobody picks them by hand. Training discovers them, the same way it discovered left field.

10How a chatbot uses all this

A chatbot is, at its core, autocomplete. It's trained on an enormous amount of text to do one thing: given some text, predict the next word. To write a whole reply, it predicts a word, sticks it on the end, and feeds the longer text back in to predict the word after that:

Predict the next word Start over

The model doing the predicting is a huge neural network: layer after layer of dot products and squashes, with the words going in as coordinates. And through training, many of those dot products come to mean something. One might light up when the input is a question. Another when it's about math. Another when it's about animals.

Here's a toy version. Each detector below is a single dot product. The text becomes a list of how many times each word appears, the detector has a number for each word, and we multiply and add. Type anything:

See each detector's numbers

In a real model nobody writes those lists of numbers. Training finds them, and the detectors it finds are rarely this tidy. But the principle holds. Grammar, facts, a sense of what's being asked, all of it ends up baked into the numbers inside billions of little dot products. Multiply, multiply, add, squash, over and over again.

What we built

1. A function takes things in and gives one result back, like distance.
2. Alignment asks whether two points face the same way, ignoring how far.
3. The dot product measures alignment with nothing but multiplying and adding, in any number of dimensions.
4. A few dot products and a max can answer a question. Drawn as circles and lines, that's a neural network.
5. The numbers inside are parameters, and training nudges them to lower the loss.
6. A squash like ReLU between layers stops the dot products collapsing into one.
7. Stack layers and you get depth, and lots of parameters. A “100B model” has 100 billion.
8. Big enough networks can learn any function, which is why they're everywhere.
9. Words become coordinates, and a chatbot is a giant network of dot products doing autocomplete.