How neural networks compute

Every layer in a neural network is arithmetic: multiply, add, squash. This page lets you watch that arithmetic happen one number at a time, starting with the layers that let a network see images, and ending with one that remembers.

1Convolution

A convolutional layer slides a small grid of weights, the kernel, across an image. At each position it multiplies every weight by the pixel underneath, adds up the nine products, and writes that one number into a new grid called a feature map.

The kernel's numbers decide what it notices. A kernel with negatives on the left and positives on the right gives a large output only where the image gets brighter from left to right, so it finds vertical edges. In a trained network nobody writes these numbers by hand; they're learned.

Input . Click and drag to draw.
Kernel. Type to change a weight.
Pixel × weight at this position
Feature map . Hover a cell.
positivenegative
Figure 1. One kernel, one image. The red square is the kernel's current position; the cell it produces is outlined in the feature map. Padding adds a border of zeros (hatched) so the kernel can sit on the edge pixels; stride 2 skips every other position and halves the output.

The same nine weights work on a real photograph. The only difference is scale: a 160×160 image means 25,600 positions instead of 64.

Input, in grayscale. Hover to inspect.
Under the kernel
Feature map
Figure 2. The kernel from Figure 1 applied at every pixel, with edge pixels repeated so the output stays 160×160. Darker ink means a stronger response. Change the kernel above and this figure updates.

2Pooling

A feature map is as large as the image it came from. Pooling shrinks it. A window slides over the map and keeps one number per window, either the largest (max pooling) or the average. Nothing is learned here; there are no weights.

Shrinking is useful. Later layers have fewer numbers to process, and a feature that shifts by a pixel usually stays in the same window, so the network cares less about exactly where something is.

Feature map
This window
Pooled . Hover a cell.
Figure 3. Pooling step by step. With max pooling, the cell that wins each window is boxed; the others are discarded. Using Figure 1's feature map applies ReLU first, setting negatives to zero, as a real CNN would.
Figure 4. Figure 2's feature map after ReLU, pooled 2×2 three times using the type chosen in Figure 3. Each step keeps a quarter of the numbers. Fine detail goes first; the strongest edges last longest.

3Dense networks

After convolution and pooling, a CNN usually ends with a few dense layers, and on their own they make up the plain neural network. Every neuron reads every output of the layer before it:

z = w1a1 + w2a2 + … + b,    a = f(z)

The weighted sum z goes through an activation function f, which bends the line. Without that bend, stacking layers would collapse into one big linear function.

Inputs
Activation f in hidden layers
Click a neuron to see its arithmetic
No neuron selected.
Figure 5. A 3–4–4–2 network. Ochre lines are positive weights, blue are negative, and thickness shows size. The two outputs pass through softmax so they read as probabilities. The weights are random, so the "classes" mean nothing yet; training is what would give them meaning.

4Recurrent networks

Every network so far sees its whole input at once. Text, speech and time series arrive in order, so a recurrent network reads one item at a time and keeps a small vector h, the hidden state, that carries what it has seen so far:

ht = tanh( Wxhxt + Whhht−1 + b )

The same weights are used at every step. Drawing it "unrolled" makes it look like a chain of copies, but it is one network applied again and again.

Step calculation
Figure 6. Each character is fed in as a one-hot column, and the red arrows show where information flows at the current step. Try cat, then bat: the last two letters match, yet the final hidden state differs, because the first letter is still in h.