Gradient Descent Explained With a Worked Example You Can Follow by Hand

The first three times I read the gradient descent formula, I nodded along and understood nothing. w = w - learning_rate * gradient looks simple enough to memorize, but memorizing it and understanding what happens to the actual numbers are two different skills. What finally made it click was sitting down with five data points, a calculator, and forcing myself to compute two full iterations by hand before writing a single line of code. This post is that walkthrough, numbers left in on purpose.

The setup: five points, one line

We want to fit a line, y_hat = w * x + b, to these five points:

If you solve this with the closed-form least-squares formula, the best-fit line turns out to be roughly y = 0.6x + 2.2. That's useful to know up front because it gives us a target to watch gradient descent creep toward. In real projects you don't get to peek at the answer first, but for a learning example it's the whole point — you get to check your work.

We'll start both parameters at zero: w = 0, b = 0. That's a deliberately bad starting guess, which is exactly what makes the first iteration instructive.

The cost function and its gradient

We're minimizing mean squared error across all five points:

MSE = (1/n) * sum((y_hat_i - y_i)^2)

To move w and b downhill, we need the partial derivative of that cost with respect to each one. For linear regression these work out to:

dw = (2/n) * sum((y_hat_i - y_i) * x_i)
db = (2/n) * sum(y_hat_i - y_i)

Every term in these formulas is something you can compute with a calculator and five rows of a spreadsheet. That's the part that got lost for me in textbook notation — it reads like abstract calculus, but it's just "average error, weighted by x" and "average error."

Iteration 1, computed by hand

With w = 0 and b = 0, every prediction is zero, so the error for each point is just 0 - y:

x=1  y=2  y_hat=0  error=-2
x=2  y=4  y_hat=0  error=-4
x=3  y=5  y_hat=0  error=-5
x=4  y=4  y_hat=0  error=-4
x=5  y=5  y_hat=0  error=-5

Now plug into the gradient formulas with n = 5:

dw = (2/5) * [(-2*1) + (-4*2) + (-5*3) + (-4*4) + (-5*5)]
= (2/5) * (-2 -8 -15 -16 -25)
= (2/5) * (-66)
= -26.4

db = (2/5) * (-2 -4 -5 -4 -5)
= (2/5) * (-20)
= -8.0

Using a learning rate of 0.01, the update rule param = param - learning_rate * gradient gives us:

w = 0 - 0.01 * (-26.4) = 0.264
b = 0 - 0.01 * (-8.0)  = 0.08

Notice the sign: the gradient was negative, so we moved w and b up, not down. That trips people up the first time — "descent" refers to descending the cost curve, not the parameter values. A negative slope on the cost curve means increasing the parameter reduces the cost, so we step in the positive direction.

Iteration 2, same process, updated numbers

Now recompute predictions with w = 0.264, b = 0.08:

x=1  y_hat=0.344  y=2  error=-1.656
x=2  y_hat=0.608  y=4  error=-3.392
x=3  y_hat=0.872  y=5  error=-4.128
x=4  y_hat=1.136  y=4  error=-2.864
x=5  y_hat=1.400  y=5  error=-3.600

dw = (2/5) * [(-1.656*1) + (-3.392*2) + (-4.128*3) + (-2.864*4) + (-3.600*5)]
= (2/5) * (-50.28)
= -20.112

db = (2/5) * (-1.656 -3.392 -4.128 -2.864 -3.600)
= (2/5) * (-15.64)
= -6.256

w = 0.264 - 0.01 * (-20.112) = 0.465
b = 0.08  - 0.01 * (-6.256)  = 0.143

After two iterations, w has moved from 0 to 0.465, already more than three-quarters of the way to the target 0.6. But b has only crawled from 0 to 0.143, nowhere near the target 2.2. That's not a bug in the math — it's the single most useful thing this hand-worked example shows.

Why b lags behind w (and why this matters more than the formula itself)

The gradient for b is just the average error. The gradient for w is the average error weighted by x, and our x values range up to 5, which amplifies that gradient. Both parameters share the same learning rate, so the parameter with the naturally larger gradient moves faster. Left alone, b would need many more iterations to catch up, and with a badly chosen learning rate it might oscillate before it gets there.

This is exactly why feature scaling (subtracting the mean, dividing by standard deviation) is standard practice before training linear models with gradient descent: it keeps the gradients for different parameters on comparable scales so a single learning rate treats them fairly. If you've ever wondered why every scikit-learn tutorial calls StandardScaler before fitting, this is the mechanical reason, not just a convention.

The same logic in Python

Here's the full iteration loop, matching the numbers above exactly so you can verify it against your hand calculation:

x = [1, 2, 3, 4, 5]
y = [2, 4, 5, 4, 5]
n = len(x)

w, b = 0.0, 0.0
lr = 0.01

for iteration in range(2):
y_hat = [w * xi + b for xi in x]
errors = [yh - yi for yh, yi in zip(y_hat, y)]

dw = (2 / n) * sum(e * xi for e, xi in zip(errors, x))
db = (2 / n) * sum(errors)

w = w - lr * dw
b = b - lr * db

print(f"iter {iteration + 1}: w={w:.4f}  b={b:.4f}  dw={dw:.4f}  db={db:.4f}")

Running this prints w=0.2640 b=0.0800 after the first iteration and w=0.4652 b=0.1426 after the second, matching the hand calculation to the fourth decimal. If you run more iterations — try 500 with the same learning rate — both parameters settle close to 0.6 and 2.2.

What actually goes wrong when people apply this

Once the mechanics are clear, the practical mistakes are almost always one of these:

• Learning rate too high. Try lr = 0.5 on this same data and w overshoots wildly on the first step and diverges instead of converging — the update jumps past the minimum and lands somewhere worse than it started.
• Learning rate too low. Set lr = 0.0001 and you'll need tens of thousands of iterations to get anywhere close to 0.6 and 2.2, burning compute for no reason.
• Unscaled features with wildly different ranges. If one feature ranges 0–1 and another ranges 0–100,000, the gradients for their weights will be on completely different scales, and a single learning rate can't serve both well.
• Stopping too early and calling it converged. Watching the cost drop for three iterations and assuming you're done is a common shortcut in a rush — check that the cost has actually flattened out, not just dropped once.

Where this fits into a real workflow

This two-point example is deliberately stripped down so the arithmetic stays tractable. In practice you'd never write your own gradient descent loop for ordinary linear regression — you'd call scikit-learn's LinearRegression or SGDRegressor and let a tested implementation handle the iteration, convergence checks, and numerical stability for you. Understanding what's happening underneath is still worth the hour it takes, because it's the same update rule sitting inside logistic regression, neural networks, and most of what people mean when they say "the model trained." If you want to see this same update rule embedded inside a complete workflow — loading data, splitting it, training, and evaluating — the walkthrough in building a tiny end-to-end ML pipeline picks up right where this one leaves off.

One more detail worth knowing: real training code almost never initializes weights at exactly zero for anything beyond a single-variable toy example like this one, because symmetric zero starts can cause certain models (especially neural networks) to update every unit identically. Instead, initial weights are usually drawn from a small random distribution. If you want a feel for how that kind of controlled randomness is generated in the first place, our random number generator is a simple way to see seeded, repeatable randomness in isolation before it's buried inside a training script.

Batch, stochastic, and mini-batch: the same update rule, different data slices

Everything worked out above is technically "batch" gradient descent — every iteration computes the gradient using all five data points before taking a single step. That's fine at five rows and painfully slow at five million, because you'd recompute the error on the entire dataset just to nudge the weights once. Stochastic gradient descent (SGD) takes the opposite extreme: compute the gradient and update the weights using just one data point at a time, cycling through the dataset in a random order each pass. It's noisier — the path toward the minimum zigzags instead of moving smoothly — but each individual update is far cheaper, and on large datasets that trade-off wins decisively.

Mini-batch gradient descent, which is what most real training code actually uses, splits the difference: compute the gradient over a small batch (32, 64, 128 rows are common choices) rather than one row or the whole dataset. It keeps updates cheap enough to iterate quickly while smoothing out some of SGD's noise. If you want to see this in our five-point example, imagine repeating the exact hand calculation above but using only points 1 and 2 for iteration one, then points 3 and 4 for iteration two — same formulas, same update rule, just a different, smaller slice of the data feeding each step.

None of this changes the core mechanic this post walked through. Batch, stochastic, and mini-batch are all the same param = param - learning_rate * gradient update; they differ only in how much data gets averaged together before that single step happens, not in the underlying rule that moves the weights.

The takeaway

Gradient descent is not mysterious once you've pushed the numbers through by hand: compute the error, weight it by how much each parameter contributed, scale it down by a learning rate, and step. Two iterations moved this toy model noticeably closer to the correct line. Run it for a few hundred more and it lands almost exactly on the least-squares answer — not because the algorithm is smart, but because repeating a simple, correct update enough times gets you there.