Gradient descent, from intuition to update rule

Gradient descent is how almost every model learns. You stand somewhere on a loss surface and repeatedly take a step downhill. The canvas sketches the loop; the note derives the rule.

The setup

We have parameters $\theta$ and a loss $L(\theta)$ that measures how wrong the model is. We want:

Why the negative gradient?

A first-order Taylor expansion around $\theta$ for a small step $\Delta$:

Among all steps of a fixed length, $\nabla L^{\top}\Delta$ is most negative when $\Delta$ points exactly opposite the gradient. That gives the update rule:

Here $\eta$ is the learning rate, the size of each step.

A worked example

Take $L(\theta) = \theta^2$, so $\nabla L = 2\theta$. Start at $\theta_0 = 4$ with $\eta = 0.25$:

The learning rate decides everything

• $\eta = 0.25$: halves the distance every step. Smooth convergence.
• $\eta = 1$: $\theta_{t+1} = -\theta_t$. Bounces between 4 and −4 forever.
• $\eta > 1$: every step overshoots further. Diverges.

For this loss, convergence needs $0 < \eta < 1$. In general the bound depends on the curvature: $\eta < 2 / \lambda_{\max}$ of the Hessian.

The training loop

flowchart LR
  A[Initialise θ] --> B[Compute loss L θ]
  B --> C[Compute gradient ∇L]
  C --> D[θ ← θ − η∇L]
  D --> E{Converged?}
  E -- no --> B
  E -- yes --> F[Done]

Ten lines of code

def gradient_descent(grad, theta, lr=0.1, steps=100, tol=1e-8):
    for _ in range(steps):
        step = lr * grad(theta)
        theta = theta - step
        if abs(step) < tol:
            break
    return theta

gradient_descent(lambda t: 2 * t, theta=4.0, lr=0.25)  # → ~0.0