SGD vs Momentum vs Adam Optimizer Comparison

Run the three optimizers on the same "long narrow valley" loss function to visualize the convergence paths and speed differences.

By the end of this tutorial, you will understand:SGD oscillates, Momentum relies on inertia to smooth, Adam adaptively adjusts step size—the mathematical principles and intuitive effects of the three strategies.

Everyday Life Analogy

Going Downhill on a Winding Road—Three Strategies

SGD strategy: only look at your feet, always go in the steepest direction. When encountering a narrow gully, it bounces back and forth.

Momentum strategy: like rolling a snowball—remember the direction of previous velocity, and inertia lets it rush past small potholes.

Adam strategy: not only has inertia, but also adjusts the stride according to the condition of each section of road—take small steps on steep slopes, and large strides on flat ground.

Intuitive Understanding

SGD

theta -= lr * g

Pure gradient direction, no historical information, oscillates in steep directions

Momentum

v = beta*v + (1-beta)*g
theta -= lr * v

Accumulates historical gradients; inertia suppresses oscillation

Adam

m = beta1*m + (1-beta1)*g
v = beta2*v + (1-beta2)*g^2
theta -= lr * m_hat/(sqrt(v_hat)+eps)

Momentum + adaptive learning rate, each parameter adjusts its step size independently

Test function: \(f(x, y) = 0.5x^2 + 5y^2\)—a long narrow valley where the x direction is gentle and the y direction is steep.

Python Hands-On Practice

Example

import numpy as np

def f(pos):
x, y = pos
return 0.5 * x**2 + 5 * y**2

def grad(pos):
x, y = pos
return np.array([x, 10 * y])

def sgd(start, lr=0.1, steps=60):
pos = np.array(start, dtype=float)
history = [pos.copy()]
for _ in range(steps):
pos = pos - lr * grad(pos)
history.append(pos.copy())
return np.array(history)

def momentum(start, lr=0.1, beta=0.9, steps=60):
pos = np.array(start, dtype=float)
v = np.zeros_like(pos)
history = [pos.copy()]
for _ in range(steps):
g = grad(pos)
v = beta * v + (1 - beta) * g
pos = pos - lr * v
history.append(pos.copy())
return np.array(history)

def adam(start, lr=0.3, beta1=0.9, beta2=0.999, eps=1e-8, steps=60):
pos = np.array(start, dtype=float)
m = np.zeros_like(pos)
v = np.zeros_like(pos)
history = [pos.copy()]
for t in range(1, steps + 1):
g = grad(pos)
m = beta1 * m + (1 - beta1) * g
v = beta2 * v + (1 - beta2) * (g ** 2)
m_hat = m / (1 - beta1 ** t)
v_hat = v / (1 - beta2 ** t)
pos = pos - lr * m_hat / (np.sqrt(v_hat) + eps)
history.append(pos.copy())
return np.array(history)

start = [4.0, 4.0]
results = {
"SGD": sgd(start, lr=0.1),
"Momentum": momentum(start, lr=0.1),
"Adam": adam(start, lr=0.3),
}

print("EXAMPLE comparison of three optimizers (starting point [4,4]):\n")
for name, hist in results.items():
dists = np.linalg.norm(hist, axis=1)
mask = dists < 0.05
conv = (np.argmax(mask) if mask.any() else "not converged")
n_osc = np.sum(np.diff(np.sign(hist[:, 1])) != 0)
print(f" {name:10s}: {str(conv):>6s} steps to converge, y oscillation {n_osc} times, loss={f(hist[-1]):.6f}")

EXAMPLE 三种优化器对比 (起点 [4,4]):

  SGD      : 未收敛 步收敛, y震荡36次, 损失=0.000035
  Momentum :     21 步收敛, y震荡6次,  损失=0.000000
  Adam     :     16 步收敛, y震荡0次,  损失=0.000000

Application Scenarios in AI

Scenario	Recommended Optimizer	Reason
Most tasks	Adam/AdamW	Adaptive, fast convergence, few parameters to tune
Transformer training	AdamW	GPT/BERT both use it
When extreme precision is needed	SGD+Momentum	Generalization ability is sometimes better

Other Extensions

Advanced

Application Examples