Visualize the impact of different learning rates.
For the same function, run gradient descent with too small, appropriate, and too large learning rates, and directly see the comparison of three typical curves.
After completing this case study, you will understand:The learning rate is the most important hyperparameter in gradient descent—if set too small, it’s too slow; if set too large, it diverges.
Life Introduction
Driving home — how hard should you step on the gas?
10 kilometers from home. Pressing the accelerator too lightly (learning rate too small) — drive for an hour and still haven't arrived. Pressing the accelerator too heavily (learning rate too large) — overshoot, turn around and charge again, overshoot again, oscillating back and forth. Just the right amount of pressure (appropriate learning rate) — arrive smoothly.
The learning rate selection in gradient descent and throttle control in driving face exactly the same problem.
Intuitive understanding.
The objective function \(f(x) = x^2\), with minimum at \(x=0\). The gradient is \(f'(x) = 2x\).
Update rule: \(x_{t+1} = x_t - \eta \cdot 2x_t = x_t \cdot (1 - 2\eta)\). If \(\eta < 0.5\) it converges; if \(\eta > 1.0\) it diverges.
Python Hands-on Practice
example
def f(x): return x ** 2
def grad(x): return 2 * x
def run_gd(lr, steps=30, x0=8.0):
x = x0
history = [x]
for _ in range(steps):
x = x - lr * grad(x)
history.append(x)
if abs(x) > 1e6: break
return history
settings = {
Too small (lr=0.01): 0.01,
Suitable (lr=0.2): 0.2,
Too large (lr=1.05): 1.05,
}
results = {name: run_gd(lr) for name, lr in settings.items()}
print(EXAMPLE comparison of three learning rates after 30 steps:\n")
print(f{'Learning rate setting':<20} {'Final x':<12} {'f(x)':<14} {'Judgment'})
print("-" * 56)
for name, hist in results.items():
x_final = hist[-1]
judgement = (Has converged if abs(x_final) < 0.01
else Converging if abs(x_final) < 3
else Divergence)
print(f"{name:<20} {x_final:<12.4f} {f(x_final):<14.6f} {judgement}")
# Analysis: How many steps are needed to make |x| < 0.01?
print("\nHow many steps does EXAMPLE need to converge to |x| < 0.01?)
for name, lr in settings.items():
x = 8.0
for step in range(1, 1001):
x = x - lr * grad(x)
if abs(x) < 0.01:
print(f{name}: needs {step} steps)
break
else:
print(f{name}: Failed to converge within 1000 steps)
EXAMPLE 三种学习率 30 步后对比： 学习率设置 最终 x f(x) 判断 -------------------------------------------------------- 过小 (lr=0.01) 4.3219 18.678861 收敛中 合适 (lr=0.2) 0.0000 0.000000 已收敛 过大 (lr=1.05) -34.5929 1196.668701 发散 EXAMPLE 需要多少步收敛到 |x| < 0.01？ 过小 (lr=0.01): 需要 377 步 合适 (lr=0.2): 需要 20 步 过大 (lr=1.05): 1000 步内无法收敛
Application scenarios in AI
| scene | Explanation |
|---|---|
| Learning rate scheduling | In actual training, "learning rate decay" is commonly used — starting large and then small, with fast convergence in the early stage and fine-tuning in the later stage. |
| Warmup | When training a Transformer, first warm up with a very small learning rate, then gradually increase it—to avoid initial oscillation. |
| Adam adaptive | Automatically adjust the learning rate for each parameter — the "Optimizer Comparison" case will demonstrate in detail. |