The Mathematical Essence of Backpropagation
This chapter will manually calculate the entire backpropagation process of a two-layer neural network.
Backpropagation = A systematic application of the chain rule on a computational graph.
Concept Analysis
Computational Graph: Forward and Backward
A simple two-layer network: input x → hidden layer (ReLU) → output → MSE loss.
Forward propagation computes the output from left to right. Backpropagationfrom right to leftcomputes the gradient of each parameter with respect to the loss.
Manually Calculate a Complete Example
Let x=2, y=4 (true label), parameters w₁=0.5, b₁=0, w₂=1, b₂=0.
Forward Propagation
| Step | Calculation | Value |
|---|---|---|
| z₁ = w₁x + b₁ | 0.5×2 + 0 | 1.0 |
| a₁ = ReLU(z₁) | max(0, 1.0) | 1.0 |
| ŷ = w₂a₁ + b₂ | 1×1.0 + 0 | 1.0 |
| L = (ŷ - y)² | (1-4)² | 9.0 |
Backpropagation (Chain Rule)
Gradient of the loss with respect to ŷ:\( \frac{\partial L}{\partial \hat{y}} = 2(\hat{y}-y) = 2(1-4) = -6 \)
Gradient of ŷ with respect to w₂ → L with respect to w₂:\( \frac{\partial \hat{y}}{\partial w_2} = a_1 = 1.0 \)，\( \frac{\partial L}{\partial w_2} = (-6) \times 1.0 = -6 \)
Gradient of L with respect to a₁ (continuing forward propagation):\( \frac{\partial L}{\partial a_1} = \frac{\partial L}{\partial \hat{y}} \cdot w_2 = -6 \times 1 = -6 \)
Through ReLU:\( \frac{\partial a_1}{\partial z_1} = 1 \) (since z₁=1>0), \( \frac{\partial L}{\partial z_1} = -6 \)
L with respect to w₁:\( \frac{\partial L}{\partial w_1} = \frac{\partial L}{\partial z_1} \cdot x = -6 \times 2 = -12 \)
Summing Up Gradients and Updating
| Parameter | Gradient | Update (lr=0.1) |
|---|---|---|
| w₁ | -12 | 0.5 + 1.2 = 1.7 |
| b₁ | -6 | 0 + 0.6 = 0.6 |
| w₂ | -6 | 1 + 0.6 = 1.6 |
| b₂ | -6 | 0 + 0.6 = 0.6 |
The gradients are all negative — this indicates the current prediction is too small (1 < 4), so all parameters need to increase.
The gradient of w₁ (-12) is larger than that of w₂ (-6): because w₁ is farther from the loss, the chain propagation path is longer, and it is multiplied by x=2.
Python Verification
Example
x = torch.tensor([2.0])
y = torch.tensor([4.0])
w1 = torch.tensor([0.5], requires_grad=True)
b1 = torch.tensor([0.0], requires_grad=True)
w2 = torch.tensor([1.0], requires_grad=True)
b2 = torch.tensor([0.0], requires_grad=True)
# Forward
a1 = torch.relu(w1 * x + b1)
y_pred = w2 * a1 + b2
loss = (y_pred - y) ** 2
loss.backward()
print("=== EXAMPLE PyTorch Verification ===")
print(f"∂L/∂w1 = {w1.grad.item():.1f} (hand-calculated: -12)")
print(f"∂L/∂b1 = {b1.grad.item():.1f} (hand calculation: -6)")
print(f"∂L/∂w2 = {w2.grad.item():.1f} (hand calculation: -6)")
print(f"∂L/∂b2 = {b2.grad.item():.1f} (hand calculation: -6)")
# One step of gradient descent
lr = 0.1
with torch.no_grad():
w1 -= lr * w1.grad; b1 -= lr * b1.grad
w2 -= lr * w2.grad; b2 -= lr * b2.grad
new_loss = ((w2 * torch.relu(w1*x+b1) + b2 - y)**2).item()
print(f"\nOriginal loss: {loss.item():.1f} → New loss: {new_loss:.2f}")
print(f"The loss decreased!")
=== EXAMPLE PyTorch 验证 === ∂L/∂w1 = -12.0 (手算: -12) ∂L/∂b1 = -6.0 (手算: -6) ∂L/∂w2 = -6.0 (手算: -6) ∂L/∂b2 = -6.0 (手算: -6) 原损失: 9.0 → 新损失: 1.03 损失减小了！
Backpropagation = Three Lines Converge
Forward propagation = matrix operations
z = Wx + b
Backpropagation = chain rule
∂L/∂w = ∂L/∂z · ∂z/∂w
Parameter update = gradient descent
θ = θ - η∇J
Loss design = MLE + cross-entropy
L = -log P(y|x)
Backpropagation essentially does three things: forward propagation uses matrix operations to compute the output, backward propagation uses the chain rule to compute gradients, and after obtaining the gradients, gradient descent is used to update parameters.
Probability and statistics, along with information theory, provide a theoretical basis for the selection of loss functions.
Other extensions