Writing a Binary Classification Neural Network from Scratch
Comprehensively applying matrix multiplication, ReLU/Sigmoid activation, cross-entropy loss, and backpropagation—connecting all knowledge points at once.
After completing this case, you will understand:The complete mathematical loop of neural network training—matrix transformation of data, activation introducing nonlinearity, cross-entropy measuring quality, and backpropagation passing gradients.
Everyday Introduction
Distinguishing Two Fruits—Size Alone Is Not Enough
Apples and oranges are mixed together. Looking only at size is not enough—some small apples are about the same size as large oranges. You need two dimensions (size + color) and a non-straight boundary—this is where neural networks come in. A single-layer linear model can only draw straight lines; adding hidden layers and nonlinear activation functions allows it to draw curved boundaries.
Intuitive Understanding
The mathematical modules corresponding to each component:
Mathematical Definition
\[ z_1 = X W_1 + b_1,\ a_1 = \text{ReLU}(z_1),\quad z_2 = a_1 W_2 + b_2,\ a_2 = \sigma(z_2) \]The gradient for Sigmoid + cross-entropy automatically simplifies to the most compact form: \(\frac{\partial L}{\partial z_2} = a_2 - y\)
Python Hands-on Practice
Example
np.random.seed(42)
def sigmoid(z):
return 1 / (1 + np.exp(-np.clip(z, -500, 500)))
def relu(z): return np.maximum(0, z)
def relu_derivative(z): return (z > 0).astype(float)
class TwoLayerNN:
def __init__(self, n_input, n_hidden, lr=0.1):
self.W1 = np.random.randn(n_input, n_hidden) * 0.5
self.b1 = np.zeros(n_hidden)
self.W2 = np.random.randn(n_hidden, 1) * 0.5
self.b2 = np.zeros(1)
self.lr = lr
self.loss_history = []
def forward(self, X):
self.z1 = X @ self.W1 + self.b1
self.a1 = relu(self.z1)
self.z2 = self.a1 @ self.W2 + self.b2
self.a2 = sigmoid(self.z2)
return self.a2
def compute_loss(self, y_pred, y_true, eps=1e-9):
y_pred = np.clip(y_pred, eps, 1 - eps)
return -np.mean(y_true * np.log(y_pred) +
(1 - y_true) * np.log(1 - y_pred))
def backward(self, X, y_true):
n = X.shape[0]
y_true = y_true.reshape(-1, 1)
dz2 = (self.a2 - y_true) / n # Sigmoid+CE simplification
dW2 = self.a1.T @ dz2
db2 = np.sum(dz2, axis=0)
da1 = dz2 @ self.W2.T # Chain rule backpropagation
dz1 = da1 * relu_derivative(self.z1)
dW1 = X.T @ dz1
db1 = np.sum(dz1, axis=0)
self.W2 -= self.lr * dW2
self.b2 -= self.lr * db2
self.W1 -= self.lr * dW1
self.b1 -= self.lr * db1
def fit(self, X, y, epochs=1000):
for epoch in range(epochs):
y_pred = self.forward(X)
loss = self.compute_loss(y_pred, y)
self.loss_history.append(loss)
self.backward(X, y)
if epoch % 200 == 0:
acc = np.mean((y_pred.flatten() > 0.5) == y)
print(f"EXAMPLE epoch {epoch:4d} loss={loss:.4f} acc={acc:.3f}")
def predict(self, X):
return (self.forward(X).flatten() > 0.5).astype(int)
# Concentric circle data (nonlinearly separable)
n = 300
theta = np.random.uniform(0, 2 * np.pi, n)
r_inner = np.random.normal(1.0, 0.15, n // 2)
r_outer = np.random.normal(2.5, 0.15, n // 2)
X = np.vstack([
np.c_[r_inner * np.cos(theta[:n//2]), r_inner * np.sin(theta[:n//2])],
np.c_[r_outer * np.cos(theta[n//2:]), r_outer * np.sin(theta[n//2:])],
])
y = np.array([0] * (n // 2) + [1] * (n // 2))
model = TwoLayerNN(n_input=2, n_hidden=8, lr=0.5)
model.fit(X, y, epochs=1000)
print(f"\nEXAMPLE Final accuracy: {np.mean(model.predict(X) == y):.2%}")
print(f"Loss: {model.loss_history[0]:.4f} -> {model.loss_history[-1]:.4f}")
EXAMPLE epoch 0 loss=0.6932 acc=0.497 EXAMPLE epoch 200 loss=0.1494 acc=0.930 EXAMPLE epoch 400 loss=0.0615 acc=0.967 EXAMPLE epoch 600 loss=0.0272 acc=0.990 EXAMPLE epoch 800 loss=0.0134 acc=1.000 EXAMPLE epoch 1000 loss=0.0075 acc=1.000 EXAMPLE 最终准确率: 100.00% 损失: 0.6932 -> 0.0075
Application Scenarios in AI
| Scenario | Relationship with This Case |
|---|---|
| Image classification | CNN = In this case, the matrix multiplication is replaced by a convolution operation. |
| Text classification | BERT fine-tuning = this case + Transformer encoder |
| Anomaly detection | Output the anomaly probability from 0 to 1—that is, the binary classification special case of this case. |