Taylor expansion -- approximating any function with polynomials.

Any smooth function can be approximated by a polynomial near a point.

First-order Taylor expansion is the basis of gradient descent, and second-order expansion leads to Newton's method. For this chapter, just understand it; no need to delve into the derivation details.

Concept Analysis

First-order Taylor = linear approximation (basis of gradient descent)

\[ f(x) \approx f(a) + f'(a)(x - a) \]

This is "draw a tangent line at point a, and use the tangent line's value to approximate the function value."

Second-order Taylor = considering curvature.

\[ f(x) \approx f(a) + f'(a)(x - a) + \frac{f''(a)}{2}(x - a)^2 \]

Adding the second-order term captures the "curvature" of the function—this is the idea behind Newton's method.

Expansions of important functions

functionExpansion at x=0
\( e^x \)\( 1 + x + \frac{x^2}{2!} + \frac{x^3}{3!} + \cdots \)
\( \sin x \)\( x - \frac{x^3}{3!} + \frac{x^5}{5!} - \cdots \)

Gradient descent uses only first-order information (direction). Newton's method uses second-order information (curvature), converging faster but at a higher computational cost.

Everyday example

Estimate sin(0.1): sin(0)=0, sin'(0)=cos(0)=1. First-order approximation: 0 + 1×0.1 = 0.1.

The true value is approximately 0.0998—using only addition and multiplication, a high-precision approximation was obtained.

Python Hands-On Practice

Example

import numpy as np

x0 = 0.5
true_val = np.sin(x0)
print(fTaylor approximation of sin({x0}) vs true value {true_val:.8f}:)

# Manually compute the first few terms of the Taylor expansion
import math
approx = 0.0
for n in range(6):
    if n % 2 == 1:  The even-numbered terms are 0.
        coef = (-1)**((n-1)//2) / math.factorial(n)
        approx += coef * x0**n
        err = abs(approx - true_val)
        print(f"{n} terms: {approx:.8f} Error: {err:.2e}")

Output:

sin(0.5) 的泰勒近似 vs 真实值 0.47942554:
  1项: 0.50000000  误差: 2.06e-02
  3项: 0.47916667  误差: 2.59e-04
  5项: 0.47942708  误差: 1.54e-06

Application scenarios in AI.

Gradient descent = first-order Taylor approximation.

Gradient descent assumes that the loss function can be locally approximated by a first-order Taylor expansion (linear approximation). Taking a step in the direction of the negative gradient corresponds to making the optimal update under the first-order approximation. This is why gradient descent requires a small learning rate—if the learning rate is too large, it goes beyond the region where the linear approximation is valid.

Newton's method = second-order Taylor optimization.

If second-order terms are added (using the Hessian matrix), the loss can be accurately approximated locally by a quadratic function, and one can jump directly to the optimum of the quadratic approximation. In theory this converges faster, but the Hessian matrix is the square of the number of parameters—a million-parameter model would require a trillion-scale Hessian, making both computation and storage impractical.

XGBoost's objective function

XGBoost uses the second-order Taylor expansion of the loss function to approximate the objective in each iteration, while also leveraging the first-order gradient and the second-order Hessian (called "hess" in XGBoost). Compared to GBDT, which only uses the first-order gradient, it converges faster and is more stable. This is the most successful application of second-order information in industrial-grade ML.

Other extensions