Introduction to Optimization Problems -- Objective Functions, Variables, and Constraints
Optimization is finding the parameters that minimize the objective function.AI training = optimizing the loss function.
Concept Analysis
Three Elements of Optimization Problems
| Element | Meaning | Correspondence in AI Training |
|---|---|---|
| Objective Function | Function to be minimized | Loss function J(θ) |
| Decision Variables | Quantities that can be adjusted | Model parameters θ |
| Constraints | Restrictions on variables | Usually unconstrained |
Convex vs Non-Convex Functions
Convex Function
Bowl-shaped; the line segment between any two points lies above the curve
Local optimum = global optimum
Such as MSE loss in linear regression
Non-convex Function
Has multiple valleys and peaks
Can only find local optima
Such as deep neural network loss
The loss function of deep neural networks is highly non-convex — but in practice, the local optima found by gradient descent are usually good enough.
Everyday Examples
Choosing the shortest route
There are three routes from home to school: 5 minutes, 8 minutes, 12 minutes.
Decision variable = which route to choose, objective function = time taken, goal = minimize time.
This is simple — just try all three routes. But AI models have millions of parameters, making it impossible to enumerate all combinations.
That's why gradient descent is needed — step by step toward the optimal solution along the direction indicated by the gradient.
Python Hands-On Practice
Example
# Convex function f(x) = x^2, global optimum at x=0
# Non-convex function f(x) = sin(x) + 0.1*x^2, with multiple local optima
def f_convex(x): return x**2
def f_nonconvex(x): return np.sin(x) + 0.1*x**2
# Performing gradient descent from different starting points (detailed in the next chapter), non-convex may converge to different local optima
print(A non-convex function may reach different local optima from different starting points)
print(This is something to pay attention to when tuning deep learning hyperparameters)
Run output:
Non-convex functions may reach different local optima from different starting points. This is something to note when tuning deep learning hyperparameters.
3D Comparison of Convex vs Non-Convex Functions
The left image below is a convex function f(x,y)=x²+y² (bowl shape, unique global optimum), the right image is a non-convex function (multiple local optima).
Rotating the viewpoint gives an intuitive sense of the difference between "convex = single valley" and "non-convex = multiple valleys":
Application Scenarios in AI
The entire training process = large-scale optimization
Training a GPT-level model means finding the optimal point in a space of hundreds of trillions of dimensions. The loss function J(θ) is a hypersurface in this space, and gradient descent gradually descends along this surface. The entire training may last several weeks, consuming millions of dollars in computing power—just to find a set of parameters that make the loss as small as possible.
Convex vs Non-Convex: Theory and Practice
Linear regression and SVM optimization problems are convex—gradient descent guarantees finding the global optimum. But the loss surface of deep neural networks is highly non-convex, with countless local optima and saddle points. In practice, however, the local optima found by SGD usually generalize well—large batches tend to converge to "sharp" local optima (poor generalization), while small batches tend to converge to "flat" local optima (good generalization).
Loss Landscape Visualization
By projecting the high-dimensional loss surface to 2D, researchers discovered that deep networks' loss landscapes have a "canyon" structure—although non-convex, following the bottom of the canyon almost always leads to a very good solution. This explains why random initialization + SGD is so robust in deep learning.
Other Extensions