Single-Variable Optimization

Q: How do you find the optimum of a single-variable function?

Find the stationary points where f'(x) = 0, then classify each with the second derivative: f''(x) > 0 means a local minimum (a candidate minimum) and f''(x) < 0 means a local maximum (a candidate maximum); when f''(x) = 0 the second-derivative test is inconclusive, so use the first-derivative test. On a closed interval [a, b], compare the interior stationary points with the endpoint values f(a) and f(b) to find the global optimum.

Q: What is golden-section search?

It is a method for efficiently locating the minimum of a one-dimensional unimodal function (a function with a single minimum) by repeatedly shrinking the search interval using the golden ratio phi = (sqrt(5)-1)/2 = 0.618. Each iteration reduces the interval by about (1 - phi) = 38.2%, which is useful when derivatives are unavailable or a numerically stable search is needed.

Q: Can minimizing a quadratic be solved in closed form?

Yes. For f(x) = ax^2 + bx + c with a > 0, f'(x) = 2ax + b = 0 gives the minimizer x* = -b/(2a). More generally, quadratic programs (a quadratic objective with linear constraints) can be solved efficiently even in many variables; least-squares regression and the basic SVM are examples.

Extrema, derivative tests, and optimization on a closed interval

Introductory Undergraduate Level

Definition of extrema

Definition: local maximum and minimum

A function $f(x)$ has a local maximum at $x = a$ if, in some neighborhood of $a$,

$$f(a) \geq f(x)$$

holds. The value $f(a)$ is then called a local maximum value.

Similarly, $f$ has a local minimum at $a$ if, in some neighborhood of $a$,

$$f(a) \leq f(x)$$

holds. Local maxima and minima together are called extrema (extremum).

Local extrema versus the global maximum/minimum

A local extremum is a local notion — a comparison within a neighborhood.

The global maximum and minimum are global notions — comparisons over the whole domain.

A local maximum need not be the global maximum, and a local minimum need not be the global minimum.

Critical points

Definition: critical point

A point $a$ where $f'(a) = 0$ or $f'(a)$ does not exist is called a critical point. A critical point with $f'(a) = 0$ is more specifically called a stationary point. For example, $x = 0$ is a critical point of $f(x) = |x|$ but not a stationary point, because $f$ is not differentiable there.

Theorem: necessary condition for an extremum (Fermat's theorem)

If $f(x)$ is differentiable at $x = a$ and has an extremum there, then $f'(a) = 0$.

Proof

Suppose $f$ has a local maximum at $x = a$ (the minimum case is analogous). In a neighborhood of $a$ we have $f(a) \geq f(x)$, so for $h > 0$

$$\dfrac{f(a+h) - f(a)}{h} \leq 0$$

Letting $h \to 0^+$ gives $f'(a) \leq 0$. Similarly, for $h < 0$

$$\dfrac{f(a+h) - f(a)}{h} \geq 0$$

Letting $h \to 0^-$ gives $f'(a) \geq 0$. Hence $f'(a) = 0$. $\square$

Note: necessary but not sufficient

Even if $f'(a) = 0$, $a$ need not be an extremum. For example, $f(x) = x^3$ has $f'(0) = 0$, but $x = 0$ is not an extremum (it is an inflection point).

First-derivative test

Theorem: first-derivative test

When $f'(a) = 0$:

$f'(x)$ changes from positive to negative at $x = a$ → $f(a)$ is a local maximum
$f'(x)$ changes from negative to positive at $x = a$ → $f(a)$ is a local minimum
$f'(x)$ does not change sign → $f(a)$ is not an extremum

Figure 1: Intuition behind the first-derivative test

Example 1: $f(x) = x^3 - 3x$

Step 1: find the critical points

$f'(x) = 3x^2 - 3 = 3(x^2 - 1) = 3(x+1)(x-1)$

$f'(x) = 0$ gives $x = -1, 1$.

Step 2: examine the sign changes

$x < -1$: $f'(x) > 0$ (increasing)
$-1 < x < 1$: $f'(x) < 0$ (decreasing)
$x > 1$: $f'(x) > 0$ (increasing)

Conclusion:

$x = -1$: local maximum $f(-1) = 2$
$x = 1$: local minimum $f(1) = -2$

Figure 2: Graph of $f(x)=x^3-3x$, with a local maximum $2$ at $x=-1$ and a local minimum $-2$ at $x=1$.

Second-derivative test

Theorem: second-derivative test

When $f'(a) = 0$ and $f''(a)$ exists:

$f''(a) < 0$ → $f(a)$ is a local maximum
$f''(a) > 0$ → $f(a)$ is a local minimum
$f''(a) = 0$ → the test is inconclusive (use the first-derivative test)

Intuition

$f''(a)$ measures the "concavity" of the curve at $x = a$.

$f''(a) < 0$: concave down ($\cap$) → local maximum
$f''(a) > 0$: concave up ($\cup$) → local minimum

Example 2: $f(x) = x^4 - 2x^2$

Step 1: find the critical points

$f'(x) = 4x^3 - 4x = 4x(x^2 - 1) = 4x(x+1)(x-1)$

$f'(x) = 0$ gives $x = -1, 0, 1$.

Step 2: classify with the second derivative

$f''(x) = 12x^2 - 4$

$f''(-1) = 12 - 4 = 8 > 0$ → local minimum $f(-1) = -1$
$f''(0) = -4 < 0$ → local maximum $f(0) = 0$
$f''(1) = 12 - 4 = 8 > 0$ → local minimum $f(1) = -1$

Figure 3: Graph of $f(x)=x^4-2x^2$, with local minima $-1$ at $x=\pm1$ and a local maximum $0$ at $x=0$.

Maximum and minimum on a closed interval

Theorem: existence of a maximum and minimum (the extreme value theorem)

If $f(x)$ is continuous on a closed interval $[a, b]$, then $f$ attains a maximum and a minimum on $[a, b]$.

How to find the maximum and minimum on a closed interval

For a function $f(x)$ continuous on $[a, b]$, the maximum and minimum lie among the following candidates:

critical points $c$ in the interior ($f'(c) = 0$ or $f'(c)$ does not exist)
the endpoints $a$, $b$

Evaluate $f$ at all of these and compare.

Example 3: maximum and minimum of $f(x) = x^3 - 3x$ on $[-2, 2]$

Step 1: find the critical points

$f'(x) = 3x^2 - 3 = 0$ gives $x = \pm 1$ (both interior).

Step 2: evaluate at the candidate points

$f(-2) = (-2)^3 - 3(-2) = -8 + 6 = -2$
$f(-1) = (-1)^3 - 3(-1) = -1 + 3 = 2$
$f(1) = 1^3 - 3(1) = 1 - 3 = -2$
$f(2) = 2^3 - 3(2) = 8 - 6 = 2$

Conclusion: the maximum is $2$ (at $x = -1, 2$) and the minimum is $-2$ (at $x = -2, 1$).

Figure 4: Graph of $f(x)=x^3-3x$ on $[-2,2]$. Maximum $2$ (at $x=-1,2$), minimum $-2$ (at $x=-2,1$); the endpoints are candidates too.

Applications

Example 4: maximizing the volume of a box

From a square sheet of side $a$ we fold an open box. We maximize $V(x) = x(a - 2x)^2$.

Here we take $a = 12$.

$V(x) = x(12 - 2x)^2 = x(144 - 48x + 4x^2) = 4x^3 - 48x^2 + 144x$

Step 1: find the critical points

$V'(x) = 12x^2 - 96x + 144 = 12(x^2 - 8x + 12) = 12(x - 2)(x - 6)$

$V'(x) = 0$ gives $x = 2, 6$.

Step 2: check the constraint

Since $x$ is the side length of the cut-out squares, we need $12 - 2x > 0$, i.e. $0 < x < a/2 = 6$. Hence $x = 6$ lies on the boundary and is excluded, and the only interior critical point is $x = 2$.

Step 3: second-derivative test

$V''(x) = 24x - 96$

$V''(2) = 48 - 96 = -48 < 0$ → local maximum.

Conclusion: the volume is maximized at $x = 2$, with $V(2) = 2 \times 64 = 128$.

Figure 5: Box volume $V(x)=4x^3-48x^2+144x$ on $0

Example 5: maximizing profit

A product has demand function $p = 100 - 2q$ ($p$: price, $q$: quantity sold; since the price is non-negative, $0 \le q \le 50$) and a unit production cost of $20$. Find the quantity that maximizes profit.

Formulation

Revenue: $R(q) = pq = (100 - 2q)q = 100q - 2q^2$
Cost: $C(q) = 20q$
Profit (objective): $\pi(q) = R(q) - C(q) = 100q - 2q^2 - 20q = 80q - 2q^2$

Step 1: find the critical point

$\pi'(q) = 80 - 4q = 0$ gives $q = 20$.

Step 2: second-derivative test

$\pi''(q) = -4 < 0$ → local maximum.

Conclusion: profit is maximized at $q = 20$, with $\pi(20) = 80 \times 20 - 2 \times 400 = 1600 - 800 = 800$.

Figure 6: Profit $\pi(q)=80q-2q^2$ on $0\le q\le 50$, with maximum profit $800$ at $q=20$ and break-even at $q=40$.

Beyond derivatives: numerical methods (golden-section search)

When $f'(x)$ is too complicated to solve analytically, or when derivatives are unavailable, numerical methods such as golden-section search are used. For a unimodal function (one with a single minimum), this method shrinks the search interval using the golden ratio $\phi = (\sqrt{5}-1)/2 \approx 0.618$, reducing it by about $(1-\phi) \approx 38.2$% per iteration to home in on the minimum. Gradient-based methods for several variables are covered in later chapters.

Summary

Critical point: a point where $f'(x) = 0$ or $f'(x)$ does not exist
Fermat's theorem: at an extremum, $f'(x) = 0$ (a necessary condition)
First-derivative test: classify extrema by the sign change of $f'(x)$
Second-derivative test: $f''(a) < 0$ → local maximum, $f''(a) > 0$ → local minimum
Optimization on a closed interval: compare the critical points and the endpoints

Frequently asked questions

How do you find the optimum of a single-variable function?

Find the stationary points where $f'(x) = 0$, then classify each with the second derivative: $f''(x) > 0$ means a local minimum (a candidate minimum) and $f''(x) < 0$ means a local maximum (a candidate maximum); when $f''(x) = 0$ the second-derivative test is inconclusive, so use the first-derivative test. On a closed interval $[a,b]$, compare the interior stationary points with the endpoint values $f(a)$ and $f(b)$ to find the global optimum.

What is golden-section search?

It is a method for efficiently locating the minimum of a one-dimensional unimodal function (a function with a single minimum) by repeatedly shrinking the search interval using the golden ratio $\phi = (\sqrt{5}-1)/2 \approx 0.618$. Each iteration reduces the interval by about $(1-\phi) \approx 38.2$%, which is useful when derivatives are unavailable or a numerically stable search is needed.

Can minimizing a quadratic be solved in closed form?

Yes. For $f(x) = ax^2 + bx + c$ (with $a > 0$), $f'(x) = 2ax + b = 0$ gives the minimizer $x^* = -b/(2a)$. More generally, quadratic programs (a quadratic objective with linear constraints) can be solved efficiently even in many variables; least-squares regression and the basic SVM are examples.