A Map of Generalized Derivatives

1. Introduction — one question and three axes

In essence, differentiation asks for the linear object that best approximates a function locally, or equivalently for a rate of change. This single question changes its appearance along three axes.

• Axis 1: which directions do we look at? — only the coordinate axes (partial derivative), or an arbitrary direction (directional derivative).
• Axis 2: what is the underlying space? — finite-dimensional $\mathbb{R}^n$ (directional derivative), or function spaces and Banach spaces (Gâteaux derivative / Fréchet derivative).
• Axis 3: how strong is the approximation, how smooth is the function? — a weak derivative taken direction by direction (Gâteaux), or a uniform best linear approximation (Fréchet). Giving up smoothness further leads to the set-valued subdifferential for non-differentiable convex functions, and to the weak derivative, which replaces differentiation by an identity between integrals.

Adding the viewpoint of duality (Legendre–Fenchel transform) and the culmination in applications (calculus of variations) completes a single map. This page is that map itself; each concept is treated in detail on its own page.

2. The whole picture (relationship diagram)

The diagram shows how each concept branches off from the central question, "the best linear approximation". The arrows indicate how the concepts are built up (widening the objects and spaces, then strengthening the requirement), and the small text beside each arrow says what has been changed. The logical implications run the other way (§3).

3. Hierarchy of strength (implications)

For real-valued maps on normed spaces, with each term used in the sense above, the following implications hold; the further right, the weaker the condition.

The arrows in the opposite direction hold only under additional assumptions:

• the partial derivatives exist and are continuous ($C^1$) $\Rightarrow$ Fréchet differentiable;
• the Gâteaux derivative exists in a neighbourhood and is continuous (in operator norm) $\Rightarrow$ Fréchet differentiable;
• counterexample 1: all directional derivatives (Gâteaux variations) exist, but the dependence on the direction is not linear, so the function is not Gâteaux differentiable — $f(x,y)=\dfrac{x^2y}{x^4+y^2}$ (directional derivative §7) and $f(x,y)=\dfrac{x^3}{x^2+y^2}$ (Gâteaux derivative §5); in both cases $D_{\mathbf h}f$ at the origin fails to be a linear function of $\mathbf h$;
• counterexample 2: Gâteaux differentiable but not Fréchet differentiable — $f(x,y)=\dfrac{x^3y}{x^6+y^2}$ (with $f(0,0)=0$). For every $\mathbf h=(a,b)$ one has $\lim_{t\to0}\dfrac{f(t\mathbf h)}{t}=0$, so the Gâteaux derivative is the zero map (linear); yet along the curve $y=x^3$ one has $f(x,x^3)=\tfrac12$, so $f$ is not even continuous at the origin, and a function that is not continuous cannot be Fréchet differentiable.

4. Comparison table

5. The story of the connections

5.1 Partial → directional derivative (from coordinates to arbitrary directions)

A partial derivative is nothing but a directional derivative with the direction fixed to a coordinate axis $\mathbf e_i$. Allowing an arbitrary unit vector $\mathbf u$, a differentiable function satisfies $D_{\mathbf u}f=\nabla f\cdot\mathbf u=\lVert\nabla f\rVert\cos\theta$, which reveals the geometry in which the gradient direction is the direction of steepest ascent. See directional derivative for details.

5.2 Directional → Gâteaux derivative (from finite dimensions to function spaces)

The defining expression uses only addition and scalar multiplication, so it remains meaningful when $\mathbf a$ and $\mathbf u$ are replaced by functions. This is the Gâteaux derivative, which for a functional is exactly the first variation $\delta J(y;\eta)$. The pitfall seen in finite dimensions — derivatives in all directions do not imply differentiability — carries over unchanged.

5.3 Gâteaux → Fréchet derivative (from weak to strong)

The Gâteaux derivative is a weak derivative that takes the limit direction by direction. The Fréchet derivative requires $\lVert h\rVert\to0$ uniformly in all directions and controls the remainder by $o(\lVert h\rVert)$: the "best linear approximation". In return for this strength, the chain rule and the mean value inequality hold cleanly, and in finite dimensions it coincides with the total derivative (the Jacobian matrix).

5.4 Extension to the nonsmooth case: the subdifferential

A convex function with a kink, such as $f(x)=|x|$, is not differentiable at the origin. Yet infinitely many lines support the graph from below (supporting hyperplanes). The set of all their slopes is the subdifferential $\partial f(x)$. At a point of differentiability it collapses to the singleton $\partial f(x)=\{\nabla f(x)\}$, so it contains the ordinary derivative. Being set-valued, it is a generalization of a rather different flavour.

5.5 Extension by moving to integrals: the weak derivative

The other route that gives up smoothness is the weak derivative. Here the pointwise limit is abandoned and the integration-by-parts formula $\int u'\varphi\,dx=-\int u\varphi'\,dx$ is adopted as the definition. The right-hand side does not differentiate $u$, so it is meaningful even when $u$ is not smooth. A function $v$ satisfying this identity for every smooth test function $\varphi$ with compact support is called the weak derivative of $u$. The weak derivative of $|x|$ is $\operatorname{sgn}(x)$, whereas a step function with a jump has no weak derivative representable by an $L^1_{\mathrm{loc}}$ function, and in the world of distributions one writes $H'=\delta$.

While the subdifferential widens the value into a set by using convexity, the weak derivative moves the way of measuring to integrals. This single step produces complete spaces in which differentiation is closed under limits (Sobolev spaces), and it is the theoretical foundation of weak solutions of partial differential equations and of the finite element method.

5.6 The dual viewpoint: the Legendre–Fenchel transform

Describing a function not by its pointwise values but by the slopes and intercepts of its tangent lines is the Legendre–Fenchel transform $f^*(p)=\sup_x(\langle p,x\rangle-f(x))$. If $f$ is proper convex and lower semicontinuous (closed proper convex), the subdifferential yields the elegant duality $p\in\partial f(x)\Leftrightarrow x\in\partial f^*(p)$ even in the nonsmooth case (Fenchel–Moreau). If in addition $f$ is strictly convex and differentiable, then $p=\nabla f(x)$ and $x=\nabla f^*(p)$ are mutually inverse maps on the corresponding domains; differentiability alone does not make $\nabla f$ injective, so the inverse need not exist. The Fenchel–Young inequality $f(x)+f^*(p)\ge\langle x,p\rangle$ ties the two together.

5.7 The culmination in applications: the calculus of variations

Setting the first variation (that is, the Gâteaux derivative) of a functional to zero yields the Euler–Lagrange equation. The brachistochrone, geodesics and the principle of least action — the basic laws of physics and geometry are written in the language of this generalized derivative. See calculus of variations.

6. Which one should you use? (quick guide)

• You only need the rate of change along one direction in finite dimensions → directional derivative.
• You want a stationarity condition in a function space (a variational problem) → Gâteaux derivative = first variation.
• You want to apply "derivative-like" theorems rigorously — chain rule, Newton's method, implicit function theorem → Fréchet derivative.
• Your objective function has kinks (L1 regularization, hinge loss, …) → subdifferential.
• You deal with differential equations whose solutions need not be smooth, or you want to justify the finite element method → weak derivative and Sobolev spaces.
• You want to swap variables (position ↔ momentum, $S$ ↔ $T$) or build a dual problem → Legendre–Fenchel transform.

7. FAQ

8. Related pages and references

Nodes of this map (dedicated pages)

• Partial derivative — rate of change along the coordinate axes
• Directional derivative — rate of change in any direction, steepest ascent
• Gâteaux derivative — the infinite-dimensional version of the directional derivative, the first variation
• Fréchet derivative — uniform best linear approximation, generalization of the total derivative
• Subdifferential — set-valued derivative of a nonsmooth convex function; the algorithm is the subgradient method
• Weak derivative — differentiation moved onto test functions by integration by parts; continues to Sobolev spaces and weak solutions and distribution theory
• Legendre transform — convex conjugate and dual variables
• Calculus of variations / Euler–Lagrange equation
• The many faces of the derivative — the finite-dimensional "faces" (complementary reading)

References

• Rockafellar, R. T. (1970). Convex Analysis. Princeton University Press.
• Zeidler, E. (1986). Nonlinear Functional Analysis and its Applications I. Springer.
• Luenberger, D. G. (1969). Optimization by Vector Space Methods. Wiley.
• Clarke, F. H. (1990). Optimization and Nonsmooth Analysis. SIAM.