Chapter 2: Fundamentals of Errors

Example: Approximation of Pi

Approximating $\pi = 3.14159265...$ by $\tilde{\pi} = 3.14$:

• Absolute error: $|3.14159... - 3.14| \approx 0.00159$
• Relative error: $\dfrac{0.00159...}{3.14159...} \approx 0.00051 = 0.051\%$

Example: Large Numbers vs. Small Numbers

Case 1: $x = 1000000$, $\tilde{x} = 1000001$

• Absolute error: $1$
• Relative error: $10^{-6} = 0.0001\%$

Case 2: $x = 0.001$, $\tilde{x} = 0.002$

• Absolute error: $0.001$
• Relative error: $1 = 100\%$

Even when absolute error is small, relative error can be large.

Example: Rounding a Decimal

Representing $\dfrac{1}{3} = 0.33333...$ with 4 digits gives $0.3333$

Absolute error: $\dfrac{1}{3} - 0.3333 = 0.00003...$

Example: Binary Representation of $0.1$

The decimal number $0.1$ becomes an infinite repeating fraction in binary:

$$0.1_{10} = 0.0001100110011..._{2}$$

Therefore, in the usual binary floating-point formats, $0.1$ cannot be represented exactly, and a small error results.

Example: Taylor Expansion of $e$

$e$ is represented by an infinite series:

$$e = \displaystyle\sum_{n=0}^{\infty} \dfrac{1}{n!} = 1 + 1 + \dfrac{1}{2} + \dfrac{1}{6} + \dfrac{1}{24} + \cdots$$

Truncating after the first 5 terms:

$$e \approx 1 + 1 + \dfrac{1}{2} + \dfrac{1}{6} + \dfrac{1}{24} = \dfrac{65}{24} = 2.70833\ldots$$

The difference from the true value $e = 2.71828...$, namely $e - \dfrac{65}{24} \approx 0.00995$, is the truncation error.

Example: Approximation of the Derivative

Approximating the limit-based definition $\displaystyle f'(x) = \lim_{h \to 0} \dfrac{f(x+h) - f(x)}{h}$ by a finite $h$:

$$f'(x) \approx \dfrac{f(x+h) - f(x)}{h}$$

If $f$ is twice differentiable, this amounts to cutting the Taylor expansion $f(x+h) = f(x) + hf'(x) + \dfrac{h^2}{2}f''(\xi)$ (with $\xi$ between $x$ and $x+h$) after the first-order term, and the remaining term leaves an error of $\dfrac{h}{2}f''(\xi)$. This is the truncation error, and its size is $O(h)$ (proportional to $h$).

However, making $h$ extremely small causes catastrophic cancellation (§5) in the subtraction of the nearly equal values $f(x+h)$ and $f(x)$: the rounding errors already contained in these values become large relative to their difference. That difference is then divided by $h$, so the effect of rounding error grows as $1/h$. Since the truncation error shrinks in proportion to $h$, this forward difference has an optimal $h$ at which the two balance (when $f$ is smooth and its values are computed with rounding error).

Example: Error Propagation in Addition

If $\tilde{x} = x + \epsilon_1$ and $\tilde{y} = y + \epsilon_2$ (where $\epsilon_1, \epsilon_2$ are signed errors, related to the absolute error of §2 by $E = |\epsilon|$), then

$$\tilde{x} + \tilde{y} = (x + y) + (\epsilon_1 + \epsilon_2)$$

The errors add as they are. Because they are signed, errors of opposite sign can cancel; the size is bounded by $|\epsilon_1 + \epsilon_2| \le |\epsilon_1| + |\epsilon_2|$.

Example: Error Propagation in Multiplication

Writing the inputs in relative-error form, $\tilde{x} = x(1+\delta_1)$ and $\tilde{y} = y(1+\delta_2)$ (where $\delta_1, \delta_2$ are signed relative errors, a different quantity from the $\epsilon$ above):

$$\tilde{x} \cdot \tilde{y} = xy\,(1 + \delta_1 + \delta_2 + \delta_1 \delta_2) \approx xy\,(1 + \delta_1 + \delta_2)$$

Dropping the second-order term $\delta_1 \delta_2$, the relative errors approximately add.

For division,

$$\dfrac{\tilde{x}}{\tilde{y}} = \dfrac{x}{y}\cdot\dfrac{1+\delta_1}{1+\delta_2} \approx \dfrac{x}{y}\,(1 + \delta_1 - \delta_2)$$

so the relative errors subtract (they cancel when $\delta_1 = \delta_2$). For both multiplication and division, the size of the relative error of the result is at most about $|\delta_1| + |\delta_2|$.

Example: Error Amplification in Subtraction

When $x$ and $y$ each carry an error and $x \approx y$, the relative error of $x - y$ becomes very large (the subtraction itself introduces no error; it raises the relative weight of the errors already in the inputs).

Example: $x = 1.0001$, $y = 1.0000$ (each with 5-digit precision; see §6 for counting significant digits)

$x - y = 0.0001$, but the number of significant digits drops to just 1. Each value has a rounding error of at most $0.00005$ (a relative error of about $0.005\%$), yet relative to the difference $0.0001$ the error can be as large as $0.0001$, a relative error of $100\%$.

This phenomenon is called catastrophic cancellation.

Example: Counting Significant Digits

• $123.45$ → 5 significant digits
• $0.00123$ → 3 significant digits (leading zeros are not counted)
• $1.23 \times 10^5$ → 3 significant digits