Chapter 2: Fundamentals of Errors
Goals
Understand the types of errors in numerical computation (absolute, relative, roundoff, truncation), and learn the basics of significant digits and error propagation.
Prerequisites
- Basic arithmetic operations
- Concepts of decimals and fractions
Table of Contents
§1 Why Errors Occur
In numerical computation, various types of "errors" arise. Understanding and managing errors is fundamental to numerical computation.
Main Sources of Error
- Representing infinite decimals with finite digits (rounding error)
- Terminating infinite computations after finitely many steps (truncation error)
- Inaccuracy of measured values or input data (input error)
§2 Absolute Error and Relative Error
Definition: Absolute Error
Let $x$ be the true value and $\tilde{x}$ the approximation. The absolute error is
$$E = |x - \tilde{x}|$$Definition: Relative Error
For a true value $x \neq 0$, the relative error is
$$E_r = \dfrac{|x - \tilde{x}|}{|x|}$$It can also be expressed as a percentage: $E_r \times 100\%$
Example: Approximation of Pi
Approximating $\pi = 3.14159265...$ by $\tilde{\pi} = 3.14$:
- Absolute error: $|3.14159... - 3.14| \approx 0.00159$
- Relative error: $\dfrac{0.00159}{3.14159} \approx 0.00051 = 0.051\%$
Example: Large Numbers vs. Small Numbers
Case 1: $x = 1000000$, $\tilde{x} = 1000001$
- Absolute error: $1$
- Relative error: $10^{-6} = 0.0001\%$
Case 2: $x = 0.001$, $\tilde{x} = 0.002$
- Absolute error: $0.001$
- Relative error: $1 = 100\%$
Even when absolute error is small, relative error can be large.
§3 Rounding Error
Rounding error arises when representing a number with a finite number of digits.
Example: Rounding a Decimal
Representing $\dfrac{1}{3} = 0.33333...$ with 4 digits gives $0.3333$
Absolute error: $\dfrac{1}{3} - 0.3333 = 0.00003...$
Rounding Methods
- Truncation: $3.146 \to 3.14$
- Round up: $3.141 \to 3.15$
- Round half up: $3.145 \to 3.15$, $3.144 \to 3.14$
- Round half to even: $3.145 \to 3.14$, $3.155 \to 3.16$ (banker's rounding)
Since computers typically perform arithmetic in binary, rounding errors can occur even with decimal fractions.
Example: Binary Representation of $0.1$
The decimal number $0.1$ becomes an infinite repeating fraction in binary:
$$0.1_{10} = 0.0001100110011..._{2}$$Therefore, a computer cannot represent $0.1$ exactly, and a small error results.
§4 Truncation Error
Truncation error arises when an infinite computation is terminated after finitely many steps.
Example: Taylor Expansion of $e$
$e$ is represented by an infinite series:
$$e = \displaystyle\sum_{n=0}^{\infty} \dfrac{1}{n!} = 1 + 1 + \dfrac{1}{2} + \dfrac{1}{6} + \dfrac{1}{24} + \cdots$$Truncating after the first 5 terms:
$$e \approx 1 + 1 + 0.5 + 0.1667 + 0.0417 = 2.7083$$The difference from the true value $e = 2.71828...$ is the truncation error.
Example: Approximation of the Derivative
Approximating the limit-based definition $\displaystyle f'(x) = \lim_{h \to 0} \dfrac{f(x+h) - f(x)}{h}$ by a finite $h$:
$$f'(x) \approx \dfrac{f(x+h) - f(x)}{h}$$Truncating the Taylor expansion at $O(h)$ produces a truncation error.
However, making $h$ extremely small triggers cancellation (rounding error) in the numerator $f(x+h) - f(x)$. Truncation error scales as $h$ while rounding error scales as $1/h$, so an optimal $h$ exists at the trade-off point.
§5 Error Propagation
When computations are repeated, errors can accumulate and amplify.
Example: Error Propagation in Addition
If $\tilde{x} = x + \epsilon_1$ and $\tilde{y} = y + \epsilon_2$, then
$$\tilde{x} + \tilde{y} = (x + y) + (\epsilon_1 + \epsilon_2)$$The errors simply add together.
Example: Error Propagation in Multiplication
Writing the inputs in relative-error form, $\tilde{x} = x(1+\epsilon_1)$ and $\tilde{y} = y(1+\epsilon_2)$:
$$\tilde{x} \cdot \tilde{y} = xy\,(1 + \epsilon_1 + \epsilon_2 + \epsilon_1 \epsilon_2) \approx xy\,(1 + \epsilon_1 + \epsilon_2)$$Dropping the second-order term $\epsilon_1 \epsilon_2$, the relative errors approximately add. Division behaves similarly.
Example: Error Amplification in Subtraction
When $x \approx y$, the relative error of $x - y$ becomes very large.
Example: $x = 1.0001$, $y = 1.0000$ (each with 4-digit precision)
$x - y = 0.0001$, but the number of significant digits drops to just 1.
This phenomenon is called catastrophic cancellation.
§6 Significant Digits
Significant digits represent the number of reliable digits in a measurement or computation.
Example: Counting Significant Digits
- $123.45$ → 5 significant digits
- $0.00123$ → 3 significant digits (leading zeros are not counted)
- $1.23 \times 10^5$ → 3 significant digits
Relationship Between Significant Digits and Relative Error
A value with $n$ significant digits has a relative error on the order of $10^{-n}$.
§7 Summary
- Absolute error $= |x - \tilde{x}|$, Relative error $= \dfrac{|x - \tilde{x}|}{|x|}$
- Rounding error: error from representing numbers with finite digits
- Truncation error: error from terminating infinite computations after finite steps
- Errors can propagate and amplify through computation
- Significant digits represent the number of reliable digits