Significant Digits and Rounding Error

Goal

Learn how to correctly count significant digits, and understand the mechanisms of catastrophic cancellation and absorption along with avoidance techniques (formula transformation, Kahan summation).

Prerequisites

Basic arithmetic of real numbers, concept of floating-point numbers

Table of Contents

1. Significant Digits

Significant digits (significant figures) are the number of meaningful (reliable) digits in a measured or computed value.

Counting Significant Digits

Count from the first nonzero digit to the last digit.

  • Leading zeros are not significant. (e.g., $0.0042$ has 2 significant digits)
  • Trailing zeros are significant. (e.g., $3.40$ has 3 significant digits)
  • Embedded zeros are significant. (e.g., $1.03$ has 3 significant digits)

An approximation $\tilde{x}$ has $n$ significant digits with respect to the true value $x$ when, in normalized form,

$$\left|\frac{\tilde{x} - x}{x}\right| \leq \frac{1}{2} \times 10^{1-n}$$

holds.

Examples

ValueSignificant digitsCount
$3.14159$3, 1, 4, 1, 5, 96 digits
$0.00340$3, 4, 03 digits
$1.200 \times 10^3$1, 2, 0, 04 digits
$5000$Ambiguous (1--4 digits)Context-dependent

2. Absolute and Relative Error

Two types of error measure how accurately an approximation $\tilde{x}$ represents the true value $x$:

$$\text{Absolute error} = |\tilde{x} - x|$$
$$\text{Relative error} = \frac{|\tilde{x} - x|}{|x|} \quad (x \neq 0)$$

Relative error is independent of magnitude, making it suitable for precision comparison. Having $n$ significant digits corresponds to relative error at most $\frac{1}{2} \times 10^{1-n}$.

3. Types of Rounding

Several methods exist for converting to floating-point (rounding).

MethodRuleExample (3 digits)
TruncationSimply discard extra digits$3.146 \to 3.14$
Round half upRound up if last digit is 5 or more$3.145 \to 3.15$
Round to nearest even (banker's rounding)Ties go to even$3.145 \to 3.14$, $3.155 \to 3.16$

The IEEE 754 default is round to nearest, ties to even, which minimizes statistical bias.

4. Catastrophic Cancellation

Catastrophic cancellation occurs when subtracting two nearly equal numbers, causing the leading significant digits to cancel and dramatically reducing the number of significant digits.

Example: Cancellation

If $a = 1.23456789$ and $b = 1.23456780$ both have 9 significant digits:

$$a - b = 0.00000009$$

The difference has only 1 significant digit -- a drastic reduction from 9.

Avoiding Catastrophic Cancellation

  • Formula transformation: Rationalize $\sqrt{x+1} - \sqrt{x}$ to $\dfrac{1}{\sqrt{x+1} + \sqrt{x}}$.
  • Taylor expansion: Use $1 - \cos x \approx \dfrac{x^2}{2}$ (for small $x$).
  • Quadratic formula: $\dfrac{-b + \sqrt{b^2-4ac}}{2a}$ suffers cancellation when $b > 0$; use $\dfrac{2c}{-b - \sqrt{b^2-4ac}}$ instead.

5. Absorption (Swamping)

Absorption (swamping) occurs when adding/subtracting two numbers of vastly different magnitudes, causing the smaller number's information to be lost.

Example: Absorption

In double precision (~16 significant digits), computing $10^{16} + 1$ ignores the $1$ since its contribution falls below the 16th digit.

$$\mathrm{fl}(10^{16} + 1) = 10^{16}$$

The problem is pronounced when sequentially adding many small values to a large one. The well-known countermeasure is Kahan's compensated summation algorithm (see Loss of Significance for pseudocode).

6. Error Propagation

When inputs $x_1, x_2, \ldots$ have errors $\Delta x_1, \Delta x_2, \ldots$, the output error of $f(x_1, x_2, \ldots)$ is approximated by first-order Taylor expansion as

$$\Delta f \approx \sum_{i} \frac{\partial f}{\partial x_i} \Delta x_i$$

Each partial derivative acts as an error amplification factor.

Error Propagation in Arithmetic

OperationAbsolute errorRelative error
$f = x \pm y$$|\Delta f| \leq |\Delta x| + |\Delta y|$Can grow (cancellation)
$f = x \cdot y$$|\Delta f| \approx |y||\Delta x| + |x||\Delta y|$$\dfrac{|\Delta f|}{|f|} \leq \dfrac{|\Delta x|}{|x|} + \dfrac{|\Delta y|}{|y|}$
$f = x / y$Compound$\dfrac{|\Delta f|}{|f|} \leq \dfrac{|\Delta x|}{|x|} + \dfrac{|\Delta y|}{|y|}$

7. Examples and Countermeasures

Example 1: Quadratic Formula (Avoiding Cancellation)

$x^2 + 10000x + 1 = 0$ Find the roots. Discriminant: $\sqrt{b^2 - 4ac} = \sqrt{99999996} \approx 9999.9998$ .

Direct computation (cancellation occurs):

$$x_1 = \frac{-10000 + 9999.9998}{2} = \frac{-0.0002}{2} = -0.0001$$

Workaround (rationalization):

$$x_1 = \frac{2c}{-b - \sqrt{b^2-4ac}} = \frac{2}{-10000 - 9999.9998} = \frac{2}{-19999.9998} \approx -0.00010000$$

The latter preserves more significant digits.

8. Frequently Asked Questions

Q1. What is catastrophic cancellation?

When two nearly equal numbers are subtracted, leading significant digits cancel, drastically reducing precision. Often avoidable through formula transformation or Taylor expansion.

Q2. What is absorption?

Information loss when adding/subtracting numbers of vastly different magnitudes. Counteracted by Kahan's compensated summation algorithm.

Q3. How to count significant digits?

Count from the first nonzero digit to the last digit.Leading zeros are excluded; trailing and embedded zeros are included.

9. References