Newton's Method: The Art of Clever Guessing

We want to find a solution of the equation $f(x) = 0$. But no matter how hard we stare at it, there seems to be no clean formula that cracks it. In such moments, one last resort still remains to us — guessing.

We start with a rough guess: “it’s probably somewhere around here.” If we miss, we replace it with a slightly better one. Repeat this, and sooner or later we close in on the solution. The root-finding methods of numerical computation all boil down, when you get right to it, to this “repeated guessing.” The question is how to re-guess cleverly. The pinnacle of that cleverness is Newton’s method. In this article, let us take a peek at the “spirit” of Newton’s method.

Letting the Tangent Line Fool You on Purpose

The idea behind Newton’s method is boldly simple. Regard the curving curve as a straight line, but only in the immediate vicinity of where you now stand. That is all.

At the current guess $x_n$, draw the line tangent to the curve $y = f(x)$. Hunting for a root on the curve itself is hard, but for a straight line the point where it crosses the x-axis is easy to compute. We adopt that intersection as the “next, better guess” $x_{n+1}$. Written as a formula, $$ x_{n+1} = x_n - \frac{f(x_n)}{f'(x_n)} $$ where $f'(x_n)$ is the slope of the tangent. It looks daunting, but what it does is “ride the tangent line like a slide and coast all the way down to the x-axis” — nothing more.

We deliberately talk ourselves into believing the curve is straight — we go and get fooled by the tangent line on purpose. But near the solution the curve really is almost perfectly straight, so this way of being fooled grows more and more on-target with every re-guess.

Why Is It So Blazingly Fast?

The true forte of Newton’s method is its speed. The reliable but plodding bisection method halves, at each step, the interval known to contain the solution. Half, then half again — admirable in its own right, but the number of correct digits grows by only about $0.3$ digits per step. It takes three or four steps to make one decimal digit more accurate.

Newton’s method grows its digits in a wholly different way. Once you draw near the solution, the number of correct digits roughly doubles with each step. Two matching digits become four next, then eight, then sixteen…. The limit of double precision (about $16$ digits) is blown clean past in a few steps. This explosive way of closing in is called quadratic convergence.

Even for the same “repeated guessing,” merely adding the one extra touch of a tangent line turns a plodding walk into rocket-like acceleration. Line up the three — Newton’s method, bisection, and the derivative-free secant method — and the “trade-off between speed and certainty” in the world of root-finding comes into clear view.

Speed Comes at a Price

If it is this fast, surely we no longer need bisection — so one is tempted to say. But in this world, tempting bargains have a catch. Newton’s method, in exchange for its speed, is hit or miss.

For instance, when the first guess lies far from the solution, the tangent line may point in some outrageous direction and carry you even further from the root. Or when the tangent strikes an almost-horizontal, flat spot, the slide stretches out nearly sideways and flings you off into the far distance (recall that the denominator $f'(x_n)$ is approaching zero). With bad luck, the iterates can bounce back and forth between two points forever, never coming to rest.

On top of that, Newton’s method needs the slope of the tangent — the derivative $f'$ — at every step. Where the derivative is not at hand, or is heavy to compute, that alone makes it awkward to use. The bisection method that reliably corners a solution; the fast but temperamental Newton’s method — this is precisely why, in the practical world, hybrid methods that take the best of both are favored. Speed and certainty, more often than not, do not get along.

The Grand Idea of “Local Linearity”

At the core of Newton’s method lies the notion that even something curved looks straight when seen from very close up. This is no mere trick of numerical analysis. The very concept of differentiation stands on this way of thinking. Replacing a complicated function, place by place, with a straight line (a first-degree expression) — this “local linearization” shows its face everywhere in applied mathematics, from optimization to machine learning to the control of robots.

That the plain act of drawing a single tangent line should connect to such a broad and deep idea. Newton’s method is the most panoramic entrance from which to meet that idea for the first time.

In Closing — Miss Cleverly, Hit Cleverly

Guessing, too, can grow this keen depending on how it is done. Let yourself be fooled by the tangent on purpose, and from the value you missed, fashion the next, better one. Newton’s method is a technique for polishing a guess in an organized way, and behind its speed lies one clear intuition: “even the curved is straight up close.”

Savor this method, precariousness and speed alike, all the way down to its nature, and the map of root-finding rises suddenly into three dimensions. Next, let us move on to the intermediate level and watch how the art of “clever hitting” blossoms on the larger stage of matrices and function approximation.