Numerical Stability of Quadratic-Equation Solvers: Avoiding Cancellation and Floating-Point Overflow

By Dr Sam 6 sources cited

The analytically exact roots of the quadratic equation ax2+bx+c = 0 are universally given by the standard algebraic formula:

x = −b ± b2 − 4ac2a

While mathematically correct in the real number field, translating this expression natively into IEEE-754 floating-point architectures exposes critical algorithmic instabilities. When engineering robust computational tools—such as evaluating arbitrary user inputs within web-based calculator portals—a naive implementation requires careful restructuring to avoid precision degradation, exponent overflow, and catastrophic cancellation.

Conditioning vs. Algorithmic Stability

A fundamental tenet of numerical analysis separates the conditioning of a mathematical problem from the stability of the computational algorithm. The conditioning of the quadratic root-finding problem depends on the coefficients and the separation and scale of the roots [2]. Even when a particular root is reasonably well-conditioned, a straightforward evaluation of the textbook formula can suffer severe numerical error because of algorithmic cancellation [5].

The Mechanism of Catastrophic Cancellation

Severe accuracy loss can occur when b2 is much larger than 4ac and one of the quadratic-formula numerators requires subtraction of nearly equal quantities. In that situation, Δ can be extremely close to |b|, making the affected numerator highly sensitive to rounding errors [1, 2].

If b > 0, calculating the positive-sign numerator −b+Δ requires the hardware arithmetic logic unit to subtract two nearly identical quantities. Although the subtraction itself is correctly rounded, the difference may contain far fewer reliable significant digits than either operand [1]. Consequently, even small rounding errors already present in the operands can become large relative errors in the resulting numerator [2].

The Stable q-Formulation

A practical way to avoid the vulnerable subtraction in one of the two textbook numerators is to choose the sign of the square-root term according to b [2, 6]. The construction chooses the square-root sign so that the two terms being combined have the same sign, avoiding the near-equal subtraction responsible for loss of significant digits. This produces the auxiliary quantity q, from which one root is computed directly and the other is recovered from the product of the roots:

q = −12(b+copysign(Δ,b))

Assuming real roots (Δ ≥ 0), this computes one root without the cancellation present in the corresponding textbook numerator. The second root is then extracted using Vieta's product property (x1x2 = c/a), which requires only one additional division and avoids the cancellation inherent in the alternative numerator [6]:

x1 = qa, x2 = cq

It is critical to note that while this formulation successfully addresses subtractive cancellation in the root numerator, it does not by itself solve every overflow, underflow, or discriminant-accuracy problem inherent in the broader quadratic evaluation.

Advanced Discriminant Evaluation and FMA

While the q-formulation resolves numerator cancellation, the evaluation of the discriminant Δ = b2 − 4ac itself remains highly vulnerable.

When b2 and 4ac are close in magnitude, their subtraction can suffer severe cancellation, potentially leaving the computed discriminant with substantially fewer accurate significant digits than either operand. Modern architectures provide a hardware-level Fused Multiply-Add (FMA) instruction [3]. An FMA can improve the evaluation of the discriminant because it computes a product and the subsequent addition/subtraction with a single final rounding. For example, fma(b,b,−4ac) avoids a separate rounding of the b2 product, although the separately computed 4ac term can still contribute error. More accurate algorithms therefore use compensated product calculations and, where appropriate, multiple FMA operations [4].

Exponent Scaling for Overflow and Underflow

Floating-point limits present a secondary failure mode. In Float32, squaring sufficiently large values of b can overflow even when the corresponding quadratic roots are representable. Since the largest finite Float32 value is approximately 3.4×1038, values of b on the order of 1019 or larger can make the intermediate computation of b2 overflow to infinity (Inf), potentially causing subsequent discriminant or root computations to produce non-finite results such as Inf or NaN. Conversely, extremely small coefficients can cause intermediate products to underflow to zero.

Scaling the coefficients, preferably by powers of two, can substantially reduce the risk of intermediate overflow and underflow while preserving the roots [6]. A production implementation may require exponent-aware scaling rather than a simple division by the maximum coefficient magnitude.

Numerical Demonstration

Consider the ill-proportioned equation x2+108x+1 = 0 evaluated natively in standard 32-bit floating-point arithmetic (Float32, ϵmach ≈ 1.19×10−7). The exact analytical roots are approximately −108 and −10−8.

Table: Illustrative binary32 results

Evaluation MethodComputed x1​ (Large)Computed x2​ (Small)Relative Error (x2​)
Standard Formula−1080.0100%
Stable q-Formula−108−10−8≈ 6.1×10−9

A straightforward IEEE-754 binary32 implementation can return 0.0 for the smaller root because the discriminant calculation and subsequent square root round in such a way that the computed square root is 108. The numerator −108+108 then evaluates to zero, causing the smaller root to be returned as zero by the naïve formula. The q-formula avoids the cancellation mechanism responsible for this loss of accuracy and, when the discriminant and intermediate operations are evaluated reliably, can preserve the small root to substantially higher relative accuracy.

Edge Cases and Implementation Limits

A robust solver must wrap the core q-formulation in defensive guards to handle special computational states:

Scholarly Literature and References

  1. Goldberg, D. (1991). What every computer scientist should know about floating-point arithmetic. ACM Computing Surveys, 23(1), 5–48.
  2. Higham, N. J. (2002). Accuracy and Stability of Numerical Algorithms (2nd ed.). Society for Industrial and Applied Mathematics (SIAM).
  3. Muller, J.-M., Brunie, N., de Dinechin, F., Jeannerod, C.-P., Joldes, M., Lefèvre, H., Melquiond, G., Revol, N., & Torres, S. (2018). Handbook of Floating-Point Arithmetic (2nd ed.). Birkhäuser.
  4. Kahan, W. M. (2004). On the Cost of Floating-Point Computation Without Extra-Precise Arithmetic. Technical note, University of California, Berkeley.
  5. Forsythe, G. E. (1970). Pitfalls in Computation, or why a Math Book isn't enough. The American Mathematical Monthly, 77(9), 931-956.
  6. Corless, R. M., & Fillion, N. (2013). A Graduate Introduction to Numerical Methods. Springer.

← All research articles How we build & check these tools