hn.today

The VHDL standard's float package gets sqrt(2) wrong

github.com15 points8 comments
Screenshot of The VHDL standard's float package gets sqrt(2) wrong

The audit tests IEEE 1076’s standard floating-point package against an exact-arithmetic reference that rounds each result once to the requested format and rounding mode. The package source is the IEEE version, also shipped unchanged with GHDL and NVC; both simulators produced matching results. The title’s example is concrete: in float64, sqrt(2) returns a bit pattern one unit below the correctly rounded value, and sqrt(14) is similarly wrong in float32. The test cases also expose errors in fused multiply-add, format conversion, reciprocal, division, scaling, and integer conversion. Some resize operations return wildly incorrect values or halt simulation.

The problem extends well beyond isolated edge cases. Across targeted tests of float32, the package produces 81,068 incorrect results and 272 simulator stops in about 860,000 calls; float64 produces 86,081 wrong results and 445 stops. Failures occur under all four tested rounding modes, especially for multiply-add, scaling, division, and narrowing conversions; widening conversions and integer-to-float conversions are among the operations that pass these tests. Exhaustive tests of tiny formats also find large error counts, including 243,520 incorrectly rounded multiply-add results among all 2,097,152 triples in a seven-bit format. One identified cause is exponent truncation during normalization: narrowing the exponent can make values such as 2^-512 become 1.0, or turn 2^600 into 2^88 when converting float64 to float32.

Read on github.com8 comments on Hacker News

Summary generated by AI from the linked article. hn.today is not affiliated with Hacker News or Y Combinator.

More in Programming

Aristotle Expands into Rust

Aristotle Expands into Rust

Aristotle has expanded its formal verification platform to Rust, allowing developers to prove code meets specifications and contains no undefined behavior, including in unsafe blocks. It can also verify that Rust rewrites behave like their original C code, targeting critical software such as smart contracts and infrastructure. (harmonic.fun)

The daily digest

Today's best Hacker News stories, summarized and screenshotted, one email a day.