Comment 14 for bug 1261048

Revision history for this message
In , Bjbraams-2 (bjbraams-2) wrote :

<<Please don't reopen this bug...>> I would not dream of it. Let me only record a possible misunderstanding. My report is not about that Excel "optimization" by which near-cancellations are replaced by exact cancellations; I mentioned that optimization only peripherally because it affects the diagnostics. Instead, my report concerns the arithmetic of the Excel and LibreOffice Calc rounding functions, by which "round to integer" of an exactly representable (IEEE-754) integer may return a different integer. In a way it is the opposite of the mentioned optimization. It replaces exact equality (which would be the correct mathematical result) by only approximate equality. It is indeed a tiny error, only about 6 bits in the mantissa in the worst case. It is just barely large enough to exceed the 5 bits of error that would be masked by that Excel optimization.