-
Romain Goyet authored
This patches avoids multiplying with negative powers-of-10 when parsing floating-point values, when those powers-of-10 can be exactly represented as a positive power. When represented as a positive power and used to divide, the resulting float will not have any rounding errors. The issue is that mp_parse_num_decimal will sometimes not give the closest floating representation of the input string. Eg for "0.3", which can't be represented exactly in floating point, mp_parse_num_decimal gives a slightly high (by 1LSB) result. This is because it computes the answer as 3 * 0.1, and since 0.1 also can't be represented exactly, multiplying by 3 multiplies up the rounding error in the 0.1. Computing it as 3 / 10, as now done by the change in this commit, gives an answer which is as close to the true value of "0.3" as possible.
Romain Goyet authoredThis patches avoids multiplying with negative powers-of-10 when parsing floating-point values, when those powers-of-10 can be exactly represented as a positive power. When represented as a positive power and used to divide, the resulting float will not have any rounding errors. The issue is that mp_parse_num_decimal will sometimes not give the closest floating representation of the input string. Eg for "0.3", which can't be represented exactly in floating point, mp_parse_num_decimal gives a slightly high (by 1LSB) result. This is because it computes the answer as 3 * 0.1, and since 0.1 also can't be represented exactly, multiplying by 3 multiplies up the rounding error in the 0.1. Computing it as 3 / 10, as now done by the change in this commit, gives an answer which is as close to the true value of "0.3" as possible.