Skip to content
Snippets Groups Projects
  • Romain Goyet's avatar
    b768cc6c
    py/parsenum: Avoid rounding errors with negative powers-of-10. · b768cc6c
    Romain Goyet authored
    This patches avoids multiplying with negative powers-of-10 when parsing
    floating-point values, when those powers-of-10 can be exactly represented
    as a positive power.  When represented as a positive power and used to
    divide, the resulting float will not have any rounding errors.
    
    The issue is that mp_parse_num_decimal will sometimes not give the closest
    floating representation of the input string.  Eg for "0.3", which can't be
    represented exactly in floating point, mp_parse_num_decimal gives a
    slightly high (by 1LSB) result.  This is because it computes the answer as
    3 * 0.1, and since 0.1 also can't be represented exactly, multiplying by 3
    multiplies up the rounding error in the 0.1.  Computing it as 3 / 10, as
    now done by the change in this commit, gives an answer which is as close to
    the true value of "0.3" as possible.
    b768cc6c
    History
    py/parsenum: Avoid rounding errors with negative powers-of-10.
    Romain Goyet authored
    This patches avoids multiplying with negative powers-of-10 when parsing
    floating-point values, when those powers-of-10 can be exactly represented
    as a positive power.  When represented as a positive power and used to
    divide, the resulting float will not have any rounding errors.
    
    The issue is that mp_parse_num_decimal will sometimes not give the closest
    floating representation of the input string.  Eg for "0.3", which can't be
    represented exactly in floating point, mp_parse_num_decimal gives a
    slightly high (by 1LSB) result.  This is because it computes the answer as
    3 * 0.1, and since 0.1 also can't be represented exactly, multiplying by 3
    multiplies up the rounding error in the 0.1.  Computing it as 3 / 10, as
    now done by the change in this commit, gives an answer which is as close to
    the true value of "0.3" as possible.