Re: float/double calculation bug ?

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
2 messages Options
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: float/double calculation bug ?

sbglasius
Hi,

This discussion does not belong on the dev mailing list, but the user mailing list ([hidden email]). Please continue your very interesting discussion there :-) Thanks.

/Søren
On Sun, 19 Mar 2017 at 16:47 James Bond <[hidden email]> wrote:
What happens if you use .toDouble() instead of .toFloat, and an explicitly double 100.0 literal?  Right now, your computation is taking an integer, dividing by a float, then multiplying by another float.  I suspect that going double precision won't fix this in all cases (floating point math is, after all, just an approximation), but at least you're processing the values as precisely as you can.

On Sun, Mar 19, 2017 at 8:28 AM, Derek Visch <[hidden email]> wrote:
Looks like floating point to me, what are you expecting?

On Mar 19, 2017 10:04 AM, "Tx. T" <[hidden email]> wrote:
Any idea why the follow code "calc" returns the "70%" of the 330000 incorrectly?
testing on: Groovy Version: 2.4.9 JVM: 1.8.0_112 Vendor: Oracle Corporation OS: Mac OS X
Mac OS X 10.12.3


groovy:000>     def calc = { amount, ttl ->
groovy:001>         double rtn
groovy:002>         if (amount[-1] != '%') rtn = amount.toDouble()
groovy:003>         else rtn = ttl / 100.0 * amount.replaceAll(/%\Z/, '').toFloat()
groovy:004> 
groovy:004>         rtn
groovy:005>     }
===> groovysh_evaluate$_run_closure1@39fcbef6
groovy:000> calc("70%", 330000)
===> 230999.99999999997
groovy:000> calc("10%", 330000)
===> 33000.0
groovy:000> calc("20%", 330000)
===> 66000.0
groovy:000> calc("30%", 330000)
===> 99000.0
groovy:000> calc("40%", 330000)
===> 132000.0
groovy:000> calc("50%", 330000)
===> 165000.0
groovy:000> calc("60%", 330000)
===> 198000.0
groovy:000> calc("70%", 330000)
===> 230999.99999999997
groovy:000> calc("80%", 330000)
===> 264000.0
groovy:000> calc("90%", 330000)
===> 297000.0
groovy:000> calc("100%", 330000)
===> 330000.0

--
Best regards / Med venlig hilsen,
Søren Berg Glasius

Hedevej 1, Gl. Rye, 8680 Ry, Denmark
Mobile: +45 40 44 91 88, Skype: sbglasius
--- Press ESC once to quit - twice to save the changes.
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: float/double calculation bug ?

sbglasius
You are running into Floating point rounding in the JVM, which is not a Groovy bug.

Your code can be salvaged by using BigDecimal:

def calc = { amount, ttl ->
    BigDecimal rtn
    if (amount[-1] != '%') rtn = amount.toBigDecimal()
    else rtn = ttl / 100.0 * amount.replaceAll(/%\Z/, '').toBigDecimal()

    rtn
}


On Sun, 19 Mar 2017 at 17:16 Søren Berg Glasius <[hidden email]> wrote:
Hi,

This discussion does not belong on the dev mailing list, but the user mailing list ([hidden email]). Please continue your very interesting discussion there :-) Thanks.

/Søren

On Sun, 19 Mar 2017 at 16:47 James Bond <[hidden email]> wrote:
What happens if you use .toDouble() instead of .toFloat, and an explicitly double 100.0 literal?  Right now, your computation is taking an integer, dividing by a float, then multiplying by another float.  I suspect that going double precision won't fix this in all cases (floating point math is, after all, just an approximation), but at least you're processing the values as precisely as you can.

On Sun, Mar 19, 2017 at 8:28 AM, Derek Visch <[hidden email]> wrote:
Looks like floating point to me, what are you expecting?

On Mar 19, 2017 10:04 AM, "Tx. T" <[hidden email]> wrote:
Any idea why the follow code "calc" returns the "70%" of the 330000 incorrectly?
testing on: Groovy Version: 2.4.9 JVM: 1.8.0_112 Vendor: Oracle Corporation OS: Mac OS X
Mac OS X 10.12.3


groovy:000>     def calc = { amount, ttl ->
groovy:001>         double rtn
groovy:002>         if (amount[-1] != '%') rtn = amount.toDouble()
groovy:003>         else rtn = ttl / 100.0 * amount.replaceAll(/%\Z/, '').toFloat()
groovy:004> 
groovy:004>         rtn
groovy:005>     }
===> groovysh_evaluate$_run_closure1@39fcbef6
groovy:000> calc("70%", 330000)
===> 230999.99999999997
groovy:000> calc("10%", 330000)
===> 33000.0
groovy:000> calc("20%", 330000)
===> 66000.0
groovy:000> calc("30%", 330000)
===> 99000.0
groovy:000> calc("40%", 330000)
===> 132000.0
groovy:000> calc("50%", 330000)
===> 165000.0
groovy:000> calc("60%", 330000)
===> 198000.0
groovy:000> calc("70%", 330000)
===> 230999.99999999997
groovy:000> calc("80%", 330000)
===> 264000.0
groovy:000> calc("90%", 330000)
===> 297000.0
groovy:000> calc("100%", 330000)
===> 330000.0

--
Best regards / Med venlig hilsen,
Søren Berg Glasius

Hedevej 1, Gl. Rye, 8680 Ry, Denmark
Mobile: <a href="tel:40%2044%2091%2088" value="+4540449188" class="gmail_msg" target="_blank">+45 40 44 91 88, Skype: sbglasius
--- Press ESC once to quit - twice to save the changes.
--
Best regards / Med venlig hilsen,
Søren Berg Glasius

Hedevej 1, Gl. Rye, 8680 Ry, Denmark
Mobile: <a href="tel:40%2044%2091%2088" value="+4540449188" class="gmail_msg" target="_blank">+45 40 44 91 88, Skype: sbglasius
--- Press ESC once to quit - twice to save the changes.
Loading...