rounding inconsistencies

["Bartley, Chris - NASHCCON" <Ch...@acs-inc.com>]

Hi,

I'm noticing some (rather frustrating) inconsistencies with how
<fmt:formatNumber> performs rounding of decimals.  I've attached a couple
files that illustrate the problem, but, in a nutshell, these three tags:

   <fmt:formatNumber type="currency" value="0.375"/>
   <fmt:formatNumber type="currency" value="5.565"/>
   <fmt:formatNumber type="currency" value="5.5651"/>

result in:

   $0.38
   $5.56
   $5.57

The first and third are what I expect, but the second value ($5.56) is
really throwing me.  Why isn't it $5.57?  Am I doing something wrong or is
there some other sorta magic I can use to get the rounding to work properly?

I'm using Standard Taglib version 1.0.4, JDK 1.3.1_07, Weblogic 7, and Win2k
sp4.

thanks heaps!

Chris

Re: rounding inconsistencies

[Jason Lea <ja...@kumachan.net.nz>]

This may or may not be the answer:

 From the JavaDocs for *java.text.DecimalFormat* which I assume it would 
use, it says it uses |"DecimalFormat| uses half-even rounding (see 
|ROUND_HALF_EVEN| <ci...@kumachan.net.nz>) for 
formatting."

ROUND_HALF_EVEN: "Rounding mode to round towards the "nearest neighbor" 
unless both neighbors are equidistant, in which case, round towards the 
even neighbor. Behaves as for ROUND_HALF_UP if the digit to the left of 
the discarded fraction is odd; behaves as for ROUND_HALF_DOWN if it's 
even. Note that this is the rounding mode that minimizes cumulative 
error when applied repeatedly over a sequence of calculations."

First line,  0.375, number to left of 5 is 7 = odd, so it rounds up to 0.38
Second line, 5.565, number to left of 5 is 6 = even, so it rounds down
Third line, 5.5651, number to left of 1 is 5 = odd, rounds up to 6, 
giving 5.566, then uses 'nearest neighbour' rounding on the last 6 to 
give 5.57

I don't know if there is any way to use a different rounding technique 
using the <fmt:formatNumber> tag.

Bartley, Chris - NASHCCON wrote:

>Hi,
>
>I'm noticing some (rather frustrating) inconsistencies with how
><fmt:formatNumber> performs rounding of decimals.  I've attached a couple
>files that illustrate the problem, but, in a nutshell, these three tags:
>
>   <fmt:formatNumber type="currency" value="0.375"/>
>   <fmt:formatNumber type="currency" value="5.565"/>
>   <fmt:formatNumber type="currency" value="5.5651"/>
>
>result in:
>
>   $0.38
>   $5.56
>   $5.57
>
>The first and third are what I expect, but the second value ($5.56) is
>really throwing me.  Why isn't it $5.57?  Am I doing something wrong or is
>there some other sorta magic I can use to get the rounding to work properly?
>
>I'm using Standard Taglib version 1.0.4, JDK 1.3.1_07, Weblogic 7, and Win2k
>sp4.
>
>thanks heaps!
>
>Chris
>
>
>  
>
>------------------------------------------------------------------------
>
>---------------------------------------------------------------------
>To unsubscribe, e-mail: taglibs-user-unsubscribe@jakarta.apache.org
>For additional commands, e-mail: taglibs-user-help@jakarta.apache.org
>


-- 
Jason Lea



---------------------------------------------------------------------
To unsubscribe, e-mail: taglibs-user-unsubscribe@jakarta.apache.org
For additional commands, e-mail: taglibs-user-help@jakarta.apache.org