You are viewing a plain text version of this content. The canonical link for it is here.
Posted to c-dev@axis.apache.org by "Fred Preston (JIRA)" <ax...@ws.apache.org> on 2005/11/23 13:45:36 UTC
[jira] Commented: (AXISCPP-838) The serialisation of 555.555 as a float produces a value of 555.554993 on the wire.
[ http://issues.apache.org/jira/browse/AXISCPP-838?page=comments#action_12358351 ]
Fred Preston commented on AXISCPP-838:
--------------------------------------
Precision problems with floats
------------------------------------------
Serialisation
-----------------
Before trying to resolve anything, I though I would do some investigation into floats... I first created lots of float values around the 555.555 problem area to see how this number was being represented...
Code "%f" Produced "%.7g" Produced "%.8g" Produced
xsd__float myFloat00 = 555.5 + 0.0001; 555.500122 555.5001 555.50012
xsd__float myFloat01 = 555.55 + 0.0001; 555.550110 555.5501 555.55011
xsd__float myFloat02 = 555.555; 555.554993 555.555 555.55499
xsd__float myFloat020 = 555.555 + 0.1; 555.655029 555.655 555.65503
xsd__float myFloat021 = 555.555 + 0.01; 555.565002 555.565 555.565
xsd__float myFloat022 = 555.555 + 0.001; 555.556030 555.556 555.55603
xsd__float myFloat023 = 555.555 + 0.0001; 555.555115 555.5551 555.55511
xsd__float myFloat024 = 555.555 + 0.00001; 555.554993 555.555 555.55499
xsd__float myFloat03 = 555.5555 + 0.0001; 555.555603 555.5556 555.5556
xsd__float myFloat04 = 555.55555 + 0.0001; 555.555664 555.5557 555.55566
xsd__float myFloat05 = 555.556 + 0.0001; 555.556091 555.5561 555.55609
xsd__float myFloat06 = -555.5 - 0.0001; -555.500122 -555.5001 -555.50012
xsd__float myFloat07 = -555.55 - 0.0001; -555.550110 -555.5501 -555.55011
xsd__float myFloat08 = -555.555 - 0.0001; -555.555115 -555.5551 -555.55511
xsd__float myFloat09 = -555.5555 - 0.0001; -555.555603 -555.5556 -555.5556
xsd__float myFloat10 = -555.55555 - 0.0001; -555.555664 -555.5557 -555.55566
xsd__float myFloat11 = -555.556 - 0.0001; -555.556091 -555.5561 -555.55609
xsd__float myFloat12 = 555.5; 555.500000 555.5 555.5
xsd__float myFloat13 = 555.55; (5) 555.55
(6) 555.55
(7) 555.55
(8) 555.54999
xsd__float myFloat14 = 555.555; (6) 555.555
(7) 555.555
(8) 555.55499
xsd__float myFloat15 = 555.5555; (7) 555.5555
xsd__float myFloat16 = 555.55499; (6) 555.555 <- This has rounded up rather than truncated.
(7) 555.555 <- This has rounded up rather than truncated.
(8) 555.55499
xsd__float myFloat17 = 555.5499; (6) 555.55 <- This has rounded up rather than truncated.
(7) 555.5499
(8) 555.54993
The best guaranteed precision without any rounding appears to be 5 digits and with rounding 6 (although I would need to confirm this).
Reading through the IEEE numeric standards, types have the following precision:-
float = sign bit, 8 bit exponent, 23 bit mantissa
double = sign bit, 11 bit exponent, 52 bit mantissa
long double = sign bit, 15 bit exponent, 64 bit mantissa
For float this would imply a precision of 1/2^22 (or 0.00000011920928955078125) or a maximum of 6 significant digits. This backs up what I've seen experimentally. I have also seen that 555.55499 (myFloat16) was rounded up by the precision specification "%.6g" to 555.555 and not truncated to 555.554.
Bits Decimal
00 0.5
01 0.25
02 0.125
03 0.0625
04 0.03125
05 0.015625
06 0.0078125
07 0.00390625
08 0.001953125
09 0.0009765625
10 0.00048828125
11 0.000244140625
12 0.0001220703125
13 0.00006103515625
14 0.000030517578125
15 0.0000152587890625
16 0.00000762939453125
17 0.000003814697265625
18 0.0000019073486328125
19 0.00000095367431640625
20 0.000000476837158203125
21 0.0000002384185791015625
22 0.00000011920928955078125
My recommendation is that for serialisation of a float (this may also be necessary for double and decimal), we use the following logic...
char * FloatToThreeDecimalPlaces( xsd__float myFloat)
{
int sign = 1;
if( myFloat < 0)
{
sign = -1;
myFloat = -myFloat;
}
int whole = (int) myFloat;
int fraction = (int) ((myFloat - (xsd__float) whole + (xsd__float) 0.00005) * (xsd__float) 1000.0);
static char szNumber[16];
sprintf( szNumber, "%d.%03d", whole * sign, fraction);
char * pZero = szNumber + strlen( szNumber) - 1;
while( *pZero != '.' && *pZero == '0')
{
*pZero = '\0';
pZero--;
}
if( *pZero == '.')
{
*pZero = '\0';
}
return szNumber;
}
Deserialisation
---------------------
We are using 'strtod' which appears to be working satisfactorily. It should be noted though that there will be some problems converting the output (which is double) to a float!
> The serialisation of 555.555 as a float produces a value of 555.554993 on the wire.
> -----------------------------------------------------------------------------------
>
> Key: AXISCPP-838
> URL: http://issues.apache.org/jira/browse/AXISCPP-838
> Project: Axis-C++
> Type: Bug
> Components: Serialization
> Environment: n/a
> Reporter: Fred Preston
> Assignee: Fred Preston
>
> Using the UnitTest_XSD_float test. If you change the code as follows:-
> Was
> [40]// Test non-nillable element
> [41] xsd__float result = ws->asNonNillableElement((xsd__float)35.353588);
> Now
> [40]// Test non-nillable element
> [41] xsd__float result = ws->asNonNillableElement((xsd__float)555.555);
> Then on the wire you get:-
> <nonNillableElement>555.554993</nonNillableElement>
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see:
http://www.atlassian.com/software/jira