You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@xalan.apache.org by bu...@apache.org on 2003/08/21 22:24:01 UTC

DO NOT REPLY [Bug 21534] - Floating point output incorrect

DO NOT REPLY TO THIS EMAIL, BUT PLEASE POST YOUR BUG 
RELATED COMMENTS THROUGH THE WEB INTERFACE AVAILABLE AT
<http://nagoya.apache.org/bugzilla/show_bug.cgi?id=21534>.
ANY REPLY MADE TO THIS MESSAGE WILL NOT BE COLLECTED AND 
INSERTED IN THE BUG DATABASE.

http://nagoya.apache.org/bugzilla/show_bug.cgi?id=21534

Floating point output incorrect





------- Additional Comments From mhoyt@ca.ibm.com  2003-08-21 20:24 -------
After some investigation, there does not seem to be a quick and efficient fix 
to this problem. 

- The brute force approach would be to simply complete the entries in the table 
i.e. add format strings for 1-9 digits precision and 36+ digits of precision 
and continue the process of doing a linear search until the correct level of 
precision is reached.
- A performance improvement on the above approach would be to implement a 
binary search. E.g. start with 10 digits of precision, if 10 is sufficient -try 
5 digits of precision, if not try 122 digits of precision, etc.  This approach 
would still require at least 2 comparisons though.
- Another solution would be to implement David M. Gay's dtoa which is an 
efficient and accurate double to string conversion routine.

- The Java processor does not have these problems because it has a different 
implementation and can rely on the system class library to convert the douuble 
to the shortest unique value.