DO NOT REPLY [Bug 43940] [PATCH] Faster method for double formatting

[bu...@apache.org]

https://issues.apache.org/bugzilla/show_bug.cgi?id=43940

--- Comment #13 from Vincent Hennebert <vh...@gmail.com> 2011-12-20 15:00:14 UTC ---
While looking at the patch something struck me as odd in the requirements.
Taking the same example of decimals being 4 and precision 8, 1.00001 will be
rounded to 1, while 0.99999999 will be kept as is. Yet the latter is much
closer to 1 than the former (1000 times, to be precise). So if rounding had to
be done, that would rather have to be the other way round, that is: keep
1.00001 as is and round 0.99999999 up to 1.

It seems to me that if drawing commands are used with numbers around 1, then
the result is likely to look sub-optimal. Especially since transformation
matrices can potentially scale up rounding errors quite a bit.

At the other end of the spectrum, 12345678901.2345 will be output as is (15
significant digits), while 1.23456789012345 will be rounded to 1.2346 (5
significant digits). What justifies to have 15 significant digits on one hand,
and only 5 on the other hand?

I understand that this exercise has to do with outputting real numbers to PDF
or PostScript (or other formats). According to the PDF Reference, Acrobat uses
single-precision floating-point numbers to represent reals (the 'float' type in
Java). Given that a float has approximately 8 significant digits —which is
actually what the PostScript Reference puts in its "Implementation Limits"
section— I suggest that we use this as the sole parameter.

Now, would we really want this parameter to be configurable? Using a smaller
number of significant digits probably makes no sense; using a higher number may
allow to achieve higher precision with those applications (viewers or printers)
that go beyond Acrobat and use doubles to represent real numbers. Such
applications may actually become more and more common with the rise of 64-bit
processors. Then again, will that increased precision make a significant
difference in the context of drawing graphics?

Any thoughts on this are welcome. As a side note, if we choose to fix the
precision to 8 then the implementation becomes almost trivial: just convert the
double into a float and 'de-scientificize' the output of toString.

Thanks,
Vincent

-- 
Configure bugmail: https://issues.apache.org/bugzilla/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the assignee for the bug.