[jira] [Commented] (CASSANDRA-9224) Figure out a better default float precision rule for cqlsh

["Stefania (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/CASSANDRA-9224?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14545142#comment-14545142 ] 

Stefania commented on CASSANDRA-9224:
-------------------------------------

The best I could figure out so far is to increase the precision to take into account the digits on the left of the decimal separator when '%g' won't use scientific notation. To work this out we can follow the rule described in the documentation (https://docs.python.org/2/library/string.html#format-specification-mini-language): if {{-4 <= exponent < precision}}  then %g will not not use scientific notation. If this is true, it will still be true with a bigger precision. So when {{-4 <= exponent < precision}} is true, we add to the precision the number of digits to the left (exponent + 1), therefore ensuring we always have precision digits after the decimal separator.


> Figure out a better default float precision rule for cqlsh
> ----------------------------------------------------------
>
>                 Key: CASSANDRA-9224
>                 URL: https://issues.apache.org/jira/browse/CASSANDRA-9224
>             Project: Cassandra
>          Issue Type: Improvement
>          Components: Tools
>            Reporter: Tyler Hobbs
>            Assignee: Stefania
>              Labels: cqlsh
>             Fix For: 3.x, 2.1.x
>
>
> We currently use a {{DEFAULT_FLOAT_PRECISION}} of 5 in cqlsh with formatting {{'%.*g' % (float_precision, val)}}.  In practice, this is way too low.  For example, 12345.5 will show up as 123456.  Since the float precision is used for cqlsh's COPY TO, it's particularly important that we maintain as much precision as is practical by default.
> There are some other tricky considerations, though.  If the precision is too high, python will do something like this:
> {noformat}
> > '%.25g' % (12345.5555555555555555,)
> '12345.55555555555474711582'
> {noformat}
> That's not terrible, but it would be nice to avoid if we can.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)