[jira] [Commented] (SPARK-25454) Division between operands with negative scale can cause precision loss

["Bruce Robbins (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/SPARK-25454?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16619911#comment-16619911 ] 

Bruce Robbins commented on SPARK-25454:
---------------------------------------

Thanks [~mgaido],

OK, so the way I understand it
- negative scales are the problem
- they were also a problem in 2.2, but it was more difficult to reproduce. For my example case (in the referenced Jira), the change to the promotion of literal 1000 in 2.3 exposed an existing issue with the handling of 1e6.

This bears out, in that when I replace 1e6 with 1000000, the issue goes away (at least for my example case):
{noformat}
scala> sql("select 26393499451/(1000000 * 1000) as c1").show
+------------+
|          c1|
+------------+
|26.393499451|
+------------+
{noformat}


> Division between operands with negative scale can cause precision loss
> ----------------------------------------------------------------------
>
>                 Key: SPARK-25454
>                 URL: https://issues.apache.org/jira/browse/SPARK-25454
>             Project: Spark
>          Issue Type: Bug
>          Components: SQL
>    Affects Versions: 2.3.0, 2.3.1
>            Reporter: Marco Gaido
>            Priority: Major
>
> The issue was originally reported by [~bersprockets] here: https://issues.apache.org/jira/browse/SPARK-22036?focusedCommentId=16618104&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-16618104.
> The problem consist in a precision loss when the second operand of the division is a decimal with a negative scale. It was present also before 2.3 but it was harder to reproduce: you had to do something like {{lit(BigDecimal(100e6))}}, while now this can happen more frequently with SQL constants.
> The problem is that our logic is taken from Hive and SQLServer where decimals with negative scales are not allowed. We might also consider enforcing this too in 3.0 eventually. Meanwhile we can fix the logic for computing the result type for a division.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org