[jira] [Updated] (SPARK-27283) BigDecimal arithmetic losing precision

["Mats (JIRA)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/SPARK-27283?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Mats updated SPARK-27283:
-------------------------
    Issue Type: Bug  (was: Question)

> BigDecimal arithmetic losing precision
> --------------------------------------
>
>                 Key: SPARK-27283
>                 URL: https://issues.apache.org/jira/browse/SPARK-27283
>             Project: Spark
>          Issue Type: Bug
>          Components: SQL
>    Affects Versions: 2.4.0
>            Reporter: Mats
>            Priority: Minor
>              Labels: decimal, float, sql
>
> When performing arithmetics between doubles and decimals, the resulting value is always a double. This is very strange to me; when an exact type is present as one of the inputs, I would expect that the inexact type is lifted and the result presented exactly, rather than lowering the exact type to the inexact and presenting a result that may contain rounding errors. The choice to use a decimal was probably taken because rounding errors were deemed an issue.
> When performing arithmetics between decimals and integers, the expected behaviour is seen; the result is a decimal.
> See the following example:
> {code:java}
> import org.apache.spark.sql.functions
> val df = sparkSession.createDataFrame(Seq(Tuple1(0L))).toDF("a")
> val decimalInt = df.select(functions.lit(BigDecimal(3.14)) + functions.lit(1) as "d")
> val decimalDouble = df.select(functions.lit(BigDecimal(3.14)) + functions.lit(1.0) as "d")
> decimalInt.schema.printTreeString()
> decimalInt.show()
> decimalDouble.schema.printTreeString()
> decimalDouble.show(){code}
> which produces this output (with possible variation on the rounding error):
> {code:java}
> root
> |-- d: decimal(4,2) (nullable = true)
> +----+
> | d  |
> +----+
> |4.14|
> +----+
> root
> |-- d: double (nullable = false)
> +-----------------+
> | d               |
> +-----------------+
> |4.140000000000001|
> +-----------------+
> {code}
>  
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org