[jira] [Created] (SPARK-27283) BigDecimal arithmetic losing precision

["Mats (JIRA)" <ji...@apache.org>]

Mats created SPARK-27283:
----------------------------

             Summary: BigDecimal arithmetic losing precision
                 Key: SPARK-27283
                 URL: https://issues.apache.org/jira/browse/SPARK-27283
             Project: Spark
          Issue Type: Question
          Components: SQL
    Affects Versions: 2.4.0
            Reporter: Mats


When performing arithmetics between doubles and decimals, the resulting value is always a double. This is very strange to me; when an exact type is present as one of the inputs, I would expect that the inexact type is lifted and the result presented exactly, rather than lowering the exact type to the inexact and presenting a result that may contain rounding errors. The choice to use a decimal was probably taken because rounding errors were deemed an issue.

When performing arithmetics between decimals and integers, the expected behaviour is seen; the result is a decimal.

See the following example:
{code:java}
import org.apache.spark.sql.functions
val df = sparkSession.createDataFrame(Seq(Tuple1(0L))).toDF("a")

val decimalInt = df.select(functions.lit(BigDecimal(3.14)) + functions.lit(1) as "d")
val decimalDouble = df.select(functions.lit(BigDecimal(3.14)) + functions.lit(1.0) as "d")

decimalInt.schema.printTreeString()
decimalInt.show()
decimalDouble.schema.printTreeString()
decimalDouble.show(){code}
which produces this output (with possible variation on the rounding error):


{code:java}
root
|-- d: decimal(4,2) (nullable = true)

+----+
| d  |
+----+
|4.14|
+----+

root
|-- d: double (nullable = false)

+-----------------+
| d               |
+-----------------+
|4.140000000000001|
+-----------------+
{code}
 

 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org