You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Josh Rosen (JIRA)" <ji...@apache.org> on 2019/06/20 20:48:00 UTC
[jira] [Updated] (SPARK-26038) Decimal
toScalaBigInt/toJavaBigInteger not work for decimals not fitting in long
[ https://issues.apache.org/jira/browse/SPARK-26038?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Josh Rosen updated SPARK-26038:
-------------------------------
Labels: correctness (was: )
We just independently rediscovered this bug. I'm adding the {{correctness}} label because the old behavior gave wrong answer (as opposed to simply crashing). Here's an example illustrating the old, buggy behavior:
{code:java}
val largeBigInt = scala.math.BigInt("1234567890123456789" * 2)
val incorrectLongValue = largeBigInt.longValue()
val defaultEncoder: org.apache.spark.sql.Encoder[scala.math.BigInt] =
org.apache.spark.sql.catalyst.encoders.ExpressionEncoder[BigInt]
val ds = spark.createDataset(Seq(largeBigInt))(defaultEncoder).map(identity(_))(defaultEncoder)
val roundtrippedThroughEncoder = ds.first()
println(roundtrippedThroughEncoder == largeBigInt){code}
In Spark 2.4.0, this outputs:
{code:java}
roundtrippedThroughEncoder: scala.math.BigInt = -3191638190864629483{code}
and the equality comparison is {{false}}.
[~smilegator] and [~juliuszsompolski], how do you feel about backporting this for 2.4.4?
> Decimal toScalaBigInt/toJavaBigInteger not work for decimals not fitting in long
> --------------------------------------------------------------------------------
>
> Key: SPARK-26038
> URL: https://issues.apache.org/jira/browse/SPARK-26038
> Project: Spark
> Issue Type: Bug
> Components: SQL
> Affects Versions: 2.2.0, 2.3.0, 2.4.0
> Reporter: Juliusz Sompolski
> Assignee: Juliusz Sompolski
> Priority: Major
> Labels: correctness
> Fix For: 3.0.0
>
>
> Decimal toScalaBigInt/toJavaBigInteger just called toLong.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org