You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Sandeep Katta (Jira)" <ji...@apache.org> on 2020/09/03 17:46:00 UTC
[jira] [Comment Edited] (SPARK-32764) compare of -0.0 < 0.0 return
true
[ https://issues.apache.org/jira/browse/SPARK-32764?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17190331#comment-17190331 ]
Sandeep Katta edited comment on SPARK-32764 at 9/3/20, 5:45 PM:
----------------------------------------------------------------
[~cloud_fan] and [~smilegator]
I did some analysis on this, it is because as a part of SPARK-30009 org.apache.spark.util.Utils.nanSafeCompareDoubles is replaced with java.lang.Double.compare. Same can be confirmed from the codegen code, I have attached those for referrence.
If you see the implementation of java.lang.Double.compare
*java.lang.Double.compare(-0.0, 0.0) < 0 evaluates to true*
*java.lang.Double.compare(0.0, -0.0) < 0 evaluates to false*
was (Author: sandeep.katta2007):
[~cloud_fan] and [~smilegator]
I did some analysis on this, it is because as a part of SPARK-30009 org.apache.spark.util.Utils.nanSafeCompareDoubles is replaced with java.lang.Double.compare. Same can be confirmed from the codegen code
> compare of -0.0 < 0.0 return true
> ---------------------------------
>
> Key: SPARK-32764
> URL: https://issues.apache.org/jira/browse/SPARK-32764
> Project: Spark
> Issue Type: Bug
> Components: SQL
> Affects Versions: 3.0.0, 3.0.1
> Reporter: Izek Greenfield
> Priority: Major
> Labels: correctness
>
> {code:scala}
> val spark: SparkSession = SparkSession
> .builder()
> .master("local")
> .appName("SparkByExamples.com")
> .getOrCreate()
> spark.sparkContext.setLogLevel("ERROR")
> import spark.sqlContext.implicits._
> val df = Seq((-0.0, 0.0)).toDF("neg", "pos")
> .withColumn("comp", col("neg") < col("pos"))
> df.show(false)
> ======
> +----+---+----+
> |neg |pos|comp|
> +----+---+----+
> |-0.0|0.0|true|
> +----+---+----+{code}
> I think that result should be false.
> **Apache Spark 2.4.6 RESULT**
> {code}
> scala> spark.version
> res0: String = 2.4.6
> scala> Seq((-0.0, 0.0)).toDF("neg", "pos").withColumn("comp", col("neg") < col("pos")).show
> +----+---+-----+
> | neg|pos| comp|
> +----+---+-----+
> |-0.0|0.0|false|
> +----+---+-----+
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org