[jira] [Created] (SPARK-39845) 0.0 and -0.0 are not consistent in set operations

["Navin Kumar (Jira)" <ji...@apache.org>]

Navin Kumar created SPARK-39845:
-----------------------------------

             Summary: 0.0 and -0.0 are not consistent in set operations 
                 Key: SPARK-39845
                 URL: https://issues.apache.org/jira/browse/SPARK-39845
             Project: Spark
          Issue Type: Bug
          Components: SQL
    Affects Versions: 3.2.1
            Reporter: Navin Kumar


This is a continuation of the issue described in SPARK-32110.

When using Array set-based functions {{array_union}}, {{array_intersect}}, {{array_except}} and {{arrays_overlap}}, {{0.0}} and {{-0.0}} have inconsistent behavior.

When parsed, {{-0.0}} is normalized to {{0.0}}. Therefore if I use {{array_union}} for example with these values directly, {{array(-0.0)}} becomes {{array(0.0)}}. See the example below using {{array_union}}:

{code:java}
scala> val df = spark.sql("SELECT array_union(array(0.0), array(-0.0))")
df: org.apache.spark.sql.DataFrame = [array_union(array(0.0), array(0.0)): array<decimal(1,1)>]
scala> df.collect()
res2: Array[org.apache.spark.sql.Row] = Array([WrappedArray(0.0)])
{code}

In this case, {{0.0}} and {{-0.0}} are considered equal and the union of the arrays produces a single value: {{0.0}}.

However, if I try this operation using a constructed dataframe, these values are not equal, and the result is an array with both {{0.0}} and {{-0.0}}.

{code:java}
scala> val df = List((Array(0.0), Array(-0.0))).toDF("a", "b")
df: org.apache.spark.sql.DataFrame = [a: array<double>, b: array<double>]

scala> df.selectExpr("array_union(a, b)").collect()
res3: Array[org.apache.spark.sql.Row] = Array([WrappedArray(0.0, -0.0)])
{code}

For {{arrays_overlap}}, here is a similar version of that inconsistency:

{code:java}
scala> val df = spark.sql("SELECT arrays_overlap(array(0.0), array(-0.0))")
df: org.apache.spark.sql.DataFrame = [arrays_overlap(array(0.0), array(0.0)): boolean]

scala> df.collect
res4: Array[org.apache.spark.sql.Row] = Array([true])
{code}

{code:java}
scala> val df = List((Array(0.0), Array(-0.0))).toDF("a", "b")
df: org.apache.spark.sql.DataFrame = [a: array<double>, b: array<double>]

scala> df.selectExpr("arrays_overlap(a, b)")
res5: org.apache.spark.sql.DataFrame = [arrays_overlap(a, b): boolean]

scala> df.selectExpr("arrays_overlap(a, b)").collect
res6: Array[org.apache.spark.sql.Row] = Array([false])
{code}

It looks like this is due to the fact that in the constructed dataframe case, the Double value is hashed by using {{java.lang.Double.doubleToLongBits}}, which will treat {{0.0}} and {{-0.0} as distinct because of the sign bit.

See here for more information: https://github.com/apache/spark/blob/master/core/src/main/scala/org/apache/spark/util/collection/OpenHashSet.scala#L312-L321 for 

I can also confirm that the same behavior occurs with FloatType and the use of {{java.lang.Float.floatToIntBits}}



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org