You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Eric Yang (Jira)" <ji...@apache.org> on 2026/08/23 20:05:00 UTC

[jira] [Commented] (SPARK-39031) NaN != NaN in pivot

    [ https://issues.apache.org/jira/browse/SPARK-39031?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18107190#comment-18107190 ] 

Eric Yang commented on SPARK-39031:
-----------------------------------

I'm working on a fix at [https://github.com/apache/spark/pull/58234] 

> NaN != NaN in pivot
> -------------------
>
>                 Key: SPARK-39031
>                 URL: https://issues.apache.org/jira/browse/SPARK-39031
>             Project: Spark
>          Issue Type: Bug
>          Components: SQL
>    Affects Versions: 3.2.0
>            Reporter: Robert Joseph Evans
>            Priority: Major
>              Labels: pull-request-available
>
> I know this is an odd corner case, but NaN != NaN in pivot which is inconsistent with other places in Spark.
>  
> {code}
> scala> Seq(Double.NaN, Double.NaN, 1.0, Double.NaN, 1.0, 1.0).toDF.groupBy("value").count.show()
> +-----+-----+                                                                   
> |value|count|
> +-----+-----+
> |  NaN|    3|
> |  1.0|    3|
> +-----+-----+
> scala> Seq(Double.NaN, Double.NaN, 1.0, Double.NaN, 1.0, 1.0).toDF.groupBy("value").pivot("value").count.show()
> +-----+----+----+
> |value| 1.0| NaN|
> +-----+----+----+
> |  NaN|null|null|
> |  1.0|   3|null|
> +-----+----+----+
> {code}
>  
> It looks like the issue is that in PivotFirst if the pivotColumn is an AtomicType a HashMap is used, but for other types a TreeMap is used with an interpretedOrdering. If we made DoubleType and FloatType use the TreeMap then the equality checks would be correct. But I am not able to really test it because if I try to pivot on an array or struct I get analysis exceptions.
>  
> {code}
> scala> Seq(Double.NaN, Double.NaN, 1.0, Double.NaN, 1.0, 1.0).toDF.selectExpr("value", "struct(value) as ar_value").groupBy("value").pivot("ar_value").count.show()
> java.lang.RuntimeException: Unsupported literal type class org.apache.spark.sql.catalyst.expressions.GenericRowWithSchema [1.0]
>   at org.apache.spark.sql.errors.QueryExecutionErrors$.literalTypeUnsupportedError(QueryExecutionErrors.scala:182)
>   at org.apache.spark.sql.catalyst.expressions.Literal$.apply(literals.scala:101)
> {code}
>  
> {code}
>  scala> Seq(Double.NaN, Double.NaN, 1.0, Double.NaN, 1.0, 1.0).toDF.selectExpr("value", "array(value) as ar_value").groupBy("value").pivot("ar_value").count.show()
> org.apache.spark.sql.AnalysisException: Invalid pivot value '[1.0]': value data type array<double> does not match pivot column data type array<double>
>   at org.apache.spark.sql.errors.QueryCompilationErrors$.pivotValDataTypeMismatchError(QueryCompilationErrors.scala:85)
>   at org.apache.spark.sql.catalyst.analysis.Analyzer$ResolvePivot$$anonfun$apply$10.$anonfun$applyOrElse$21(Analyzer.scala:762)
>   at scala.collection.TraversableLike.$anonfun$map$1(TraversableLike.scala:286)
>   at scala.collection.mutable.ResizableArray.foreach(ResizableArray.scala:62)
>   at scala.collection.mutable.ResizableArray.foreach$(ResizableArray.scala:55)
> {code}



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org