[jira] [Commented] (SPARK-13731) expression evaluation for NaN in select statement

["Ian (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/SPARK-13731?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15183869#comment-15183869 ] 

Ian commented on SPARK-13731:
-----------------------------

The expression in select essentially defined a transformation from data residing on storage or even another RDD.
We are seeing that the transformation result is now null for both NaN and Infinity.
We saw SPARK-9076, which seemed addressing only how NaN value in RDD can be handled(comparing,ordering, ...), 
but our concerned case is more on the another aspect and more fundamental about the expressions that might produce NaN.

 
 


> expression evaluation for NaN in select statement
> -------------------------------------------------
>
>                 Key: SPARK-13731
>                 URL: https://issues.apache.org/jira/browse/SPARK-13731
>             Project: Spark
>          Issue Type: Bug
>          Components: SQL
>    Affects Versions: 1.6.0
>            Reporter: Ian
>
> We are expecting arithmetic expression a/b should be:
> 1. returning NaN if a=0 and b=0
> 2. returning Infinity if a=1 and b=0
> Is the expectation reasonable? 
> The following is a simple test case snippet that read from storage and evaluate arithmetic in select.
> It si assuming org.apache.spark.sql.hive.execution.SQLQuerySuite: 
> {code}
>   test("Expression should be evaluated to Nan/Infinity in Select") {
>     withTable("testNan") {
>       withTempTable("src") {
>         Seq((1d, 0d), (0d, 0d)).toDF().registerTempTable("src")
>         sql("CREATE TABLE testNan(a double, b double) STORED AS PARQUET AS SELECT * FROM src")
>       }
>       checkAnswer(sql(
>         """
>           |SELECT a/b FROM testNan
>         """.stripMargin),
>         Seq(
>           Row(Double.PositiveInfinity),
>           Row(Double.NaN)
>         )
>       )
>     }
>   }
> == Physical Plan ==
> Project [(a#28 / b#29) AS _c0#30]
> +- Scan ParquetRelation: default.testnan[a#28,b#29] InputPaths: file:/private/var/folders/dy/19y6pfm92pj9s40mbs8xd9hm0000gp/T/warehouse--5b617080-e909-4812-90e8-63d2dd0aef5a/testnan
> == Results ==
> !== Correct Answer - 2 ==   == Spark Answer - 2 ==
> ![Infinity]                 [null]
> ![NaN]                      [null]
>       
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org