[jira] [Commented] (SPARK-32640) Spark 3.1 log(NaN) returns null instead of NaN

["Sean R. Owen (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/SPARK-32640?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17179712#comment-17179712 ] 

Sean R. Owen commented on SPARK-32640:
--------------------------------------

I wouldn't expect a behavior change. But I also don't know what the right answer is meant to be in this case, so it might be a 'bug fix'. But then I don't know of any changes along these lines. [~cloud_fan] do you have any ideas about the right answer here?

Under the hood we use StrictMath.log1p, which would return NaN for NaN.

BTW I'd expect log1p(-1) to be negative infinity? Just like log(0) should.

> Spark 3.1 log(NaN) returns null instead of NaN
> ----------------------------------------------
>
>                 Key: SPARK-32640
>                 URL: https://issues.apache.org/jira/browse/SPARK-32640
>             Project: Spark
>          Issue Type: Bug
>          Components: SQL
>    Affects Versions: 3.1.0
>            Reporter: Thomas Graves
>            Priority: Major
>
> I was testing Spark 3.1.0 and I noticed that if you take the log(NaN) it now returns a null whereas in Spark 3.0 it returned a NaN.  I'm not an expert in this but I thought NaN was correct.
> Spark 3.1.0 Example:
> >>> df.selectExpr(["value", "log1p(value)"]).show()
> +--------------+-----------------+
> |        value|      LOG1P(value)|
> +--------------+-----------------+
> |-3.4028235E38|              null|
> |3.4028235E38|88.72283906194683|
> |          0.0|               0.0|
> |         -0.0|              -0.0|
> |          1.0|0.6931471805599453|
> |         -1.0|              null|
> |          NaN|              null|
> +--------------+-----------------+
>  
> Spark 3.0.0 example:
>  
> +-------------+------------------+
> | value| LOG1P(value)|
> +-------------+------------------+
> |-3.4028235E38| null|
> | 3.4028235E38| 88.72283906194683|
> | 0.0| 0.0|
> | -0.0| -0.0|
> | 1.0|0.6931471805599453|
> | -1.0| null|
> | NaN| NaN|
> +-------------+------------------+
>  
> Note it also does the same for log1p, log2, log10



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org