You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Thomas Graves (Jira)" <ji...@apache.org> on 2020/08/19 19:15:00 UTC
[jira] [Comment Edited] (SPARK-32640) Spark 3.1 log(NaN) returns
null instead of NaN
[ https://issues.apache.org/jira/browse/SPARK-32640?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17180772#comment-17180772 ]
Thomas Graves edited comment on SPARK-32640 at 8/19/20, 7:14 PM:
-----------------------------------------------------------------
hmm, interesting, this is how my test was reproducing with paysark:
{code:java}
special_cases = [0.0, -0.0, 1.0, -1.0]
special_cases.append(float('nan'))
from pyspark.sql.types import *
df = spark.createDataFrame(special_cases, DoubleType())
df.selectExpr('log(value)').show()
{code}
+---------------+
|LOG(E(), value)|
+---------------+
|null|
+---------------+
>>> df.show()
+-----+
|value|
+-----+
|0.0|
|-0.0|
|1.0|
|-1.0|
|NaN|
+-----+
>>> df.printSchema()
root
|– value: double (nullable = true)|
was (Author: tgraves):
hmm, interesting, this is how my test was reproducing with paysark:
'''special_cases = [0.0, -0.0, 1.0, -1.0]
special_cases.append(float('nan'))
from pyspark.sql.types import *
df = spark.createDataFrame(special_cases, DoubleType())
df.selectExpr('log(value)').show()
'''
+---------------+
|LOG(E(), value)|
+---------------+
|null|
+---------------+
>>> df.show()
+-----+
|value|
+-----+
|0.0|
|-0.0|
|1.0|
|-1.0|
|NaN|
+-----+
>>> df.printSchema()
root
|– value: double (nullable = true)|
> Spark 3.1 log(NaN) returns null instead of NaN
> ----------------------------------------------
>
> Key: SPARK-32640
> URL: https://issues.apache.org/jira/browse/SPARK-32640
> Project: Spark
> Issue Type: Bug
> Components: SQL
> Affects Versions: 3.1.0
> Reporter: Thomas Graves
> Priority: Major
>
> I was testing Spark 3.1.0 and I noticed that if you take the log(NaN) it now returns a null whereas in Spark 3.0 it returned a NaN. I'm not an expert in this but I thought NaN was correct.
> Spark 3.1.0 Example:
> >>> df.selectExpr(["value", "log1p(value)"]).show()
> +--------------+-----------------+
> | value| LOG1P(value)|
> +--------------+-----------------+
> |-3.4028235E38| null|
> |3.4028235E38|88.72283906194683|
> | 0.0| 0.0|
> | -0.0| -0.0|
> | 1.0|0.6931471805599453|
> | -1.0| null|
> | NaN| null|
> +--------------+-----------------+
>
> Spark 3.0.0 example:
>
> +-------------+------------------+
> | value| LOG1P(value)|
> +-------------+------------------+
> |-3.4028235E38| null|
> | 3.4028235E38| 88.72283906194683|
> | 0.0| 0.0|
> | -0.0| -0.0|
> | 1.0|0.6931471805599453|
> | -1.0| null|
> | NaN| NaN|
> +-------------+------------------+
>
> Note it also does the same for log1p, log2, log10
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org