You are viewing a plain text version of this content. The canonical link for it is here.

Posted to reviews@spark.apache.org by ueshin <gi...@git.apache.org> on 2018/04/18 06:01:05 UTC

[GitHub] spark pull request #21094: [SPARK-24007][SQL] EqualNullSafe for FloatType an...

GitHub user ueshin opened a pull request:

    https://github.com/apache/spark/pull/21094

    [SPARK-24007][SQL] EqualNullSafe for FloatType and DoubleType might generate a wrong result by codegen.

    ## What changes were proposed in this pull request?
    
    `EqualNullSafe` for `FloatType` and `DoubleType` might generate a wrong result by codegen.
    
    ```scala
    scala> val df = Seq((Some(-1.0d), None), (None, Some(-1.0d))).toDF()
    df: org.apache.spark.sql.DataFrame = [_1: double, _2: double]
    
    scala> df.show()
    +----+----+
    |  _1|  _2|
    +----+----+
    |-1.0|null|
    |null|-1.0|
    +----+----+
    
    
    scala> df.filter("_1 <=> _2").show()
    +----+----+
    |  _1|  _2|
    +----+----+
    |-1.0|null|
    |null|-1.0|
    +----+----+
    ```
    
    The result should be empty but the result remains two rows.
    
    ## How was this patch tested?
    
    Added a test.


You can merge this pull request into a Git repository by running:

    $ git pull https://github.com/ueshin/apache-spark issues/SPARK-24007/equalnullsafe

Alternatively you can review and apply these changes as the patch at:

    https://github.com/apache/spark/pull/21094.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

    This closes #21094
    
----
commit ad74513002f610a9c82a18eeff8b4218a1c81116
Author: Takuya UESHIN <ue...@...>
Date:   2018-04-18T05:12:11Z

    Fix EqualNullSafe for FloatType and DoubleType.

----


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark issue #21094: [SPARK-24007][SQL] EqualNullSafe for FloatType and Doubl...

Posted by rednaxelafx <gi...@git.apache.org>.

Github user rednaxelafx commented on the issue:

    https://github.com/apache/spark/pull/21094
  
    LGTM
    Wow, that's a hideous bug...


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark issue #21094: [SPARK-24007][SQL] EqualNullSafe for FloatType and Doubl...

Posted by ueshin <gi...@git.apache.org>.

Github user ueshin commented on the issue:

    https://github.com/apache/spark/pull/21094
  
    Jenkins, retest this please.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark issue #21094: [SPARK-24007][SQL] EqualNullSafe for FloatType and Doubl...

Posted by ueshin <gi...@git.apache.org>.

Github user ueshin commented on the issue:

    https://github.com/apache/spark/pull/21094
  
    cc @gatorsmile @cloud-fan 


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark issue #21094: [SPARK-24007][SQL] EqualNullSafe for FloatType and Doubl...

Posted by AmplabJenkins <gi...@git.apache.org>.

Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/21094
  
    Merged build finished. Test FAILed.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark issue #21094: [SPARK-24007][SQL] EqualNullSafe for FloatType and Doubl...

Posted by AmplabJenkins <gi...@git.apache.org>.

Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/21094
  
    Test PASSed.
    Refer to this link for build results (access rights to CI server needed): 
    https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution/2421/
    Test PASSed.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark issue #21094: [SPARK-24007][SQL] EqualNullSafe for FloatType and Doubl...

Posted by gatorsmile <gi...@git.apache.org>.

Github user gatorsmile commented on the issue:

    https://github.com/apache/spark/pull/21094
  
    Thanks! Merged to master/2.3/2.2



---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark issue #21094: [SPARK-24007][SQL] EqualNullSafe for FloatType and Doubl...

Posted by AmplabJenkins <gi...@git.apache.org>.

Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/21094
  
    Test PASSed.
    Refer to this link for build results (access rights to CI server needed): 
    https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution/2424/
    Test PASSed.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark issue #21094: [SPARK-24007][SQL] EqualNullSafe for FloatType and Doubl...

Posted by SparkQA <gi...@git.apache.org>.

Github user SparkQA commented on the issue:

    https://github.com/apache/spark/pull/21094
  
    **[Test build #89498 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/89498/testReport)** for PR 21094 at commit [`ad74513`](https://github.com/apache/spark/commit/ad74513002f610a9c82a18eeff8b4218a1c81116).


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark issue #21094: [SPARK-24007][SQL] EqualNullSafe for FloatType and Doubl...

Posted by AmplabJenkins <gi...@git.apache.org>.

Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/21094
  
    Test PASSed.
    Refer to this link for build results (access rights to CI server needed): 
    https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution/2419/
    Test PASSed.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark issue #21094: [SPARK-24007][SQL] EqualNullSafe for FloatType and Doubl...

Posted by ueshin <gi...@git.apache.org>.

Github user ueshin commented on the issue:

    https://github.com/apache/spark/pull/21094
  
    Jenkins, retest this please.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark issue #21094: [SPARK-24007][SQL] EqualNullSafe for FloatType and Doubl...

Posted by AmplabJenkins <gi...@git.apache.org>.

Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/21094
  
    Merged build finished. Test PASSed.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark issue #21094: [SPARK-24007][SQL] EqualNullSafe for FloatType and Doubl...

Posted by SparkQA <gi...@git.apache.org>.

Github user SparkQA commented on the issue:

    https://github.com/apache/spark/pull/21094
  
    **[Test build #89491 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/89491/testReport)** for PR 21094 at commit [`ad74513`](https://github.com/apache/spark/commit/ad74513002f610a9c82a18eeff8b4218a1c81116).
     * This patch **fails Spark unit tests**.
     * This patch merges cleanly.
     * This patch adds no public classes.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark issue #21094: [SPARK-24007][SQL] EqualNullSafe for FloatType and Doubl...

Posted by kiszk <gi...@git.apache.org>.

Github user kiszk commented on the issue:

    https://github.com/apache/spark/pull/21094
  
    LGTM


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark issue #21094: [SPARK-24007][SQL] EqualNullSafe for FloatType and Doubl...

Posted by cloud-fan <gi...@git.apache.org>.

Github user cloud-fan commented on the issue:

    https://github.com/apache/spark/pull/21094
  
    good catch! LGTM


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark issue #21094: [SPARK-24007][SQL] EqualNullSafe for FloatType and Doubl...

Posted by SparkQA <gi...@git.apache.org>.

Github user SparkQA commented on the issue:

    https://github.com/apache/spark/pull/21094
  
    **[Test build #89491 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/89491/testReport)** for PR 21094 at commit [`ad74513`](https://github.com/apache/spark/commit/ad74513002f610a9c82a18eeff8b4218a1c81116).


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark issue #21094: [SPARK-24007][SQL] EqualNullSafe for FloatType and Doubl...

Posted by AmplabJenkins <gi...@git.apache.org>.

Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/21094
  
    Test FAILed.
    Refer to this link for build results (access rights to CI server needed): 
    https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/89491/
    Test FAILed.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark issue #21094: [SPARK-24007][SQL] EqualNullSafe for FloatType and Doubl...

Posted by SparkQA <gi...@git.apache.org>.

Github user SparkQA commented on the issue:

    https://github.com/apache/spark/pull/21094
  
    **[Test build #89498 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/89498/testReport)** for PR 21094 at commit [`ad74513`](https://github.com/apache/spark/commit/ad74513002f610a9c82a18eeff8b4218a1c81116).
     * This patch passes all tests.
     * This patch merges cleanly.
     * This patch adds no public classes.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark issue #21094: [SPARK-24007][SQL] EqualNullSafe for FloatType and Doubl...

Posted by AmplabJenkins <gi...@git.apache.org>.

Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/21094
  
    Merged build finished. Test PASSed.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark issue #21094: [SPARK-24007][SQL] EqualNullSafe for FloatType and Doubl...

Posted by AmplabJenkins <gi...@git.apache.org>.

Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/21094
  
    Test FAILed.
    Refer to this link for build results (access rights to CI server needed): 
    https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/89486/
    Test FAILed.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark issue #21094: [SPARK-24007][SQL] EqualNullSafe for FloatType and Doubl...

Posted by AmplabJenkins <gi...@git.apache.org>.

Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/21094
  
    Merged build finished. Test PASSed.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark issue #21094: [SPARK-24007][SQL] EqualNullSafe for FloatType and Doubl...

Posted by hvanhovell <gi...@git.apache.org>.

Github user hvanhovell commented on the issue:

    https://github.com/apache/spark/pull/21094
  
    @gatorsmile we are in the process of adding more structure to code generation, see https://github.com/apache/spark/commit/7c1654e2159662e7e663ba141719d755002f770a. This should eliminate this class of bug (expressions will always be wrapped with parenthesis).



---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark issue #21094: [SPARK-24007][SQL] EqualNullSafe for FloatType and Doubl...

Posted by AmplabJenkins <gi...@git.apache.org>.

Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/21094
  
    Test PASSed.
    Refer to this link for build results (access rights to CI server needed): 
    https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/89498/
    Test PASSed.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org