You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by gatorsmile <gi...@git.apache.org> on 2018/09/12 17:56:30 UTC

[GitHub] spark pull request #22403: [SPARK-25402][SQL][BACKPORT-2.2] Null handling in...

GitHub user gatorsmile opened a pull request:

    https://github.com/apache/spark/pull/22403

    [SPARK-25402][SQL][BACKPORT-2.2] Null handling in BooleanSimplification

    ## What changes were proposed in this pull request?
    This PR is to fix the null handling in BooleanSimplification. In the rule BooleanSimplification, there are two cases that do not properly handle null values. The optimization is not right if either side is null. This PR is to fix them.
    
    ## How was this patch tested?
    Added test cases


You can merge this pull request into a Git repository by running:

    $ git pull https://github.com/gatorsmile/spark backportSpark25402

Alternatively you can review and apply these changes as the patch at:

    https://github.com/apache/spark/pull/22403.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

    This closes #22403
    
----
commit 7088371330591b5278ae4e6083cfcbedc5f0fda6
Author: gatorsmile <ga...@...>
Date:   2018-09-12T13:11:22Z

    [SPARK-25402][SQL] Null handling in BooleanSimplification
    
    This PR is to fix the null handling in BooleanSimplification. In the rule BooleanSimplification, there are two cases that do not properly handle null values. The optimization is not right if either side is null. This PR is to fix them.
    
    Added test cases
    
    Closes #22390 from gatorsmile/fixBooleanSimplification.
    
    Authored-by: gatorsmile <ga...@gmail.com>
    Signed-off-by: Wenchen Fan <we...@databricks.com>
    (cherry picked from commit 79cc59718fdf7785bdc37a26bb8df4c6151114a6)
    Signed-off-by: Wenchen Fan <we...@databricks.com>

----


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark issue #22403: [SPARK-25402][SQL][BACKPORT-2.2] Null handling in Boolea...

Posted by dongjoon-hyun <gi...@git.apache.org>.
Github user dongjoon-hyun commented on the issue:

    https://github.com/apache/spark/pull/22403
  
    Merged to branch-2.2.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark issue #22403: [SPARK-25402][SQL][BACKPORT-2.2] Null handling in Boolea...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:

    https://github.com/apache/spark/pull/22403
  
    **[Test build #96033 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/96033/testReport)** for PR 22403 at commit [`ec43727`](https://github.com/apache/spark/commit/ec437276d4be8bb905a2fd2f119f4ea372d03d1c).


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark pull request #22403: [SPARK-25402][SQL][BACKPORT-2.2] Null handling in...

Posted by dongjoon-hyun <gi...@git.apache.org>.
Github user dongjoon-hyun commented on a diff in the pull request:

    https://github.com/apache/spark/pull/22403#discussion_r217171729
  
    --- Diff: sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/optimizer/BooleanSimplificationSuite.scala ---
    @@ -37,6 +38,7 @@ class BooleanSimplificationSuite extends PlanTest with PredicateHelper {
           Batch("Constant Folding", FixedPoint(50),
             NullPropagation(conf),
             ConstantFolding,
    +        SimplifyConditionals,
             BooleanSimplification,
             PruneFilters(conf)) :: Nil
       }
    --- End diff --
    
    At line 46, SPARK-17851 changed like the following. We need that.
    ```scala
    - val testRelation = LocalRelation('a.int, 'b.int, 'c.int, 'd.string)
    + val testRelation = LocalRelation('a.int, 'b.int, 'c.int, 'd.string,
       'e.boolean, 'f.boolean, 'g.boolean, 'h.boolean)
    ```



---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark issue #22403: [SPARK-25402][SQL][BACKPORT-2.2] Null handling in Boolea...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/22403
  
    Test FAILed.
    Refer to this link for build results (access rights to CI server needed): 
    https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/95999/
    Test FAILed.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark issue #22403: [SPARK-25402][SQL][BACKPORT-2.2] Null handling in Boolea...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:

    https://github.com/apache/spark/pull/22403
  
    **[Test build #95999 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/95999/testReport)** for PR 22403 at commit [`7088371`](https://github.com/apache/spark/commit/7088371330591b5278ae4e6083cfcbedc5f0fda6).
     * This patch **fails Spark unit tests**.
     * This patch merges cleanly.
     * This patch adds no public classes.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark issue #22403: [SPARK-25402][SQL][BACKPORT-2.2] Null handling in Boolea...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:

    https://github.com/apache/spark/pull/22403
  
    **[Test build #96033 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/96033/testReport)** for PR 22403 at commit [`ec43727`](https://github.com/apache/spark/commit/ec437276d4be8bb905a2fd2f119f4ea372d03d1c).
     * This patch passes all tests.
     * This patch merges cleanly.
     * This patch adds no public classes.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark issue #22403: [SPARK-25402][SQL][BACKPORT-2.2] Null handling in Boolea...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/22403
  
    Merged build finished. Test PASSed.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark issue #22403: [SPARK-25402][SQL][BACKPORT-2.2] Null handling in Boolea...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/22403
  
    Merged build finished. Test PASSed.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark issue #22403: [SPARK-25402][SQL][BACKPORT-2.2] Null handling in Boolea...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/22403
  
    Test PASSed.
    Refer to this link for build results (access rights to CI server needed): 
    https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/96033/
    Test PASSed.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark pull request #22403: [SPARK-25402][SQL][BACKPORT-2.2] Null handling in...

Posted by gatorsmile <gi...@git.apache.org>.
Github user gatorsmile commented on a diff in the pull request:

    https://github.com/apache/spark/pull/22403#discussion_r217265214
  
    --- Diff: sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/optimizer/BooleanSimplificationSuite.scala ---
    @@ -37,6 +38,7 @@ class BooleanSimplificationSuite extends PlanTest with PredicateHelper {
           Batch("Constant Folding", FixedPoint(50),
             NullPropagation(conf),
             ConstantFolding,
    +        SimplifyConditionals,
             BooleanSimplification,
             PruneFilters(conf)) :: Nil
       }
    --- End diff --
    
    I see. Thanks!



---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark issue #22403: [SPARK-25402][SQL][BACKPORT-2.2] Null handling in Boolea...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/22403
  
    Test PASSed.
    Refer to this link for build results (access rights to CI server needed): 
    https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/3054/
    Test PASSed.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark issue #22403: [SPARK-25402][SQL][BACKPORT-2.2] Null handling in Boolea...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/22403
  
    Test PASSed.
    Refer to this link for build results (access rights to CI server needed): 
    https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/3082/
    Test PASSed.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark pull request #22403: [SPARK-25402][SQL][BACKPORT-2.2] Null handling in...

Posted by gatorsmile <gi...@git.apache.org>.
Github user gatorsmile closed the pull request at:

    https://github.com/apache/spark/pull/22403


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark issue #22403: [SPARK-25402][SQL][BACKPORT-2.2] Null handling in Boolea...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/22403
  
    Merged build finished. Test PASSed.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark issue #22403: [SPARK-25402][SQL][BACKPORT-2.2] Null handling in Boolea...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:

    https://github.com/apache/spark/pull/22403
  
    **[Test build #95999 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/95999/testReport)** for PR 22403 at commit [`7088371`](https://github.com/apache/spark/commit/7088371330591b5278ae4e6083cfcbedc5f0fda6).


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark issue #22403: [SPARK-25402][SQL][BACKPORT-2.2] Null handling in Boolea...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/22403
  
    Merged build finished. Test FAILed.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org