You are viewing a plain text version of this content. The canonical link for it is here.

Posted to reviews@spark.apache.org by dilipbiswal <gi...@git.apache.org> on 2018/08/01 05:49:58 UTC

[GitHub] spark pull request #21941: [SPARK-24966][SQL] Implement precedence rules for...

GitHub user dilipbiswal opened a pull request:

    https://github.com/apache/spark/pull/21941

    [SPARK-24966][SQL] Implement precedence rules for set operations.

    ## What changes were proposed in this pull request?
    
    Currently the set operations INTERSECT, UNION and EXCEPT are assigned the same precedence. This PR fixes the problem by giving INTERSECT  higher precedence than UNION and EXCEPT. UNION and EXCEPT operators are evaluated in the order in which they appear in the query from left to right.
    
    This results in change in behavior because of the change in order of evaluations of set operators in a query. The old behavior is still preserved under a newly added config parameter.
    
    Query `:`
    ```
    SELECT * FROM t1
    UNION 
    SELECT * FROM t2
    EXCEPT
    SELECT * FROM t3
    INTERSECT
    SELECT * FROM t4
    ```
    Parsed plan before the change `:`
    ```
    == Parsed Logical Plan ==
    'Intersect false
    :- 'Except false
    :  :- 'Distinct
    :  :  +- 'Union
    :  :     :- 'Project [*]
    :  :     :  +- 'UnresolvedRelation `t1`
    :  :     +- 'Project [*]
    :  :        +- 'UnresolvedRelation `t2`
    :  +- 'Project [*]
    :     +- 'UnresolvedRelation `t3`
    +- 'Project [*]
       +- 'UnresolvedRelation `t4`
    ```
    Parsed plan after the change `:`
    ```
    == Parsed Logical Plan ==
    'Except false
    :- 'Distinct
    :  +- 'Union
    :     :- 'Project [*]
    :     :  +- 'UnresolvedRelation `t1`
    :     +- 'Project [*]
    :        +- 'UnresolvedRelation `t2`
    +- 'Intersect false
       :- 'Project [*]
       :  +- 'UnresolvedRelation `t3`
       +- 'Project [*]
          +- 'UnresolvedRelation `t4`
    ```
    ## How was this patch tested?
    Added tests in PlanParserSuite, SQLQueryTestSuite.
    
    Please review http://spark.apache.org/contributing.html before opening a pull request.


You can merge this pull request into a Git repository by running:

    $ git pull https://github.com/dilipbiswal/spark SPARK-24966

Alternatively you can review and apply these changes as the patch at:

    https://github.com/apache/spark/pull/21941.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

    This closes #21941
    
----
commit c0821b6dd8e713edf2bd1ddd9a27f1999970d8f8
Author: Dilip Biswal <db...@...>
Date:   2018-07-30T05:10:29Z

    [SPARK-24966] Implement precedence rules for set operations.

----


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark issue #21941: [SPARK-24966][SQL] Implement precedence rules for set op...

Posted by AmplabJenkins <gi...@git.apache.org>.

Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/21941
  
    Test FAILed.
    Refer to this link for build results (access rights to CI server needed): 
    https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/94046/
    Test FAILed.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark issue #21941: [SPARK-24966][SQL] Implement precedence rules for set op...

Posted by SparkQA <gi...@git.apache.org>.

Github user SparkQA commented on the issue:

    https://github.com/apache/spark/pull/21941
  
    **[Test build #94046 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/94046/testReport)** for PR 21941 at commit [`bc6df75`](https://github.com/apache/spark/commit/bc6df7517ed71c49955e42590888eb29af600118).
     * This patch **fails to generate documentation**.
     * This patch merges cleanly.
     * This patch adds no public classes.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark issue #21941: [SPARK-24966][SQL] Implement precedence rules for set op...

Posted by SparkQA <gi...@git.apache.org>.

Github user SparkQA commented on the issue:

    https://github.com/apache/spark/pull/21941
  
    **[Test build #93871 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/93871/testReport)** for PR 21941 at commit [`c0821b6`](https://github.com/apache/spark/commit/c0821b6dd8e713edf2bd1ddd9a27f1999970d8f8).
     * This patch **fails Spark unit tests**.
     * This patch merges cleanly.
     * This patch adds no public classes.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark issue #21941: [SPARK-24966][SQL] Implement precedence rules for set op...

Posted by AmplabJenkins <gi...@git.apache.org>.

Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/21941
  
    Test PASSed.
    Refer to this link for build results (access rights to CI server needed): 
    https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/1626/
    Test PASSed.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark issue #21941: [SPARK-24966][SQL] Implement precedence rules for set op...

Posted by AmplabJenkins <gi...@git.apache.org>.

Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/21941
  
    Merged build finished. Test PASSed.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark issue #21941: [SPARK-24966][SQL] Implement precedence rules for set op...

Posted by dilipbiswal <gi...@git.apache.org>.

Github user dilipbiswal commented on the issue:

    https://github.com/apache/spark/pull/21941
  
    @gatorsmile rebasing in a hope that the test result changes :-)


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark issue #21941: [SPARK-24966][SQL] Implement precedence rules for set op...

Posted by dilipbiswal <gi...@git.apache.org>.

Github user dilipbiswal commented on the issue:

    https://github.com/apache/spark/pull/21941
  
    retest this please


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] spark issue #21941: [SPARK-24966][SQL] Implement precedence rules for set op...

Posted by AmplabJenkins <gi...@git.apache.org>.

Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/21941
  
    Test PASSed.
    Refer to this link for build results (access rights to CI server needed): 
    https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/1695/
    Test PASSed.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org