You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by tejasapatil <gi...@git.apache.org> on 2016/08/27 03:14:22 UTC

[GitHub] spark pull request #14841: [SPARK-17271] [SQL] Planner adds un-necessary Sor...

GitHub user tejasapatil opened a pull request:

    https://github.com/apache/spark/pull/14841

    [SPARK-17271] [SQL] Planner adds un-necessary Sort even if child ordering is semantically same as required ordering

    ## What changes were proposed in this pull request?
    
    Jira : https://issues.apache.org/jira/browse/SPARK-17271
    
    This PR includes following changes:
    - Added a `semanticEquals` method to `SortOrder` so that it can compare underlying child expressions semantically (and not using default Object.equals)
    - Fixed `EnsureRequirements` to use semantic comparison of SortOrder
    
    ## How was this patch tested?
    
    - Added a test case to `PlannerSuite`. Ran rest tests in `PlannerSuite`

You can merge this pull request into a Git repository by running:

    $ git pull https://github.com/tejasapatil/spark SPARK-17271_sort_order_equals_bug

Alternatively you can review and apply these changes as the patch at:

    https://github.com/apache/spark/pull/14841.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

    This closes #14841
    
----
commit 7192418b3a26a14642fc04fc92bf496a954ffa5d
Author: Tejas Patil <te...@fb.com>
Date:   2016-08-27T02:59:28Z

    [SPARK-17271] [SQL] Planner adds un-necessary Sort even if child ordering is semantically same as required ordering

----


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #14841: [SPARK-17271] [SQL] Planner adds un-necessary Sort even ...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:

    https://github.com/apache/spark/pull/14841
  
    **[Test build #64512 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/64512/consoleFull)** for PR 14841 at commit [`7192418`](https://github.com/apache/spark/commit/7192418b3a26a14642fc04fc92bf496a954ffa5d).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #14841: [SPARK-17271] [SQL] Planner adds un-necessary Sort even ...

Posted by tejasapatil <gi...@git.apache.org>.
Github user tejasapatil commented on the issue:

    https://github.com/apache/spark/pull/14841
  
    cc'ing @rxin and @hvanhovell for review


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #14841: [SPARK-17271] [SQL] Planner adds un-necessary Sort even ...

Posted by hvanhovell <gi...@git.apache.org>.
Github user hvanhovell commented on the issue:

    https://github.com/apache/spark/pull/14841
  
    LGTM - merging to master. Thanks!


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #14841: [SPARK-17271] [SQL] Planner adds un-necessary Sort even ...

Posted by cloud-fan <gi...@git.apache.org>.
Github user cloud-fan commented on the issue:

    https://github.com/apache/spark/pull/14841
  
    yea, I think so, it's pretty safe. @tejasapatil do you mind sending a new PR against branch 2.0 to backport this PR and the follow-up one? thanks!


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #14841: [SPARK-17271] [SQL] Planner adds un-necessary Sort even ...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/14841
  
    Test PASSed.
    Refer to this link for build results (access rights to CI server needed): 
    https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/64512/
    Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #14841: [SPARK-17271] [SQL] Planner adds un-necessary Sort even ...

Posted by rxin <gi...@git.apache.org>.
Github user rxin commented on the issue:

    https://github.com/apache/spark/pull/14841
  
    Should this go into 2.0?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #14841: [SPARK-17271] [SQL] Planner adds un-necessary Sort even ...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:

    https://github.com/apache/spark/pull/14841
  
    Merged build finished. Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request #14841: [SPARK-17271] [SQL] Planner adds un-necessary Sor...

Posted by cloud-fan <gi...@git.apache.org>.
Github user cloud-fan commented on a diff in the pull request:

    https://github.com/apache/spark/pull/14841#discussion_r77106801
  
    --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/SortOrder.scala ---
    @@ -61,6 +61,9 @@ case class SortOrder(child: Expression, direction: SortDirection)
       override def sql: String = child.sql + " " + direction.sql
     
       def isAscending: Boolean = direction == Ascending
    +
    +  def semanticEquals(other: SortOrder): Boolean =
    --- End diff --
    
    `Expression` has a default implementation of `semanticEquals`, doesn't it work?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request #14841: [SPARK-17271] [SQL] Planner adds un-necessary Sor...

Posted by cloud-fan <gi...@git.apache.org>.
Github user cloud-fan commented on a diff in the pull request:

    https://github.com/apache/spark/pull/14841#discussion_r77114998
  
    --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/SortOrder.scala ---
    @@ -61,6 +61,9 @@ case class SortOrder(child: Expression, direction: SortDirection)
       override def sql: String = child.sql + " " + direction.sql
     
       def isAscending: Boolean = direction == Ascending
    +
    +  def semanticEquals(other: SortOrder): Boolean =
    --- End diff --
    
    yea I understand in `EnsureRequirements` we should use `semanticEquals` instead of `==` to compare `SortOrder`, but why we need to implement `samanticEquals` again in `SortOrder`? What's wrong with the default implementation?
    
    I mean, there is no need to "introduce" a `semanticEquals` in `SortOrder`, it already has.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #14841: [SPARK-17271] [SQL] Planner adds un-necessary Sort even ...

Posted by tejasapatil <gi...@git.apache.org>.
Github user tejasapatil commented on the issue:

    https://github.com/apache/spark/pull/14841
  
    ok to test


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark issue #14841: [SPARK-17271] [SQL] Planner adds un-necessary Sort even ...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:

    https://github.com/apache/spark/pull/14841
  
    **[Test build #64512 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/64512/consoleFull)** for PR 14841 at commit [`7192418`](https://github.com/apache/spark/commit/7192418b3a26a14642fc04fc92bf496a954ffa5d).
     * This patch passes all tests.
     * This patch merges cleanly.
     * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request #14841: [SPARK-17271] [SQL] Planner adds un-necessary Sor...

Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:

    https://github.com/apache/spark/pull/14841


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request #14841: [SPARK-17271] [SQL] Planner adds un-necessary Sor...

Posted by tejasapatil <gi...@git.apache.org>.
Github user tejasapatil commented on a diff in the pull request:

    https://github.com/apache/spark/pull/14841#discussion_r77117090
  
    --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/SortOrder.scala ---
    @@ -61,6 +61,9 @@ case class SortOrder(child: Expression, direction: SortDirection)
       override def sql: String = child.sql + " " + direction.sql
     
       def isAscending: Boolean = direction == Ascending
    +
    +  def semanticEquals(other: SortOrder): Boolean =
    --- End diff --
    
    @cloud-fan : I see what you were trying to say before. I tried that and it worked. I have created a PR to clean it up : https://github.com/apache/spark/pull/14910 Thanks for pointing this out !!


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request #14841: [SPARK-17271] [SQL] Planner adds un-necessary Sor...

Posted by tejasapatil <gi...@git.apache.org>.
Github user tejasapatil commented on a diff in the pull request:

    https://github.com/apache/spark/pull/14841#discussion_r77113690
  
    --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/SortOrder.scala ---
    @@ -61,6 +61,9 @@ case class SortOrder(child: Expression, direction: SortDirection)
       override def sql: String = child.sql + " " + direction.sql
     
       def isAscending: Boolean = direction == Ascending
    +
    +  def semanticEquals(other: SortOrder): Boolean =
    --- End diff --
    
    @cloud-fan : If you look at the old version of `EnsureRequirements` below at L253, it compared raw `SortOrder` objects which will use `equals()` generated for it. In scala, `equals()` for case classes is merely doing `equals()` over all its fields so that lead to `Expression`'s `equals()` being used instead of its `semanticEquals()`.
    
    My fix here was to introduce a `semanticEquals` in `SortOrder` which compares the underlying `Expression` semantically.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org