You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by GitBox <gi...@apache.org> on 2021/03/05 02:42:05 UTC
[GitHub] [spark] wangyum opened a new pull request #31750: [SPARK-34628][SQL] Remove GlobalLimit operator if its child max rows not larger than limit number
wangyum opened a new pull request #31750:
URL: https://github.com/apache/spark/pull/31750
### What changes were proposed in this pull request?
This pr remove `GlobalLimit` operator if its child max rows not larger than limit number. For example:
```
val testRelation = LocalRelation.fromExternalRows(Seq("a".attr.int, "b".attr.int, "c".attr.int), 1.to(10).map(_ => Row(1, 2, 3)) )
val query = GlobalLimit(100, testRelation)
```
We can remove this `GlobalLimit`.
### Why are the changes needed?
Further optimize the query.
### Does this PR introduce _any_ user-facing change?
No.
### How was this patch tested?
Unit test.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #31750: [SPARK-34628][SQL] Remove GlobalLimit operator if its child max rows not larger than limit number
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #31750:
URL: https://github.com/apache/spark/pull/31750#issuecomment-791972169
**[Test build #135830 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/135830/testReport)** for PR 31750 at commit [`22957b4`](https://github.com/apache/spark/commit/22957b441749c8e0bd1b0a52605a7bc133b8828d).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #31750: [SPARK-34628][SQL] Remove GlobalLimit operator if its child max rows not larger than limit number
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #31750:
URL: https://github.com/apache/spark/pull/31750#issuecomment-791234753
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/135787/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] wangyum commented on pull request #31750: [SPARK-34628][SQL] Remove GlobalLimit operator if its child max rows not larger than limit number
Posted by GitBox <gi...@apache.org>.
wangyum commented on pull request #31750:
URL: https://github.com/apache/spark/pull/31750#issuecomment-791910031
cc @cloud-fan This is a part of https://github.com/apache/spark/pull/31691. To make that pr more clear.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #31750: [SPARK-34628][SQL] Remove GlobalLimit operator if its child max rows not larger than limit number
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #31750:
URL: https://github.com/apache/spark/pull/31750#issuecomment-792043501
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/135830/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #31750: [SPARK-34628][SQL] Remove GlobalLimit operator if its child max rows not larger than limit number
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #31750:
URL: https://github.com/apache/spark/pull/31750#issuecomment-791990344
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/40412/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #31750: [SPARK-34628][SQL] Remove GlobalLimit operator if its child max rows not larger than limit number
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #31750:
URL: https://github.com/apache/spark/pull/31750#issuecomment-792043501
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/135830/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #31750: [SPARK-34628][SQL] Remove GlobalLimit operator if its child max rows not larger than limit number
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #31750:
URL: https://github.com/apache/spark/pull/31750#issuecomment-791122068
**[Test build #135787 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/135787/testReport)** for PR 31750 at commit [`f9ee999`](https://github.com/apache/spark/commit/f9ee999f2cfc749a2f16fa49191397bbdc103a0d).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #31750: [SPARK-34628][SQL] Remove GlobalLimit operator if its child max rows not larger than limit number
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #31750:
URL: https://github.com/apache/spark/pull/31750#issuecomment-791979770
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/40412/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #31750: [SPARK-34628][SQL] Remove GlobalLimit operator if its child max rows not larger than limit number
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #31750:
URL: https://github.com/apache/spark/pull/31750#issuecomment-791221845
**[Test build #135787 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/135787/testReport)** for PR 31750 at commit [`f9ee999`](https://github.com/apache/spark/commit/f9ee999f2cfc749a2f16fa49191397bbdc103a0d).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #31750: [SPARK-34628][SQL] Remove GlobalLimit operator if its child max rows not larger than limit number
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #31750:
URL: https://github.com/apache/spark/pull/31750#issuecomment-791987294
Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/40412/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #31750: [SPARK-34628][SQL] Remove GlobalLimit operator if its child max rows not larger than limit number
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #31750:
URL: https://github.com/apache/spark/pull/31750#issuecomment-791990344
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/40412/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #31750: [SPARK-34628][SQL] Remove GlobalLimit operator if its child max rows not larger than limit number
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #31750:
URL: https://github.com/apache/spark/pull/31750#issuecomment-791153863
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/40369/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] wangyum commented on a change in pull request #31750: [SPARK-34628][SQL] Remove GlobalLimit operator if its child max rows not larger than limit number
Posted by GitBox <gi...@apache.org>.
wangyum commented on a change in pull request #31750:
URL: https://github.com/apache/spark/pull/31750#discussion_r588891807
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/Optimizer.scala
##########
@@ -1559,7 +1559,8 @@ object PushPredicateThroughJoin extends Rule[LogicalPlan] with PredicateHelper {
/**
* This rule optimizes Limit operators by:
* 1. Eliminate [[Limit]] operators if it's child max row <= limit.
- * 2. Combines two adjacent [[Limit]] operators into one, merging the
+ * 2. Eliminate [[GlobalLimit]] operators if it's child max row <= limit.
Review comment:
fixed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] maropu commented on a change in pull request #31750: [SPARK-34628][SQL] Remove GlobalLimit operator if its child max rows not larger than limit number
Posted by GitBox <gi...@apache.org>.
maropu commented on a change in pull request #31750:
URL: https://github.com/apache/spark/pull/31750#discussion_r588883847
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/Optimizer.scala
##########
@@ -1559,7 +1559,8 @@ object PushPredicateThroughJoin extends Rule[LogicalPlan] with PredicateHelper {
/**
* This rule optimizes Limit operators by:
* 1. Eliminate [[Limit]] operators if it's child max row <= limit.
- * 2. Combines two adjacent [[Limit]] operators into one, merging the
+ * 2. Eliminate [[GlobalLimit]] operators if it's child max row <= limit.
Review comment:
nit: how about merging `1.` and `2.` like `Eliminate [[Limit]]/[[GlobalLimit]] ...`?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #31750: [SPARK-34628][SQL] Remove GlobalLimit operator if its child max rows not larger than limit number
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #31750:
URL: https://github.com/apache/spark/pull/31750#issuecomment-791122068
**[Test build #135787 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/135787/testReport)** for PR 31750 at commit [`f9ee999`](https://github.com/apache/spark/commit/f9ee999f2cfc749a2f16fa49191397bbdc103a0d).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #31750: [SPARK-34628][SQL] Remove GlobalLimit operator if its child max rows not larger than limit number
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #31750:
URL: https://github.com/apache/spark/pull/31750#issuecomment-792040005
**[Test build #135830 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/135830/testReport)** for PR 31750 at commit [`22957b4`](https://github.com/apache/spark/commit/22957b441749c8e0bd1b0a52605a7bc133b8828d).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #31750: [SPARK-34628][SQL] Remove GlobalLimit operator if its child max rows not larger than limit number
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #31750:
URL: https://github.com/apache/spark/pull/31750#issuecomment-791153863
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/40369/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on pull request #31750: [SPARK-34628][SQL] Remove GlobalLimit operator if its child max rows not larger than limit number
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on pull request #31750:
URL: https://github.com/apache/spark/pull/31750#issuecomment-792457702
late LGTM
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #31750: [SPARK-34628][SQL] Remove GlobalLimit operator if its child max rows not larger than limit number
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #31750:
URL: https://github.com/apache/spark/pull/31750#issuecomment-791234753
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/135787/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dongjoon-hyun closed pull request #31750: [SPARK-34628][SQL] Remove GlobalLimit operator if its child max rows not larger than limit number
Posted by GitBox <gi...@apache.org>.
dongjoon-hyun closed pull request #31750:
URL: https://github.com/apache/spark/pull/31750
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #31750: [SPARK-34628][SQL] Remove GlobalLimit operator if its child max rows not larger than limit number
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #31750:
URL: https://github.com/apache/spark/pull/31750#issuecomment-791972169
**[Test build #135830 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/135830/testReport)** for PR 31750 at commit [`22957b4`](https://github.com/apache/spark/commit/22957b441749c8e0bd1b0a52605a7bc133b8828d).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org