You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by GitBox <gi...@apache.org> on 2021/06/26 14:25:41 UTC
[GitHub] [spark] wangyum opened a new pull request #33100: [SPARK-35906][SQL] Remove order by if the maximum number of rows less than or equal to 1
wangyum opened a new pull request #33100:
URL: https://github.com/apache/spark/pull/33100
### What changes were proposed in this pull request?
This PR removes order by if the maximum number of rows less than or equal to 1. For example:
```scala
spark.sql("select count(*) from range(1, 10, 2, 2) order by 1 limit 10").explain("cost")
```
Before this pr:
```
== Optimized Logical Plan ==
Sort [count(1)#2L ASC NULLS FIRST], true, Statistics(sizeInBytes=16.0 B)
+- Aggregate [count(1) AS count(1)#2L], Statistics(sizeInBytes=16.0 B, rowCount=1)
+- Project, Statistics(sizeInBytes=20.0 B)
+- Range (1, 10, step=2, splits=Some(2)), Statistics(sizeInBytes=40.0 B, rowCount=5)
```
After this pr:
```
== Optimized Logical Plan ==
Aggregate [count(1) AS count(1)#2L], Statistics(sizeInBytes=16.0 B, rowCount=1)
+- Project, Statistics(sizeInBytes=20.0 B)
+- Range (1, 10, step=2, splits=Some(2)), Statistics(sizeInBytes=40.0 B, rowCount=5)
```
### Why are the changes needed?
Improve query performance.
### Does this PR introduce _any_ user-facing change?
No.
### How was this patch tested?
Unit test.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] wangyum commented on pull request #33100: [SPARK-35906][SQL] Remove order by if the maximum number of rows less than or equal to 1
Posted by GitBox <gi...@apache.org>.
wangyum commented on pull request #33100:
URL: https://github.com/apache/spark/pull/33100#issuecomment-875241720
I have a PR to fix it: https://github.com/apache/spark/pull/33240
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #33100: [SPARK-35906][SQL] Remove order by if the maximum number of rows less than or equal to 1
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #33100:
URL: https://github.com/apache/spark/pull/33100#issuecomment-870824450
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/140365/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on pull request #33100: [SPARK-35906][SQL] Remove order by if the maximum number of rows less than or equal to 1
Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on pull request #33100:
URL: https://github.com/apache/spark/pull/33100#issuecomment-870806007
cc @cloud-fan
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] wangyum commented on pull request #33100: [SPARK-35906][SQL] Remove order by if the maximum number of rows less than or equal to 1
Posted by GitBox <gi...@apache.org>.
wangyum commented on pull request #33100:
URL: https://github.com/apache/spark/pull/33100#issuecomment-875237884
Thank you @allisonwang-db.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #33100: [SPARK-35906][SQL] Remove order by if the maximum number of rows less than or equal to 1
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #33100:
URL: https://github.com/apache/spark/pull/33100#issuecomment-870824450
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/140365/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on a change in pull request #33100: [SPARK-35906][SQL] Remove order by if the maximum number of rows less than or equal to 1
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on a change in pull request #33100:
URL: https://github.com/apache/spark/pull/33100#discussion_r659455547
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/Optimizer.scala
##########
@@ -1218,6 +1219,7 @@ object EliminateSorts extends Rule[LogicalPlan] {
_.containsPattern(SORT))(applyLocally)
private val applyLocally: PartialFunction[LogicalPlan, LogicalPlan] = {
+ case Sort(_, _, child) if child.maxRows.exists(_ <= 1L) => child
Review comment:
The change here looks good. I wonder if we should better create a separate file to group similar stuff though .... e.g.) we can remove deduplicate too.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dongjoon-hyun closed pull request #33100: [SPARK-35906][SQL] Remove order by if the maximum number of rows less than or equal to 1
Posted by GitBox <gi...@apache.org>.
dongjoon-hyun closed pull request #33100:
URL: https://github.com/apache/spark/pull/33100
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on pull request #33100: [SPARK-35906][SQL] Remove order by if the maximum number of rows less than or equal to 1
Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on pull request #33100:
URL: https://github.com/apache/spark/pull/33100#issuecomment-869126313
It seems that Jenkins is down. Could you re-trigger your GitHub Action?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #33100: [SPARK-35906][SQL] Remove order by if the maximum number of rows less than or equal to 1
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #33100:
URL: https://github.com/apache/spark/pull/33100#issuecomment-870826301
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/44896/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on pull request #33100: [SPARK-35906][SQL] Remove order by if the maximum number of rows less than or equal to 1
Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on pull request #33100:
URL: https://github.com/apache/spark/pull/33100#issuecomment-869059372
Could you check the UT failure?
```
SPARK-24556: always rewrite output partitioning in ReusedExchangeExec and InMemoryTableScanExec *** FAILED *** (40 milliseconds)
ArrayBuffer() had size 0 instead of expected size 1 (PlannerSuite.scala:746)
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] allisonwang-db commented on pull request #33100: [SPARK-35906][SQL] Remove order by if the maximum number of rows less than or equal to 1
Posted by GitBox <gi...@apache.org>.
allisonwang-db commented on pull request #33100:
URL: https://github.com/apache/spark/pull/33100#issuecomment-875074883
Hi @wangyum It seems that the `maxRow` check should be added in `recursiveRemoveSort` instead of `applyLocally`. Current the Once strategy's idempotency can be broken if the plan has two consecutive sorts that are removable:
```scala
Sort [a ASC]
+- Sort [a ASC]
```
Example query:
```sql
select a from (select a from values(0, 1) t(a, b) order by a) order by a
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on pull request #33100: [SPARK-35906][SQL] Remove order by if the maximum number of rows less than or equal to 1
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on pull request #33100:
URL: https://github.com/apache/spark/pull/33100#issuecomment-870859902
late LGTM
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #33100: [SPARK-35906][SQL] Remove order by if the maximum number of rows less than or equal to 1
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #33100:
URL: https://github.com/apache/spark/pull/33100#issuecomment-870826301
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/44896/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on pull request #33100: [SPARK-35906][SQL] Remove order by if the maximum number of rows less than or equal to 1
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on pull request #33100:
URL: https://github.com/apache/spark/pull/33100#issuecomment-875240629
Should we revert this, @wangyum and @allisonwang-db ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on pull request #33100: [SPARK-35906][SQL] Remove order by if the maximum number of rows less than or equal to 1
Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on pull request #33100:
URL: https://github.com/apache/spark/pull/33100#issuecomment-870804816
Thank you, @wangyum , @HyukjinKwon , @maropu , @huaxingao .
For the above comment (https://github.com/apache/spark/pull/33100#discussion_r659455547), we can discuss later since it's about refactoring.
Merged to master for Apache Spark 3.2.0.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on pull request #33100: [SPARK-35906][SQL] Remove order by if the maximum number of rows less than or equal to 1
Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on pull request #33100:
URL: https://github.com/apache/spark/pull/33100#issuecomment-869125794
Retest this please
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org