You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by GitBox <gi...@apache.org> on 2020/08/05 10:23:00 UTC
[GitHub] [spark] Karl-WangSK opened a new pull request #29360: [SPARK-32542][SQL]Add a batch for optimizing logicalPlan
Karl-WangSK opened a new pull request #29360:
URL: https://github.com/apache/spark/pull/29360
<!--
Thanks for sending a pull request! Here are some tips for you:
1. If this is your first time, please read our contributor guidelines: https://spark.apache.org/contributing.html
2. Ensure you have added or run the appropriate tests for your PR: https://spark.apache.org/developer-tools.html
3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][SPARK-XXXX] Your PR title ...'.
4. Be sure to keep the PR description updated to reflect all changes.
5. Please write your PR title to summarize what this PR proposes.
6. If possible, provide a concise example to reproduce the issue for a faster review.
7. If you want to add a new configuration, please read the guideline first for naming configurations in
'core/src/main/scala/org/apache/spark/internal/config/ConfigEntry.scala'.
-->
### What changes were proposed in this pull request?
This pr adds a batch in Optimizer for multidimensional analysis sql
### Why are the changes needed?
<!--
Please clarify why the changes are needed. For instance,
1. If you propose a new API, clarify the use case for a new API.
2. If you fix a bug, you can clarify why it is a bug.
-->
Split an expand into several smallExpand,which contains the Specified number of projections.
For instance,like this sql.select a,b,c,d,count(1) from table1 group by a,b,c,d with cube. It can expand 2^4 times of original data size.
Now we specify the spark.sql.optimizer.projections.size=4.The Expand will be split into 2^4/4 smallExpand.It can reduce the shuffle pressure and improve performance in multidimensional analysis when data is huge.
### Does this PR introduce _any_ user-facing change?
no
### How was this patch tested?
unit test
Authored-by: Karl-WangSK <ws...@qq.com>
Signed-off-by: Karl-WangSK <ws...@qq.com>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #29360: [SPARK-32542][SQL]Add a Batch in Optimizer to improve performance in multidimensional analysis
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29360:
URL: https://github.com/apache/spark/pull/29360#issuecomment-673352065
**[Test build #127404 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/127404/testReport)** for PR 29360 at commit [`4a1350c`](https://github.com/apache/spark/commit/4a1350c42a73721c392a5ff655e4fc8b639304f0).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #29360: [SPARK-32542][SQL]Add a Batch in Optimizer to improve performance in multidimensional analysis
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #29360:
URL: https://github.com/apache/spark/pull/29360#issuecomment-670384881
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #29360: [SPARK-32542][SQL]Add a Batch in Optimizer to improve performance in multidimensional analysis
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29360:
URL: https://github.com/apache/spark/pull/29360#issuecomment-670350289
Merged build finished. Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] maropu commented on pull request #29360: [SPARK-32542][SQL]Add a batch for optimizing logicalPlan
Posted by GitBox <gi...@apache.org>.
maropu commented on pull request #29360:
URL: https://github.com/apache/spark/pull/29360#issuecomment-669648276
Thanks for the contribution, @Karl-WangSK. btw, could you show us performance numbers with/without this fix in the PR description first?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #29360: [SPARK-32542][SQL]Add a batch for optimizing logicalPlan
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #29360:
URL: https://github.com/apache/spark/pull/29360#issuecomment-669667434
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] Karl-WangSK commented on pull request #29360: [SPARK-32542][SQL]Add a batch for optimizing logicalPlan
Posted by GitBox <gi...@apache.org>.
Karl-WangSK commented on pull request #29360:
URL: https://github.com/apache/spark/pull/29360#issuecomment-669963736
> @Karl-WangSK Why did you close this? If you gave up working on this, could we close the correspoinding jira, too?
sry for closing this pr. sql test failed. didn't find the reason. need time to fix the bugs.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #29360: [SPARK-32542][SQL]Add a Batch in Optimizer to improve performance in multidimensional analysis
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #29360:
URL: https://github.com/apache/spark/pull/29360#issuecomment-670334926
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #29360: [SPARK-32542][SQL]Add a batch for optimizing logicalPlan
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #29360:
URL: https://github.com/apache/spark/pull/29360#issuecomment-669113041
Can one of the admins verify this patch?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #29360: [SPARK-32542][SQL]Add a Batch in Optimizer to improve performance in multidimensional analysis
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #29360:
URL: https://github.com/apache/spark/pull/29360#issuecomment-671013827
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #29360: [SPARK-32542][SQL]Add a Batch in Optimizer to improve performance in multidimensional analysis
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29360:
URL: https://github.com/apache/spark/pull/29360#issuecomment-671194078
Merged build finished. Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #29360: [SPARK-32542][SQL]Add a Batch in Optimizer to improve performance in multidimensional analysis
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29360:
URL: https://github.com/apache/spark/pull/29360#issuecomment-670451551
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/127190/
Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #29360: [SPARK-32542][SQL]Add a Batch in Optimizer to improve performance in multidimensional analysis
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29360:
URL: https://github.com/apache/spark/pull/29360#issuecomment-671366334
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #29360: [SPARK-32542][SQL]Add a Batch in Optimizer to improve performance in multidimensional analysis
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #29360:
URL: https://github.com/apache/spark/pull/29360#issuecomment-670970688
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #29360: [SPARK-32542][SQL]Add a Batch in Optimizer to improve performance in multidimensional analysis
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #29360:
URL: https://github.com/apache/spark/pull/29360#issuecomment-671427770
**[Test build #127285 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/127285/testReport)** for PR 29360 at commit [`01670c2`](https://github.com/apache/spark/commit/01670c27c80b4c9de4c15c51c6ea84c5bb818a7e).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #29360: [SPARK-32542][SQL] Add an optimizer rule to split an Expand into multiple Expands for aggregates
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29360:
URL: https://github.com/apache/spark/pull/29360#issuecomment-675234148
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] Karl-WangSK commented on pull request #29360: [SPARK-32542][SQL] Add an optimizer rule to split an Expand into multiple Expands for aggregates
Posted by GitBox <gi...@apache.org>.
Karl-WangSK commented on pull request #29360:
URL: https://github.com/apache/spark/pull/29360#issuecomment-675515887
@cloud-fan @maropu Please take a look, thanks!
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #29360: [SPARK-32542][SQL]Add a batch for optimizing logicalPlan
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29360:
URL: https://github.com/apache/spark/pull/29360#issuecomment-669667063
**[Test build #127123 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/127123/testReport)** for PR 29360 at commit [`7f8680c`](https://github.com/apache/spark/commit/7f8680c2b80138fd4a97e4dd985f24ff35a6a540).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #29360: [SPARK-32542][SQL]Add a Batch in Optimizer to improve performance in multidimensional analysis
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29360:
URL: https://github.com/apache/spark/pull/29360#issuecomment-673352784
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #29360: [SPARK-32542][SQL] Add an optimizer rule to split an Expand into multiple Expands for aggregates
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #29360:
URL: https://github.com/apache/spark/pull/29360#issuecomment-675234148
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #29360: [SPARK-32542][SQL]Add a batch for optimizing logicalPlan
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29360:
URL: https://github.com/apache/spark/pull/29360#issuecomment-669649790
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #29360: [SPARK-32542][SQL]Add a Batch in Optimizer to improve performance in multidimensional analysis
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #29360:
URL: https://github.com/apache/spark/pull/29360#issuecomment-672004353
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #29360: [SPARK-32542][SQL]Add a Batch in Optimizer to improve performance in multidimensional analysis
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #29360:
URL: https://github.com/apache/spark/pull/29360#issuecomment-672003736
**[Test build #127332 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/127332/testReport)** for PR 29360 at commit [`5bda4ad`](https://github.com/apache/spark/commit/5bda4adbd94fd2f33f40ac7b210c6d6ab1774e08).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] Karl-WangSK commented on a change in pull request #29360: [SPARK-32542][SQL]Add a Batch in Optimizer to improve performance in multidimensional analysis
Posted by GitBox <gi...@apache.org>.
Karl-WangSK commented on a change in pull request #29360:
URL: https://github.com/apache/spark/pull/29360#discussion_r467544123
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/Optimizer.scala
##########
@@ -1823,3 +1824,32 @@ object OptimizeLimitZero extends Rule[LogicalPlan] {
empty(ll)
}
}
+
+/**
+ * Split [[Expand]] into several splitExpand if the projection size of Expand is larger
Review comment:
ok
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #29360: [SPARK-32542][SQL] Add an optimizer rule to split an Expand into multiple Expands for aggregates
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29360:
URL: https://github.com/apache/spark/pull/29360#issuecomment-679845465
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/127866/
Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #29360: [SPARK-32542][SQL] Add an optimizer rule to split an Expand into multiple Expands for aggregates
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29360:
URL: https://github.com/apache/spark/pull/29360#issuecomment-679470385
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #29360: [SPARK-32542][SQL] Add an optimizer rule to split an Expand into multiple Expands for aggregates
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #29360:
URL: https://github.com/apache/spark/pull/29360#issuecomment-678468481
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #29360: [SPARK-32542][SQL]Add a Batch in Optimizer to improve performance in multidimensional analysis
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29360:
URL: https://github.com/apache/spark/pull/29360#issuecomment-670394871
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #29360: [SPARK-32542][SQL]Add a batch for optimizing logicalPlan
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #29360:
URL: https://github.com/apache/spark/pull/29360#issuecomment-669671155
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] Karl-WangSK commented on pull request #29360: [SPARK-32542][SQL]Add a Batch in Optimizer to improve performance in multidimensional analysis
Posted by GitBox <gi...@apache.org>.
Karl-WangSK commented on pull request #29360:
URL: https://github.com/apache/spark/pull/29360#issuecomment-673382458
retest this please
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #29360: [SPARK-32542][SQL] Add an optimizer rule to split an Expand into multiple Expands for aggregates
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #29360:
URL: https://github.com/apache/spark/pull/29360#issuecomment-675546591
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #29360: [SPARK-32542][SQL]Add a Batch in Optimizer to improve performance in multidimensional analysis
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29360:
URL: https://github.com/apache/spark/pull/29360#issuecomment-670944935
**[Test build #127226 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/127226/testReport)** for PR 29360 at commit [`4e62422`](https://github.com/apache/spark/commit/4e6242271b3312cc5dff4c9fdf0ab98432e01c6e).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #29360: [SPARK-32542][SQL]Add a Batch in Optimizer to improve performance in multidimensional analysis
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29360:
URL: https://github.com/apache/spark/pull/29360#issuecomment-670459340
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/127198/
Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #29360: [SPARK-32542][SQL]Add a batch for optimizing logicalPlan
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #29360:
URL: https://github.com/apache/spark/pull/29360#issuecomment-669968776
Can one of the admins verify this patch?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #29360: [SPARK-32542][SQL]Add a Batch in Optimizer to improve performance in multidimensional analysis
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29360:
URL: https://github.com/apache/spark/pull/29360#issuecomment-671014588
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #29360: [SPARK-32542][SQL]Add a Batch in Optimizer to improve performance in multidimensional analysis
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #29360:
URL: https://github.com/apache/spark/pull/29360#issuecomment-671022082
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #29360: [SPARK-32542][SQL]Add a Batch in Optimizer to improve performance in multidimensional analysis
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #29360:
URL: https://github.com/apache/spark/pull/29360#issuecomment-670465144
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] Karl-WangSK closed pull request #29360: [SPARK-32542][SQL]Add a batch for optimizing logicalPlan
Posted by GitBox <gi...@apache.org>.
Karl-WangSK closed pull request #29360:
URL: https://github.com/apache/spark/pull/29360
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #29360: [SPARK-32542][SQL] Add an optimizer rule to split an Expand into multiple Expands for aggregates
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29360:
URL: https://github.com/apache/spark/pull/29360#issuecomment-679845460
Merged build finished. Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #29360: [SPARK-32542][SQL]Add a Batch in Optimizer to improve performance in multidimensional analysis
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29360:
URL: https://github.com/apache/spark/pull/29360#issuecomment-670384159
**[Test build #127189 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/127189/testReport)** for PR 29360 at commit [`e04d513`](https://github.com/apache/spark/commit/e04d5139163897a7fade1fc3a79b36a414744335).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #29360: [SPARK-32542][SQL]Add a Batch in Optimizer to improve performance in multidimensional analysis
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29360:
URL: https://github.com/apache/spark/pull/29360#issuecomment-670459320
**[Test build #127198 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/127198/testReport)** for PR 29360 at commit [`ce20579`](https://github.com/apache/spark/commit/ce20579f7f1e924f3f6d8a66f005c77aa0fc689f).
* This patch **fails Scala style tests**.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #29360: [SPARK-32542][SQL]Add a Batch in Optimizer to improve performance in multidimensional analysis
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #29360:
URL: https://github.com/apache/spark/pull/29360#issuecomment-673568012
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #29360: [SPARK-32542][SQL]Add a batch for optimizing logicalPlan
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29360:
URL: https://github.com/apache/spark/pull/29360#issuecomment-669667430
**[Test build #127123 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/127123/testReport)** for PR 29360 at commit [`7f8680c`](https://github.com/apache/spark/commit/7f8680c2b80138fd4a97e4dd985f24ff35a6a540).
* This patch **fails Scala style tests**.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #29360: [SPARK-32542][SQL]Add a Batch in Optimizer to improve performance in multidimensional analysis
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29360:
URL: https://github.com/apache/spark/pull/29360#issuecomment-673468488
**[Test build #127404 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/127404/testReport)** for PR 29360 at commit [`4a1350c`](https://github.com/apache/spark/commit/4a1350c42a73721c392a5ff655e4fc8b639304f0).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #29360: [SPARK-32542][SQL]Add a Batch in Optimizer to improve performance in multidimensional analysis
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29360:
URL: https://github.com/apache/spark/pull/29360#issuecomment-670930230
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #29360: [SPARK-32542][SQL]Add a Batch in Optimizer to improve performance in multidimensional analysis
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29360:
URL: https://github.com/apache/spark/pull/29360#issuecomment-670334926
Merged build finished. Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #29360: [SPARK-32542][SQL]Add a Batch in Optimizer to improve performance in multidimensional analysis
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29360:
URL: https://github.com/apache/spark/pull/29360#issuecomment-670459118
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #29360: [SPARK-32542][SQL]Add a Batch in Optimizer to improve performance in multidimensional analysis
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #29360:
URL: https://github.com/apache/spark/pull/29360#issuecomment-673432002
**[Test build #127409 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/127409/testReport)** for PR 29360 at commit [`5c2e630`](https://github.com/apache/spark/commit/5c2e630fd83a9b32af4e471401d7bf47823750a5).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #29360: [SPARK-32542][SQL]Add a Batch in Optimizer to improve performance in multidimensional analysis
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29360:
URL: https://github.com/apache/spark/pull/29360#issuecomment-671164411
**[Test build #127264 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/127264/testReport)** for PR 29360 at commit [`f904eab`](https://github.com/apache/spark/commit/f904eabd3fa986b907e0d0d32976f9d815d27b9c).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #29360: [SPARK-32542][SQL]Add a Batch in Optimizer to improve performance in multidimensional analysis
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29360:
URL: https://github.com/apache/spark/pull/29360#issuecomment-670465144
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #29360: [SPARK-32542][SQL]Add a Batch in Optimizer to improve performance in multidimensional analysis
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #29360:
URL: https://github.com/apache/spark/pull/29360#issuecomment-670930106
**[Test build #127223 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/127223/testReport)** for PR 29360 at commit [`bf73a5c`](https://github.com/apache/spark/commit/bf73a5c997ae3eeaeff712cd368c076e38b12684).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #29360: [SPARK-32542][SQL]Add a Batch in Optimizer to improve performance in multidimensional analysis
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29360:
URL: https://github.com/apache/spark/pull/29360#issuecomment-671022082
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #29360: [SPARK-32542][SQL]Add a Batch in Optimizer to improve performance in multidimensional analysis
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29360:
URL: https://github.com/apache/spark/pull/29360#issuecomment-671013934
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #29360: [SPARK-32542][SQL]Add a Batch in Optimizer to improve performance in multidimensional analysis
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #29360:
URL: https://github.com/apache/spark/pull/29360#issuecomment-669977687
**[Test build #127147 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/127147/testReport)** for PR 29360 at commit [`db8290c`](https://github.com/apache/spark/commit/db8290ccdbc2a4b82309b2d10b5e6f26a1041a36).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #29360: [SPARK-32542][SQL]Add a Batch in Optimizer to improve performance in multidimensional analysis
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29360:
URL: https://github.com/apache/spark/pull/29360#issuecomment-671412696
Merged build finished. Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #29360: [SPARK-32542][SQL] Add an optimizer rule to split an Expand into multiple Expands for aggregates
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #29360:
URL: https://github.com/apache/spark/pull/29360#issuecomment-678335806
**[Test build #127745 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/127745/testReport)** for PR 29360 at commit [`039ba05`](https://github.com/apache/spark/commit/039ba05ca3b2c5d328290d9c67b5a629df6d9687).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #29360: [SPARK-32542][SQL]Add a Batch in Optimizer to improve performance in multidimensional analysis
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #29360:
URL: https://github.com/apache/spark/pull/29360#issuecomment-673352784
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] Karl-WangSK commented on pull request #29360: [SPARK-32542][SQL] Add an optimizer rule to split an Expand into multiple Expands for aggregates
Posted by GitBox <gi...@apache.org>.
Karl-WangSK commented on pull request #29360:
URL: https://github.com/apache/spark/pull/29360#issuecomment-686560052
ready to merge if no other problems @LuciferYang Thanks!
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #29360: [SPARK-32542][SQL] Add an optimizer rule to split an Expand into multiple Expands for aggregates
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29360:
URL: https://github.com/apache/spark/pull/29360#issuecomment-675297167
**[Test build #127526 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/127526/testReport)** for PR 29360 at commit [`87b9a82`](https://github.com/apache/spark/commit/87b9a825359168eb07fe5f9791e1dc26ce138046).
* This patch **fails due to an unknown error code, -9**.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #29360: [SPARK-32542][SQL]Add a Batch in Optimizer to improve performance in multidimensional analysis
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29360:
URL: https://github.com/apache/spark/pull/29360#issuecomment-672012047
Merged build finished. Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #29360: [SPARK-32542][SQL]Add a Batch in Optimizer to improve performance in multidimensional analysis
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29360:
URL: https://github.com/apache/spark/pull/29360#issuecomment-672012063
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/127333/
Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #29360: [SPARK-32542][SQL]Add a Batch in Optimizer to improve performance in multidimensional analysis
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #29360:
URL: https://github.com/apache/spark/pull/29360#issuecomment-671045408
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #29360: [SPARK-32542][SQL]Add a Batch in Optimizer to improve performance in multidimensional analysis
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #29360:
URL: https://github.com/apache/spark/pull/29360#issuecomment-670426547
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #29360: [SPARK-32542][SQL]Add a Batch in Optimizer to improve performance in multidimensional analysis
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29360:
URL: https://github.com/apache/spark/pull/29360#issuecomment-671370232
**[Test build #127282 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/127282/testReport)** for PR 29360 at commit [`3a2901e`](https://github.com/apache/spark/commit/3a2901e2b4deef99decfc811a9d00ca7135bff39).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #29360: [SPARK-32542][SQL]Add a Batch in Optimizer to improve performance in multidimensional analysis
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29360:
URL: https://github.com/apache/spark/pull/29360#issuecomment-672008431
**[Test build #127333 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/127333/testReport)** for PR 29360 at commit [`7a748c2`](https://github.com/apache/spark/commit/7a748c284a54d0aa4a776215254f98d89d25cb33).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #29360: [SPARK-32542][SQL]Add a Batch in Optimizer to improve performance in multidimensional analysis
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29360:
URL: https://github.com/apache/spark/pull/29360#issuecomment-670470687
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] Karl-WangSK commented on pull request #29360: [SPARK-32542][SQL] Add an optimizer rule to split an Expand into multiple Expands for aggregates
Posted by GitBox <gi...@apache.org>.
Karl-WangSK commented on pull request #29360:
URL: https://github.com/apache/spark/pull/29360#issuecomment-674625909
hi. Can anyone deal with this pr?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #29360: [SPARK-32542][SQL]Add a Batch in Optimizer to improve performance in multidimensional analysis
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29360:
URL: https://github.com/apache/spark/pull/29360#issuecomment-671456297
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/127285/
Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #29360: [SPARK-32542][SQL]Add a Batch in Optimizer to improve performance in multidimensional analysis
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29360:
URL: https://github.com/apache/spark/pull/29360#issuecomment-672248376
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #29360: [SPARK-32542][SQL]Add a Batch in Optimizer to improve performance in multidimensional analysis
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29360:
URL: https://github.com/apache/spark/pull/29360#issuecomment-670384890
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/127189/
Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] Karl-WangSK commented on pull request #29360: [SPARK-32542][SQL]Add a Batch in Optimizer to improve performance in multidimensional analysis
Posted by GitBox <gi...@apache.org>.
Karl-WangSK commented on pull request #29360:
URL: https://github.com/apache/spark/pull/29360#issuecomment-673495331
@viirya @maropu
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #29360: [SPARK-32542][SQL] Add an optimizer rule to split an Expand into multiple Expands for aggregates
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #29360:
URL: https://github.com/apache/spark/pull/29360#issuecomment-675233817
**[Test build #127526 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/127526/testReport)** for PR 29360 at commit [`87b9a82`](https://github.com/apache/spark/commit/87b9a825359168eb07fe5f9791e1dc26ce138046).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #29360: [SPARK-32542][SQL]Add a batch for optimizing logicalPlan
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #29360:
URL: https://github.com/apache/spark/pull/29360#issuecomment-669669258
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #29360: [SPARK-32542][SQL]Add a Batch in Optimizer to improve performance in multidimensional analysis
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #29360:
URL: https://github.com/apache/spark/pull/29360#issuecomment-670381299
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #29360: [SPARK-32542][SQL]Add a Batch in Optimizer to improve performance in multidimensional analysis
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #29360:
URL: https://github.com/apache/spark/pull/29360#issuecomment-670930872
**[Test build #127224 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/127224/testReport)** for PR 29360 at commit [`6cf396d`](https://github.com/apache/spark/commit/6cf396d874ba004a0694aa7d66c639afab873173).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #29360: [SPARK-32542][SQL]Add a Batch in Optimizer to improve performance in multidimensional analysis
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29360:
URL: https://github.com/apache/spark/pull/29360#issuecomment-670394338
**[Test build #127190 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/127190/testReport)** for PR 29360 at commit [`d34b089`](https://github.com/apache/spark/commit/d34b08961a2da012cd01ddfa01febc84115b66cf).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #29360: [SPARK-32542][SQL]Add a Batch in Optimizer to improve performance in multidimensional analysis
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29360:
URL: https://github.com/apache/spark/pull/29360#issuecomment-670425426
**[Test build #127194 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/127194/testReport)** for PR 29360 at commit [`34a76d1`](https://github.com/apache/spark/commit/34a76d1633cdb53c70cb20f89d403816317d46cf).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #29360: [SPARK-32542][SQL]Add a Batch in Optimizer to improve performance in multidimensional analysis
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #29360:
URL: https://github.com/apache/spark/pull/29360#issuecomment-671194078
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #29360: [SPARK-32542][SQL]Add a Batch in Optimizer to improve performance in multidimensional analysis
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #29360:
URL: https://github.com/apache/spark/pull/29360#issuecomment-671370232
**[Test build #127282 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/127282/testReport)** for PR 29360 at commit [`3a2901e`](https://github.com/apache/spark/commit/3a2901e2b4deef99decfc811a9d00ca7135bff39).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #29360: [SPARK-32542][SQL]Add a Batch in Optimizer to improve performance in multidimensional analysis
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29360:
URL: https://github.com/apache/spark/pull/29360#issuecomment-670930872
**[Test build #127224 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/127224/testReport)** for PR 29360 at commit [`6cf396d`](https://github.com/apache/spark/commit/6cf396d874ba004a0694aa7d66c639afab873173).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #29360: [SPARK-32542][SQL]Add a Batch in Optimizer to improve performance in multidimensional analysis
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29360:
URL: https://github.com/apache/spark/pull/29360#issuecomment-672009312
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] maropu commented on pull request #29360: [SPARK-32542][SQL]Add a Batch in Optimizer to improve performance in multidimensional analysis
Posted by GitBox <gi...@apache.org>.
maropu commented on pull request #29360:
URL: https://github.com/apache/spark/pull/29360#issuecomment-673753587
> But shuffle is happened during Aggregate here, right? By splitting, the total amount of shuffled data is not changed, but split into several ones. Does it really result significant improvement?
As @viirya said above, I think the same. Why can this reduce the amount of shuffle writes? In the case of `expand -> partial aggregates`, the aggregates seem to have the same **total** amount of output size.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #29360: [SPARK-32542][SQL]Add a Batch in Optimizer to improve performance in multidimensional analysis
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29360:
URL: https://github.com/apache/spark/pull/29360#issuecomment-670338755
**[Test build #127178 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/127178/testReport)** for PR 29360 at commit [`396e16d`](https://github.com/apache/spark/commit/396e16daae99e8c764217ce0a69eb9c80459e631).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #29360: [SPARK-32542][SQL]Add a Batch in Optimizer to improve performance in multidimensional analysis
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #29360:
URL: https://github.com/apache/spark/pull/29360#issuecomment-670384159
**[Test build #127189 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/127189/testReport)** for PR 29360 at commit [`e04d513`](https://github.com/apache/spark/commit/e04d5139163897a7fade1fc3a79b36a414744335).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #29360: [SPARK-32542][SQL]Add a Batch in Optimizer to improve performance in multidimensional analysis
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29360:
URL: https://github.com/apache/spark/pull/29360#issuecomment-670467876
**[Test build #127201 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/127201/testReport)** for PR 29360 at commit [`b87ec37`](https://github.com/apache/spark/commit/b87ec370957a008984bdc0db7f62d9783e21f4d8).
* This patch **fails to build**.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #29360: [SPARK-32542][SQL]Add a Batch in Optimizer to improve performance in multidimensional analysis
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29360:
URL: https://github.com/apache/spark/pull/29360#issuecomment-673429679
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #29360: [SPARK-32542][SQL]Add a Batch in Optimizer to improve performance in multidimensional analysis
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #29360:
URL: https://github.com/apache/spark/pull/29360#issuecomment-670062420
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #29360: [SPARK-32542][SQL]Add a Batch in Optimizer to improve performance in multidimensional analysis
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29360:
URL: https://github.com/apache/spark/pull/29360#issuecomment-671016255
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #29360: [SPARK-32542][SQL]Add a Batch in Optimizer to improve performance in multidimensional analysis
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #29360:
URL: https://github.com/apache/spark/pull/29360#issuecomment-673469505
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #29360: [SPARK-32542][SQL]Add a batch for optimizing logicalPlan
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #29360:
URL: https://github.com/apache/spark/pull/29360#issuecomment-669667063
**[Test build #127123 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/127123/testReport)** for PR 29360 at commit [`7f8680c`](https://github.com/apache/spark/commit/7f8680c2b80138fd4a97e4dd985f24ff35a6a540).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #29360: [SPARK-32542][SQL]Add a Batch in Optimizer to improve performance in multidimensional analysis
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29360:
URL: https://github.com/apache/spark/pull/29360#issuecomment-670533469
**[Test build #127202 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/127202/testReport)** for PR 29360 at commit [`3899943`](https://github.com/apache/spark/commit/3899943e679c5db56ff907619bb01759b70025b6).
* This patch **fails Spark unit tests**.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #29360: [SPARK-32542][SQL]Add a Batch in Optimizer to improve performance in multidimensional analysis
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29360:
URL: https://github.com/apache/spark/pull/29360#issuecomment-670334913
**[Test build #127177 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/127177/testReport)** for PR 29360 at commit [`758ead7`](https://github.com/apache/spark/commit/758ead769659508731e724b52437cf6811f3aeed).
* This patch **fails Scala style tests**.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #29360: [SPARK-32542][SQL]Add a batch for optimizing logicalPlan
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #29360:
URL: https://github.com/apache/spark/pull/29360#issuecomment-669672688
**[Test build #127126 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/127126/testReport)** for PR 29360 at commit [`db8290c`](https://github.com/apache/spark/commit/db8290ccdbc2a4b82309b2d10b5e6f26a1041a36).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #29360: [SPARK-32542][SQL]Add a Batch in Optimizer to improve performance in multidimensional analysis
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #29360:
URL: https://github.com/apache/spark/pull/29360#issuecomment-671196967
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] maropu commented on a change in pull request #29360: [SPARK-32542][SQL]Add a batch for optimizing logicalPlan
Posted by GitBox <gi...@apache.org>.
maropu commented on a change in pull request #29360:
URL: https://github.com/apache/spark/pull/29360#discussion_r466111474
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/Optimizer.scala
##########
@@ -1823,3 +1824,33 @@ object OptimizeLimitZero extends Rule[LogicalPlan] {
empty(ll)
}
}
+
Review comment:
nit: remove this single blank.
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala
##########
@@ -2699,6 +2712,7 @@ object SQLConf {
.booleanConf
.createWithDefault(false)
+
Review comment:
nit: remove this.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #29360: [SPARK-32542][SQL]Add a Batch in Optimizer to improve performance in multidimensional analysis
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29360:
URL: https://github.com/apache/spark/pull/29360#issuecomment-670384862
**[Test build #127189 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/127189/testReport)** for PR 29360 at commit [`e04d513`](https://github.com/apache/spark/commit/e04d5139163897a7fade1fc3a79b36a414744335).
* This patch **fails Scala style tests**.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #29360: [SPARK-32542][SQL]Add a Batch in Optimizer to improve performance in multidimensional analysis
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #29360:
URL: https://github.com/apache/spark/pull/29360#issuecomment-670394871
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #29360: [SPARK-32542][SQL]Add a Batch in Optimizer to improve performance in multidimensional analysis
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #29360:
URL: https://github.com/apache/spark/pull/29360#issuecomment-671267047
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #29360: [SPARK-32542][SQL]Add a batch for optimizing logicalPlan
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29360:
URL: https://github.com/apache/spark/pull/29360#issuecomment-669113698
Can one of the admins verify this patch?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #29360: [SPARK-32542][SQL]Add a Batch in Optimizer to improve performance in multidimensional analysis
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #29360:
URL: https://github.com/apache/spark/pull/29360#issuecomment-671173465
**[Test build #127266 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/127266/testReport)** for PR 29360 at commit [`82a1d4c`](https://github.com/apache/spark/commit/82a1d4c91f52a6ce523f1c0aa3703172a42e72b6).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #29360: [SPARK-32542][SQL]Add a Batch in Optimizer to improve performance in multidimensional analysis
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #29360:
URL: https://github.com/apache/spark/pull/29360#issuecomment-671456287
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #29360: [SPARK-32542][SQL]Add a Batch in Optimizer to improve performance in multidimensional analysis
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29360:
URL: https://github.com/apache/spark/pull/29360#issuecomment-671014481
**[Test build #127236 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/127236/testReport)** for PR 29360 at commit [`c1b318a`](https://github.com/apache/spark/commit/c1b318a3b6cc458f3b3ab4d4a0e2ec7dabb3014c).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #29360: [SPARK-32542][SQL]Add a Batch in Optimizer to improve performance in multidimensional analysis
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #29360:
URL: https://github.com/apache/spark/pull/29360#issuecomment-670945092
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #29360: [SPARK-32542][SQL]Add a Batch in Optimizer to improve performance in multidimensional analysis
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29360:
URL: https://github.com/apache/spark/pull/29360#issuecomment-673469505
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #29360: [SPARK-32542][SQL]Add a Batch in Optimizer to improve performance in multidimensional analysis
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #29360:
URL: https://github.com/apache/spark/pull/29360#issuecomment-670944579
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #29360: [SPARK-32542][SQL]Add a Batch in Optimizer to improve performance in multidimensional analysis
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29360:
URL: https://github.com/apache/spark/pull/29360#issuecomment-670952749
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/127223/
Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #29360: [SPARK-32542][SQL]Add a Batch in Optimizer to improve performance in multidimensional analysis
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29360:
URL: https://github.com/apache/spark/pull/29360#issuecomment-670944579
Merged build finished. Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] maropu commented on a change in pull request #29360: [SPARK-32542][SQL]Add a batch for optimizing logicalPlan
Posted by GitBox <gi...@apache.org>.
maropu commented on a change in pull request #29360:
URL: https://github.com/apache/spark/pull/29360#discussion_r466111375
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala
##########
@@ -2690,6 +2690,19 @@ object SQLConf {
.booleanConf
.createWithDefault(true)
+
+ val GROUPING_WITH_UNION = buildConf("spark.sql.optimizer.grouping")
+ .doc("When true,enable grouping with union.")
+ .booleanConf
+ .createWithDefault(false)
+
+ val GROUPING_EXPAND_PROJECTIONS = buildConf("spark.sql.optimizer.projections.size")
+ .doc("Split a expand into several smallExpand,which contains the size of projections.It can" +
+ "reduce the shuffle pressure and improve performance in multidimensional analysis " +
+ "when data is huge.")
+ .intConf
+ .createWithDefault(4)
Review comment:
Could you check if all the existing tests pass if `spark.sql.optimizer.grouping`=true and `spark.sql.optimizer.projections.size`=1?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #29360: [SPARK-32542][SQL]Add a Batch in Optimizer to improve performance in multidimensional analysis
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #29360:
URL: https://github.com/apache/spark/pull/29360#issuecomment-672057796
**[Test build #127335 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/127335/testReport)** for PR 29360 at commit [`2c1de30`](https://github.com/apache/spark/commit/2c1de3089b0a16c9119688ebd4487a9db2377e47).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #29360: [SPARK-32542][SQL]Add a Batch in Optimizer to improve performance in multidimensional analysis
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #29360:
URL: https://github.com/apache/spark/pull/29360#issuecomment-670464688
**[Test build #127201 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/127201/testReport)** for PR 29360 at commit [`b87ec37`](https://github.com/apache/spark/commit/b87ec370957a008984bdc0db7f62d9783e21f4d8).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] Karl-WangSK removed a comment on pull request #29360: [SPARK-32542][SQL] Add an optimizer rule to split an Expand into multiple Expands for aggregates
Posted by GitBox <gi...@apache.org>.
Karl-WangSK removed a comment on pull request #29360:
URL: https://github.com/apache/spark/pull/29360#issuecomment-680740923
retest this please
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #29360: [SPARK-32542][SQL]Add a Batch in Optimizer to improve performance in multidimensional analysis
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29360:
URL: https://github.com/apache/spark/pull/29360#issuecomment-671456109
**[Test build #127285 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/127285/testReport)** for PR 29360 at commit [`01670c2`](https://github.com/apache/spark/commit/01670c27c80b4c9de4c15c51c6ea84c5bb818a7e).
* This patch **fails Spark unit tests**.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #29360: [SPARK-32542][SQL] Add an optimizer rule to split an Expand into multiple Expands for aggregates
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29360:
URL: https://github.com/apache/spark/pull/29360#issuecomment-678340352
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #29360: [SPARK-32542][SQL]Add a Batch in Optimizer to improve performance in multidimensional analysis
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29360:
URL: https://github.com/apache/spark/pull/29360#issuecomment-671194083
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/127266/
Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #29360: [SPARK-32542][SQL] Add an optimizer rule to split an Expand into multiple Expands for aggregates
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #29360:
URL: https://github.com/apache/spark/pull/29360#issuecomment-675392085
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #29360: [SPARK-32542][SQL] Add an optimizer rule to split an Expand into multiple Expands for aggregates
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29360:
URL: https://github.com/apache/spark/pull/29360#issuecomment-675391574
**[Test build #127564 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/127564/testReport)** for PR 29360 at commit [`e17fa60`](https://github.com/apache/spark/commit/e17fa605684ae114973a3c34a58239d3a9f8dc52).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #29360: [SPARK-32542][SQL]Add a Batch in Optimizer to improve performance in multidimensional analysis
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29360:
URL: https://github.com/apache/spark/pull/29360#issuecomment-673567010
**[Test build #127409 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/127409/testReport)** for PR 29360 at commit [`5c2e630`](https://github.com/apache/spark/commit/5c2e630fd83a9b32af4e471401d7bf47823750a5).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #29360: [SPARK-32542][SQL]Add a Batch in Optimizer to improve performance in multidimensional analysis
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29360:
URL: https://github.com/apache/spark/pull/29360#issuecomment-672247240
**[Test build #127332 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/127332/testReport)** for PR 29360 at commit [`5bda4ad`](https://github.com/apache/spark/commit/5bda4adbd94fd2f33f40ac7b210c6d6ab1774e08).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #29360: [SPARK-32542][SQL]Add a Batch in Optimizer to improve performance in multidimensional analysis
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29360:
URL: https://github.com/apache/spark/pull/29360#issuecomment-672233491
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/127335/
Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #29360: [SPARK-32542][SQL]Add a Batch in Optimizer to improve performance in multidimensional analysis
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #29360:
URL: https://github.com/apache/spark/pull/29360#issuecomment-672248376
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #29360: [SPARK-32542][SQL]Add a Batch in Optimizer to improve performance in multidimensional analysis
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #29360:
URL: https://github.com/apache/spark/pull/29360#issuecomment-670944935
**[Test build #127226 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/127226/testReport)** for PR 29360 at commit [`4e62422`](https://github.com/apache/spark/commit/4e6242271b3312cc5dff4c9fdf0ab98432e01c6e).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #29360: [SPARK-32542][SQL]Add a Batch in Optimizer to improve performance in multidimensional analysis
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29360:
URL: https://github.com/apache/spark/pull/29360#issuecomment-671162815
**[Test build #127263 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/127263/testReport)** for PR 29360 at commit [`7519afc`](https://github.com/apache/spark/commit/7519afcac4bd712b9a017d1fe56ad5a6547eddf0).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] viirya commented on a change in pull request #29360: [SPARK-32542][SQL]Add a Batch in Optimizer to improve performance in multidimensional analysis
Posted by GitBox <gi...@apache.org>.
viirya commented on a change in pull request #29360:
URL: https://github.com/apache/spark/pull/29360#discussion_r467544147
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala
##########
@@ -2690,6 +2690,18 @@ object SQLConf {
.booleanConf
.createWithDefault(true)
+ val GROUPING_WITH_UNION = buildConf("spark.sql.optimizer.grouping")
+ .doc("When true, enable grouping with union.")
+ .booleanConf
+ .createWithDefault(false)
+
+ val GROUPING_EXPAND_PROJECTIONS = buildConf("spark.sql.optimizer.projections.size")
+ .doc("Split a expand into several smallExpand, which contains the size of projections. It " +
Review comment:
expand -> Expand
smallExpand -> small Expand
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #29360: [SPARK-32542][SQL]Add a Batch in Optimizer to improve performance in multidimensional analysis
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29360:
URL: https://github.com/apache/spark/pull/29360#issuecomment-670334251
**[Test build #127177 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/127177/testReport)** for PR 29360 at commit [`758ead7`](https://github.com/apache/spark/commit/758ead769659508731e724b52437cf6811f3aeed).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] Karl-WangSK removed a comment on pull request #29360: [SPARK-32542][SQL]Add a Batch in Optimizer to improve performance in multidimensional analysis
Posted by GitBox <gi...@apache.org>.
Karl-WangSK removed a comment on pull request #29360:
URL: https://github.com/apache/spark/pull/29360#issuecomment-673426929
retest this please
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #29360: [SPARK-32542][SQL]Add a Batch in Optimizer to improve performance in multidimensional analysis
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #29360:
URL: https://github.com/apache/spark/pull/29360#issuecomment-672059164
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #29360: [SPARK-32542][SQL]Add a Batch in Optimizer to improve performance in multidimensional analysis
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29360:
URL: https://github.com/apache/spark/pull/29360#issuecomment-672059164
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #29360: [SPARK-32542][SQL]Add a Batch in Optimizer to improve performance in multidimensional analysis
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29360:
URL: https://github.com/apache/spark/pull/29360#issuecomment-670426557
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/127194/
Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #29360: [SPARK-32542][SQL]Add a Batch in Optimizer to improve performance in multidimensional analysis
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #29360:
URL: https://github.com/apache/spark/pull/29360#issuecomment-671021931
**[Test build #127237 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/127237/testReport)** for PR 29360 at commit [`f904eab`](https://github.com/apache/spark/commit/f904eabd3fa986b907e0d0d32976f9d815d27b9c).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #29360: [SPARK-32542][SQL]Add a Batch in Optimizer to improve performance in multidimensional analysis
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #29360:
URL: https://github.com/apache/spark/pull/29360#issuecomment-670334251
**[Test build #127177 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/127177/testReport)** for PR 29360 at commit [`758ead7`](https://github.com/apache/spark/commit/758ead769659508731e724b52437cf6811f3aeed).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #29360: [SPARK-32542][SQL]Add a Batch in Optimizer to improve performance in multidimensional analysis
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29360:
URL: https://github.com/apache/spark/pull/29360#issuecomment-670458611
**[Test build #127198 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/127198/testReport)** for PR 29360 at commit [`ce20579`](https://github.com/apache/spark/commit/ce20579f7f1e924f3f6d8a66f005c77aa0fc689f).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #29360: [SPARK-32542][SQL]Add a Batch in Optimizer to improve performance in multidimensional analysis
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29360:
URL: https://github.com/apache/spark/pull/29360#issuecomment-670451541
Merged build finished. Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #29360: [SPARK-32542][SQL]Add a Batch in Optimizer to improve performance in multidimensional analysis
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29360:
URL: https://github.com/apache/spark/pull/29360#issuecomment-671267047
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #29360: [SPARK-32542][SQL]Add a batch for optimizing logicalPlan
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #29360:
URL: https://github.com/apache/spark/pull/29360#issuecomment-669668870
**[Test build #127124 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/127124/testReport)** for PR 29360 at commit [`a71e96b`](https://github.com/apache/spark/commit/a71e96bf3c22d979d15b8b373c262bdf6c6e4cf9).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #29360: [SPARK-32542][SQL]Add a Batch in Optimizer to improve performance in multidimensional analysis
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29360:
URL: https://github.com/apache/spark/pull/29360#issuecomment-671427770
**[Test build #127285 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/127285/testReport)** for PR 29360 at commit [`01670c2`](https://github.com/apache/spark/commit/01670c27c80b4c9de4c15c51c6ea84c5bb818a7e).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #29360: [SPARK-32542][SQL]Add a Batch in Optimizer to improve performance in multidimensional analysis
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29360:
URL: https://github.com/apache/spark/pull/29360#issuecomment-671456287
Merged build finished. Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #29360: [SPARK-32542][SQL] Add an optimizer rule to split an Expand into multiple Expands for aggregates
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #29360:
URL: https://github.com/apache/spark/pull/29360#issuecomment-678340352
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #29360: [SPARK-32542][SQL] Add an optimizer rule to split an Expand into multiple Expands for aggregates
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29360:
URL: https://github.com/apache/spark/pull/29360#issuecomment-675297814
Merged build finished. Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #29360: [SPARK-32542][SQL]Add a Batch in Optimizer to improve performance in multidimensional analysis
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29360:
URL: https://github.com/apache/spark/pull/29360#issuecomment-671412707
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/127282/
Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #29360: [SPARK-32542][SQL]Add a Batch in Optimizer to improve performance in multidimensional analysis
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #29360:
URL: https://github.com/apache/spark/pull/29360#issuecomment-672009312
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #29360: [SPARK-32542][SQL]Add a Batch in Optimizer to improve performance in multidimensional analysis
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29360:
URL: https://github.com/apache/spark/pull/29360#issuecomment-670464688
**[Test build #127201 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/127201/testReport)** for PR 29360 at commit [`b87ec37`](https://github.com/apache/spark/commit/b87ec370957a008984bdc0db7f62d9783e21f4d8).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #29360: [SPARK-32542][SQL] Add an optimizer rule to split an Expand into multiple Expands for aggregates
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29360:
URL: https://github.com/apache/spark/pull/29360#issuecomment-675544580
**[Test build #127564 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/127564/testReport)** for PR 29360 at commit [`e17fa60`](https://github.com/apache/spark/commit/e17fa605684ae114973a3c34a58239d3a9f8dc52).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #29360: [SPARK-32542][SQL]Add a batch for optimizing logicalPlan
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29360:
URL: https://github.com/apache/spark/pull/29360#issuecomment-669667436
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/127123/
Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #29360: [SPARK-32542][SQL]Add a Batch in Optimizer to improve performance in multidimensional analysis
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29360:
URL: https://github.com/apache/spark/pull/29360#issuecomment-669977687
**[Test build #127147 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/127147/testReport)** for PR 29360 at commit [`db8290c`](https://github.com/apache/spark/commit/db8290ccdbc2a4b82309b2d10b5e6f26a1041a36).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #29360: [SPARK-32542][SQL]Add a batch for optimizing logicalPlan
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29360:
URL: https://github.com/apache/spark/pull/29360#issuecomment-669669247
**[Test build #127124 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/127124/testReport)** for PR 29360 at commit [`a71e96b`](https://github.com/apache/spark/commit/a71e96bf3c22d979d15b8b373c262bdf6c6e4cf9).
* This patch **fails Scala style tests**.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #29360: [SPARK-32542][SQL]Add a Batch in Optimizer to improve performance in multidimensional analysis
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29360:
URL: https://github.com/apache/spark/pull/29360#issuecomment-670944582
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/127224/
Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #29360: [SPARK-32542][SQL]Add a Batch in Optimizer to improve performance in multidimensional analysis
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29360:
URL: https://github.com/apache/spark/pull/29360#issuecomment-670534019
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/127202/
Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] viirya commented on a change in pull request #29360: [SPARK-32542][SQL]Add a Batch in Optimizer to improve performance in multidimensional analysis
Posted by GitBox <gi...@apache.org>.
viirya commented on a change in pull request #29360:
URL: https://github.com/apache/spark/pull/29360#discussion_r467543947
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/Optimizer.scala
##########
@@ -1823,3 +1824,32 @@ object OptimizeLimitZero extends Rule[LogicalPlan] {
empty(ll)
}
}
+
+/**
+ * Split [[Expand]] into several splitExpand if the projection size of Expand is larger
Review comment:
I think you mean "Split [[Expand]] into several [[Expand]] ...". There isn't a class called `splitExpand`.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] Karl-WangSK removed a comment on pull request #29360: [SPARK-32542][SQL] Add an optimizer rule to split an Expand into multiple Expands for aggregates
Posted by GitBox <gi...@apache.org>.
Karl-WangSK removed a comment on pull request #29360:
URL: https://github.com/apache/spark/pull/29360#issuecomment-686560052
ready to merge if no other problems @LuciferYang Thanks!
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] Karl-WangSK commented on a change in pull request #29360: [SPARK-32542][SQL]Add a Batch in Optimizer to improve performance in multidimensional analysis
Posted by GitBox <gi...@apache.org>.
Karl-WangSK commented on a change in pull request #29360:
URL: https://github.com/apache/spark/pull/29360#discussion_r468955369
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/Optimizer.scala
##########
@@ -1823,3 +1824,32 @@ object OptimizeLimitZero extends Rule[LogicalPlan] {
empty(ll)
}
}
+
+/**
+ * Split [[Expand]] into several Expand if the projection size of Expand is larger
+ * than default projection size.
+ */
+object SplitAggregateWithExpand extends Rule[LogicalPlan] {
+ private def splitExpand(expand: Expand, num: Int): Seq[Expand] = {
+ val groupedProjections = expand.projections.grouped(num).toList
+ val expands: Seq[Expand] = groupedProjections.map {
+ projectionSeq => Expand(projectionSeq, expand.output, expand.child)
+ }
+ expands
+ }
+
+ def apply(plan: LogicalPlan): LogicalPlan = plan transform {
+ case a @ Aggregate(_, _, e @ Expand(projections, _, _)) =>
+ if (SQLConf.get.groupingWithUnion && projections.length
+ > SQLConf.get.groupingExpandProjections) {
+ val num = SQLConf.get.groupingExpandProjections
+ val subExpands = splitExpand(e, num)
+ val aggregates: Seq[Aggregate] = subExpands.map { expand =>
+ Aggregate(a.groupingExpressions, a.aggregateExpressions, expand)
+ }
Review comment:
not the same data, they have passed the sql tests, the output is the same as expected
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] Karl-WangSK commented on pull request #29360: [SPARK-32542][SQL] Add an optimizer rule to split an Expand into multiple Expands for aggregates
Posted by GitBox <gi...@apache.org>.
Karl-WangSK commented on pull request #29360:
URL: https://github.com/apache/spark/pull/29360#issuecomment-675322813
retest this please
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #29360: [SPARK-32542][SQL]Add a Batch in Optimizer to improve performance in multidimensional analysis
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29360:
URL: https://github.com/apache/spark/pull/29360#issuecomment-670945092
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #29360: [SPARK-32542][SQL] Add an optimizer rule to split an Expand into multiple Expands for aggregates
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29360:
URL: https://github.com/apache/spark/pull/29360#issuecomment-673941594
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #29360: [SPARK-32542][SQL]Add a Batch in Optimizer to improve performance in multidimensional analysis
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #29360:
URL: https://github.com/apache/spark/pull/29360#issuecomment-671196871
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #29360: [SPARK-32542][SQL]Add a Batch in Optimizer to improve performance in multidimensional analysis
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29360:
URL: https://github.com/apache/spark/pull/29360#issuecomment-671196871
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] LuciferYang commented on a change in pull request #29360: [SPARK-32542][SQL] Add an optimizer rule to split an Expand into multiple Expands for aggregates
Posted by GitBox <gi...@apache.org>.
LuciferYang commented on a change in pull request #29360:
URL: https://github.com/apache/spark/pull/29360#discussion_r474739747
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/Optimizer.scala
##########
@@ -1823,3 +1824,32 @@ object OptimizeLimitZero extends Rule[LogicalPlan] {
empty(ll)
}
}
+
+/**
+ * Split [[Expand]] into several Expand if the projection size of Expand is larger
+ * than default projection size.
+ */
+object SplitAggregateWithExpand extends Rule[LogicalPlan] {
+ private def splitExpand(expand: Expand, num: Int): Seq[Expand] = {
+ val groupedProjections = expand.projections.grouped(num).toList
+ val expands: Seq[Expand] = groupedProjections.map {
Review comment:
maybe we don't need this intermediate variable `val expands`
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #29360: [SPARK-32542][SQL]Add a Batch in Optimizer to improve performance in multidimensional analysis
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29360:
URL: https://github.com/apache/spark/pull/29360#issuecomment-671163512
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #29360: [SPARK-32542][SQL] Add an optimizer rule to split an Expand into multiple Expands for aggregates
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29360:
URL: https://github.com/apache/spark/pull/29360#issuecomment-678336689
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/127745/
Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #29360: [SPARK-32542][SQL]Add a Batch in Optimizer to improve performance in multidimensional analysis
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29360:
URL: https://github.com/apache/spark/pull/29360#issuecomment-670334551
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #29360: [SPARK-32542][SQL]Add a Batch in Optimizer to improve performance in multidimensional analysis
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29360:
URL: https://github.com/apache/spark/pull/29360#issuecomment-672231939
**[Test build #127335 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/127335/testReport)** for PR 29360 at commit [`2c1de30`](https://github.com/apache/spark/commit/2c1de3089b0a16c9119688ebd4487a9db2377e47).
* This patch **fails Spark unit tests**.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #29360: [SPARK-32542][SQL]Add a batch for optimizing logicalPlan
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #29360:
URL: https://github.com/apache/spark/pull/29360#issuecomment-669649456
**[Test build #127117 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/127117/testReport)** for PR 29360 at commit [`e5d0d0d`](https://github.com/apache/spark/commit/e5d0d0d2221f8a9374edf35e2298bdc38dfc85fd).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #29360: [SPARK-32542][SQL]Add a Batch in Optimizer to improve performance in multidimensional analysis
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29360:
URL: https://github.com/apache/spark/pull/29360#issuecomment-670062181
**[Test build #127147 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/127147/testReport)** for PR 29360 at commit [`db8290c`](https://github.com/apache/spark/commit/db8290ccdbc2a4b82309b2d10b5e6f26a1041a36).
* This patch **fails Spark unit tests**.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] Karl-WangSK commented on a change in pull request #29360: [SPARK-32542][SQL]Add a batch for optimizing logicalPlan
Posted by GitBox <gi...@apache.org>.
Karl-WangSK commented on a change in pull request #29360:
URL: https://github.com/apache/spark/pull/29360#discussion_r466129980
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala
##########
@@ -2690,6 +2690,19 @@ object SQLConf {
.booleanConf
.createWithDefault(true)
+
+ val GROUPING_WITH_UNION = buildConf("spark.sql.optimizer.grouping")
+ .doc("When true,enable grouping with union.")
+ .booleanConf
+ .createWithDefault(false)
+
+ val GROUPING_EXPAND_PROJECTIONS = buildConf("spark.sql.optimizer.projections.size")
+ .doc("Split a expand into several smallExpand,which contains the size of projections.It can" +
+ "reduce the shuffle pressure and improve performance in multidimensional analysis " +
+ "when data is huge.")
+ .intConf
+ .createWithDefault(4)
Review comment:
ok!
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #29360: [SPARK-32542][SQL] Add an optimizer rule to split an Expand into multiple Expands for aggregates
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29360:
URL: https://github.com/apache/spark/pull/29360#issuecomment-678343295
**[Test build #127747 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/127747/testReport)** for PR 29360 at commit [`4632f40`](https://github.com/apache/spark/commit/4632f40c262fe4aa52a30fe1a2817f994e6ad72d).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #29360: [SPARK-32542][SQL]Add a Batch in Optimizer to improve performance in multidimensional analysis
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #29360:
URL: https://github.com/apache/spark/pull/29360#issuecomment-670336836
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] Karl-WangSK commented on pull request #29360: [SPARK-32542][SQL]Add a Batch in Optimizer to improve performance in multidimensional analysis
Posted by GitBox <gi...@apache.org>.
Karl-WangSK commented on pull request #29360:
URL: https://github.com/apache/spark/pull/29360#issuecomment-673496112
please review the benchmark
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #29360: [SPARK-32542][SQL]Add a Batch in Optimizer to improve performance in multidimensional analysis
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29360:
URL: https://github.com/apache/spark/pull/29360#issuecomment-670334933
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/127177/
Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #29360: [SPARK-32542][SQL] Add an optimizer rule to split an Expand into multiple Expands for aggregates
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29360:
URL: https://github.com/apache/spark/pull/29360#issuecomment-674048872
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #29360: [SPARK-32542][SQL]Add a batch for optimizing logicalPlan
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29360:
URL: https://github.com/apache/spark/pull/29360#issuecomment-669672688
**[Test build #127126 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/127126/testReport)** for PR 29360 at commit [`db8290c`](https://github.com/apache/spark/commit/db8290ccdbc2a4b82309b2d10b5e6f26a1041a36).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #29360: [SPARK-32542][SQL]Add a Batch in Optimizer to improve performance in multidimensional analysis
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29360:
URL: https://github.com/apache/spark/pull/29360#issuecomment-672003736
**[Test build #127332 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/127332/testReport)** for PR 29360 at commit [`5bda4ad`](https://github.com/apache/spark/commit/5bda4adbd94fd2f33f40ac7b210c6d6ab1774e08).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #29360: [SPARK-32542][SQL]Add a batch for optimizing logicalPlan
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #29360:
URL: https://github.com/apache/spark/pull/29360#issuecomment-669741593
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #29360: [SPARK-32542][SQL] Add an optimizer rule to split an Expand into multiple Expands for aggregates
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #29360:
URL: https://github.com/apache/spark/pull/29360#issuecomment-674048872
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #29360: [SPARK-32542][SQL]Add a Batch in Optimizer to improve performance in multidimensional analysis
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #29360:
URL: https://github.com/apache/spark/pull/29360#issuecomment-671266434
**[Test build #127279 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/127279/testReport)** for PR 29360 at commit [`eb94158`](https://github.com/apache/spark/commit/eb94158def8ef49f5f3e6ea1062a453d8dc60f01).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #29360: [SPARK-32542][SQL]Add a Batch in Optimizer to improve performance in multidimensional analysis
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29360:
URL: https://github.com/apache/spark/pull/29360#issuecomment-673432002
**[Test build #127409 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/127409/testReport)** for PR 29360 at commit [`5c2e630`](https://github.com/apache/spark/commit/5c2e630fd83a9b32af4e471401d7bf47823750a5).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #29360: [SPARK-32542][SQL]Add a Batch in Optimizer to improve performance in multidimensional analysis
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #29360:
URL: https://github.com/apache/spark/pull/29360#issuecomment-671169977
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #29360: [SPARK-32542][SQL]Add a Batch in Optimizer to improve performance in multidimensional analysis
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #29360:
URL: https://github.com/apache/spark/pull/29360#issuecomment-670459336
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #29360: [SPARK-32542][SQL]Add a Batch in Optimizer to improve performance in multidimensional analysis
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29360:
URL: https://github.com/apache/spark/pull/29360#issuecomment-670470213
**[Test build #127202 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/127202/testReport)** for PR 29360 at commit [`3899943`](https://github.com/apache/spark/commit/3899943e679c5db56ff907619bb01759b70025b6).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] maropu commented on pull request #29360: [SPARK-32542][SQL]Add a batch for optimizing logicalPlan
Posted by GitBox <gi...@apache.org>.
maropu commented on pull request #29360:
URL: https://github.com/apache/spark/pull/29360#issuecomment-669955751
@Karl-WangSK Why did you close this? If you gave up working on this, could we close the correspoinding jira, too?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #29360: [SPARK-32542][SQL]Add a Batch in Optimizer to improve performance in multidimensional analysis
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29360:
URL: https://github.com/apache/spark/pull/29360#issuecomment-669972828
Can one of the admins verify this patch?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #29360: [SPARK-32542][SQL]Add a batch for optimizing logicalPlan
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29360:
URL: https://github.com/apache/spark/pull/29360#issuecomment-669649456
**[Test build #127117 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/127117/testReport)** for PR 29360 at commit [`e5d0d0d`](https://github.com/apache/spark/commit/e5d0d0d2221f8a9374edf35e2298bdc38dfc85fd).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #29360: [SPARK-32542][SQL] Add an optimizer rule to split an Expand into multiple Expands for aggregates
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #29360:
URL: https://github.com/apache/spark/pull/29360#issuecomment-675391574
**[Test build #127564 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/127564/testReport)** for PR 29360 at commit [`e17fa60`](https://github.com/apache/spark/commit/e17fa605684ae114973a3c34a58239d3a9f8dc52).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] maropu commented on pull request #29360: [SPARK-32542][SQL]Add a batch for optimizing logicalPlan
Posted by GitBox <gi...@apache.org>.
maropu commented on pull request #29360:
URL: https://github.com/apache/spark/pull/29360#issuecomment-669968604
If you're still working on this, its okay to keep this open.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #29360: [SPARK-32542][SQL] Add an optimizer rule to split an Expand into multiple Expands for aggregates
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29360:
URL: https://github.com/apache/spark/pull/29360#issuecomment-675546591
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #29360: [SPARK-32542][SQL]Add a Batch in Optimizer to improve performance in multidimensional analysis
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #29360:
URL: https://github.com/apache/spark/pull/29360#issuecomment-670952746
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] Karl-WangSK commented on pull request #29360: [SPARK-32542][SQL]Add a Batch in Optimizer to improve performance in multidimensional analysis
Posted by GitBox <gi...@apache.org>.
Karl-WangSK commented on pull request #29360:
URL: https://github.com/apache/spark/pull/29360#issuecomment-673426929
retest this please
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #29360: [SPARK-32542][SQL]Add a Batch in Optimizer to improve performance in multidimensional analysis
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29360:
URL: https://github.com/apache/spark/pull/29360#issuecomment-669978465
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #29360: [SPARK-32542][SQL]Add a Batch in Optimizer to improve performance in multidimensional analysis
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29360:
URL: https://github.com/apache/spark/pull/29360#issuecomment-671174613
**[Test build #127267 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/127267/testReport)** for PR 29360 at commit [`f7bdea0`](https://github.com/apache/spark/commit/f7bdea013bf868b84a9c9339897ea45bba56c32e).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #29360: [SPARK-32542][SQL]Add a Batch in Optimizer to improve performance in multidimensional analysis
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29360:
URL: https://github.com/apache/spark/pull/29360#issuecomment-671173465
**[Test build #127266 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/127266/testReport)** for PR 29360 at commit [`82a1d4c`](https://github.com/apache/spark/commit/82a1d4c91f52a6ce523f1c0aa3703172a42e72b6).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #29360: [SPARK-32542][SQL]Add a batch for optimizing logicalPlan
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #29360:
URL: https://github.com/apache/spark/pull/29360#issuecomment-669747638
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] Karl-WangSK removed a comment on pull request #29360: [SPARK-32542][SQL]Add a Batch in Optimizer to improve performance in multidimensional analysis
Posted by GitBox <gi...@apache.org>.
Karl-WangSK removed a comment on pull request #29360:
URL: https://github.com/apache/spark/pull/29360#issuecomment-673382458
retest this please
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #29360: [SPARK-32542][SQL]Add a Batch in Optimizer to improve performance in multidimensional analysis
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29360:
URL: https://github.com/apache/spark/pull/29360#issuecomment-671193976
**[Test build #127266 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/127266/testReport)** for PR 29360 at commit [`82a1d4c`](https://github.com/apache/spark/commit/82a1d4c91f52a6ce523f1c0aa3703172a42e72b6).
* This patch **fails Spark unit tests**.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #29360: [SPARK-32542][SQL]Add a Batch in Optimizer to improve performance in multidimensional analysis
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29360:
URL: https://github.com/apache/spark/pull/29360#issuecomment-671045241
**[Test build #127237 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/127237/testReport)** for PR 29360 at commit [`f904eab`](https://github.com/apache/spark/commit/f904eabd3fa986b907e0d0d32976f9d815d27b9c).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #29360: [SPARK-32542][SQL]Add a Batch in Optimizer to improve performance in multidimensional analysis
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #29360:
URL: https://github.com/apache/spark/pull/29360#issuecomment-670467903
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #29360: [SPARK-32542][SQL]Add a Batch in Optimizer to improve performance in multidimensional analysis
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #29360:
URL: https://github.com/apache/spark/pull/29360#issuecomment-670459118
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #29360: [SPARK-32542][SQL]Add a Batch in Optimizer to improve performance in multidimensional analysis
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #29360:
URL: https://github.com/apache/spark/pull/29360#issuecomment-670534006
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #29360: [SPARK-32542][SQL]Add a Batch in Optimizer to improve performance in multidimensional analysis
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29360:
URL: https://github.com/apache/spark/pull/29360#issuecomment-670425817
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #29360: [SPARK-32542][SQL] Add an optimizer rule to split an Expand into multiple Expands for aggregates
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29360:
URL: https://github.com/apache/spark/pull/29360#issuecomment-678336371
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] Karl-WangSK commented on a change in pull request #29360: [SPARK-32542][SQL] Add an optimizer rule to split an Expand into multiple Expands for aggregates
Posted by GitBox <gi...@apache.org>.
Karl-WangSK commented on a change in pull request #29360:
URL: https://github.com/apache/spark/pull/29360#discussion_r474749994
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/Optimizer.scala
##########
@@ -1823,3 +1824,32 @@ object OptimizeLimitZero extends Rule[LogicalPlan] {
empty(ll)
}
}
+
+/**
+ * Split [[Expand]] into several Expand if the projection size of Expand is larger
+ * than default projection size.
+ */
+object SplitAggregateWithExpand extends Rule[LogicalPlan] {
+ private def splitExpand(expand: Expand, num: Int): Seq[Expand] = {
+ val groupedProjections = expand.projections.grouped(num).toList
+ val expands: Seq[Expand] = groupedProjections.map {
Review comment:
ok
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #29360: [SPARK-32542][SQL]Add a Batch in Optimizer to improve performance in multidimensional analysis
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29360:
URL: https://github.com/apache/spark/pull/29360#issuecomment-670426530
**[Test build #127194 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/127194/testReport)** for PR 29360 at commit [`34a76d1`](https://github.com/apache/spark/commit/34a76d1633cdb53c70cb20f89d403816317d46cf).
* This patch **fails Scala style tests**.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #29360: [SPARK-32542][SQL] Add an optimizer rule to split an Expand into multiple Expands for aggregates
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29360:
URL: https://github.com/apache/spark/pull/29360#issuecomment-678335806
**[Test build #127745 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/127745/testReport)** for PR 29360 at commit [`039ba05`](https://github.com/apache/spark/commit/039ba05ca3b2c5d328290d9c67b5a629df6d9687).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] Karl-WangSK removed a comment on pull request #29360: [SPARK-32542][SQL] Add an optimizer rule to split an Expand into multiple Expands for aggregates
Posted by GitBox <gi...@apache.org>.
Karl-WangSK removed a comment on pull request #29360:
URL: https://github.com/apache/spark/pull/29360#issuecomment-675322813
retest this please
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #29360: [SPARK-32542][SQL]Add a batch for optimizing logicalPlan
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29360:
URL: https://github.com/apache/spark/pull/29360#issuecomment-669668870
**[Test build #127124 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/127124/testReport)** for PR 29360 at commit [`a71e96b`](https://github.com/apache/spark/commit/a71e96bf3c22d979d15b8b373c262bdf6c6e4cf9).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #29360: [SPARK-32542][SQL]Add a Batch in Optimizer to improve performance in multidimensional analysis
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #29360:
URL: https://github.com/apache/spark/pull/29360#issuecomment-673352065
**[Test build #127404 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/127404/testReport)** for PR 29360 at commit [`4a1350c`](https://github.com/apache/spark/commit/4a1350c42a73721c392a5ff655e4fc8b639304f0).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #29360: [SPARK-32542][SQL]Add a Batch in Optimizer to improve performance in multidimensional analysis
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #29360:
URL: https://github.com/apache/spark/pull/29360#issuecomment-672233431
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #29360: [SPARK-32542][SQL]Add a Batch in Optimizer to improve performance in multidimensional analysis
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29360:
URL: https://github.com/apache/spark/pull/29360#issuecomment-672233431
Merged build finished. Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #29360: [SPARK-32542][SQL]Add a Batch in Optimizer to improve performance in multidimensional analysis
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #29360:
URL: https://github.com/apache/spark/pull/29360#issuecomment-672012047
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #29360: [SPARK-32542][SQL]Add a Batch in Optimizer to improve performance in multidimensional analysis
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29360:
URL: https://github.com/apache/spark/pull/29360#issuecomment-671196616
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #29360: [SPARK-32542][SQL]Add a batch for optimizing logicalPlan
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29360:
URL: https://github.com/apache/spark/pull/29360#issuecomment-669747638
Merged build finished. Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #29360: [SPARK-32542][SQL]Add a Batch in Optimizer to improve performance in multidimensional analysis
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29360:
URL: https://github.com/apache/spark/pull/29360#issuecomment-672057796
**[Test build #127335 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/127335/testReport)** for PR 29360 at commit [`2c1de30`](https://github.com/apache/spark/commit/2c1de3089b0a16c9119688ebd4487a9db2377e47).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #29360: [SPARK-32542][SQL] Add an optimizer rule to split an Expand into multiple Expands for aggregates
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29360:
URL: https://github.com/apache/spark/pull/29360#issuecomment-675233817
**[Test build #127526 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/127526/testReport)** for PR 29360 at commit [`87b9a82`](https://github.com/apache/spark/commit/87b9a825359168eb07fe5f9791e1dc26ce138046).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #29360: [SPARK-32542][SQL]Add a Batch in Optimizer to improve performance in multidimensional analysis
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29360:
URL: https://github.com/apache/spark/pull/29360#issuecomment-670970447
**[Test build #127226 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/127226/testReport)** for PR 29360 at commit [`4e62422`](https://github.com/apache/spark/commit/4e6242271b3312cc5dff4c9fdf0ab98432e01c6e).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] Karl-WangSK removed a comment on pull request #29360: [SPARK-32542][SQL] Add an optimizer rule to split an Expand into multiple Expands for aggregates
Posted by GitBox <gi...@apache.org>.
Karl-WangSK removed a comment on pull request #29360:
URL: https://github.com/apache/spark/pull/29360#issuecomment-675202769
@cloud-fan
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #29360: [SPARK-32542][SQL]Add a Batch in Optimizer to improve performance in multidimensional analysis
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #29360:
URL: https://github.com/apache/spark/pull/29360#issuecomment-671168061
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #29360: [SPARK-32542][SQL]Add a Batch in Optimizer to improve performance in multidimensional analysis
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29360:
URL: https://github.com/apache/spark/pull/29360#issuecomment-671277969
**[Test build #127279 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/127279/testReport)** for PR 29360 at commit [`eb94158`](https://github.com/apache/spark/commit/eb94158def8ef49f5f3e6ea1062a453d8dc60f01).
* This patch **fails Spark unit tests**.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #29360: [SPARK-32542][SQL]Add a Batch in Optimizer to improve performance in multidimensional analysis
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #29360:
URL: https://github.com/apache/spark/pull/29360#issuecomment-671163512
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #29360: [SPARK-32542][SQL]Add a Batch in Optimizer to improve performance in multidimensional analysis
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29360:
URL: https://github.com/apache/spark/pull/29360#issuecomment-670952610
**[Test build #127223 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/127223/testReport)** for PR 29360 at commit [`bf73a5c`](https://github.com/apache/spark/commit/bf73a5c997ae3eeaeff712cd368c076e38b12684).
* This patch **fails Spark unit tests**.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #29360: [SPARK-32542][SQL]Add a Batch in Optimizer to improve performance in multidimensional analysis
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #29360:
URL: https://github.com/apache/spark/pull/29360#issuecomment-671174613
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #29360: [SPARK-32542][SQL]Add a batch for optimizing logicalPlan
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #29360:
URL: https://github.com/apache/spark/pull/29360#issuecomment-669669158
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] viirya commented on a change in pull request #29360: [SPARK-32542][SQL]Add a Batch in Optimizer to improve performance in multidimensional analysis
Posted by GitBox <gi...@apache.org>.
viirya commented on a change in pull request #29360:
URL: https://github.com/apache/spark/pull/29360#discussion_r467543192
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala
##########
@@ -2690,6 +2690,18 @@ object SQLConf {
.booleanConf
.createWithDefault(true)
+ val GROUPING_WITH_UNION = buildConf("spark.sql.optimizer.grouping")
+ .doc("When true,enable grouping with union.")
Review comment:
For text style. Please add a space after punctuation marks like , or .
For example, "When true,enable grouping with union." -> "When true, enable grouping with union."
There are several places with this style issue.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #29360: [SPARK-32542][SQL]Add a Batch in Optimizer to improve performance in multidimensional analysis
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #29360:
URL: https://github.com/apache/spark/pull/29360#issuecomment-670350289
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #29360: [SPARK-32542][SQL]Add a Batch in Optimizer to improve performance in multidimensional analysis
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29360:
URL: https://github.com/apache/spark/pull/29360#issuecomment-670381299
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #29360: [SPARK-32542][SQL]Add a Batch in Optimizer to improve performance in multidimensional analysis
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29360:
URL: https://github.com/apache/spark/pull/29360#issuecomment-670336836
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #29360: [SPARK-32542][SQL]Add a Batch in Optimizer to improve performance in multidimensional analysis
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #29360:
URL: https://github.com/apache/spark/pull/29360#issuecomment-671163039
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #29360: [SPARK-32542][SQL]Add a Batch in Optimizer to improve performance in multidimensional analysis
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #29360:
URL: https://github.com/apache/spark/pull/29360#issuecomment-670425817
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] viirya commented on a change in pull request #29360: [SPARK-32542][SQL]Add a Batch in Optimizer to improve performance in multidimensional analysis
Posted by GitBox <gi...@apache.org>.
viirya commented on a change in pull request #29360:
URL: https://github.com/apache/spark/pull/29360#discussion_r467601888
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/Optimizer.scala
##########
@@ -1823,3 +1824,32 @@ object OptimizeLimitZero extends Rule[LogicalPlan] {
empty(ll)
}
}
+
+/**
+ * Split [[Expand]] into several Expand if the projection size of Expand is larger
+ * than default projection size.
+ */
+object SplitAggregateWithExpand extends Rule[LogicalPlan] {
Review comment:
I'm curious about how this reduces shuffled data. I think the original idea is, in the cases of Expand will produce huge data, splitting it to small ones.
But shuffle is happened during Aggregate here, right? By splitting, the total amount of shuffled data is not changed, but split into several ones. Does it really result significant improvement?
I'm curious to look at the benchmark results.
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/Optimizer.scala
##########
@@ -1823,3 +1824,32 @@ object OptimizeLimitZero extends Rule[LogicalPlan] {
empty(ll)
}
}
+
+/**
+ * Split [[Expand]] into several Expand if the projection size of Expand is larger
+ * than default projection size.
+ */
+object SplitAggregateWithExpand extends Rule[LogicalPlan] {
+ private def splitExpand(expand: Expand, num: Int): Seq[Expand] = {
+ val groupedProjections = expand.projections.grouped(num).toList
+ val expands: Seq[Expand] = groupedProjections.map {
+ projectionSeq => Expand(projectionSeq, expand.output, expand.child)
+ }
+ expands
+ }
+
+ def apply(plan: LogicalPlan): LogicalPlan = plan transform {
+ case a @ Aggregate(_, _, e @ Expand(projections, _, _)) =>
+ if (SQLConf.get.groupingWithUnion && projections.length
+ > SQLConf.get.groupingExpandProjections) {
+ val num = SQLConf.get.groupingExpandProjections
+ val subExpands = splitExpand(e, num)
+ val aggregates: Seq[Aggregate] = subExpands.map { expand =>
+ Aggregate(a.groupingExpressions, a.aggregateExpressions, expand)
+ }
Review comment:
These split expand + aggregate read same data. That's said this optimization will read same data multiple times.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #29360: [SPARK-32542][SQL]Add a Batch in Optimizer to improve performance in multidimensional analysis
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #29360:
URL: https://github.com/apache/spark/pull/29360#issuecomment-670334551
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #29360: [SPARK-32542][SQL] Add an optimizer rule to split an Expand into multiple Expands for aggregates
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #29360:
URL: https://github.com/apache/spark/pull/29360#issuecomment-679470062
**[Test build #127866 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/127866/testReport)** for PR 29360 at commit [`b30d2ba`](https://github.com/apache/spark/commit/b30d2ba400af5e2a46fad11cc2131542db1ab246).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #29360: [SPARK-32542][SQL] Add an optimizer rule to split an Expand into multiple Expands for aggregates
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #29360:
URL: https://github.com/apache/spark/pull/29360#issuecomment-678336371
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #29360: [SPARK-32542][SQL] Add an optimizer rule to split an Expand into multiple Expands for aggregates
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #29360:
URL: https://github.com/apache/spark/pull/29360#issuecomment-678343295
**[Test build #127747 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/127747/testReport)** for PR 29360 at commit [`4632f40`](https://github.com/apache/spark/commit/4632f40c262fe4aa52a30fe1a2817f994e6ad72d).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #29360: [SPARK-32542][SQL]Add a Batch in Optimizer to improve performance in multidimensional analysis
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #29360:
URL: https://github.com/apache/spark/pull/29360#issuecomment-673429679
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #29360: [SPARK-32542][SQL] Add an optimizer rule to split an Expand into multiple Expands for aggregates
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #29360:
URL: https://github.com/apache/spark/pull/29360#issuecomment-673941158
**[Test build #127446 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/127446/testReport)** for PR 29360 at commit [`b0a7c6e`](https://github.com/apache/spark/commit/b0a7c6eb7aa47f68629024ad9eb0836ce9ea97e4).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] Karl-WangSK commented on pull request #29360: [SPARK-32542][SQL] Add an optimizer rule to split an Expand into multiple Expands for aggregates
Posted by GitBox <gi...@apache.org>.
Karl-WangSK commented on pull request #29360:
URL: https://github.com/apache/spark/pull/29360#issuecomment-673868646
yes.The shuffle output is the same, because the size of the data is the same.
As you can see the benchmark:
cube 7 fields k1, k2, k3, k4, k5, k6, k7(128x projections) and cube 6 fields k1, k2, k3, k4, k5, k6(64x projections) with grouping off
data size is double ,but the time ,one is 2.4min ,the another one is 8.7min, not just double time .It will be affected by data size Especially when the memory is limited.
The original data I created is about 20M, executor memory is 1g. when it expands to 64x or 128x. It will have big impact on shuffle performance.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #29360: [SPARK-32542][SQL]Add a Batch in Optimizer to improve performance in multidimensional analysis
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29360:
URL: https://github.com/apache/spark/pull/29360#issuecomment-670534006
Merged build finished. Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] maropu commented on pull request #29360: [SPARK-32542][SQL]Add a Batch in Optimizer to improve performance in multidimensional analysis
Posted by GitBox <gi...@apache.org>.
maropu commented on pull request #29360:
URL: https://github.com/apache/spark/pull/29360#issuecomment-669975734
ok to test
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] Karl-WangSK commented on a change in pull request #29360: [SPARK-32542][SQL]Add a Batch in Optimizer to improve performance in multidimensional analysis
Posted by GitBox <gi...@apache.org>.
Karl-WangSK commented on a change in pull request #29360:
URL: https://github.com/apache/spark/pull/29360#discussion_r467552369
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala
##########
@@ -2690,6 +2690,18 @@ object SQLConf {
.booleanConf
.createWithDefault(true)
+ val GROUPING_WITH_UNION = buildConf("spark.sql.optimizer.grouping")
+ .doc("When true, enable grouping with union.")
+ .booleanConf
+ .createWithDefault(false)
Review comment:
yes. The default spark.sql.optimizer.projections.size=4,but if your sql's projections excceed 4,and you still don't want to turn on this function. you can set spark.sql.optimizer.grouping=false
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #29360: [SPARK-32542][SQL]Add a Batch in Optimizer to improve performance in multidimensional analysis
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29360:
URL: https://github.com/apache/spark/pull/29360#issuecomment-671021931
**[Test build #127237 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/127237/testReport)** for PR 29360 at commit [`f904eab`](https://github.com/apache/spark/commit/f904eabd3fa986b907e0d0d32976f9d815d27b9c).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #29360: [SPARK-32542][SQL]Add a batch for optimizing logicalPlan
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29360:
URL: https://github.com/apache/spark/pull/29360#issuecomment-669669158
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #29360: [SPARK-32542][SQL]Add a batch for optimizing logicalPlan
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29360:
URL: https://github.com/apache/spark/pull/29360#issuecomment-669671155
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #29360: [SPARK-32542][SQL]Add a Batch in Optimizer to improve performance in multidimensional analysis
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #29360:
URL: https://github.com/apache/spark/pull/29360#issuecomment-670470687
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #29360: [SPARK-32542][SQL]Add a Batch in Optimizer to improve performance in multidimensional analysis
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29360:
URL: https://github.com/apache/spark/pull/29360#issuecomment-670384881
Merged build finished. Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #29360: [SPARK-32542][SQL]Add a Batch in Optimizer to improve performance in multidimensional analysis
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29360:
URL: https://github.com/apache/spark/pull/29360#issuecomment-672004353
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #29360: [SPARK-32542][SQL] Add an optimizer rule to split an Expand into multiple Expands for aggregates
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #29360:
URL: https://github.com/apache/spark/pull/29360#issuecomment-679845460
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #29360: [SPARK-32542][SQL]Add a Batch in Optimizer to improve performance in multidimensional analysis
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #29360:
URL: https://github.com/apache/spark/pull/29360#issuecomment-671366334
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #29360: [SPARK-32542][SQL]Add a batch for optimizing logicalPlan
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #29360:
URL: https://github.com/apache/spark/pull/29360#issuecomment-669113698
Can one of the admins verify this patch?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #29360: [SPARK-32542][SQL]Add a Batch in Optimizer to improve performance in multidimensional analysis
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29360:
URL: https://github.com/apache/spark/pull/29360#issuecomment-670350295
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/127178/
Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #29360: [SPARK-32542][SQL]Add a Batch in Optimizer to improve performance in multidimensional analysis
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29360:
URL: https://github.com/apache/spark/pull/29360#issuecomment-671016267
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] Karl-WangSK commented on pull request #29360: [SPARK-32542][SQL] Add an optimizer rule to split an Expand into multiple Expands for aggregates
Posted by GitBox <gi...@apache.org>.
Karl-WangSK commented on pull request #29360:
URL: https://github.com/apache/spark/pull/29360#issuecomment-675202769
@cloud-fan
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #29360: [SPARK-32542][SQL]Add a Batch in Optimizer to improve performance in multidimensional analysis
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29360:
URL: https://github.com/apache/spark/pull/29360#issuecomment-671045408
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #29360: [SPARK-32542][SQL]Add a Batch in Optimizer to improve performance in multidimensional analysis
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #29360:
URL: https://github.com/apache/spark/pull/29360#issuecomment-672008431
**[Test build #127333 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/127333/testReport)** for PR 29360 at commit [`7a748c2`](https://github.com/apache/spark/commit/7a748c284a54d0aa4a776215254f98d89d25cb33).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #29360: [SPARK-32542][SQL]Add a Batch in Optimizer to improve performance in multidimensional analysis
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29360:
URL: https://github.com/apache/spark/pull/29360#issuecomment-671266434
**[Test build #127279 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/127279/testReport)** for PR 29360 at commit [`eb94158`](https://github.com/apache/spark/commit/eb94158def8ef49f5f3e6ea1062a453d8dc60f01).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #29360: [SPARK-32542][SQL]Add a Batch in Optimizer to improve performance in multidimensional analysis
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #29360:
URL: https://github.com/apache/spark/pull/29360#issuecomment-671423884
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org