You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by GitBox <gi...@apache.org> on 2021/02/27 22:59:44 UTC
[GitHub] [spark] tanelk opened a new pull request #31677: [SPARK-34565][SQL] Collapse Window nodes with Project between them
tanelk opened a new pull request #31677:
URL: https://github.com/apache/spark/pull/31677
<!--
Thanks for sending a pull request! Here are some tips for you:
1. If this is your first time, please read our contributor guidelines: https://spark.apache.org/contributing.html
2. Ensure you have added or run the appropriate tests for your PR: https://spark.apache.org/developer-tools.html
3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][SPARK-XXXX] Your PR title ...'.
4. Be sure to keep the PR description updated to reflect all changes.
5. Please write your PR title to summarize what this PR proposes.
6. If possible, provide a concise example to reproduce the issue for a faster review.
7. If you want to add a new configuration, please read the guideline first for naming configurations in
'core/src/main/scala/org/apache/spark/internal/config/ConfigEntry.scala'.
-->
### What changes were proposed in this pull request?
<!--
Please clarify what changes you are proposing. The purpose of this section is to outline the changes and how this PR fixes the issue.
If possible, please consider writing useful notes for better and faster reviews in your PR. See the examples below.
1. If you refactor some codes with changing classes, showing the class hierarchy will help reviewers.
2. If you fix some SQL features, you can provide some references of other DBMSes.
3. If there is design documentation, please add the link.
4. If there is a discussion in the mailing list, please add the link.
-->
Extend the `CollapseWindow` rule to collapse `Window` nodes, that have `Project` between them.
### Why are the changes needed?
<!--
Please clarify why the changes are needed. For instance,
1. If you propose a new API, clarify the use case for a new API.
2. If you fix a bug, you can clarify why it is a bug.
-->
The analyzer will turn a `dataset.withColumn("colName", expressionWithWindowFunction)` method call to a `Project - Window - Project` chain in the logical plan. When this method is called multiple times in a row, then the projects can block the `Window` nodes from being collapsed by the current `CollapseWindow` rule.
### Does this PR introduce _any_ user-facing change?
<!--
Note that it means *any* user-facing change including all aspects such as the documentation fix.
If yes, please clarify the previous behavior and the change this PR proposes - provide the console output, description and/or an example to show the behavior difference if possible.
If possible, please also clarify if this is a user-facing change compared to the released Spark versions or within the unreleased branches such as master.
If no, write 'No'.
-->
No
### How was this patch tested?
<!--
If tests were added, say they were added here. Please make sure to add some test cases that check the changes thoroughly including negative and positive cases if possible.
If it was tested in a way different from regular unit tests, please clarify how you tested step by step, ideally copy and paste-able, so that other reviewers can test and check, and descendants can verify in the future.
If tests were not added, please describe why they were not added and/or why it was difficult to add.
-->
UT
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on pull request #31677: [SPARK-34565][SQL] Collapse Window nodes with Project between them
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on pull request #31677:
URL: https://github.com/apache/spark/pull/31677#issuecomment-860132059
retest this please
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] maropu commented on a change in pull request #31677: [SPARK-34565][SQL] Collapse Window nodes with Project between them
Posted by GitBox <gi...@apache.org>.
maropu commented on a change in pull request #31677:
URL: https://github.com/apache/spark/pull/31677#discussion_r589125172
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/Optimizer.scala
##########
@@ -914,13 +914,27 @@ object OptimizeWindowFunctions extends Rule[LogicalPlan] {
*/
object CollapseWindow extends Rule[LogicalPlan] {
def apply(plan: LogicalPlan): LogicalPlan = plan transformUp {
- case w1 @ Window(we1, ps1, os1, w2 @ Window(we2, ps2, os2, grandChild))
- if ps1 == ps2 && os1 == os2 && w1.references.intersect(w2.windowOutputSet).isEmpty &&
- we1.nonEmpty && we2.nonEmpty &&
- // This assumes Window contains the same type of window expressions. This is ensured
- // by ExtractWindowFunctions.
- WindowFunctionType.functionType(we1.head) == WindowFunctionType.functionType(we2.head) =>
+ case w1 @ Window(we1, _, _, w2 @ Window(we2, _, _, grandChild))
+ if windowsCompatible(w1, w2) =>
w1.copy(windowExpressions = we2 ++ we1, child = grandChild)
+
+ case w1 @ Window(we1, _, _, Project(pl, w2 @ Window(we2, _, _, grandChild)))
+ if windowsCompatible(w1, w2) && w1.references.subsetOf(grandChild.outputSet) =>
+ Project(
+ pl ++ we1.map(_.toAttribute),
+ w1.copy(windowExpressions = we2 ++ we1, child = grandChild)
+ )
+ }
+
+ def windowsCompatible(w1: Window, w2: Window): Boolean = {
Review comment:
nit: private
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #31677: [SPARK-34565][SQL] Collapse Window nodes with Project between them
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #31677:
URL: https://github.com/apache/spark/pull/31677#issuecomment-864348544
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/44539/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] wangyum commented on pull request #31677: [SPARK-34565][SQL] Collapse Window nodes with Project between them
Posted by GitBox <gi...@apache.org>.
wangyum commented on pull request #31677:
URL: https://github.com/apache/spark/pull/31677#issuecomment-864338820
retest this please.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #31677: [SPARK-34565][SQL] Collapse Window nodes with Project between them
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #31677:
URL: https://github.com/apache/spark/pull/31677#issuecomment-792242978
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/40422/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] wangyum commented on a change in pull request #31677: [SPARK-34565][SQL] Collapse Window nodes with Project between them
Posted by GitBox <gi...@apache.org>.
wangyum commented on a change in pull request #31677:
URL: https://github.com/apache/spark/pull/31677#discussion_r602666684
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/Optimizer.scala
##########
@@ -914,13 +914,27 @@ object OptimizeWindowFunctions extends Rule[LogicalPlan] {
*/
object CollapseWindow extends Rule[LogicalPlan] {
def apply(plan: LogicalPlan): LogicalPlan = plan transformUp {
- case w1 @ Window(we1, ps1, os1, w2 @ Window(we2, ps2, os2, grandChild))
- if ps1 == ps2 && os1 == os2 && w1.references.intersect(w2.windowOutputSet).isEmpty &&
- we1.nonEmpty && we2.nonEmpty &&
- // This assumes Window contains the same type of window expressions. This is ensured
- // by ExtractWindowFunctions.
- WindowFunctionType.functionType(we1.head) == WindowFunctionType.functionType(we2.head) =>
+ case w1 @ Window(we1, _, _, w2 @ Window(we2, _, _, grandChild))
+ if windowsCompatible(w1, w2) =>
w1.copy(windowExpressions = we2 ++ we1, child = grandChild)
+
+ case w1 @ Window(we1, _, _, Project(pl, w2 @ Window(we2, _, _, grandChild)))
+ if windowsCompatible(w1, w2) && w1.references.subsetOf(grandChild.outputSet) =>
+ Project(
+ pl ++ we1.map(_.toAttribute),
+ w1.copy(windowExpressions = we2 ++ we1, child = grandChild)
+ )
+ }
Review comment:
cc @dongjoon-hyun
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #31677: [SPARK-34565][SQL] Collapse Window nodes with Project between them
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #31677:
URL: https://github.com/apache/spark/pull/31677#issuecomment-847310473
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/138874/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #31677: [SPARK-34565][SQL] Collapse Window nodes with Project between them
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #31677:
URL: https://github.com/apache/spark/pull/31677#issuecomment-859238226
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #31677: [SPARK-34565][SQL] Collapse Window nodes with Project between them
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #31677:
URL: https://github.com/apache/spark/pull/31677#issuecomment-847125255
**[Test build #138874 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/138874/testReport)** for PR 31677 at commit [`de3a7f1`](https://github.com/apache/spark/commit/de3a7f1cc864261429fd19e69d9f1119a5bc000d).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #31677: [SPARK-34565][SQL] Collapse Window nodes with Project between them
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #31677:
URL: https://github.com/apache/spark/pull/31677#issuecomment-808849650
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/41188/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #31677: [SPARK-34565][SQL] Collapse Window nodes with Project between them
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #31677:
URL: https://github.com/apache/spark/pull/31677#issuecomment-808743142
Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/41177/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] wangyum commented on a change in pull request #31677: [SPARK-34565][SQL] Collapse Window nodes with Project between them
Posted by GitBox <gi...@apache.org>.
wangyum commented on a change in pull request #31677:
URL: https://github.com/apache/spark/pull/31677#discussion_r602817987
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/Optimizer.scala
##########
@@ -914,13 +914,27 @@ object OptimizeWindowFunctions extends Rule[LogicalPlan] {
*/
object CollapseWindow extends Rule[LogicalPlan] {
def apply(plan: LogicalPlan): LogicalPlan = plan transformUp {
- case w1 @ Window(we1, ps1, os1, w2 @ Window(we2, ps2, os2, grandChild))
- if ps1 == ps2 && os1 == os2 && w1.references.intersect(w2.windowOutputSet).isEmpty &&
- we1.nonEmpty && we2.nonEmpty &&
- // This assumes Window contains the same type of window expressions. This is ensured
- // by ExtractWindowFunctions.
- WindowFunctionType.functionType(we1.head) == WindowFunctionType.functionType(we2.head) =>
+ case w1 @ Window(we1, _, _, w2 @ Window(we2, _, _, grandChild))
+ if windowsCompatible(w1, w2) =>
w1.copy(windowExpressions = we2 ++ we1, child = grandChild)
+
+ case w1 @ Window(we1, _, _, Project(pl, w2 @ Window(we2, _, _, grandChild)))
+ if windowsCompatible(w1, w2) && w1.references.subsetOf(grandChild.outputSet) =>
+ Project(
+ pl ++ w1.windowOutputSet,
+ w1.copy(windowExpressions = we2 ++ we1, child = grandChild)
+ )
+ }
+
+ private def windowsCompatible(w1: Window, w2: Window): Boolean = {
Review comment:
Could we move `windowsCompatible` before `apply`. We usually put the private function before the `apply`. See `TransposeWindow`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] wangyum commented on a change in pull request #31677: [SPARK-34565][SQL] Collapse Window nodes with Project between them
Posted by GitBox <gi...@apache.org>.
wangyum commented on a change in pull request #31677:
URL: https://github.com/apache/spark/pull/31677#discussion_r602877074
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/Optimizer.scala
##########
@@ -913,14 +913,27 @@ object OptimizeWindowFunctions extends Rule[LogicalPlan] {
* independent and are of the same window function type, collapse into the parent.
*/
object CollapseWindow extends Rule[LogicalPlan] {
+ private def windowsCompatible(w1: Window, w2: Window): Boolean = {
+ w1.partitionSpec == w2.partitionSpec &&
+ w1.orderSpec == w2.orderSpec &&
+ w1.references.intersect(w2.windowOutputSet).isEmpty &&
+ w1.windowExpressions.nonEmpty && w2.windowExpressions.nonEmpty &&
+ // This assumes Window contains the same type of window expressions. This is ensured
+ // by ExtractWindowFunctions.
+ WindowFunctionType.functionType(w1.windowExpressions.head) ==
+ WindowFunctionType.functionType(w2.windowExpressions.head)
+ }
+
def apply(plan: LogicalPlan): LogicalPlan = plan transformUp {
- case w1 @ Window(we1, ps1, os1, w2 @ Window(we2, ps2, os2, grandChild))
- if ps1 == ps2 && os1 == os2 && w1.references.intersect(w2.windowOutputSet).isEmpty &&
- we1.nonEmpty && we2.nonEmpty &&
- // This assumes Window contains the same type of window expressions. This is ensured
- // by ExtractWindowFunctions.
- WindowFunctionType.functionType(we1.head) == WindowFunctionType.functionType(we2.head) =>
+ case w1 @ Window(we1, _, _, w2 @ Window(we2, _, _, grandChild))
+ if windowsCompatible(w1, w2) =>
Review comment:
Two more spaces?
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/Optimizer.scala
##########
@@ -913,14 +913,27 @@ object OptimizeWindowFunctions extends Rule[LogicalPlan] {
* independent and are of the same window function type, collapse into the parent.
*/
object CollapseWindow extends Rule[LogicalPlan] {
+ private def windowsCompatible(w1: Window, w2: Window): Boolean = {
+ w1.partitionSpec == w2.partitionSpec &&
+ w1.orderSpec == w2.orderSpec &&
+ w1.references.intersect(w2.windowOutputSet).isEmpty &&
+ w1.windowExpressions.nonEmpty && w2.windowExpressions.nonEmpty &&
+ // This assumes Window contains the same type of window expressions. This is ensured
+ // by ExtractWindowFunctions.
+ WindowFunctionType.functionType(w1.windowExpressions.head) ==
+ WindowFunctionType.functionType(w2.windowExpressions.head)
+ }
+
def apply(plan: LogicalPlan): LogicalPlan = plan transformUp {
- case w1 @ Window(we1, ps1, os1, w2 @ Window(we2, ps2, os2, grandChild))
- if ps1 == ps2 && os1 == os2 && w1.references.intersect(w2.windowOutputSet).isEmpty &&
- we1.nonEmpty && we2.nonEmpty &&
- // This assumes Window contains the same type of window expressions. This is ensured
- // by ExtractWindowFunctions.
- WindowFunctionType.functionType(we1.head) == WindowFunctionType.functionType(we2.head) =>
+ case w1 @ Window(we1, _, _, w2 @ Window(we2, _, _, grandChild))
+ if windowsCompatible(w1, w2) =>
w1.copy(windowExpressions = we2 ++ we1, child = grandChild)
+
+ case w1 @ Window(we1, _, _, Project(pl, w2 @ Window(we2, _, _, grandChild)))
+ if windowsCompatible(w1, w2) && w1.references.subsetOf(grandChild.outputSet) =>
Review comment:
Two more spaces?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #31677: [SPARK-34565][SQL] Collapse Window nodes with Project between them
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #31677:
URL: https://github.com/apache/spark/pull/31677#issuecomment-792561644
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/135859/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #31677: [SPARK-34565][SQL] Collapse Window nodes with Project between them
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #31677:
URL: https://github.com/apache/spark/pull/31677#issuecomment-847299937
**[Test build #138874 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/138874/testReport)** for PR 31677 at commit [`de3a7f1`](https://github.com/apache/spark/commit/de3a7f1cc864261429fd19e69d9f1119a5bc000d).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #31677: [SPARK-34565][SQL] Collapse Window nodes with Project between them
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #31677:
URL: https://github.com/apache/spark/pull/31677#issuecomment-864368267
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/140013/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] tanelk commented on pull request #31677: [SPARK-34565][SQL] Collapse Window nodes with Project between them
Posted by GitBox <gi...@apache.org>.
tanelk commented on pull request #31677:
URL: https://github.com/apache/spark/pull/31677#issuecomment-800864784
@HyukjinKwon, this has been aproved for a while now, any change, that we can merge this?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #31677: [SPARK-34565][SQL] Collapse Window nodes with Project between them
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #31677:
URL: https://github.com/apache/spark/pull/31677#issuecomment-792590404
Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/40441/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] tanelk commented on pull request #31677: [SPARK-34565][SQL] Collapse Window nodes with Project between them
Posted by GitBox <gi...@apache.org>.
tanelk commented on pull request #31677:
URL: https://github.com/apache/spark/pull/31677#issuecomment-858317298
This and a very similar PR #31980 have been approved for a while now. @maropu , could you take another look, maybe we can merge this?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] wangyum commented on pull request #31677: [SPARK-34565][SQL] Collapse Window nodes with Project between them
Posted by GitBox <gi...@apache.org>.
wangyum commented on pull request #31677:
URL: https://github.com/apache/spark/pull/31677#issuecomment-810283420
Another similar issue: https://github.com/apache/spark/pull/31980.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #31677: [SPARK-34565][SQL] Collapse Window nodes with Project between them
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #31677:
URL: https://github.com/apache/spark/pull/31677#issuecomment-787206719
**[Test build #135548 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/135548/testReport)** for PR 31677 at commit [`6205110`](https://github.com/apache/spark/commit/620511036fe137be29a909029181d381aea9b1dc).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #31677: [SPARK-34565][SQL] Collapse Window nodes with Project between them
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #31677:
URL: https://github.com/apache/spark/pull/31677#issuecomment-859238226
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #31677: [SPARK-34565][SQL] Collapse Window nodes with Project between them
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #31677:
URL: https://github.com/apache/spark/pull/31677#issuecomment-808842815
**[Test build #136605 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/136605/testReport)** for PR 31677 at commit [`3ddd0d3`](https://github.com/apache/spark/commit/3ddd0d3d7a80531a1ec35fceeb4321c2c7fab56e).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] tanelk commented on a change in pull request #31677: [SPARK-34565][SQL] Collapse Window nodes with Project between them
Posted by GitBox <gi...@apache.org>.
tanelk commented on a change in pull request #31677:
URL: https://github.com/apache/spark/pull/31677#discussion_r585783079
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/Optimizer.scala
##########
@@ -914,13 +914,27 @@ object OptimizeWindowFunctions extends Rule[LogicalPlan] {
*/
object CollapseWindow extends Rule[LogicalPlan] {
def apply(plan: LogicalPlan): LogicalPlan = plan transformUp {
- case w1 @ Window(we1, ps1, os1, w2 @ Window(we2, ps2, os2, grandChild))
- if ps1 == ps2 && os1 == os2 && w1.references.intersect(w2.windowOutputSet).isEmpty &&
- we1.nonEmpty && we2.nonEmpty &&
- // This assumes Window contains the same type of window expressions. This is ensured
- // by ExtractWindowFunctions.
- WindowFunctionType.functionType(we1.head) == WindowFunctionType.functionType(we2.head) =>
+ case w1 @ Window(we1, _, _, w2 @ Window(we2, _, _, grandChild))
+ if windowsCompatible(w1, w2) =>
w1.copy(windowExpressions = we2 ++ we1, child = grandChild)
+
+ case w1 @ Window(we1, _, _, Project(pl, w2 @ Window(we2, _, _, grandChild)))
Review comment:
Ah, you are correct. I added a test case for that.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #31677: [SPARK-34565][SQL] Collapse Window nodes with Project between them
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #31677:
URL: https://github.com/apache/spark/pull/31677#issuecomment-808911068
Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/41199/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #31677: [SPARK-34565][SQL] Collapse Window nodes with Project between them
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #31677:
URL: https://github.com/apache/spark/pull/31677#issuecomment-808743146
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/41177/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #31677: [SPARK-34565][SQL] Collapse Window nodes with Project between them
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #31677:
URL: https://github.com/apache/spark/pull/31677#issuecomment-792276293
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/135840/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #31677: [SPARK-34565][SQL] Collapse Window nodes with Project between them
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #31677:
URL: https://github.com/apache/spark/pull/31677#issuecomment-787385430
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/135548/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #31677: [SPARK-34565][SQL] Collapse Window nodes with Project between them
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #31677:
URL: https://github.com/apache/spark/pull/31677#issuecomment-808947114
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/136617/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #31677: [SPARK-34565][SQL] Collapse Window nodes with Project between them
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #31677:
URL: https://github.com/apache/spark/pull/31677#issuecomment-808842815
**[Test build #136605 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/136605/testReport)** for PR 31677 at commit [`3ddd0d3`](https://github.com/apache/spark/commit/3ddd0d3d7a80531a1ec35fceeb4321c2c7fab56e).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #31677: [SPARK-34565][SQL] Collapse Window nodes with Project between them
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #31677:
URL: https://github.com/apache/spark/pull/31677#issuecomment-808912536
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/41199/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #31677: [SPARK-34565][SQL] Collapse Window nodes with Project between them
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #31677:
URL: https://github.com/apache/spark/pull/31677#issuecomment-808732564
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/41177/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #31677: [SPARK-34565][SQL] Collapse Window nodes with Project between them
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #31677:
URL: https://github.com/apache/spark/pull/31677#issuecomment-864352441
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/44539/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #31677: [SPARK-34565][SQL] Collapse Window nodes with Project between them
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #31677:
URL: https://github.com/apache/spark/pull/31677#issuecomment-808725259
**[Test build #136593 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/136593/testReport)** for PR 31677 at commit [`77e93a6`](https://github.com/apache/spark/commit/77e93a6ebbf92cc7714b8c3c24007129691b2a0a).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #31677: [SPARK-34565][SQL] Collapse Window nodes with Project between them
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #31677:
URL: https://github.com/apache/spark/pull/31677#issuecomment-808871043
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/136605/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #31677: [SPARK-34565][SQL] Collapse Window nodes with Project between them
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #31677:
URL: https://github.com/apache/spark/pull/31677#issuecomment-808725259
**[Test build #136593 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/136593/testReport)** for PR 31677 at commit [`77e93a6`](https://github.com/apache/spark/commit/77e93a6ebbf92cc7714b8c3c24007129691b2a0a).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #31677: [SPARK-34565][SQL] Collapse Window nodes with Project between them
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #31677:
URL: https://github.com/apache/spark/pull/31677#issuecomment-787206719
**[Test build #135548 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/135548/testReport)** for PR 31677 at commit [`6205110`](https://github.com/apache/spark/commit/620511036fe137be29a909029181d381aea9b1dc).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #31677: [SPARK-34565][SQL] Collapse Window nodes with Project between them
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #31677:
URL: https://github.com/apache/spark/pull/31677#issuecomment-847310473
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/138874/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #31677: [SPARK-34565][SQL] Collapse Window nodes with Project between them
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #31677:
URL: https://github.com/apache/spark/pull/31677#issuecomment-787366341
**[Test build #135548 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/135548/testReport)** for PR 31677 at commit [`6205110`](https://github.com/apache/spark/commit/620511036fe137be29a909029181d381aea9b1dc).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #31677: [SPARK-34565][SQL] Collapse Window nodes with Project between them
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #31677:
URL: https://github.com/apache/spark/pull/31677#issuecomment-808912536
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/41199/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #31677: [SPARK-34565][SQL] Collapse Window nodes with Project between them
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #31677:
URL: https://github.com/apache/spark/pull/31677#issuecomment-792568568
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/40441/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] maropu closed pull request #31677: [SPARK-34565][SQL] Collapse Window nodes with Project between them
Posted by GitBox <gi...@apache.org>.
maropu closed pull request #31677:
URL: https://github.com/apache/spark/pull/31677
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #31677: [SPARK-34565][SQL] Collapse Window nodes with Project between them
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #31677:
URL: https://github.com/apache/spark/pull/31677#issuecomment-792238811
**[Test build #135840 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/135840/testReport)** for PR 31677 at commit [`b5558c6`](https://github.com/apache/spark/commit/b5558c6735acdea8bd5a10c527e057ec388987d4).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #31677: [SPARK-34565][SQL] Collapse Window nodes with Project between them
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #31677:
URL: https://github.com/apache/spark/pull/31677#issuecomment-808743146
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/41177/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #31677: [SPARK-34565][SQL] Collapse Window nodes with Project between them
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #31677:
URL: https://github.com/apache/spark/pull/31677#issuecomment-787217143
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/40129/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #31677: [SPARK-34565][SQL] Collapse Window nodes with Project between them
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #31677:
URL: https://github.com/apache/spark/pull/31677#issuecomment-792602445
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/40441/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #31677: [SPARK-34565][SQL] Collapse Window nodes with Project between them
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #31677:
URL: https://github.com/apache/spark/pull/31677#issuecomment-808846261
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/41188/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on pull request #31677: [SPARK-34565][SQL] Collapse Window nodes with Project between them
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on pull request #31677:
URL: https://github.com/apache/spark/pull/31677#issuecomment-792234653
retest this please
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #31677: [SPARK-34565][SQL] Collapse Window nodes with Project between them
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #31677:
URL: https://github.com/apache/spark/pull/31677#issuecomment-808910143
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/41199/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #31677: [SPARK-34565][SQL] Collapse Window nodes with Project between them
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #31677:
URL: https://github.com/apache/spark/pull/31677#issuecomment-808945108
**[Test build #136617 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/136617/testReport)** for PR 31677 at commit [`80c9a9a`](https://github.com/apache/spark/commit/80c9a9aa1a415c4b4de8a7cbf246a48256652583).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #31677: [SPARK-34565][SQL] Collapse Window nodes with Project between them
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #31677:
URL: https://github.com/apache/spark/pull/31677#issuecomment-789110803
**[Test build #135670 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/135670/testReport)** for PR 31677 at commit [`b5558c6`](https://github.com/apache/spark/commit/b5558c6735acdea8bd5a10c527e057ec388987d4).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #31677: [SPARK-34565][SQL] Collapse Window nodes with Project between them
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #31677:
URL: https://github.com/apache/spark/pull/31677#issuecomment-864368267
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/140013/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #31677: [SPARK-34565][SQL] Collapse Window nodes with Project between them
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #31677:
URL: https://github.com/apache/spark/pull/31677#issuecomment-792238811
**[Test build #135840 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/135840/testReport)** for PR 31677 at commit [`b5558c6`](https://github.com/apache/spark/commit/b5558c6735acdea8bd5a10c527e057ec388987d4).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #31677: [SPARK-34565][SQL] Collapse Window nodes with Project between them
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #31677:
URL: https://github.com/apache/spark/pull/31677#issuecomment-864352441
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/44539/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon removed a comment on pull request #31677: [SPARK-34565][SQL] Collapse Window nodes with Project between them
Posted by GitBox <gi...@apache.org>.
HyukjinKwon removed a comment on pull request #31677:
URL: https://github.com/apache/spark/pull/31677#issuecomment-860132059
retest this please
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #31677: [SPARK-34565][SQL] Collapse Window nodes with Project between them
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #31677:
URL: https://github.com/apache/spark/pull/31677#issuecomment-792276293
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/135840/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] wangyum commented on a change in pull request #31677: [SPARK-34565][SQL] Collapse Window nodes with Project between them
Posted by GitBox <gi...@apache.org>.
wangyum commented on a change in pull request #31677:
URL: https://github.com/apache/spark/pull/31677#discussion_r602817891
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/Optimizer.scala
##########
@@ -914,13 +914,27 @@ object OptimizeWindowFunctions extends Rule[LogicalPlan] {
*/
object CollapseWindow extends Rule[LogicalPlan] {
def apply(plan: LogicalPlan): LogicalPlan = plan transformUp {
- case w1 @ Window(we1, ps1, os1, w2 @ Window(we2, ps2, os2, grandChild))
- if ps1 == ps2 && os1 == os2 && w1.references.intersect(w2.windowOutputSet).isEmpty &&
- we1.nonEmpty && we2.nonEmpty &&
- // This assumes Window contains the same type of window expressions. This is ensured
- // by ExtractWindowFunctions.
- WindowFunctionType.functionType(we1.head) == WindowFunctionType.functionType(we2.head) =>
+ case w1 @ Window(we1, _, _, w2 @ Window(we2, _, _, grandChild))
+ if windowsCompatible(w1, w2) =>
w1.copy(windowExpressions = we2 ++ we1, child = grandChild)
+
+ case w1 @ Window(we1, _, _, Project(pl, w2 @ Window(we2, _, _, grandChild)))
+ if windowsCompatible(w1, w2) && w1.references.subsetOf(grandChild.outputSet) =>
+ Project(
+ pl ++ w1.windowOutputSet,
+ w1.copy(windowExpressions = we2 ++ we1, child = grandChild)
+ )
Review comment:
```scala
Project(
pl ++ w1.windowOutputSet,
w1.copy(windowExpressions = we2 ++ we1, child = grandChild)
)
```
to
```scala
Project(
pl ++ w1.windowOutputSet,
w1.copy(windowExpressions = we2 ++ we1, child = grandChild))
```
?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #31677: [SPARK-34565][SQL] Collapse Window nodes with Project between them
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #31677:
URL: https://github.com/apache/spark/pull/31677#issuecomment-864364846
**[Test build #140013 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/140013/testReport)** for PR 31677 at commit [`7097ec3`](https://github.com/apache/spark/commit/7097ec3c724356a0d4c226b1222845b6df738e39).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #31677: [SPARK-34565][SQL] Collapse Window nodes with Project between them
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #31677:
URL: https://github.com/apache/spark/pull/31677#issuecomment-808905230
**[Test build #136617 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/136617/testReport)** for PR 31677 at commit [`80c9a9a`](https://github.com/apache/spark/commit/80c9a9aa1a415c4b4de8a7cbf246a48256652583).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #31677: [SPARK-34565][SQL] Collapse Window nodes with Project between them
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #31677:
URL: https://github.com/apache/spark/pull/31677#issuecomment-808765973
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/136593/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #31677: [SPARK-34565][SQL] Collapse Window nodes with Project between them
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #31677:
URL: https://github.com/apache/spark/pull/31677#issuecomment-789168375
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #31677: [SPARK-34565][SQL] Collapse Window nodes with Project between them
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #31677:
URL: https://github.com/apache/spark/pull/31677#issuecomment-847160184
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/43396/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] maropu commented on pull request #31677: [SPARK-34565][SQL] Collapse Window nodes with Project between them
Posted by GitBox <gi...@apache.org>.
maropu commented on pull request #31677:
URL: https://github.com/apache/spark/pull/31677#issuecomment-865020771
Thank you, @tanelk . Merged to master.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #31677: [SPARK-34565][SQL] Collapse Window nodes with Project between them
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #31677:
URL: https://github.com/apache/spark/pull/31677#issuecomment-792250323
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/40422/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] hvanhovell commented on a change in pull request #31677: [SPARK-34565][SQL] Collapse Window nodes with Project between them
Posted by GitBox <gi...@apache.org>.
hvanhovell commented on a change in pull request #31677:
URL: https://github.com/apache/spark/pull/31677#discussion_r585381065
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/Optimizer.scala
##########
@@ -914,13 +914,27 @@ object OptimizeWindowFunctions extends Rule[LogicalPlan] {
*/
object CollapseWindow extends Rule[LogicalPlan] {
def apply(plan: LogicalPlan): LogicalPlan = plan transformUp {
- case w1 @ Window(we1, ps1, os1, w2 @ Window(we2, ps2, os2, grandChild))
- if ps1 == ps2 && os1 == os2 && w1.references.intersect(w2.windowOutputSet).isEmpty &&
- we1.nonEmpty && we2.nonEmpty &&
- // This assumes Window contains the same type of window expressions. This is ensured
- // by ExtractWindowFunctions.
- WindowFunctionType.functionType(we1.head) == WindowFunctionType.functionType(we2.head) =>
+ case w1 @ Window(we1, _, _, w2 @ Window(we2, _, _, grandChild))
+ if windowsCompatible(w1, w2) =>
w1.copy(windowExpressions = we2 ++ we1, child = grandChild)
+
+ case w1 @ Window(we1, _, _, Project(pl, w2 @ Window(we2, _, _, grandChild)))
Review comment:
What happens if `w1` uses an attribute by produced by the project?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #31677: [SPARK-34565][SQL] Collapse Window nodes with Project between them
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #31677:
URL: https://github.com/apache/spark/pull/31677#issuecomment-864352104
Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/44539/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #31677: [SPARK-34565][SQL] Collapse Window nodes with Project between them
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #31677:
URL: https://github.com/apache/spark/pull/31677#issuecomment-792250323
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/40422/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #31677: [SPARK-34565][SQL] Collapse Window nodes with Project between them
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #31677:
URL: https://github.com/apache/spark/pull/31677#issuecomment-789168372
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #31677: [SPARK-34565][SQL] Collapse Window nodes with Project between them
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #31677:
URL: https://github.com/apache/spark/pull/31677#issuecomment-847157677
Kubernetes integration test unable to build dist.
exiting with code: 1
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/43396/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #31677: [SPARK-34565][SQL] Collapse Window nodes with Project between them
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #31677:
URL: https://github.com/apache/spark/pull/31677#issuecomment-864341819
**[Test build #140013 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/140013/testReport)** for PR 31677 at commit [`7097ec3`](https://github.com/apache/spark/commit/7097ec3c724356a0d4c226b1222845b6df738e39).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #31677: [SPARK-34565][SQL] Collapse Window nodes with Project between them
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #31677:
URL: https://github.com/apache/spark/pull/31677#issuecomment-787214989
Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/40129/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on pull request #31677: [SPARK-34565][SQL] Collapse Window nodes with Project between them
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on pull request #31677:
URL: https://github.com/apache/spark/pull/31677#issuecomment-810131974
@hvanhovell does it look good to you too?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #31677: [SPARK-34565][SQL] Collapse Window nodes with Project between them
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #31677:
URL: https://github.com/apache/spark/pull/31677#issuecomment-792246588
Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/40422/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #31677: [SPARK-34565][SQL] Collapse Window nodes with Project between them
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #31677:
URL: https://github.com/apache/spark/pull/31677#issuecomment-864341819
**[Test build #140013 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/140013/testReport)** for PR 31677 at commit [`7097ec3`](https://github.com/apache/spark/commit/7097ec3c724356a0d4c226b1222845b6df738e39).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] maropu commented on pull request #31677: [SPARK-34565][SQL] Collapse Window nodes with Project between them
Posted by GitBox <gi...@apache.org>.
maropu commented on pull request #31677:
URL: https://github.com/apache/spark/pull/31677#issuecomment-865020771
Thank you, @tanelk . Merged to master.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #31677: [SPARK-34565][SQL] Collapse Window nodes with Project between them
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #31677:
URL: https://github.com/apache/spark/pull/31677#issuecomment-808760219
**[Test build #136593 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/136593/testReport)** for PR 31677 at commit [`77e93a6`](https://github.com/apache/spark/commit/77e93a6ebbf92cc7714b8c3c24007129691b2a0a).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #31677: [SPARK-34565][SQL] Collapse Window nodes with Project between them
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #31677:
URL: https://github.com/apache/spark/pull/31677#issuecomment-808905230
**[Test build #136617 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/136617/testReport)** for PR 31677 at commit [`80c9a9a`](https://github.com/apache/spark/commit/80c9a9aa1a415c4b4de8a7cbf246a48256652583).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] maropu commented on pull request #31677: [SPARK-34565][SQL] Collapse Window nodes with Project between them
Posted by GitBox <gi...@apache.org>.
maropu commented on pull request #31677:
URL: https://github.com/apache/spark/pull/31677#issuecomment-859231193
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #31677: [SPARK-34565][SQL] Collapse Window nodes with Project between them
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #31677:
URL: https://github.com/apache/spark/pull/31677#issuecomment-847125255
**[Test build #138874 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/138874/testReport)** for PR 31677 at commit [`de3a7f1`](https://github.com/apache/spark/commit/de3a7f1cc864261429fd19e69d9f1119a5bc000d).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #31677: [SPARK-34565][SQL] Collapse Window nodes with Project between them
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #31677:
URL: https://github.com/apache/spark/pull/31677#issuecomment-808849650
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/41188/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #31677: [SPARK-34565][SQL] Collapse Window nodes with Project between them
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #31677:
URL: https://github.com/apache/spark/pull/31677#issuecomment-787217143
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/40129/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] wangyum commented on pull request #31677: [SPARK-34565][SQL] Collapse Window nodes with Project between them
Posted by GitBox <gi...@apache.org>.
wangyum commented on pull request #31677:
URL: https://github.com/apache/spark/pull/31677#issuecomment-860338233
retest this please
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #31677: [SPARK-34565][SQL] Collapse Window nodes with Project between them
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #31677:
URL: https://github.com/apache/spark/pull/31677#issuecomment-859239766
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] maropu closed pull request #31677: [SPARK-34565][SQL] Collapse Window nodes with Project between them
Posted by GitBox <gi...@apache.org>.
maropu closed pull request #31677:
URL: https://github.com/apache/spark/pull/31677
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #31677: [SPARK-34565][SQL] Collapse Window nodes with Project between them
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #31677:
URL: https://github.com/apache/spark/pull/31677#issuecomment-808765973
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/136593/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #31677: [SPARK-34565][SQL] Collapse Window nodes with Project between them
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #31677:
URL: https://github.com/apache/spark/pull/31677#issuecomment-808871043
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/136605/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #31677: [SPARK-34565][SQL] Collapse Window nodes with Project between them
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #31677:
URL: https://github.com/apache/spark/pull/31677#issuecomment-808867075
**[Test build #136605 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/136605/testReport)** for PR 31677 at commit [`3ddd0d3`](https://github.com/apache/spark/commit/3ddd0d3d7a80531a1ec35fceeb4321c2c7fab56e).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] tanelk commented on pull request #31677: [SPARK-34565][SQL] Collapse Window nodes with Project between them
Posted by GitBox <gi...@apache.org>.
tanelk commented on pull request #31677:
URL: https://github.com/apache/spark/pull/31677#issuecomment-847125608
@HyukjinKwon, this has few aprovals, but is there anyone else you would like to take a look at it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on pull request #31677: [SPARK-34565][SQL] Collapse Window nodes with Project between them
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on pull request #31677:
URL: https://github.com/apache/spark/pull/31677#issuecomment-793213226
@cloud-fan and @hvanhovell, do you have any comment on this? If there's no comments in few days, I will merge this in.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #31677: [SPARK-34565][SQL] Collapse Window nodes with Project between them
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #31677:
URL: https://github.com/apache/spark/pull/31677#issuecomment-792561644
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/135859/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #31677: [SPARK-34565][SQL] Collapse Window nodes with Project between them
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #31677:
URL: https://github.com/apache/spark/pull/31677#issuecomment-847160184
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/43396/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #31677: [SPARK-34565][SQL] Collapse Window nodes with Project between them
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #31677:
URL: https://github.com/apache/spark/pull/31677#issuecomment-808947114
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/136617/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #31677: [SPARK-34565][SQL] Collapse Window nodes with Project between them
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #31677:
URL: https://github.com/apache/spark/pull/31677#issuecomment-787211168
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/40129/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #31677: [SPARK-34565][SQL] Collapse Window nodes with Project between them
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #31677:
URL: https://github.com/apache/spark/pull/31677#issuecomment-787385430
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/135548/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on pull request #31677: [SPARK-34565][SQL] Collapse Window nodes with Project between them
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on pull request #31677:
URL: https://github.com/apache/spark/pull/31677#issuecomment-800894675
`Aggregate` kinds of embed a project list and it's easier to write rules to deal with it. We probably should do the same for `Window`. We hit similar issues in https://github.com/apache/spark/pull/31691/files#diff-ce15d5a4cc87dff76686e359221425fb5df32b114a9b2aec5ea18b66d60c0148R49 that we have to take extra care of Project + Window.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #31677: [SPARK-34565][SQL] Collapse Window nodes with Project between them
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #31677:
URL: https://github.com/apache/spark/pull/31677#issuecomment-808847969
Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/41188/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #31677: [SPARK-34565][SQL] Collapse Window nodes with Project between them
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #31677:
URL: https://github.com/apache/spark/pull/31677#issuecomment-859239766
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #31677: [SPARK-34565][SQL] Collapse Window nodes with Project between them
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #31677:
URL: https://github.com/apache/spark/pull/31677#issuecomment-792602445
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/40441/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #31677: [SPARK-34565][SQL] Collapse Window nodes with Project between them
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #31677:
URL: https://github.com/apache/spark/pull/31677#issuecomment-792275043
**[Test build #135840 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/135840/testReport)** for PR 31677 at commit [`b5558c6`](https://github.com/apache/spark/commit/b5558c6735acdea8bd5a10c527e057ec388987d4).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org