You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by GitBox <gi...@apache.org> on 2021/04/01 23:19:09 UTC
[GitHub] [spark] dbaliafroozeh opened a new pull request #32030: [WIP] Improve map children
dbaliafroozeh opened a new pull request #32030:
URL: https://github.com/apache/spark/pull/32030
<!--
Thanks for sending a pull request! Here are some tips for you:
1. If this is your first time, please read our contributor guidelines: https://spark.apache.org/contributing.html
2. Ensure you have added or run the appropriate tests for your PR: https://spark.apache.org/developer-tools.html
3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][SPARK-XXXX] Your PR title ...'.
4. Be sure to keep the PR description updated to reflect all changes.
5. Please write your PR title to summarize what this PR proposes.
6. If possible, provide a concise example to reproduce the issue for a faster review.
7. If you want to add a new configuration, please read the guideline first for naming configurations in
'core/src/main/scala/org/apache/spark/internal/config/ConfigEntry.scala'.
-->
### What changes were proposed in this pull request?
<!--
Please clarify what changes you are proposing. The purpose of this section is to outline the changes and how this PR fixes the issue.
If possible, please consider writing useful notes for better and faster reviews in your PR. See the examples below.
1. If you refactor some codes with changing classes, showing the class hierarchy will help reviewers.
2. If you fix some SQL features, you can provide some references of other DBMSes.
3. If there is design documentation, please add the link.
4. If there is a discussion in the mailing list, please add the link.
-->
### Why are the changes needed?
<!--
Please clarify why the changes are needed. For instance,
1. If you propose a new API, clarify the use case for a new API.
2. If you fix a bug, you can clarify why it is a bug.
-->
### Does this PR introduce _any_ user-facing change?
<!--
Note that it means *any* user-facing change including all aspects such as the documentation fix.
If yes, please clarify the previous behavior and the change this PR proposes - provide the console output, description and/or an example to show the behavior difference if possible.
If possible, please also clarify if this is a user-facing change compared to the released Spark versions or within the unreleased branches such as master.
If no, write 'No'.
-->
### How was this patch tested?
<!--
If tests were added, say they were added here. Please make sure to add some test cases that check the changes thoroughly including negative and positive cases if possible.
If it was tested in a way different from regular unit tests, please clarify how you tested step by step, ideally copy and paste-able, so that other reviewers can test and check, and descendants can verify in the future.
If tests were not added, please describe why they were not added and/or why it was difficult to add.
-->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32030: [SPARK-34989] Improve the performance of mapChildren and withNewChildren methods
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32030:
URL: https://github.com/apache/spark/pull/32030#issuecomment-816773548
**[Test build #137127 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/137127/testReport)** for PR 32030 at commit [`2bb22ee`](https://github.com/apache/spark/commit/2bb22ee8a072606816c71433920d47eaac889932).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #32030: [SPARK-34989] Improve the performance of mapChildren and withNewChildren methods
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #32030:
URL: https://github.com/apache/spark/pull/32030#issuecomment-815813621
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/137070/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] sigmod commented on a change in pull request #32030: [SPARK-34989] Improve the performance of mapChildren and withNewChildren methods
Posted by GitBox <gi...@apache.org>.
sigmod commented on a change in pull request #32030:
URL: https://github.com/apache/spark/pull/32030#discussion_r609857814
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/trees/TreeNode.scala
##########
@@ -246,11 +246,50 @@ abstract class TreeNode[BaseType <: TreeNode[BaseType]] extends Product {
arr
}
+ private def childrenTheSame(
+ originalChildren: IndexedSeq[BaseType], newChildren: IndexedSeq[BaseType]): Boolean = {
+ val size = originalChildren.size
+ var i = 0
+ var childrenTheSame = true
+ while (i < size && childrenTheSame) {
Review comment:
can we use early return in the loop body?
(it seems more readable to me)
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/trees/TreeNode.scala
##########
@@ -246,11 +246,50 @@ abstract class TreeNode[BaseType <: TreeNode[BaseType]] extends Product {
arr
}
+ private def childrenTheSame(
+ originalChildren: IndexedSeq[BaseType], newChildren: IndexedSeq[BaseType]): Boolean = {
+ val size = originalChildren.size
+ var i = 0
+ var childrenTheSame = true
+ while (i < size && childrenTheSame) {
+ childrenTheSame &&= originalChildren(i) fastEquals newChildren(i)
+ i += 1
+ }
+ childrenTheSame
+ }
+
+ // This is a temporary solution, we will change the type of children to IndexedSeq in a
+ // followup PR
+ private def asIndexedSeq(seq: Seq[BaseType]): IndexedSeq[BaseType] = {
+ if (seq.isInstanceOf[IndexedSeq[BaseType]]) {
+ seq.asInstanceOf[IndexedSeq[BaseType]]
+ } else {
+ seq.toIndexedSeq
+ }
+ }
+
+ def withNewChildren(newChildren: Seq[BaseType]): BaseType = {
Review comment:
Mark this function as final?
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/trees/TreeNode.scala
##########
@@ -246,11 +246,50 @@ abstract class TreeNode[BaseType <: TreeNode[BaseType]] extends Product {
arr
}
+ private def childrenTheSame(
+ originalChildren: IndexedSeq[BaseType], newChildren: IndexedSeq[BaseType]): Boolean = {
+ val size = originalChildren.size
+ var i = 0
+ var childrenTheSame = true
+ while (i < size && childrenTheSame) {
+ childrenTheSame &&= originalChildren(i) fastEquals newChildren(i)
+ i += 1
+ }
+ childrenTheSame
+ }
+
+ // This is a temporary solution, we will change the type of children to IndexedSeq in a
+ // followup PR
+ private def asIndexedSeq(seq: Seq[BaseType]): IndexedSeq[BaseType] = {
+ if (seq.isInstanceOf[IndexedSeq[BaseType]]) {
+ seq.asInstanceOf[IndexedSeq[BaseType]]
+ } else {
+ seq.toIndexedSeq
+ }
+ }
+
+ def withNewChildren(newChildren: Seq[BaseType]): BaseType = {
+ val childrenIndexedSeq = asIndexedSeq(children)
+ val newChildrenIndexedSeq = asIndexedSeq(newChildren)
+ assert(newChildrenIndexedSeq.size == childrenIndexedSeq.size, "Incorrect number of children")
+ if (childrenIndexedSeq.isEmpty || childrenTheSame(newChildrenIndexedSeq, childrenIndexedSeq)) {
+ this
+ } else {
+ CurrentOrigin.withOrigin(origin) {
+ val res = withNewChildrenInternal(newChildrenIndexedSeq)
+ res.copyTagsFrom(this)
+ res
+ }
+ }
+ }
+
+ protected def withNewChildrenInternal(newChildren: IndexedSeq[BaseType]): BaseType
+
/**
* Returns a copy of this node with the children replaced.
* TODO: Validate somewhere (in debug mode?) that children are ordered correctly.
*/
- def withNewChildren(newChildren: Seq[BaseType]): BaseType = {
+ def legacyWithNewChildren(newChildren: Seq[BaseType]): BaseType = {
Review comment:
mark this function as final?
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/trees/TreeNode.scala
##########
@@ -246,11 +246,50 @@ abstract class TreeNode[BaseType <: TreeNode[BaseType]] extends Product {
arr
}
+ private def childrenTheSame(
Review comment:
Nit:
childrenFastEquals
(so as to be consistent with the use of "fastEquals")
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #32030: [SPARK-34989] Improve the performance of mapChildren and withNewChildren methods
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #32030:
URL: https://github.com/apache/spark/pull/32030#issuecomment-816611642
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/41705/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] hvanhovell commented on pull request #32030: [SPARK-34989] Improve the performance of mapChildren and withNewChildren methods
Posted by GitBox <gi...@apache.org>.
hvanhovell commented on pull request #32030:
URL: https://github.com/apache/spark/pull/32030#issuecomment-816668103
Merging to master. Thanks!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32030: [SPARK-34989] Improve the performance of mapChildren and withNewChildren methods
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32030:
URL: https://github.com/apache/spark/pull/32030#issuecomment-815847802
Kubernetes integration test unable to build dist.
exiting with code: 1
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/41658/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] hvanhovell closed pull request #32030: [SPARK-34989] Improve the performance of mapChildren and withNewChildren methods
Posted by GitBox <gi...@apache.org>.
hvanhovell closed pull request #32030:
URL: https://github.com/apache/spark/pull/32030
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #32030: [SPARK-34989] Improve the performance of mapChildren and withNewChildren methods
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #32030:
URL: https://github.com/apache/spark/pull/32030#issuecomment-815962659
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #32030: [WIP] Improve map children
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #32030:
URL: https://github.com/apache/spark/pull/32030#issuecomment-814515791
**[Test build #136975 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/136975/testReport)** for PR 32030 at commit [`b526852`](https://github.com/apache/spark/commit/b526852c5133c09f52729ce55bbb2e87b51c915e).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32030: [WIP] Improve the performance of mapChildren and withNewChildren methods
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32030:
URL: https://github.com/apache/spark/pull/32030#issuecomment-815205303
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/41609/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32030: [SPARK-34989] Improve the performance of mapChildren and withNewChildren methods
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32030:
URL: https://github.com/apache/spark/pull/32030#issuecomment-816146755
**[Test build #137098 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/137098/testReport)** for PR 32030 at commit [`8a27cb1`](https://github.com/apache/spark/commit/8a27cb1921bf332d1754dfb7d72a0b5c57fbbda9).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32030: [WIP] Improve map children
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32030:
URL: https://github.com/apache/spark/pull/32030#issuecomment-814402920
**[Test build #136957 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/136957/testReport)** for PR 32030 at commit [`8a19f14`](https://github.com/apache/spark/commit/8a19f14d7583b39eca94a305d2d2a16e29e7aadf).
* This patch **fails Spark unit tests**.
* This patch merges cleanly.
* This patch adds no public classes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #32030: [SPARK-34989] Improve the performance of mapChildren and withNewChildren methods
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #32030:
URL: https://github.com/apache/spark/pull/32030#issuecomment-816799602
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/137133/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #32030: [WIP] Improve the performance of mapChildren and withNewChildren methods
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #32030:
URL: https://github.com/apache/spark/pull/32030#issuecomment-815081268
**[Test build #137029 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/137029/testReport)** for PR 32030 at commit [`b6c563d`](https://github.com/apache/spark/commit/b6c563d79876bc0d1e7f47d37623c3836397fa38).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #32030: [SPARK-34989] Improve the performance of mapChildren and withNewChildren methods
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #32030:
URL: https://github.com/apache/spark/pull/32030#issuecomment-816315747
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/41683/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #32030: [WIP] Improve map children
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #32030:
URL: https://github.com/apache/spark/pull/32030#issuecomment-814444532
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/136961/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dbaliafroozeh commented on a change in pull request #32030: [WIP] Improve the performance of mapChildren and withNewChildren methods
Posted by GitBox <gi...@apache.org>.
dbaliafroozeh commented on a change in pull request #32030:
URL: https://github.com/apache/spark/pull/32030#discussion_r609141398
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/aggregate/interfaces.scala
##########
@@ -164,6 +164,9 @@ case class AggregateExpression(
case _ => aggFuncStr
}
}
+
+ override protected def withNewChildrenInternal(newChildren: IndexedSeq[Expression]): Expression =
+ super.legacyWithNewChildren(newChildren)
Review comment:
Fixed it. Probably it makes sense to have a special node, e.g., `UnaryWithOptional` for nodes with one child and one optional child. There are a number of some nodes. Manually implementing them is tricky. I'd like to ideally only have to implement the generic version like this:
```
override protected def withNewChildrenInternal(newChildren: IndexedSeq[Expression]): Expression =
copy(children = newChildren)
```
That is, only for nodes with a list of children. We'll do these tweaks in a followup.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #32030: [SPARK-34989] Improve the performance of mapChildren and withNewChildren methods
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #32030:
URL: https://github.com/apache/spark/pull/32030#issuecomment-816613181
**[Test build #137133 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/137133/testReport)** for PR 32030 at commit [`5bc0ff7`](https://github.com/apache/spark/commit/5bc0ff7e88ec11caa06b3c786bcdea9a01ddfcaf).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #32030: [SPARK-34989] Improve the performance of mapChildren and withNewChildren methods
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #32030:
URL: https://github.com/apache/spark/pull/32030#issuecomment-816641515
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/41711/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #32030: [SPARK-34989] Improve the performance of mapChildren and withNewChildren methods
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #32030:
URL: https://github.com/apache/spark/pull/32030#issuecomment-815957896
**[Test build #137088 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/137088/testReport)** for PR 32030 at commit [`9c033d6`](https://github.com/apache/spark/commit/9c033d6210cb8a8287b989d46abed7c142d3fd4d).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #32030: [WIP] Improve the performance of mapChildren and withNewChildren methods
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #32030:
URL: https://github.com/apache/spark/pull/32030#issuecomment-815374761
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/41616/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on a change in pull request #32030: [SPARK-34989] Improve the performance of mapChildren and withNewChildren methods
Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on a change in pull request #32030:
URL: https://github.com/apache/spark/pull/32030#discussion_r610327961
##########
File path: mllib/src/main/scala/org/apache/spark/ml/stat/Summarizer.scala
##########
@@ -349,8 +349,7 @@ private[spark] object SummaryBuilderImpl extends Logging {
weightExpr: Expression,
mutableAggBufferOffset: Int,
inputAggBufferOffset: Int)
- extends TypedImperativeAggregate[SummarizerBuffer]
- with ImplicitCastInputTypes
+ extends TypedImperativeAggregate[SummarizerBuffer] with ImplicitCastInputTypes
Review comment:
Could you revert these two line change, @dbaliafroozeh ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] sigmod commented on a change in pull request #32030: [SPARK-34989] Improve the performance of mapChildren and withNewChildren methods
Posted by GitBox <gi...@apache.org>.
sigmod commented on a change in pull request #32030:
URL: https://github.com/apache/spark/pull/32030#discussion_r609940832
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/trees/TreeNode.scala
##########
@@ -246,11 +246,50 @@ abstract class TreeNode[BaseType <: TreeNode[BaseType]] extends Product {
arr
}
+ private def childrenTheSame(
+ originalChildren: IndexedSeq[BaseType], newChildren: IndexedSeq[BaseType]): Boolean = {
+ val size = originalChildren.size
+ var i = 0
+ var childrenTheSame = true
+ while (i < size && childrenTheSame) {
+ childrenTheSame &&= originalChildren(i) fastEquals newChildren(i)
+ i += 1
+ }
+ childrenTheSame
+ }
+
+ // This is a temporary solution, we will change the type of children to IndexedSeq in a
+ // followup PR
+ private def asIndexedSeq(seq: Seq[BaseType]): IndexedSeq[BaseType] = {
+ if (seq.isInstanceOf[IndexedSeq[BaseType]]) {
+ seq.asInstanceOf[IndexedSeq[BaseType]]
+ } else {
+ seq.toIndexedSeq
+ }
+ }
+
+ def withNewChildren(newChildren: Seq[BaseType]): BaseType = {
+ val childrenIndexedSeq = asIndexedSeq(children)
+ val newChildrenIndexedSeq = asIndexedSeq(newChildren)
+ assert(newChildrenIndexedSeq.size == childrenIndexedSeq.size, "Incorrect number of children")
+ if (childrenIndexedSeq.isEmpty || childrenTheSame(newChildrenIndexedSeq, childrenIndexedSeq)) {
+ this
+ } else {
+ CurrentOrigin.withOrigin(origin) {
+ val res = withNewChildrenInternal(newChildrenIndexedSeq)
+ res.copyTagsFrom(this)
+ res
+ }
+ }
+ }
+
+ protected def withNewChildrenInternal(newChildren: IndexedSeq[BaseType]): BaseType
Review comment:
Ok, makes sense. Thanks for the elaboration!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #32030: [SPARK-34989] Improve the performance of mapChildren and withNewChildren methods
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #32030:
URL: https://github.com/apache/spark/pull/32030#issuecomment-816221728
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/41677/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #32030: [WIP] Improve map children
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #32030:
URL: https://github.com/apache/spark/pull/32030#issuecomment-814283666
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/41534/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on a change in pull request #32030: [SPARK-34989] Improve the performance of mapChildren and withNewChildren methods
Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on a change in pull request #32030:
URL: https://github.com/apache/spark/pull/32030#discussion_r610329219
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/DynamicPruning.scala
##########
@@ -78,6 +78,9 @@ case class DynamicPruningSubquery(
buildKeys = buildKeys.map(_.canonicalized),
exprId = ExprId(0))
}
+
+ override protected def withNewChildInternal(newChild: Expression): Expression =
Review comment:
Just a question, why the `return type` is n`Expression` instead of `DynamicPruningSubquery`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32030: [WIP] Improve the performance of mapChildren and withNewChildren methods
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32030:
URL: https://github.com/apache/spark/pull/32030#issuecomment-815213937
Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/41609/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #32030: [WIP] Improve map children
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #32030:
URL: https://github.com/apache/spark/pull/32030#issuecomment-815042176
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/137028/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #32030: [SPARK-34989] Improve the performance of mapChildren and withNewChildren methods
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #32030:
URL: https://github.com/apache/spark/pull/32030#issuecomment-815963645
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/137088/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #32030: [SPARK-34989] Improve the performance of mapChildren and withNewChildren methods
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #32030:
URL: https://github.com/apache/spark/pull/32030#issuecomment-816146755
**[Test build #137098 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/137098/testReport)** for PR 32030 at commit [`8a27cb1`](https://github.com/apache/spark/commit/8a27cb1921bf332d1754dfb7d72a0b5c57fbbda9).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32030: [SPARK-34989] Improve the performance of mapChildren and withNewChildren methods
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32030:
URL: https://github.com/apache/spark/pull/32030#issuecomment-816640406
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32030: [WIP] Improve map children
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32030:
URL: https://github.com/apache/spark/pull/32030#issuecomment-814286421
**[Test build #136961 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/136961/testReport)** for PR 32030 at commit [`359db0a`](https://github.com/apache/spark/commit/359db0a8fc10626d43583245394099f14f8645be).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #32030: [WIP] Improve map children
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #32030:
URL: https://github.com/apache/spark/pull/32030#issuecomment-814328874
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/41538/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32030: [SPARK-34989] Improve the performance of mapChildren and withNewChildren methods
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32030:
URL: https://github.com/apache/spark/pull/32030#issuecomment-816266209
**[Test build #137102 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/137102/testReport)** for PR 32030 at commit [`85e236c`](https://github.com/apache/spark/commit/85e236ce24ffa37fc12b85d64a4d586a376f20a7).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32030: [SPARK-34989] Improve the performance of mapChildren and withNewChildren methods
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32030:
URL: https://github.com/apache/spark/pull/32030#issuecomment-816613181
**[Test build #137133 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/137133/testReport)** for PR 32030 at commit [`5bc0ff7`](https://github.com/apache/spark/commit/5bc0ff7e88ec11caa06b3c786bcdea9a01ddfcaf).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32030: [WIP] Improve map children
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32030:
URL: https://github.com/apache/spark/pull/32030#issuecomment-814244160
**[Test build #136957 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/136957/testReport)** for PR 32030 at commit [`8a19f14`](https://github.com/apache/spark/commit/8a19f14d7583b39eca94a305d2d2a16e29e7aadf).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #32030: [SPARK-34989] Improve the performance of mapChildren and withNewChildren methods
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #32030:
URL: https://github.com/apache/spark/pull/32030#issuecomment-816413714
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/137105/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32030: [WIP] Improve map children
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32030:
URL: https://github.com/apache/spark/pull/32030#issuecomment-814443761
**[Test build #136961 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/136961/testReport)** for PR 32030 at commit [`359db0a`](https://github.com/apache/spark/commit/359db0a8fc10626d43583245394099f14f8645be).
* This patch **fails Spark unit tests**.
* This patch merges cleanly.
* This patch adds no public classes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #32030: [SPARK-34989] Improve the performance of mapChildren and withNewChildren methods
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #32030:
URL: https://github.com/apache/spark/pull/32030#issuecomment-816082181
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/137080/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dbaliafroozeh commented on a change in pull request #32030: [SPARK-34989] Improve the performance of mapChildren and withNewChildren methods
Posted by GitBox <gi...@apache.org>.
dbaliafroozeh commented on a change in pull request #32030:
URL: https://github.com/apache/spark/pull/32030#discussion_r609889192
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/trees/TreeNode.scala
##########
@@ -246,11 +246,50 @@ abstract class TreeNode[BaseType <: TreeNode[BaseType]] extends Product {
arr
}
+ private def childrenTheSame(
+ originalChildren: IndexedSeq[BaseType], newChildren: IndexedSeq[BaseType]): Boolean = {
+ val size = originalChildren.size
+ var i = 0
+ var childrenTheSame = true
+ while (i < size && childrenTheSame) {
Review comment:
And also `break` in Scala doesn't translate to Java break, throws an exception and is expensive.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32030: [WIP] Improve map children
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32030:
URL: https://github.com/apache/spark/pull/32030#issuecomment-814167899
**[Test build #136944 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/136944/testReport)** for PR 32030 at commit [`698d89d`](https://github.com/apache/spark/commit/698d89d60aabcfe08724091b80e2c86cc5f2f5c7).
* This patch **fails Spark unit tests**.
* This patch merges cleanly.
* This patch adds no public classes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32030: [SPARK-34989] Improve the performance of mapChildren and withNewChildren methods
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32030:
URL: https://github.com/apache/spark/pull/32030#issuecomment-815815938
**[Test build #137080 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/137080/testReport)** for PR 32030 at commit [`27832a2`](https://github.com/apache/spark/commit/27832a2fdd0ee77a020f714514dfd106b9a5083a).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] hvanhovell commented on a change in pull request #32030: [WIP] Improve the performance of mapChildren and withNewChildren methods
Posted by GitBox <gi...@apache.org>.
hvanhovell commented on a change in pull request #32030:
URL: https://github.com/apache/spark/pull/32030#discussion_r609093782
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/aggregate/interfaces.scala
##########
@@ -164,6 +164,9 @@ case class AggregateExpression(
case _ => aggFuncStr
}
}
+
+ override protected def withNewChildrenInternal(newChildren: IndexedSeq[Expression]): Expression =
+ super.legacyWithNewChildren(newChildren)
Review comment:
Why not implement this one?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #32030: [SPARK-34989] Improve the performance of mapChildren and withNewChildren methods
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #32030:
URL: https://github.com/apache/spark/pull/32030#issuecomment-816799602
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/137133/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] sigmod commented on a change in pull request #32030: [SPARK-34989] Improve the performance of mapChildren and withNewChildren methods
Posted by GitBox <gi...@apache.org>.
sigmod commented on a change in pull request #32030:
URL: https://github.com/apache/spark/pull/32030#discussion_r609870502
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/trees/TreeNode.scala
##########
@@ -246,11 +246,50 @@ abstract class TreeNode[BaseType <: TreeNode[BaseType]] extends Product {
arr
}
+ private def childrenTheSame(
+ originalChildren: IndexedSeq[BaseType], newChildren: IndexedSeq[BaseType]): Boolean = {
+ val size = originalChildren.size
+ var i = 0
+ var childrenTheSame = true
+ while (i < size && childrenTheSame) {
+ childrenTheSame &&= originalChildren(i) fastEquals newChildren(i)
+ i += 1
+ }
+ childrenTheSame
+ }
+
+ // This is a temporary solution, we will change the type of children to IndexedSeq in a
+ // followup PR
+ private def asIndexedSeq(seq: Seq[BaseType]): IndexedSeq[BaseType] = {
+ if (seq.isInstanceOf[IndexedSeq[BaseType]]) {
+ seq.asInstanceOf[IndexedSeq[BaseType]]
+ } else {
+ seq.toIndexedSeq
+ }
+ }
+
+ def withNewChildren(newChildren: Seq[BaseType]): BaseType = {
+ val childrenIndexedSeq = asIndexedSeq(children)
+ val newChildrenIndexedSeq = asIndexedSeq(newChildren)
+ assert(newChildrenIndexedSeq.size == childrenIndexedSeq.size, "Incorrect number of children")
+ if (childrenIndexedSeq.isEmpty || childrenTheSame(newChildrenIndexedSeq, childrenIndexedSeq)) {
+ this
+ } else {
+ CurrentOrigin.withOrigin(origin) {
+ val res = withNewChildrenInternal(newChildrenIndexedSeq)
+ res.copyTagsFrom(this)
+ res
+ }
+ }
+ }
+
+ protected def withNewChildrenInternal(newChildren: IndexedSeq[BaseType]): BaseType
+
/**
* Returns a copy of this node with the children replaced.
* TODO: Validate somewhere (in debug mode?) that children are ordered correctly.
*/
- def withNewChildren(newChildren: Seq[BaseType]): BaseType = {
+ def legacyWithNewChildren(newChildren: Seq[BaseType]): BaseType = {
Review comment:
and protected as well?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #32030: [WIP] Improve the performance of mapChildren and withNewChildren methods
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #32030:
URL: https://github.com/apache/spark/pull/32030#issuecomment-815121062
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/137030/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #32030: [SPARK-34989] Improve the performance of mapChildren and withNewChildren methods
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #32030:
URL: https://github.com/apache/spark/pull/32030#issuecomment-816208079
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/41676/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #32030: [WIP] Improve map children
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #32030:
URL: https://github.com/apache/spark/pull/32030#issuecomment-814286421
**[Test build #136961 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/136961/testReport)** for PR 32030 at commit [`359db0a`](https://github.com/apache/spark/commit/359db0a8fc10626d43583245394099f14f8645be).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32030: [SPARK-34989] Improve the performance of mapChildren and withNewChildren methods
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32030:
URL: https://github.com/apache/spark/pull/32030#issuecomment-815957896
**[Test build #137088 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/137088/testReport)** for PR 32030 at commit [`9c033d6`](https://github.com/apache/spark/commit/9c033d6210cb8a8287b989d46abed7c142d3fd4d).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #32030: [WIP] Improve map children
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #32030:
URL: https://github.com/apache/spark/pull/32030#issuecomment-812236261
**[Test build #136828 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/136828/testReport)** for PR 32030 at commit [`7045e7a`](https://github.com/apache/spark/commit/7045e7a8bb844e6a5d48fda0ab06926f67c9f4ca).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #32030: [WIP] Improve the performance of mapChildren and withNewChildren methods
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #32030:
URL: https://github.com/apache/spark/pull/32030#issuecomment-815118758
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/41607/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32030: [SPARK-34989] Improve the performance of mapChildren and withNewChildren methods
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32030:
URL: https://github.com/apache/spark/pull/32030#issuecomment-816149078
**[Test build #137098 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/137098/testReport)** for PR 32030 at commit [`8a27cb1`](https://github.com/apache/spark/commit/8a27cb1921bf332d1754dfb7d72a0b5c57fbbda9).
* This patch **fails Scala style tests**.
* This patch merges cleanly.
* This patch adds no public classes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dbaliafroozeh commented on a change in pull request #32030: [SPARK-34989] Improve the performance of mapChildren and withNewChildren methods
Posted by GitBox <gi...@apache.org>.
dbaliafroozeh commented on a change in pull request #32030:
URL: https://github.com/apache/spark/pull/32030#discussion_r610490212
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/DynamicPruning.scala
##########
@@ -94,4 +97,7 @@ case class DynamicPruningExpression(child: Expression)
override protected def doGenCode(ctx: CodegenContext, ev: ExprCode): ExprCode = {
child.genCode(ctx)
}
+
+ override protected def withNewChildInternal(newChild: Expression): Expression =
Review comment:
Fixed those two cases, thanks for spotting them!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32030: [WIP] Improve map children
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32030:
URL: https://github.com/apache/spark/pull/32030#issuecomment-812254032
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/41408/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dbaliafroozeh commented on a change in pull request #32030: [WIP] Improve the performance of mapChildren and withNewChildren methods
Posted by GitBox <gi...@apache.org>.
dbaliafroozeh commented on a change in pull request #32030:
URL: https://github.com/apache/spark/pull/32030#discussion_r609141398
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/aggregate/interfaces.scala
##########
@@ -164,6 +164,9 @@ case class AggregateExpression(
case _ => aggFuncStr
}
}
+
+ override protected def withNewChildrenInternal(newChildren: IndexedSeq[Expression]): Expression =
+ super.legacyWithNewChildren(newChildren)
Review comment:
Fixed it. Probably it makes sense to have a special node with nodes with a child and an optional child. There are a number of some nodes. Manually implementing them is tricky. I'd like to ideally only have to implement this method like this:
```
override protected def withNewChildrenInternal(newChildren: IndexedSeq[Expression]): Expression =
copy(children = newChildren)
```
That is, only for nodes with a list of children. We'll do these tweaks in a followup.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32030: [SPARK-34989] Improve the performance of mapChildren and withNewChildren methods
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32030:
URL: https://github.com/apache/spark/pull/32030#issuecomment-816341178
**[Test build #137099 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/137099/testReport)** for PR 32030 at commit [`060a098`](https://github.com/apache/spark/commit/060a09890919cbfee410326f825b537099872810).
* This patch **fails PySpark unit tests**.
* This patch merges cleanly.
* This patch adds no public classes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #32030: [SPARK-34989] Improve the performance of mapChildren and withNewChildren methods
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #32030:
URL: https://github.com/apache/spark/pull/32030#issuecomment-816298157
**[Test build #137105 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/137105/testReport)** for PR 32030 at commit [`4130602`](https://github.com/apache/spark/commit/413060242f2ef9f56706076d4fe74ad37b239a7d).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32030: [WIP] Improve the performance of mapChildren and withNewChildren methods
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32030:
URL: https://github.com/apache/spark/pull/32030#issuecomment-815119456
**[Test build #137030 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/137030/testReport)** for PR 32030 at commit [`413ef4f`](https://github.com/apache/spark/commit/413ef4f31ca496707a55c93f8eaf1e4f888e6feb).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32030: [WIP] Improve map children
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32030:
URL: https://github.com/apache/spark/pull/32030#issuecomment-814076465
**[Test build #136944 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/136944/testReport)** for PR 32030 at commit [`698d89d`](https://github.com/apache/spark/commit/698d89d60aabcfe08724091b80e2c86cc5f2f5c7).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #32030: [WIP] Improve map children
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #32030:
URL: https://github.com/apache/spark/pull/32030#issuecomment-812271783
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #32030: [WIP] Improve the performance of mapChildren and withNewChildren methods
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #32030:
URL: https://github.com/apache/spark/pull/32030#issuecomment-815219135
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #32030: [WIP] Improve map children
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #32030:
URL: https://github.com/apache/spark/pull/32030#issuecomment-815042176
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/137028/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #32030: [WIP] Improve map children
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #32030:
URL: https://github.com/apache/spark/pull/32030#issuecomment-814244160
**[Test build #136957 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/136957/testReport)** for PR 32030 at commit [`8a19f14`](https://github.com/apache/spark/commit/8a19f14d7583b39eca94a305d2d2a16e29e7aadf).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #32030: [WIP] Improve map children
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #32030:
URL: https://github.com/apache/spark/pull/32030#issuecomment-814548464
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/41552/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #32030: [SPARK-34989] Improve the performance of mapChildren and withNewChildren methods
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #32030:
URL: https://github.com/apache/spark/pull/32030#issuecomment-816388958
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/137102/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32030: [SPARK-34989] Improve the performance of mapChildren and withNewChildren methods
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32030:
URL: https://github.com/apache/spark/pull/32030#issuecomment-816160395
**[Test build #137099 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/137099/testReport)** for PR 32030 at commit [`060a098`](https://github.com/apache/spark/commit/060a09890919cbfee410326f825b537099872810).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #32030: [SPARK-34989] Improve the performance of mapChildren and withNewChildren methods
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #32030:
URL: https://github.com/apache/spark/pull/32030#issuecomment-816208079
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/41676/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #32030: [WIP] Improve map children
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #32030:
URL: https://github.com/apache/spark/pull/32030#issuecomment-814549884
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/136975/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32030: [WIP] Improve the performance of mapChildren and withNewChildren methods
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32030:
URL: https://github.com/apache/spark/pull/32030#issuecomment-815638652
**[Test build #137070 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/137070/testReport)** for PR 32030 at commit [`498d248`](https://github.com/apache/spark/commit/498d24893270fa0c45182a6c9677981e2ba4b742).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #32030: [WIP] Improve map children
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #32030:
URL: https://github.com/apache/spark/pull/32030#issuecomment-812271782
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32030: [WIP] Improve the performance of mapChildren and withNewChildren methods
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32030:
URL: https://github.com/apache/spark/pull/32030#issuecomment-815369752
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32030: [SPARK-34989] Improve the performance of mapChildren and withNewChildren methods
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32030:
URL: https://github.com/apache/spark/pull/32030#issuecomment-816797775
**[Test build #137133 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/137133/testReport)** for PR 32030 at commit [`5bc0ff7`](https://github.com/apache/spark/commit/5bc0ff7e88ec11caa06b3c786bcdea9a01ddfcaf).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #32030: [SPARK-34989] Improve the performance of mapChildren and withNewChildren methods
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #32030:
URL: https://github.com/apache/spark/pull/32030#issuecomment-816315747
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/41683/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #32030: [SPARK-34989] Improve the performance of mapChildren and withNewChildren methods
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #32030:
URL: https://github.com/apache/spark/pull/32030#issuecomment-816221728
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/41677/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #32030: [WIP] Improve the performance of mapChildren and withNewChildren methods
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #32030:
URL: https://github.com/apache/spark/pull/32030#issuecomment-815157997
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/41608/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32030: [SPARK-34989] Improve the performance of mapChildren and withNewChildren methods
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32030:
URL: https://github.com/apache/spark/pull/32030#issuecomment-816293693
Kubernetes integration test unable to build dist.
exiting with code: 1
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/41680/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #32030: [WIP] Improve map children
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #32030:
URL: https://github.com/apache/spark/pull/32030#issuecomment-814444532
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/136961/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #32030: [WIP] Improve the performance of mapChildren and withNewChildren methods
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #32030:
URL: https://github.com/apache/spark/pull/32030#issuecomment-815413211
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/137038/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32030: [WIP] Improve map children
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32030:
URL: https://github.com/apache/spark/pull/32030#issuecomment-814106864
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #32030: [SPARK-34989] Improve the performance of mapChildren and withNewChildren methods
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #32030:
URL: https://github.com/apache/spark/pull/32030#issuecomment-816082181
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/137080/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32030: [SPARK-34989] Improve the performance of mapChildren and withNewChildren methods
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32030:
URL: https://github.com/apache/spark/pull/32030#issuecomment-816221664
Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/41677/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #32030: [SPARK-34989] Improve the performance of mapChildren and withNewChildren methods
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #32030:
URL: https://github.com/apache/spark/pull/32030#issuecomment-816294150
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/41680/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #32030: [SPARK-34989] Improve the performance of mapChildren and withNewChildren methods
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #32030:
URL: https://github.com/apache/spark/pull/32030#issuecomment-815638652
**[Test build #137070 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/137070/testReport)** for PR 32030 at commit [`498d248`](https://github.com/apache/spark/commit/498d24893270fa0c45182a6c9677981e2ba4b742).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #32030: [SPARK-34989] Improve the performance of mapChildren and withNewChildren methods
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #32030:
URL: https://github.com/apache/spark/pull/32030#issuecomment-816795744
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/137127/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32030: [SPARK-34989] Improve the performance of mapChildren and withNewChildren methods
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32030:
URL: https://github.com/apache/spark/pull/32030#issuecomment-815963594
**[Test build #137088 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/137088/testReport)** for PR 32030 at commit [`9c033d6`](https://github.com/apache/spark/commit/9c033d6210cb8a8287b989d46abed7c142d3fd4d).
* This patch **fails to build**.
* This patch merges cleanly.
* This patch adds no public classes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #32030: [WIP] Improve the performance of mapChildren and withNewChildren methods
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #32030:
URL: https://github.com/apache/spark/pull/32030#issuecomment-815675137
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/41648/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32030: [WIP] Improve the performance of mapChildren and withNewChildren methods
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32030:
URL: https://github.com/apache/spark/pull/32030#issuecomment-815158638
**[Test build #137031 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/137031/testReport)** for PR 32030 at commit [`ceef366`](https://github.com/apache/spark/commit/ceef366fe2701ab697213c5ae03659e78afdb40c).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #32030: [WIP] Improve map children
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #32030:
URL: https://github.com/apache/spark/pull/32030#issuecomment-814106905
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/41520/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32030: [WIP] Improve map children
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32030:
URL: https://github.com/apache/spark/pull/32030#issuecomment-814515791
**[Test build #136975 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/136975/testReport)** for PR 32030 at commit [`b526852`](https://github.com/apache/spark/commit/b526852c5133c09f52729ce55bbb2e87b51c915e).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] hvanhovell commented on pull request #32030: [SPARK-34989] Improve the performance of mapChildren and withNewChildren methods
Posted by GitBox <gi...@apache.org>.
hvanhovell commented on pull request #32030:
URL: https://github.com/apache/spark/pull/32030#issuecomment-816884661
Yikes, jumped the gun on the merge... Sorry about that.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #32030: [SPARK-34989] Improve the performance of mapChildren and withNewChildren methods
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #32030:
URL: https://github.com/apache/spark/pull/32030#issuecomment-816149111
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/137098/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #32030: [WIP] Improve map children
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #32030:
URL: https://github.com/apache/spark/pull/32030#issuecomment-814196074
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/136944/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #32030: [SPARK-34989] Improve the performance of mapChildren and withNewChildren methods
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #32030:
URL: https://github.com/apache/spark/pull/32030#issuecomment-816413714
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/137105/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32030: [SPARK-34989] Improve the performance of mapChildren and withNewChildren methods
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32030:
URL: https://github.com/apache/spark/pull/32030#issuecomment-816206879
Kubernetes integration test unable to build dist.
exiting with code: 1
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/41676/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #32030: [SPARK-34989] Improve the performance of mapChildren and withNewChildren methods
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #32030:
URL: https://github.com/apache/spark/pull/32030#issuecomment-816294150
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/41680/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32030: [SPARK-34989] Improve the performance of mapChildren and withNewChildren methods
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32030:
URL: https://github.com/apache/spark/pull/32030#issuecomment-816298157
**[Test build #137105 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/137105/testReport)** for PR 32030 at commit [`4130602`](https://github.com/apache/spark/commit/413060242f2ef9f56706076d4fe74ad37b239a7d).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #32030: [SPARK-34989] Improve the performance of mapChildren and withNewChildren methods
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #32030:
URL: https://github.com/apache/spark/pull/32030#issuecomment-816641515
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/41711/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32030: [WIP] Improve map children
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32030:
URL: https://github.com/apache/spark/pull/32030#issuecomment-815040941
**[Test build #137028 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/137028/testReport)** for PR 32030 at commit [`4e9dc01`](https://github.com/apache/spark/commit/4e9dc0144251406d299c6d22aea8e6e7371bcb3e).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32030: [SPARK-34989] Improve the performance of mapChildren and withNewChildren methods
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32030:
URL: https://github.com/apache/spark/pull/32030#issuecomment-816611609
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32030: [WIP] Improve the performance of mapChildren and withNewChildren methods
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32030:
URL: https://github.com/apache/spark/pull/32030#issuecomment-815664046
Kubernetes integration test unable to build dist.
exiting with code: 1
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/41648/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #32030: [SPARK-34989] Improve the performance of mapChildren and withNewChildren methods
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #32030:
URL: https://github.com/apache/spark/pull/32030#issuecomment-815962659
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/41666/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #32030: [SPARK-34989] Improve the performance of mapChildren and withNewChildren methods
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #32030:
URL: https://github.com/apache/spark/pull/32030#issuecomment-816149111
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/137098/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32030: [SPARK-34989] Improve the performance of mapChildren and withNewChildren methods
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32030:
URL: https://github.com/apache/spark/pull/32030#issuecomment-816214415
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/41677/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #32030: [SPARK-34989] Improve the performance of mapChildren and withNewChildren methods
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #32030:
URL: https://github.com/apache/spark/pull/32030#issuecomment-815815938
**[Test build #137080 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/137080/testReport)** for PR 32030 at commit [`27832a2`](https://github.com/apache/spark/commit/27832a2fdd0ee77a020f714514dfd106b9a5083a).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dbaliafroozeh commented on pull request #32030: [WIP] Improve the performance of mapChildren and withNewChildren methods
Posted by GitBox <gi...@apache.org>.
dbaliafroozeh commented on pull request #32030:
URL: https://github.com/apache/spark/pull/32030#issuecomment-815711733
@hvanhovell @sigmod @maryannxue @cloud-fan
This PR is ready for review. Please take a look
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #32030: [WIP] Improve map children
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #32030:
URL: https://github.com/apache/spark/pull/32030#issuecomment-814405594
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/136957/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32030: [WIP] Improve map children
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32030:
URL: https://github.com/apache/spark/pull/32030#issuecomment-815081268
**[Test build #137029 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/137029/testReport)** for PR 32030 at commit [`b6c563d`](https://github.com/apache/spark/commit/b6c563d79876bc0d1e7f47d37623c3836397fa38).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32030: [WIP] Improve map children
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32030:
URL: https://github.com/apache/spark/pull/32030#issuecomment-814534437
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/41552/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #32030: [WIP] Improve the performance of mapChildren and withNewChildren methods
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #32030:
URL: https://github.com/apache/spark/pull/32030#issuecomment-815119456
**[Test build #137030 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/137030/testReport)** for PR 32030 at commit [`413ef4f`](https://github.com/apache/spark/commit/413ef4f31ca496707a55c93f8eaf1e4f888e6feb).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32030: [WIP] Improve the performance of mapChildren and withNewChildren methods
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32030:
URL: https://github.com/apache/spark/pull/32030#issuecomment-815121043
**[Test build #137030 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/137030/testReport)** for PR 32030 at commit [`413ef4f`](https://github.com/apache/spark/commit/413ef4f31ca496707a55c93f8eaf1e4f888e6feb).
* This patch **fails Scala style tests**.
* This patch merges cleanly.
* This patch adds no public classes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #32030: [SPARK-34989] Improve the performance of mapChildren and withNewChildren methods
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #32030:
URL: https://github.com/apache/spark/pull/32030#issuecomment-815859380
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/41658/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32030: [WIP] Improve map children
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32030:
URL: https://github.com/apache/spark/pull/32030#issuecomment-815042146
**[Test build #137028 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/137028/testReport)** for PR 32030 at commit [`4e9dc01`](https://github.com/apache/spark/commit/4e9dc0144251406d299c6d22aea8e6e7371bcb3e).
* This patch **fails Scala style tests**.
* This patch merges cleanly.
* This patch adds no public classes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #32030: [SPARK-34989] Improve the performance of mapChildren and withNewChildren methods
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #32030:
URL: https://github.com/apache/spark/pull/32030#issuecomment-816352289
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/137099/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #32030: [WIP] Improve the performance of mapChildren and withNewChildren methods
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #32030:
URL: https://github.com/apache/spark/pull/32030#issuecomment-815675137
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/41648/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on pull request #32030: [SPARK-34989] Improve the performance of mapChildren and withNewChildren methods
Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on pull request #32030:
URL: https://github.com/apache/spark/pull/32030#issuecomment-816800864
Thank you, @dbaliafroozeh and all.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32030: [WIP] Improve the performance of mapChildren and withNewChildren methods
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32030:
URL: https://github.com/apache/spark/pull/32030#issuecomment-815115363
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32030: [WIP] Improve map children
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32030:
URL: https://github.com/apache/spark/pull/32030#issuecomment-814549784
**[Test build #136975 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/136975/testReport)** for PR 32030 at commit [`b526852`](https://github.com/apache/spark/commit/b526852c5133c09f52729ce55bbb2e87b51c915e).
* This patch **fails Spark unit tests**.
* This patch merges cleanly.
* This patch adds no public classes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #32030: [WIP] Improve map children
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #32030:
URL: https://github.com/apache/spark/pull/32030#issuecomment-814076465
**[Test build #136944 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/136944/testReport)** for PR 32030 at commit [`698d89d`](https://github.com/apache/spark/commit/698d89d60aabcfe08724091b80e2c86cc5f2f5c7).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #32030: [WIP] Improve the performance of mapChildren and withNewChildren methods
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #32030:
URL: https://github.com/apache/spark/pull/32030#issuecomment-815219134
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #32030: [WIP] Improve map children
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #32030:
URL: https://github.com/apache/spark/pull/32030#issuecomment-814196074
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/136944/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #32030: [WIP] Improve map children
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #32030:
URL: https://github.com/apache/spark/pull/32030#issuecomment-814106905
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/41520/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32030: [WIP] Improve map children
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32030:
URL: https://github.com/apache/spark/pull/32030#issuecomment-812270113
**[Test build #136828 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/136828/testReport)** for PR 32030 at commit [`7045e7a`](https://github.com/apache/spark/commit/7045e7a8bb844e6a5d48fda0ab06926f67c9f4ca).
* This patch **fails Spark unit tests**.
* This patch merges cleanly.
* This patch adds no public classes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #32030: [SPARK-34989] Improve the performance of mapChildren and withNewChildren methods
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #32030:
URL: https://github.com/apache/spark/pull/32030#issuecomment-816388958
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/137102/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #32030: [SPARK-34989] Improve the performance of mapChildren and withNewChildren methods
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #32030:
URL: https://github.com/apache/spark/pull/32030#issuecomment-816611642
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/41705/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #32030: [WIP] Improve the performance of mapChildren and withNewChildren methods
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #32030:
URL: https://github.com/apache/spark/pull/32030#issuecomment-815353635
**[Test build #137038 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/137038/testReport)** for PR 32030 at commit [`1d42fb7`](https://github.com/apache/spark/commit/1d42fb7024b44f3f3debbacf64e19e1c6f61d47a).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32030: [WIP] Improve map children
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32030:
URL: https://github.com/apache/spark/pull/32030#issuecomment-814535940
Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/41552/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #32030: [SPARK-34989] Improve the performance of mapChildren and withNewChildren methods
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #32030:
URL: https://github.com/apache/spark/pull/32030#issuecomment-816352289
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/137099/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #32030: [WIP] Improve map children
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #32030:
URL: https://github.com/apache/spark/pull/32030#issuecomment-815079929
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/41606/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32030: [WIP] Improve map children
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32030:
URL: https://github.com/apache/spark/pull/32030#issuecomment-812262654
Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/41408/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] sigmod commented on a change in pull request #32030: [SPARK-34989] Improve the performance of mapChildren and withNewChildren methods
Posted by GitBox <gi...@apache.org>.
sigmod commented on a change in pull request #32030:
URL: https://github.com/apache/spark/pull/32030#discussion_r609924205
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/trees/TreeNode.scala
##########
@@ -246,11 +246,50 @@ abstract class TreeNode[BaseType <: TreeNode[BaseType]] extends Product {
arr
}
+ private def childrenTheSame(
+ originalChildren: IndexedSeq[BaseType], newChildren: IndexedSeq[BaseType]): Boolean = {
+ val size = originalChildren.size
+ var i = 0
+ var childrenTheSame = true
+ while (i < size && childrenTheSame) {
+ childrenTheSame &&= originalChildren(i) fastEquals newChildren(i)
+ i += 1
+ }
+ childrenTheSame
+ }
+
+ // This is a temporary solution, we will change the type of children to IndexedSeq in a
+ // followup PR
+ private def asIndexedSeq(seq: Seq[BaseType]): IndexedSeq[BaseType] = {
+ if (seq.isInstanceOf[IndexedSeq[BaseType]]) {
+ seq.asInstanceOf[IndexedSeq[BaseType]]
+ } else {
+ seq.toIndexedSeq
+ }
+ }
+
+ def withNewChildren(newChildren: Seq[BaseType]): BaseType = {
+ val childrenIndexedSeq = asIndexedSeq(children)
+ val newChildrenIndexedSeq = asIndexedSeq(newChildren)
+ assert(newChildrenIndexedSeq.size == childrenIndexedSeq.size, "Incorrect number of children")
+ if (childrenIndexedSeq.isEmpty || childrenTheSame(newChildrenIndexedSeq, childrenIndexedSeq)) {
+ this
+ } else {
+ CurrentOrigin.withOrigin(origin) {
+ val res = withNewChildrenInternal(newChildrenIndexedSeq)
+ res.copyTagsFrom(this)
+ res
+ }
+ }
+ }
+
+ protected def withNewChildrenInternal(newChildren: IndexedSeq[BaseType]): BaseType
Review comment:
Does it make sense to call legacyWithNewChildren or use its code in the default implementation so that subclasses do not have to call legacyWithNewChildren in withNewChildrenInternal?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #32030: [WIP] Improve map children
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #32030:
URL: https://github.com/apache/spark/pull/32030#issuecomment-814548464
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/41552/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dbaliafroozeh commented on a change in pull request #32030: [SPARK-34989] Improve the performance of mapChildren and withNewChildren methods
Posted by GitBox <gi...@apache.org>.
dbaliafroozeh commented on a change in pull request #32030:
URL: https://github.com/apache/spark/pull/32030#discussion_r609934635
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/trees/TreeNode.scala
##########
@@ -246,11 +246,50 @@ abstract class TreeNode[BaseType <: TreeNode[BaseType]] extends Product {
arr
}
+ private def childrenTheSame(
+ originalChildren: IndexedSeq[BaseType], newChildren: IndexedSeq[BaseType]): Boolean = {
+ val size = originalChildren.size
+ var i = 0
+ var childrenTheSame = true
+ while (i < size && childrenTheSame) {
+ childrenTheSame &&= originalChildren(i) fastEquals newChildren(i)
+ i += 1
+ }
+ childrenTheSame
+ }
+
+ // This is a temporary solution, we will change the type of children to IndexedSeq in a
+ // followup PR
+ private def asIndexedSeq(seq: Seq[BaseType]): IndexedSeq[BaseType] = {
+ if (seq.isInstanceOf[IndexedSeq[BaseType]]) {
+ seq.asInstanceOf[IndexedSeq[BaseType]]
+ } else {
+ seq.toIndexedSeq
+ }
+ }
+
+ def withNewChildren(newChildren: Seq[BaseType]): BaseType = {
+ val childrenIndexedSeq = asIndexedSeq(children)
+ val newChildrenIndexedSeq = asIndexedSeq(newChildren)
+ assert(newChildrenIndexedSeq.size == childrenIndexedSeq.size, "Incorrect number of children")
+ if (childrenIndexedSeq.isEmpty || childrenTheSame(newChildrenIndexedSeq, childrenIndexedSeq)) {
+ this
+ } else {
+ CurrentOrigin.withOrigin(origin) {
+ val res = withNewChildrenInternal(newChildrenIndexedSeq)
+ res.copyTagsFrom(this)
+ res
+ }
+ }
+ }
+
+ protected def withNewChildrenInternal(newChildren: IndexedSeq[BaseType]): BaseType
Review comment:
Having a default implementation will lead to people who add new expressions don't implement `withNewChildrenInternal` and we again be back to the same situation having many slow `withNewChildren` implementations, so I prefer to make have it like this to enforce `withNewChildrenInternal` implementation. Actually, even now, there are two expressions added to the master and I need to update this PR to implement the `withNewChildrenInternal` for them. The `legacyWithNewChildren` is here for a transition period, we have some expressions that are a bit hard to write `withNewChildrenInternal` for and probably need some refactoring. The goal is to remove `legacyWithNewChildren` altogether at some point.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #32030: [SPARK-34989] Improve the performance of mapChildren and withNewChildren methods
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #32030:
URL: https://github.com/apache/spark/pull/32030#issuecomment-816582945
**[Test build #137127 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/137127/testReport)** for PR 32030 at commit [`2bb22ee`](https://github.com/apache/spark/commit/2bb22ee8a072606816c71433920d47eaac889932).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #32030: [SPARK-34989] Improve the performance of mapChildren and withNewChildren methods
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #32030:
URL: https://github.com/apache/spark/pull/32030#issuecomment-816160395
**[Test build #137099 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/137099/testReport)** for PR 32030 at commit [`060a098`](https://github.com/apache/spark/commit/060a09890919cbfee410326f825b537099872810).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #32030: [SPARK-34989] Improve the performance of mapChildren and withNewChildren methods
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #32030:
URL: https://github.com/apache/spark/pull/32030#issuecomment-815859380
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/41658/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #32030: [WIP] Improve the performance of mapChildren and withNewChildren methods
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #32030:
URL: https://github.com/apache/spark/pull/32030#issuecomment-815082456
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32030: [WIP] Improve map children
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32030:
URL: https://github.com/apache/spark/pull/32030#issuecomment-814322425
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on a change in pull request #32030: [SPARK-34989] Improve the performance of mapChildren and withNewChildren methods
Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on a change in pull request #32030:
URL: https://github.com/apache/spark/pull/32030#discussion_r610329571
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/DynamicPruning.scala
##########
@@ -94,4 +97,7 @@ case class DynamicPruningExpression(child: Expression)
override protected def doGenCode(ctx: CodegenContext, ev: ExprCode): ExprCode = {
child.genCode(ctx)
}
+
+ override protected def withNewChildInternal(newChild: Expression): Expression =
Review comment:
The same question here: `Expression` instead of `DynamicPruningExpression`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #32030: [SPARK-34989] Improve the performance of mapChildren and withNewChildren methods
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #32030:
URL: https://github.com/apache/spark/pull/32030#issuecomment-815813621
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/137070/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #32030: [WIP] Improve map children
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #32030:
URL: https://github.com/apache/spark/pull/32030#issuecomment-814405594
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/136957/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32030: [SPARK-34989] Improve the performance of mapChildren and withNewChildren methods
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32030:
URL: https://github.com/apache/spark/pull/32030#issuecomment-816582945
**[Test build #137127 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/137127/testReport)** for PR 32030 at commit [`2bb22ee`](https://github.com/apache/spark/commit/2bb22ee8a072606816c71433920d47eaac889932).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32030: [WIP] Improve the performance of mapChildren and withNewChildren methods
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32030:
URL: https://github.com/apache/spark/pull/32030#issuecomment-815406859
**[Test build #137038 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/137038/testReport)** for PR 32030 at commit [`1d42fb7`](https://github.com/apache/spark/commit/1d42fb7024b44f3f3debbacf64e19e1c6f61d47a).
* This patch **fails Spark unit tests**.
* This patch merges cleanly.
* This patch adds no public classes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #32030: [WIP] Improve map children
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #32030:
URL: https://github.com/apache/spark/pull/32030#issuecomment-815079929
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/41606/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32030: [WIP] Improve map children
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32030:
URL: https://github.com/apache/spark/pull/32030#issuecomment-815075097
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #32030: [SPARK-34989] Improve the performance of mapChildren and withNewChildren methods
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #32030:
URL: https://github.com/apache/spark/pull/32030#issuecomment-816795744
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/137127/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #32030: [WIP] Improve the performance of mapChildren and withNewChildren methods
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #32030:
URL: https://github.com/apache/spark/pull/32030#issuecomment-815157997
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/41608/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #32030: [WIP] Improve map children
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #32030:
URL: https://github.com/apache/spark/pull/32030#issuecomment-815040941
**[Test build #137028 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/137028/testReport)** for PR 32030 at commit [`4e9dc01`](https://github.com/apache/spark/commit/4e9dc0144251406d299c6d22aea8e6e7371bcb3e).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32030: [SPARK-34989] Improve the performance of mapChildren and withNewChildren methods
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32030:
URL: https://github.com/apache/spark/pull/32030#issuecomment-815796873
**[Test build #137070 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/137070/testReport)** for PR 32030 at commit [`498d248`](https://github.com/apache/spark/commit/498d24893270fa0c45182a6c9677981e2ba4b742).
* This patch **fails Spark unit tests**.
* This patch merges cleanly.
* This patch adds no public classes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #32030: [WIP] Improve map children
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #32030:
URL: https://github.com/apache/spark/pull/32030#issuecomment-814549884
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/136975/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dbaliafroozeh commented on a change in pull request #32030: [WIP] Improve the performance of mapChildren and withNewChildren methods
Posted by GitBox <gi...@apache.org>.
dbaliafroozeh commented on a change in pull request #32030:
URL: https://github.com/apache/spark/pull/32030#discussion_r608801196
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/DeduplicateRelations.scala
##########
@@ -95,9 +95,9 @@ object DeduplicateRelations extends Rule[LogicalPlan] {
.flatMap(_.output).zip(newChildren.flatMap(_.output))
.filter { case (a1, a2) => a1.exprId != a2.exprId }
)
- plan.withNewChildren(newChildren.toSeq).rewriteAttrs(attrMap)
+ plan.withNewChildren(newChildren.toList).rewriteAttrs(attrMap)
Review comment:
Note to reviewers: this change is necessary to prevent tpc-ds plan stability tests to pass. It was complaining that a list of children is an `ArrayBuffer` instead of a `List`. The reason for the difference is that the new `withNewChildren` just replaces the children with the given new children and doesn't care about the exact list type.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32030: [WIP] Improve the performance of mapChildren and withNewChildren methods
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32030:
URL: https://github.com/apache/spark/pull/32030#issuecomment-815152748
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on a change in pull request #32030: [SPARK-34989] Improve the performance of mapChildren and withNewChildren methods
Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on a change in pull request #32030:
URL: https://github.com/apache/spark/pull/32030#discussion_r610329571
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/DynamicPruning.scala
##########
@@ -94,4 +97,7 @@ case class DynamicPruningExpression(child: Expression)
override protected def doGenCode(ctx: CodegenContext, ev: ExprCode): ExprCode = {
child.genCode(ctx)
}
+
+ override protected def withNewChildInternal(newChild: Expression): Expression =
Review comment:
The same question here: Is there a reason to use `Expression` instead of `DynamicPruningExpression`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dbaliafroozeh commented on a change in pull request #32030: [WIP] Improve the performance of mapChildren and withNewChildren methods
Posted by GitBox <gi...@apache.org>.
dbaliafroozeh commented on a change in pull request #32030:
URL: https://github.com/apache/spark/pull/32030#discussion_r609141398
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/aggregate/interfaces.scala
##########
@@ -164,6 +164,9 @@ case class AggregateExpression(
case _ => aggFuncStr
}
}
+
+ override protected def withNewChildrenInternal(newChildren: IndexedSeq[Expression]): Expression =
+ super.legacyWithNewChildren(newChildren)
Review comment:
Fixed it. Probably it makes sense to have a special node, e.g., `UnaryWithOptional` for nodes with one child and one optional child. There are a number of some nodes. Manually implementing them is tricky. I'd like to ideally only have to implement this method like this:
```
override protected def withNewChildrenInternal(newChildren: IndexedSeq[Expression]): Expression =
copy(children = newChildren)
```
That is, only for nodes with a list of children. We'll do these tweaks in a followup.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32030: [SPARK-34989] Improve the performance of mapChildren and withNewChildren methods
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32030:
URL: https://github.com/apache/spark/pull/32030#issuecomment-816401935
**[Test build #137105 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/137105/testReport)** for PR 32030 at commit [`4130602`](https://github.com/apache/spark/commit/413060242f2ef9f56706076d4fe74ad37b239a7d).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dbaliafroozeh commented on a change in pull request #32030: [WIP] Improve map children
Posted by GitBox <gi...@apache.org>.
dbaliafroozeh commented on a change in pull request #32030:
URL: https://github.com/apache/spark/pull/32030#discussion_r608801196
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/DeduplicateRelations.scala
##########
@@ -95,9 +95,9 @@ object DeduplicateRelations extends Rule[LogicalPlan] {
.flatMap(_.output).zip(newChildren.flatMap(_.output))
.filter { case (a1, a2) => a1.exprId != a2.exprId }
)
- plan.withNewChildren(newChildren.toSeq).rewriteAttrs(attrMap)
+ plan.withNewChildren(newChildren.toList).rewriteAttrs(attrMap)
Review comment:
Note to reviewers: this change is necessary to prevent tpc-ds plan stability tests to pass. It was complaining that a list of children is an `ArrayBuffer` instead of a `List`. The reason for the difference is that the new `withNewChildren` just replaces the children with the given new children and doesn't care about the exact list type.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32030: [SPARK-34989] Improve the performance of mapChildren and withNewChildren methods
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32030:
URL: https://github.com/apache/spark/pull/32030#issuecomment-816047763
**[Test build #137080 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/137080/testReport)** for PR 32030 at commit [`27832a2`](https://github.com/apache/spark/commit/27832a2fdd0ee77a020f714514dfd106b9a5083a).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32030: [WIP] Improve map children
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32030:
URL: https://github.com/apache/spark/pull/32030#issuecomment-812236261
**[Test build #136828 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/136828/testReport)** for PR 32030 at commit [`7045e7a`](https://github.com/apache/spark/commit/7045e7a8bb844e6a5d48fda0ab06926f67c9f4ca).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #32030: [WIP] Improve the performance of mapChildren and withNewChildren methods
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #32030:
URL: https://github.com/apache/spark/pull/32030#issuecomment-815374761
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/41616/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #32030: [WIP] Improve map children
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #32030:
URL: https://github.com/apache/spark/pull/32030#issuecomment-814283666
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/41534/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #32030: [SPARK-34989] Improve the performance of mapChildren and withNewChildren methods
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on a change in pull request #32030:
URL: https://github.com/apache/spark/pull/32030#discussion_r609884966
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/trees/TreeNode.scala
##########
@@ -246,11 +246,50 @@ abstract class TreeNode[BaseType <: TreeNode[BaseType]] extends Product {
arr
}
+ private def childrenTheSame(
+ originalChildren: IndexedSeq[BaseType], newChildren: IndexedSeq[BaseType]): Boolean = {
+ val size = originalChildren.size
+ var i = 0
+ var childrenTheSame = true
+ while (i < size && childrenTheSame) {
Review comment:
early return in loop is not recommended in Scala...
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #32030: [WIP] Improve map children
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #32030:
URL: https://github.com/apache/spark/pull/32030#issuecomment-814328874
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/41538/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #32030: [WIP] Improve the performance of mapChildren and withNewChildren methods
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #32030:
URL: https://github.com/apache/spark/pull/32030#issuecomment-815121062
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/137030/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #32030: [WIP] Improve the performance of mapChildren and withNewChildren methods
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #32030:
URL: https://github.com/apache/spark/pull/32030#issuecomment-815158638
**[Test build #137031 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/137031/testReport)** for PR 32030 at commit [`ceef366`](https://github.com/apache/spark/commit/ceef366fe2701ab697213c5ae03659e78afdb40c).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #32030: [WIP] Improve the performance of mapChildren and withNewChildren methods
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #32030:
URL: https://github.com/apache/spark/pull/32030#issuecomment-815413211
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/137038/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #32030: [SPARK-34989] Improve the performance of mapChildren and withNewChildren methods
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #32030:
URL: https://github.com/apache/spark/pull/32030#issuecomment-816266209
**[Test build #137102 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/137102/testReport)** for PR 32030 at commit [`85e236c`](https://github.com/apache/spark/commit/85e236ce24ffa37fc12b85d64a4d586a376f20a7).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #32030: [WIP] Improve the performance of mapChildren and withNewChildren methods
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #32030:
URL: https://github.com/apache/spark/pull/32030#issuecomment-815082456
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/137029/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32030: [WIP] Improve the performance of mapChildren and withNewChildren methods
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32030:
URL: https://github.com/apache/spark/pull/32030#issuecomment-815180531
**[Test build #137031 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/137031/testReport)** for PR 32030 at commit [`ceef366`](https://github.com/apache/spark/commit/ceef366fe2701ab697213c5ae03659e78afdb40c).
* This patch **fails Spark unit tests**.
* This patch merges cleanly.
* This patch adds no public classes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32030: [WIP] Improve map children
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32030:
URL: https://github.com/apache/spark/pull/32030#issuecomment-814278142
Kubernetes integration test unable to build dist.
exiting with code: 1
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/41534/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on pull request #32030: [SPARK-34989] Improve the performance of mapChildren and withNewChildren methods
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on pull request #32030:
URL: https://github.com/apache/spark/pull/32030#issuecomment-815903141
This reminds me of `TimeZoneAwareExpression.withTimeZone` and `ImperativeAggregate.withNewMutableAggBufferOffset`. They can be simply implemented with case class copy, but unfortunately, Scala is not flexible enough to allow you to implement it in a generic way :(
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32030: [SPARK-34989] Improve the performance of mapChildren and withNewChildren methods
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32030:
URL: https://github.com/apache/spark/pull/32030#issuecomment-815962623
Kubernetes integration test unable to build dist.
exiting with code: 1
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/41666/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] sigmod commented on a change in pull request #32030: [SPARK-34989] Improve the performance of mapChildren and withNewChildren methods
Posted by GitBox <gi...@apache.org>.
sigmod commented on a change in pull request #32030:
URL: https://github.com/apache/spark/pull/32030#discussion_r609917045
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/trees/TreeNode.scala
##########
@@ -246,11 +246,50 @@ abstract class TreeNode[BaseType <: TreeNode[BaseType]] extends Product {
arr
}
+ private def childrenTheSame(
+ originalChildren: IndexedSeq[BaseType], newChildren: IndexedSeq[BaseType]): Boolean = {
+ val size = originalChildren.size
+ var i = 0
+ var childrenTheSame = true
+ while (i < size && childrenTheSame) {
Review comment:
Maybe I'm missing sth., but early return is preferred in the style guide?
https://github.com/databricks/scala-style-guide#return-statements
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32030: [SPARK-34989] Improve the performance of mapChildren and withNewChildren methods
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32030:
URL: https://github.com/apache/spark/pull/32030#issuecomment-816315715
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32030: [WIP] Improve the performance of mapChildren and withNewChildren methods
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32030:
URL: https://github.com/apache/spark/pull/32030#issuecomment-815353635
**[Test build #137038 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/137038/testReport)** for PR 32030 at commit [`1d42fb7`](https://github.com/apache/spark/commit/1d42fb7024b44f3f3debbacf64e19e1c6f61d47a).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32030: [WIP] Improve map children
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32030:
URL: https://github.com/apache/spark/pull/32030#issuecomment-815082439
**[Test build #137029 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/137029/testReport)** for PR 32030 at commit [`b6c563d`](https://github.com/apache/spark/commit/b6c563d79876bc0d1e7f47d37623c3836397fa38).
* This patch **fails Scala style tests**.
* This patch merges cleanly.
* This patch adds no public classes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32030: [SPARK-34989] Improve the performance of mapChildren and withNewChildren methods
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32030:
URL: https://github.com/apache/spark/pull/32030#issuecomment-816378150
**[Test build #137102 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/137102/testReport)** for PR 32030 at commit [`85e236c`](https://github.com/apache/spark/commit/85e236ce24ffa37fc12b85d64a4d586a376f20a7).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org