You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by GitBox <gi...@apache.org> on 2021/08/04 20:25:21 UTC
[GitHub] [spark] akpatnam25 opened a new pull request #33644: [SPARK-36419][CORE]: Optionally move final aggregation in RDD.treeAggregate to executor
akpatnam25 opened a new pull request #33644:
URL: https://github.com/apache/spark/pull/33644
What changes were proposed in this pull request?
Move final iteration of aggregation of RDD.treeAggregate to an executor with one partition and fetch that result to the driver
Why are the changes needed?
RDD.fold pulls all shuffle partitions to the driver to merge the result
a. Driver becomes a single point of failure in the case that there are a lot of partitions to do the final aggregation on
Shuffle machinery at executors is much more robust/fault tolerant compared to fetching results to driver.
Does this PR introduce any user-facing change?
The previous behavior always did the final aggregation in the driver. The user can now (optionally) provide a boolean config (default = false) ENABLE_EXECUTOR_TREE_AGGREGATE to do that final aggregation in a single partition executor before fetching the results to the driver. The only additional cost is that the user will see an extra stage in their job.
How was this patch tested?
This patch was tested via unit tests, and also tested on a cluster.
The screenshots showing the extra stage on a cluster are attached below (before vs after).


--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] akpatnam25 commented on a change in pull request #33644: [SPARK-36419][CORE] Optionally move final aggregation in RDD.treeAggregate to executor
Posted by GitBox <gi...@apache.org>.
akpatnam25 commented on a change in pull request #33644:
URL: https://github.com/apache/spark/pull/33644#discussion_r690532532
##########
File path: core/src/main/scala/org/apache/spark/rdd/RDD.scala
##########
@@ -1233,6 +1233,22 @@ abstract class RDD[T: ClassTag](
(i, iter) => iter.map((i % curNumPartitions, _))
Review comment:
Overloading also does not work due to the reasons given in the above thread
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] srowen commented on a change in pull request #33644: [SPARK-36419][CORE] Optionally move final aggregation in RDD.treeAggregate to executor
Posted by GitBox <gi...@apache.org>.
srowen commented on a change in pull request #33644:
URL: https://github.com/apache/spark/pull/33644#discussion_r688075038
##########
File path: project/MimaExcludes.scala
##########
@@ -104,6 +104,10 @@ object MimaExcludes {
ProblemFilters.exclude[DirectMissingMethodProblem]("org.apache.spark.shuffle.api.SingleSpillShuffleMapOutputWriter.transferMapSpillFile"),
ProblemFilters.exclude[ReversedMissingMethodProblem]("org.apache.spark.shuffle.api.SingleSpillShuffleMapOutputWriter.transferMapSpillFile"),
ProblemFilters.exclude[ReversedMissingMethodProblem]("org.apache.spark.shuffle.api.ShuffleMapOutputWriter.commitAllPartitions")
+
+ // [SPARK-36419][CORE] Optionally move final aggregation in RDD.treeAggregate to executor
+ ProblemFilters.exclude[ReversedMissingMethodProblem]("org.apache.spark.api.java.JavaRDDLike.treeAggregate")
Review comment:
Hm. Two separate methods won't work here b/c of the currying (or whatever it's called)? the thing is we need to not break API compatibility over this. If there's no apparent way around it ... hm. We could make a differently named new method which is kind of ugly, or fall back to a config param?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on pull request #33644: [SPARK-36419][CORE] Optionally move final aggregation in RDD.treeAggregate to executor
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on pull request #33644:
URL: https://github.com/apache/spark/pull/33644#issuecomment-897406866
The idea looks fine. In Spark SQL, global aggregate (no grouping key) also launches a single reduce task at the executor side to do the aggregation work, and send back the result to the driver later. It's useful when you have many concurrent SQL queries, as you have many reduce tasks to do the computation work instead of a single driver node.
And also +1 to use parameters instead of configs.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] mridulm edited a comment on pull request #33644: [SPARK-36419][CORE]: Optionally move final aggregation in RDD.treeAggregate to executor
Posted by GitBox <gi...@apache.org>.
mridulm edited a comment on pull request #33644:
URL: https://github.com/apache/spark/pull/33644#issuecomment-894871776
Makes sense @srowen, thanks for taking a look !
+CC @mengxr, @HyukjinKwon, @cloud-fan, @dongjoon-hyun - would be great if you can take a look (Tagging based on git history).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] venkata91 commented on a change in pull request #33644: [SPARK-36419][CORE] Optionally move final aggregation in RDD.treeAggregate to executor
Posted by GitBox <gi...@apache.org>.
venkata91 commented on a change in pull request #33644:
URL: https://github.com/apache/spark/pull/33644#discussion_r688082284
##########
File path: project/MimaExcludes.scala
##########
@@ -104,6 +104,10 @@ object MimaExcludes {
ProblemFilters.exclude[DirectMissingMethodProblem]("org.apache.spark.shuffle.api.SingleSpillShuffleMapOutputWriter.transferMapSpillFile"),
ProblemFilters.exclude[ReversedMissingMethodProblem]("org.apache.spark.shuffle.api.SingleSpillShuffleMapOutputWriter.transferMapSpillFile"),
ProblemFilters.exclude[ReversedMissingMethodProblem]("org.apache.spark.shuffle.api.ShuffleMapOutputWriter.commitAllPartitions")
+
+ // [SPARK-36419][CORE] Optionally move final aggregation in RDD.treeAggregate to executor
+ ProblemFilters.exclude[ReversedMissingMethodProblem]("org.apache.spark.api.java.JavaRDDLike.treeAggregate")
Review comment:
Yes it doesn't work because of the currying here I think. `config` seems like an easy workaround rather than creating a new method.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] venkata91 commented on a change in pull request #33644: [SPARK-36419][CORE]: Optionally move final aggregation in RDD.treeAggregate to executor
Posted by GitBox <gi...@apache.org>.
venkata91 commented on a change in pull request #33644:
URL: https://github.com/apache/spark/pull/33644#discussion_r682991221
##########
File path: core/src/main/scala/org/apache/spark/rdd/RDD.scala
##########
@@ -1233,6 +1233,21 @@ abstract class RDD[T: ClassTag](
(i, iter) => iter.map((i % curNumPartitions, _))
}.foldByKey(zeroValue, new HashPartitioner(curNumPartitions))(cleanCombOp).values
}
+ if (conf.get(ENABLE_EXECUTOR_TREE_AGGREGATE) && partiallyAggregated.partitions.length > 1) {
+ // define a new partitioner that results in only 1 partition
+ val constantPartitioner = new Partitioner {
+ override def numPartitions: Int = 1
+
+ override def getPartition(key: Any): Int = 0
+ }
+ // map the partially aggregated rdd into a key-value rdd
+ // do the computation in the single executor with one partition
+ // get the new RDD[U]
+ partiallyAggregated = partiallyAggregated
+ .map(v => (0.toByte, v))
+ .foldByKey(zeroValue, constantPartitioner)(cleanCombOp)
+ .values
+ }
val copiedZeroValue = Utils.clone(zeroValue, sc.env.closureSerializer.newInstance())
Review comment:
wouldn't this submit another job after the `foldByKey` again on the `fold` eventually to just return the only value?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33644: [SPARK-36419][CORE] Optionally move final aggregation in RDD.treeAggregate to executor
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33644:
URL: https://github.com/apache/spark/pull/33644#issuecomment-898728954
**[Test build #142451 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/142451/testReport)** for PR 33644 at commit [`75945bc`](https://github.com/apache/spark/commit/75945bc49e2090a8d7f9e3dfa67e82d545edca3f).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] akpatnam25 commented on a change in pull request #33644: [SPARK-36419][CORE] Optionally move final aggregation in RDD.treeAggregate to executor
Posted by GitBox <gi...@apache.org>.
akpatnam25 commented on a change in pull request #33644:
URL: https://github.com/apache/spark/pull/33644#discussion_r690532188
##########
File path: core/src/main/scala/org/apache/spark/rdd/RDD.scala
##########
@@ -1233,6 +1233,22 @@ abstract class RDD[T: ClassTag](
(i, iter) => iter.map((i % curNumPartitions, _))
Review comment:
i don't think we can do that as we did not want to cause binary incompabilities. that was the whole reason i had to revert back to the config approach. refer to this thread: https://github.com/apache/spark/pull/33644#discussion_r687122613
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33644: [SPARK-36419][CORE] Optionally move final aggregation in RDD.treeAggregate to executor
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33644:
URL: https://github.com/apache/spark/pull/33644#issuecomment-902199999
Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/47151/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] akpatnam25 commented on a change in pull request #33644: [SPARK-36419][CORE] Optionally move final aggregation in RDD.treeAggregate to executor
Posted by GitBox <gi...@apache.org>.
akpatnam25 commented on a change in pull request #33644:
URL: https://github.com/apache/spark/pull/33644#discussion_r688081365
##########
File path: project/MimaExcludes.scala
##########
@@ -104,6 +104,10 @@ object MimaExcludes {
ProblemFilters.exclude[DirectMissingMethodProblem]("org.apache.spark.shuffle.api.SingleSpillShuffleMapOutputWriter.transferMapSpillFile"),
ProblemFilters.exclude[ReversedMissingMethodProblem]("org.apache.spark.shuffle.api.SingleSpillShuffleMapOutputWriter.transferMapSpillFile"),
ProblemFilters.exclude[ReversedMissingMethodProblem]("org.apache.spark.shuffle.api.ShuffleMapOutputWriter.commitAllPartitions")
+
+ // [SPARK-36419][CORE] Optionally move final aggregation in RDD.treeAggregate to executor
+ ProblemFilters.exclude[ReversedMissingMethodProblem]("org.apache.spark.api.java.JavaRDDLike.treeAggregate")
Review comment:
since we already decided that using parameters was the better option compared to configs, let's make a new method?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon edited a comment on pull request #33644: [SPARK-36419][CORE]: Optionally move final aggregation in RDD.treeAggregate to executor
Posted by GitBox <gi...@apache.org>.
HyukjinKwon edited a comment on pull request #33644:
URL: https://github.com/apache/spark/pull/33644#issuecomment-896413705
This is a new behaviour and this change won't be beneficial always given the discussion above so I would think it might be better for users to selectively choose the behaviour. But I really don't have a strong preference - it does require application side change that is also a valid point. I see no objection on this change so I would defer to @mridulm on how to proceed further - he has a much better insight in core than me.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] srowen commented on pull request #33644: [SPARK-36419][CORE]: Optionally move final aggregation in RDD.treeAggregate to executor
Posted by GitBox <gi...@apache.org>.
srowen commented on pull request #33644:
URL: https://github.com/apache/spark/pull/33644#issuecomment-894385278
Hm. Still feels weird. What's the overhead of the extra hop in 'normal' cases? This case seems solvable with sufficient driver memory
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] akpatnam25 removed a comment on pull request #33644: [SPARK-36419][CORE]: Optionally move final aggregation in RDD.treeAggregate to executor
Posted by GitBox <gi...@apache.org>.
akpatnam25 removed a comment on pull request #33644:
URL: https://github.com/apache/spark/pull/33644#issuecomment-893594625
retest this please
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] venkata91 commented on a change in pull request #33644: [SPARK-36419][CORE] Optionally move final aggregation in RDD.treeAggregate to executor
Posted by GitBox <gi...@apache.org>.
venkata91 commented on a change in pull request #33644:
URL: https://github.com/apache/spark/pull/33644#discussion_r692368891
##########
File path: core/src/main/scala/org/apache/spark/api/java/JavaRDDLike.scala
##########
@@ -450,6 +450,19 @@ trait JavaRDDLike[T, This <: JavaRDDLike[T, This]] extends Serializable {
treeAggregate(zeroValue, seqOp, combOp, 2)
}
+ /**
+ * `org.apache.spark.api.java.JavaRDDLike.treeAggregate` with a parameter to do the
+ * final aggregation on the executor.
+ */
+ def treeAggregate[U](
+ zeroValue: U,
Review comment:
nit: indent 4 spaces.
##########
File path: core/src/main/scala/org/apache/spark/rdd/RDD.scala
##########
@@ -1201,16 +1201,25 @@ abstract class RDD[T: ClassTag](
jobResult
}
+ def treeAggregate[U: ClassTag](zeroValue: U)(
+ seqOp: (U, T) => U,
+ combOp: (U, U) => U,
Review comment:
nit: same here indent 4 spaces.
##########
File path: core/src/main/scala/org/apache/spark/rdd/RDD.scala
##########
@@ -1201,16 +1201,25 @@ abstract class RDD[T: ClassTag](
jobResult
}
+ def treeAggregate[U: ClassTag](zeroValue: U)(
+ seqOp: (U, T) => U,
+ combOp: (U, U) => U,
+ depth: Int = 2): U = withScope {
+ treeAggregate(zeroValue, seqOp, combOp, depth, finalAggregateOnExecutor = false)
+ }
+
/**
* Aggregates the elements of this RDD in a multi-level tree pattern.
* This method is semantically identical to [[org.apache.spark.rdd.RDD#aggregate]].
*
* @param depth suggested depth of the tree (default: 2)
*/
- def treeAggregate[U: ClassTag](zeroValue: U)(
- seqOp: (U, T) => U,
- combOp: (U, U) => U,
- depth: Int = 2): U = withScope {
+ def treeAggregate[U: ClassTag](
+ zeroValue: U,
+ seqOp: (U, T) => U,
Review comment:
nit: indent 4 spaces.
##########
File path: core/src/main/scala/org/apache/spark/rdd/RDD.scala
##########
@@ -1201,16 +1201,25 @@ abstract class RDD[T: ClassTag](
jobResult
}
+ def treeAggregate[U: ClassTag](zeroValue: U)(
+ seqOp: (U, T) => U,
+ combOp: (U, U) => U,
+ depth: Int = 2): U = withScope {
+ treeAggregate(zeroValue, seqOp, combOp, depth, finalAggregateOnExecutor = false)
+ }
+
/**
* Aggregates the elements of this RDD in a multi-level tree pattern.
* This method is semantically identical to [[org.apache.spark.rdd.RDD#aggregate]].
*
Review comment:
Isn't this method doc for the above API? Also can we add a new scala doc for the newly added method accordingly?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] venkata91 commented on pull request #33644: [SPARK-36419][CORE]: Optionally move final aggregation in RDD.treeAggregate to executor
Posted by GitBox <gi...@apache.org>.
venkata91 commented on pull request #33644:
URL: https://github.com/apache/spark/pull/33644#issuecomment-896148614
@HyukjinKwon Adding as parameter instead of config makes sense, but that would require application code change - Not sure if users would adopt it this way. Thoughts?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] mridulm edited a comment on pull request #33644: [SPARK-36419][CORE]: Optionally move final aggregation in RDD.treeAggregate to executor
Posted by GitBox <gi...@apache.org>.
mridulm edited a comment on pull request #33644:
URL: https://github.com/apache/spark/pull/33644#issuecomment-894320564
@srowen You are right that the total computation is slightly worse (an additional fold with zero value at driver and ofcourse the extra shuffle).
The main issue is what @venkata91 mentioned - at executor, it is done in context of shuffle, where memory is better managed (not necessarily because executor has more memory).
At driver, it is essentially fetching all partitions (single value per partition) as part of the `fold`'s `runJob` : for nontrivial values, this causes OOM at driver, failing the application (memory usage for io + N serde in addition to the fold).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] akpatnam25 commented on a change in pull request #33644: [SPARK-36419][CORE] Optionally move final aggregation in RDD.treeAggregate to executor
Posted by GitBox <gi...@apache.org>.
akpatnam25 commented on a change in pull request #33644:
URL: https://github.com/apache/spark/pull/33644#discussion_r692533918
##########
File path: core/src/main/scala/org/apache/spark/rdd/RDD.scala
##########
@@ -1233,6 +1233,22 @@ abstract class RDD[T: ClassTag](
(i, iter) => iter.map((i % curNumPartitions, _))
Review comment:
i just made the change. i think i did not read @HyukjinKwon's suggestion properly earlier, as I did not realize that he was proposing a method that was not curried. i think this approach is the best way, and also does not break binary/source incompatibility.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on pull request #33644: [SPARK-36419][CORE]: Optionally move final aggregation in RDD.treeAggregate to executor
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on pull request #33644:
URL: https://github.com/apache/spark/pull/33644#issuecomment-894889792
I am okay with this in general. We have some places like this e.g., listing files or single partition when operating a global window in Spark SQL. Not sure if the configuration is the best way though. Let me add some more people who might be interested in here: @jiangxb1987 @Ngone51
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon edited a comment on pull request #33644: [SPARK-36419][CORE]: Optionally move final aggregation in RDD.treeAggregate to executor
Posted by GitBox <gi...@apache.org>.
HyukjinKwon edited a comment on pull request #33644:
URL: https://github.com/apache/spark/pull/33644#issuecomment-894889792
I am okay with this in general. We have some places like this (launching separate a job to compute on executor) e.g., listing files or single partition when operating a global window in Spark SQL. Not sure if the configuration is the best way though. Let me add some more people who might be interested in here: @jiangxb1987 @Ngone51
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on pull request #33644: [SPARK-36419][CORE] Optionally move final aggregation in RDD.treeAggregate to executor
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on pull request #33644:
URL: https://github.com/apache/spark/pull/33644#issuecomment-900811204
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #33644: [SPARK-36419][CORE]: Optionally move final aggregation in RDD.treeAggregate to executor
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #33644:
URL: https://github.com/apache/spark/pull/33644#issuecomment-892967870
Can one of the admins verify this patch?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] akpatnam25 commented on a change in pull request #33644: [SPARK-36419][CORE] Optionally move final aggregation in RDD.treeAggregate to executor
Posted by GitBox <gi...@apache.org>.
akpatnam25 commented on a change in pull request #33644:
URL: https://github.com/apache/spark/pull/33644#discussion_r688081365
##########
File path: project/MimaExcludes.scala
##########
@@ -104,6 +104,10 @@ object MimaExcludes {
ProblemFilters.exclude[DirectMissingMethodProblem]("org.apache.spark.shuffle.api.SingleSpillShuffleMapOutputWriter.transferMapSpillFile"),
ProblemFilters.exclude[ReversedMissingMethodProblem]("org.apache.spark.shuffle.api.SingleSpillShuffleMapOutputWriter.transferMapSpillFile"),
ProblemFilters.exclude[ReversedMissingMethodProblem]("org.apache.spark.shuffle.api.ShuffleMapOutputWriter.commitAllPartitions")
+
+ // [SPARK-36419][CORE] Optionally move final aggregation in RDD.treeAggregate to executor
+ ProblemFilters.exclude[ReversedMissingMethodProblem]("org.apache.spark.api.java.JavaRDDLike.treeAggregate")
Review comment:
since we already decided that using parameters was the better option compared to configs, let's make a new function?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] venkata91 commented on pull request #33644: [SPARK-36419][CORE] Optionally move final aggregation in RDD.treeAggregate to executor
Posted by GitBox <gi...@apache.org>.
venkata91 commented on pull request #33644:
URL: https://github.com/apache/spark/pull/33644#issuecomment-901275284
@akpatnam25 Also update the PR with the latest changes from master, so that GA would run again probably that can fix the linters issue.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #33644: [SPARK-36419][CORE]: Optionally move final aggregation in RDD.treeAggregate to executor
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #33644:
URL: https://github.com/apache/spark/pull/33644#issuecomment-892967870
Can one of the admins verify this patch?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] akpatnam25 commented on pull request #33644: [SPARK-36419][CORE] Optionally move final aggregation in RDD.treeAggregate to executor
Posted by GitBox <gi...@apache.org>.
akpatnam25 commented on pull request #33644:
URL: https://github.com/apache/spark/pull/33644#issuecomment-900921583
@HyukjinKwon can we merge this then now? seems like we have consensus from everyone.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on a change in pull request #33644: [SPARK-36419][CORE] Optionally move final aggregation in RDD.treeAggregate to executor
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on a change in pull request #33644:
URL: https://github.com/apache/spark/pull/33644#discussion_r690900300
##########
File path: core/src/main/scala/org/apache/spark/rdd/RDD.scala
##########
@@ -1233,6 +1233,22 @@ abstract class RDD[T: ClassTag](
(i, iter) => iter.map((i % curNumPartitions, _))
Review comment:
Okie, got the problem now. Configuration SGTM
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #33644: [SPARK-36419][CORE] Optionally move final aggregation in RDD.treeAggregate to executor
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #33644:
URL: https://github.com/apache/spark/pull/33644#issuecomment-898652298
**[Test build #142451 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/142451/testReport)** for PR 33644 at commit [`75945bc`](https://github.com/apache/spark/commit/75945bc49e2090a8d7f9e3dfa67e82d545edca3f).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] mridulm edited a comment on pull request #33644: [SPARK-36419][CORE]: Optionally move final aggregation in RDD.treeAggregate to executor
Posted by GitBox <gi...@apache.org>.
mridulm edited a comment on pull request #33644:
URL: https://github.com/apache/spark/pull/33644#issuecomment-894320564
@srowen You are right that the total computation is slightly worse (an additional fold with zero value at driver and ofcourse the extra shuffle).
The main issue is what @venkata91 mentioned - at executor, it is done in context of shuffle, where memory is better managed (not necessarily because executor has more memory).
At driver, it is essentially fetching all partitions (single value per partition) as part of the `fold`'s `runJob` : for nontrivial values, this causes OOM at driver, failing the application (memory usage for N io fetches + N serde in addition to the fold).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33644: [SPARK-36419][CORE] Optionally move final aggregation in RDD.treeAggregate to executor
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33644:
URL: https://github.com/apache/spark/pull/33644#issuecomment-898641008
**[Test build #142445 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/142445/testReport)** for PR 33644 at commit [`722cbed`](https://github.com/apache/spark/commit/722cbede07f72644092e906530bce1213b0e1901).
* This patch **fails Scala style tests**.
* This patch merges cleanly.
* This patch adds no public classes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33644: [SPARK-36419][CORE] Optionally move final aggregation in RDD.treeAggregate to executor
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33644:
URL: https://github.com/apache/spark/pull/33644#issuecomment-902186886
**[Test build #142651 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/142651/testReport)** for PR 33644 at commit [`f728a02`](https://github.com/apache/spark/commit/f728a02407c31faf4a13665c4763c13344e4036d).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33644: [SPARK-36419][CORE] Optionally move final aggregation in RDD.treeAggregate to executor
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33644:
URL: https://github.com/apache/spark/pull/33644#issuecomment-901329417
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/47122/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] mridulm commented on pull request #33644: [SPARK-36419][CORE] Optionally move final aggregation in RDD.treeAggregate to executor
Posted by GitBox <gi...@apache.org>.
mridulm commented on pull request #33644:
URL: https://github.com/apache/spark/pull/33644#issuecomment-897002819
I agree with @HyukjinKwon, let us make it a method param disabled by default.
Even with the config option, given it is disabled by default, it will require app config change in order to leverage it anyway - and we have finer control with this than doing it app wide as a config option.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #33644: [SPARK-36419][CORE] Optionally move final aggregation in RDD.treeAggregate to executor
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #33644:
URL: https://github.com/apache/spark/pull/33644#issuecomment-901353282
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/47122/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33644: [SPARK-36419][CORE] Optionally move final aggregation in RDD.treeAggregate to executor
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33644:
URL: https://github.com/apache/spark/pull/33644#issuecomment-901302037
**[Test build #142621 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/142621/testReport)** for PR 33644 at commit [`d50601d`](https://github.com/apache/spark/commit/d50601d26b8347255d9890d0c8671a0e2c89ae89).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33644: [SPARK-36419][CORE] Optionally move final aggregation in RDD.treeAggregate to executor
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33644:
URL: https://github.com/apache/spark/pull/33644#issuecomment-898668782
Kubernetes integration test unable to build dist.
exiting with code: 1
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/46951/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] akpatnam25 commented on a change in pull request #33644: [SPARK-36419][CORE]: Optionally move final aggregation in RDD.treeAggregate to executor
Posted by GitBox <gi...@apache.org>.
akpatnam25 commented on a change in pull request #33644:
URL: https://github.com/apache/spark/pull/33644#discussion_r683850825
##########
File path: core/src/main/scala/org/apache/spark/rdd/RDD.scala
##########
@@ -1233,6 +1233,21 @@ abstract class RDD[T: ClassTag](
(i, iter) => iter.map((i % curNumPartitions, _))
}.foldByKey(zeroValue, new HashPartitioner(curNumPartitions))(cleanCombOp).values
}
+ if (conf.get(ENABLE_EXECUTOR_TREE_AGGREGATE) && partiallyAggregated.partitions.length > 1) {
+ // define a new partitioner that results in only 1 partition
+ val constantPartitioner = new Partitioner {
+ override def numPartitions: Int = 1
+
+ override def getPartition(key: Any): Int = 0
+ }
+ // map the partially aggregated rdd into a key-value rdd
+ // do the computation in the single executor with one partition
+ // get the new RDD[U]
+ partiallyAggregated = partiallyAggregated
Review comment:
As a temporary fix for the internal issues we saw, we increased driver memory and also lowered `spark.maxRemoteBlockSizeFetchToMem` so that it prevents the driver from trying to put the large buffer it is fetching into memory. Despite the slight overhead, it should prevent the driver from being a single point of failure.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] akpatnam25 commented on a change in pull request #33644: [SPARK-36419][CORE]: Optionally move final aggregation in RDD.treeAggregate to executor
Posted by GitBox <gi...@apache.org>.
akpatnam25 commented on a change in pull request #33644:
URL: https://github.com/apache/spark/pull/33644#discussion_r683850825
##########
File path: core/src/main/scala/org/apache/spark/rdd/RDD.scala
##########
@@ -1233,6 +1233,21 @@ abstract class RDD[T: ClassTag](
(i, iter) => iter.map((i % curNumPartitions, _))
}.foldByKey(zeroValue, new HashPartitioner(curNumPartitions))(cleanCombOp).values
}
+ if (conf.get(ENABLE_EXECUTOR_TREE_AGGREGATE) && partiallyAggregated.partitions.length > 1) {
+ // define a new partitioner that results in only 1 partition
+ val constantPartitioner = new Partitioner {
+ override def numPartitions: Int = 1
+
+ override def getPartition(key: Any): Int = 0
+ }
+ // map the partially aggregated rdd into a key-value rdd
+ // do the computation in the single executor with one partition
+ // get the new RDD[U]
+ partiallyAggregated = partiallyAggregated
Review comment:
As a temporary fix for the internal issues we saw, we increased driver memory and also lowered `spark.maxRemoteBlockSizeFetchToMem` so that it prevents the driver from trying to put the large buffer it is fetching into memory. Despite the slight overhead, this change should prevent the driver from being a single point of failure in such cases.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] akpatnam25 commented on a change in pull request #33644: [SPARK-36419][CORE]: Optionally move final aggregation in RDD.treeAggregate to executor
Posted by GitBox <gi...@apache.org>.
akpatnam25 commented on a change in pull request #33644:
URL: https://github.com/apache/spark/pull/33644#discussion_r683853088
##########
File path: core/src/main/scala/org/apache/spark/rdd/RDD.scala
##########
@@ -1233,6 +1233,21 @@ abstract class RDD[T: ClassTag](
(i, iter) => iter.map((i % curNumPartitions, _))
}.foldByKey(zeroValue, new HashPartitioner(curNumPartitions))(cleanCombOp).values
}
+ if (conf.get(ENABLE_EXECUTOR_TREE_AGGREGATE) && partiallyAggregated.partitions.length > 1) {
+ // define a new partitioner that results in only 1 partition
+ val constantPartitioner = new Partitioner {
+ override def numPartitions: Int = 1
+
+ override def getPartition(key: Any): Int = 0
+ }
+ // map the partially aggregated rdd into a key-value rdd
+ // do the computation in the single executor with one partition
+ // get the new RDD[U]
+ partiallyAggregated = partiallyAggregated
Review comment:
it would not be any less memory intensive. It is just that the executors tend to be configured with higher memory, so there is a lesser chance of this happening on executors. Additionally, executors can retry the task several times, opposed to just 1 try and failing on the driver
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan closed pull request #33644: [SPARK-36419][CORE] Optionally move final aggregation in RDD.treeAggregate to executor
Posted by GitBox <gi...@apache.org>.
cloud-fan closed pull request #33644:
URL: https://github.com/apache/spark/pull/33644
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] akpatnam25 commented on pull request #33644: [SPARK-36419][CORE] Optionally move final aggregation in RDD.treeAggregate to executor
Posted by GitBox <gi...@apache.org>.
akpatnam25 commented on pull request #33644:
URL: https://github.com/apache/spark/pull/33644#issuecomment-904288270
can this be merged now? We now have consensus on both the reason for doing this change and also have implemented a parameter based approach. @mridulm @venkata91 @HyukjinKwon
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #33644: [SPARK-36419][CORE] Optionally move final aggregation in RDD.treeAggregate to executor
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #33644:
URL: https://github.com/apache/spark/pull/33644#issuecomment-897195152
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/46855/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #33644: [SPARK-36419][CORE] Optionally move final aggregation in RDD.treeAggregate to executor
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #33644:
URL: https://github.com/apache/spark/pull/33644#issuecomment-902423682
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/142651/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #33644: [SPARK-36419][CORE] Optionally move final aggregation in RDD.treeAggregate to executor
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #33644:
URL: https://github.com/apache/spark/pull/33644#issuecomment-898647249
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] srowen commented on a change in pull request #33644: [SPARK-36419][CORE] Optionally move final aggregation in RDD.treeAggregate to executor
Posted by GitBox <gi...@apache.org>.
srowen commented on a change in pull request #33644:
URL: https://github.com/apache/spark/pull/33644#discussion_r687289070
##########
File path: core/src/test/java/test/org/apache/spark/JavaAPISuite.java
##########
@@ -541,7 +541,7 @@ public void treeAggregate() {
JavaRDD<Integer> rdd = sc.parallelize(Arrays.asList(-5, -4, -3, -2, -1, 1, 2, 3, 4), 10);
Function2<Integer, Integer, Integer> add = (a, b) -> a + b;
for (int depth = 1; depth <= 10; depth++) {
- int sum = rdd.treeAggregate(0, add, add, depth);
+ int sum = rdd.treeAggregate(0, add, add, depth, false);
Review comment:
Right. in the bytecode it is not optional as there is no such thing in the bytecode. For Java callers it's required; Scala compiler inserts the default value when compiling Scala code.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33644: [SPARK-36419][CORE] Optionally move final aggregation in RDD.treeAggregate to executor
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33644:
URL: https://github.com/apache/spark/pull/33644#issuecomment-902411198
**[Test build #142651 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/142651/testReport)** for PR 33644 at commit [`f728a02`](https://github.com/apache/spark/commit/f728a02407c31faf4a13665c4763c13344e4036d).
* This patch **fails from timeout after a configured wait of `500m`**.
* This patch merges cleanly.
* This patch adds no public classes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #33644: [SPARK-36419][CORE] Optionally move final aggregation in RDD.treeAggregate to executor
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #33644:
URL: https://github.com/apache/spark/pull/33644#issuecomment-898641048
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/142445/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #33644: [SPARK-36419][CORE] Optionally move final aggregation in RDD.treeAggregate to executor
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #33644:
URL: https://github.com/apache/spark/pull/33644#issuecomment-902200052
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/47151/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33644: [SPARK-36419][CORE] Optionally move final aggregation in RDD.treeAggregate to executor
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33644:
URL: https://github.com/apache/spark/pull/33644#issuecomment-898652298
**[Test build #142451 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/142451/testReport)** for PR 33644 at commit [`75945bc`](https://github.com/apache/spark/commit/75945bc49e2090a8d7f9e3dfa67e82d545edca3f).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #33644: [SPARK-36419][CORE] Optionally move final aggregation in RDD.treeAggregate to executor
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #33644:
URL: https://github.com/apache/spark/pull/33644#issuecomment-901353282
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] venkata91 commented on pull request #33644: [SPARK-36419][CORE]: Optionally move final aggregation in RDD.treeAggregate to executor
Posted by GitBox <gi...@apache.org>.
venkata91 commented on pull request #33644:
URL: https://github.com/apache/spark/pull/33644#issuecomment-894400752
Extra shuffle and the additional scheduling overhead for the 1 task foldByKey operation. This can be slightly high for trivial `treeAggregate` operations, in that case it makes sense to control this behavior under a flag. Thoughts?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33644: [SPARK-36419][CORE] Optionally move final aggregation in RDD.treeAggregate to executor
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33644:
URL: https://github.com/apache/spark/pull/33644#issuecomment-898649298
**[Test build #142450 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/142450/testReport)** for PR 33644 at commit [`2ab08c4`](https://github.com/apache/spark/commit/2ab08c4e054b922c0fe450761fbaf7407294ef33).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33644: [SPARK-36419][CORE] Optionally move final aggregation in RDD.treeAggregate to executor
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33644:
URL: https://github.com/apache/spark/pull/33644#issuecomment-902176951
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/47151/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33644: [SPARK-36419][CORE] Optionally move final aggregation in RDD.treeAggregate to executor
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33644:
URL: https://github.com/apache/spark/pull/33644#issuecomment-897176588
Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/46855/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] venkata91 commented on a change in pull request #33644: [SPARK-36419][CORE]: Optionally move final aggregation in RDD.treeAggregate to executor
Posted by GitBox <gi...@apache.org>.
venkata91 commented on a change in pull request #33644:
URL: https://github.com/apache/spark/pull/33644#discussion_r682991221
##########
File path: core/src/main/scala/org/apache/spark/rdd/RDD.scala
##########
@@ -1233,6 +1233,21 @@ abstract class RDD[T: ClassTag](
(i, iter) => iter.map((i % curNumPartitions, _))
}.foldByKey(zeroValue, new HashPartitioner(curNumPartitions))(cleanCombOp).values
}
+ if (conf.get(ENABLE_EXECUTOR_TREE_AGGREGATE) && partiallyAggregated.partitions.length > 1) {
+ // define a new partitioner that results in only 1 partition
+ val constantPartitioner = new Partitioner {
+ override def numPartitions: Int = 1
+
+ override def getPartition(key: Any): Int = 0
+ }
+ // map the partially aggregated rdd into a key-value rdd
+ // do the computation in the single executor with one partition
+ // get the new RDD[U]
+ partiallyAggregated = partiallyAggregated
+ .map(v => (0.toByte, v))
+ .foldByKey(zeroValue, constantPartitioner)(cleanCombOp)
+ .values
+ }
val copiedZeroValue = Utils.clone(zeroValue, sc.env.closureSerializer.newInstance())
Review comment:
wouldn't this submit another job after the `foldByKey` again on the `fold` eventually to just return the only value?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] venkata91 commented on a change in pull request #33644: [SPARK-36419][CORE] Optionally move final aggregation in RDD.treeAggregate to executor
Posted by GitBox <gi...@apache.org>.
venkata91 commented on a change in pull request #33644:
URL: https://github.com/apache/spark/pull/33644#discussion_r688045988
##########
File path: project/MimaExcludes.scala
##########
@@ -104,6 +104,10 @@ object MimaExcludes {
ProblemFilters.exclude[DirectMissingMethodProblem]("org.apache.spark.shuffle.api.SingleSpillShuffleMapOutputWriter.transferMapSpillFile"),
ProblemFilters.exclude[ReversedMissingMethodProblem]("org.apache.spark.shuffle.api.SingleSpillShuffleMapOutputWriter.transferMapSpillFile"),
ProblemFilters.exclude[ReversedMissingMethodProblem]("org.apache.spark.shuffle.api.ShuffleMapOutputWriter.commitAllPartitions")
+
+ // [SPARK-36419][CORE] Optionally move final aggregation in RDD.treeAggregate to executor
+ ProblemFilters.exclude[ReversedMissingMethodProblem]("org.apache.spark.api.java.JavaRDDLike.treeAggregate")
Review comment:
I think the issue is because of `zeroValue:U` is only inspected by the compiler to resolve the overloaded methods. Due to that the compiler throws `ambiguous reference to overloaded definition`. Something like this cannot be resolved - `val func = rdd.treeAggregate(0L) _` properly with the newly added overloaded method. @srowen What do you think?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on a change in pull request #33644: [SPARK-36419][CORE]: Optionally move final aggregation in RDD.treeAggregate to executor
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on a change in pull request #33644:
URL: https://github.com/apache/spark/pull/33644#discussion_r684855590
##########
File path: core/src/main/scala/org/apache/spark/internal/config/package.scala
##########
@@ -315,6 +315,12 @@ package object config {
.bytesConf(ByteUnit.MiB)
.createOptional
+ private[spark] val ENABLE_EXECUTOR_TREE_AGGREGATE = ConfigBuilder("spark.executor.treeAggregate")
Review comment:
I have a similar feeling w/ Sean's comment here. Should we maybe just add it as a parameter instead of adding it as a configuration? Not sure which way is the best. One concern might be that this is a static configuration that cannot be changed.
##########
File path: core/src/test/scala/org/apache/spark/rdd/RDDSuite.scala
##########
@@ -275,6 +275,18 @@ class RDDSuite extends SparkFunSuite with SharedSparkContext with Eventually {
}
}
+ test("treeAggregate with ENABLE_EXECUTOR_TREE_AGGREGATE true") {
Review comment:
nit .. but I would add a JIRA prefix
##########
File path: core/src/main/scala/org/apache/spark/internal/config/package.scala
##########
@@ -315,6 +315,12 @@ package object config {
.bytesConf(ByteUnit.MiB)
.createOptional
+ private[spark] val ENABLE_EXECUTOR_TREE_AGGREGATE = ConfigBuilder("spark.executor.treeAggregate")
Review comment:
If we will add it as a configuration, I would rename it something like `spark.rdd.treeAggregate.finalAggregateOnExecutor` (feel free to change the wording).
##########
File path: core/src/test/scala/org/apache/spark/rdd/RDDSuite.scala
##########
@@ -275,6 +275,18 @@ class RDDSuite extends SparkFunSuite with SharedSparkContext with Eventually {
}
}
+ test("treeAggregate with ENABLE_EXECUTOR_TREE_AGGREGATE true") {
Review comment:
and should probably fix the test title (`ENABLE_EXECUTOR_TREE_AGGREGATE`) too.
##########
File path: core/src/main/scala/org/apache/spark/internal/config/package.scala
##########
@@ -315,6 +315,12 @@ package object config {
.bytesConf(ByteUnit.MiB)
.createOptional
+ private[spark] val ENABLE_EXECUTOR_TREE_AGGREGATE = ConfigBuilder("spark.executor.treeAggregate")
Review comment:
(If we'd add it as a parameter, we will have to add it to `JavaRDDLike` too)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33644: [SPARK-36419][CORE] Optionally move final aggregation in RDD.treeAggregate to executor
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33644:
URL: https://github.com/apache/spark/pull/33644#issuecomment-901353230
Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/47122/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #33644: [SPARK-36419][CORE] Optionally move final aggregation in RDD.treeAggregate to executor
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #33644:
URL: https://github.com/apache/spark/pull/33644#issuecomment-902242086
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] srowen commented on pull request #33644: [SPARK-36419][CORE]: Optionally move final aggregation in RDD.treeAggregate to executor
Posted by GitBox <gi...@apache.org>.
srowen commented on pull request #33644:
URL: https://github.com/apache/spark/pull/33644#issuecomment-894361123
Hm, I wonder if this is better solved with a slightly higher tree depth - or is it not controllable in the context in question?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #33644: [SPARK-36419][CORE] Optionally move final aggregation in RDD.treeAggregate to executor
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #33644:
URL: https://github.com/apache/spark/pull/33644#issuecomment-898729809
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/142451/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] mridulm commented on a change in pull request #33644: [SPARK-36419][CORE] Optionally move final aggregation in RDD.treeAggregate to executor
Posted by GitBox <gi...@apache.org>.
mridulm commented on a change in pull request #33644:
URL: https://github.com/apache/spark/pull/33644#discussion_r692293917
##########
File path: core/src/main/scala/org/apache/spark/rdd/RDD.scala
##########
@@ -1233,6 +1233,22 @@ abstract class RDD[T: ClassTag](
(i, iter) => iter.map((i % curNumPartitions, _))
Review comment:
Actually, I like @HyukjinKwon's proposal.
Add a new `treeAggregate` method to RDD which has all the parameters explicitly specified (no defaults, no currying).
And have everything else delegate to it.
This should take care of @srowen's concerns about binary/source compatibility (existing methods remain as is), while introducing a new method in scala api which allows for per method customization (and not global config affecting all aggregates).
Thoughts ?
In RDD:
```
def treeAggregate[U: ClassTag](zeroValue: U)(
seqOp: (U, T) => U,
combOp: (U, U) => U,
depth: Int = 2): U = withScope {
treeAggregate(zeroValue, seqOp, combOp, depth, false)
}
def treeAggregate[U: ClassTag](
zeroValue: U,
seqOp: (U, T) => U,
combOp: (U, U) => U,
depth: Int,
finalAggregateOnExecutor: Boolean): U = withScope {
// modified method taking finalAggregateOnExecutor into account
}
```
java api to mirror and delegate to the scala api as appropriate.
Thoughts @HyukjinKwon, @srowen, @akpatnam25, @venkata91 ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] mridulm commented on pull request #33644: [SPARK-36419][CORE]: Optionally move final aggregation in RDD.treeAggregate to executor
Posted by GitBox <gi...@apache.org>.
mridulm commented on pull request #33644:
URL: https://github.com/apache/spark/pull/33644#issuecomment-893761008
+CC @srowen, @dongjoon-hyun
The original folks who worked on the api dont seem to be active anymore.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #33644: [SPARK-36419][CORE] Optionally move final aggregation in RDD.treeAggregate to executor
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #33644:
URL: https://github.com/apache/spark/pull/33644#issuecomment-898724156
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/46956/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] akpatnam25 commented on a change in pull request #33644: [SPARK-36419][CORE] Optionally move final aggregation in RDD.treeAggregate to executor
Posted by GitBox <gi...@apache.org>.
akpatnam25 commented on a change in pull request #33644:
URL: https://github.com/apache/spark/pull/33644#discussion_r688119953
##########
File path: project/MimaExcludes.scala
##########
@@ -104,6 +104,10 @@ object MimaExcludes {
ProblemFilters.exclude[DirectMissingMethodProblem]("org.apache.spark.shuffle.api.SingleSpillShuffleMapOutputWriter.transferMapSpillFile"),
ProblemFilters.exclude[ReversedMissingMethodProblem]("org.apache.spark.shuffle.api.SingleSpillShuffleMapOutputWriter.transferMapSpillFile"),
ProblemFilters.exclude[ReversedMissingMethodProblem]("org.apache.spark.shuffle.api.ShuffleMapOutputWriter.commitAllPartitions")
+
+ // [SPARK-36419][CORE] Optionally move final aggregation in RDD.treeAggregate to executor
+ ProblemFilters.exclude[ReversedMissingMethodProblem]("org.apache.spark.api.java.JavaRDDLike.treeAggregate")
Review comment:
okay, so i think the best option we have now is to revert back to the earlier version that used a config. not sure if making a new method is worth it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] srowen commented on pull request #33644: [SPARK-36419][CORE]: Optionally move final aggregation in RDD.treeAggregate to executor
Posted by GitBox <gi...@apache.org>.
srowen commented on pull request #33644:
URL: https://github.com/apache/spark/pull/33644#issuecomment-894403593
Can it be quantified at all? Trying to get a sense of whether it affects most normal sized cases non trivially or really adds very little except for trivial cases. Users are not going to find the flag in general so question is how much this impacts users vs solves a problem that already has a solution of sorts
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] mridulm commented on a change in pull request #33644: [SPARK-36419][CORE] Optionally move final aggregation in RDD.treeAggregate to executor
Posted by GitBox <gi...@apache.org>.
mridulm commented on a change in pull request #33644:
URL: https://github.com/apache/spark/pull/33644#discussion_r692293917
##########
File path: core/src/main/scala/org/apache/spark/rdd/RDD.scala
##########
@@ -1233,6 +1233,22 @@ abstract class RDD[T: ClassTag](
(i, iter) => iter.map((i % curNumPartitions, _))
Review comment:
Actually, I like @HyukjinKwon's proposal.
A variant of it would be add a new `treeAggregate` method to RDD which has all the parameters explicitly specified (no defaults, no currying).
And have everything else delegate to it.
This should take care of @srowen's concerns about binary/source compatibility (existing methods remain as is), while introducing a new method in scala api which allows for per method customization (and not global config affecting all aggregates).
Thoughts ?
In RDD:
```
def treeAggregate[U: ClassTag](zeroValue: U)(
seqOp: (U, T) => U,
combOp: (U, U) => U,
depth: Int = 2): U = withScope {
treeAggregate(zeroValue, seqOp, combOp, depth, false)
}
def treeAggregate[U: ClassTag](
zeroValue: U,
seqOp: (U, T) => U,
combOp: (U, U) => U,
depth: Int,
finalAggregateOnExecutor: Boolean): U = withScope {
// modified method taking finalAggregateOnExecutor into account
}
```
java api to mirror and delegate to the scala api as appropriate.
Thoughts @HyukjinKwon, @srowen, @akpatnam25, @venkata91 ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] akpatnam25 commented on a change in pull request #33644: [SPARK-36419][CORE] Optionally move final aggregation in RDD.treeAggregate to executor
Posted by GitBox <gi...@apache.org>.
akpatnam25 commented on a change in pull request #33644:
URL: https://github.com/apache/spark/pull/33644#discussion_r688106700
##########
File path: project/MimaExcludes.scala
##########
@@ -104,6 +104,10 @@ object MimaExcludes {
ProblemFilters.exclude[DirectMissingMethodProblem]("org.apache.spark.shuffle.api.SingleSpillShuffleMapOutputWriter.transferMapSpillFile"),
ProblemFilters.exclude[ReversedMissingMethodProblem]("org.apache.spark.shuffle.api.SingleSpillShuffleMapOutputWriter.transferMapSpillFile"),
ProblemFilters.exclude[ReversedMissingMethodProblem]("org.apache.spark.shuffle.api.ShuffleMapOutputWriter.commitAllPartitions")
+
+ // [SPARK-36419][CORE] Optionally move final aggregation in RDD.treeAggregate to executor
+ ProblemFilters.exclude[ReversedMissingMethodProblem]("org.apache.spark.api.java.JavaRDDLike.treeAggregate")
Review comment:
@srowen i dont think we break api compatibility on the scala side because we use a default param anyways, so the user would not have to adapt to the new signature. So i am proposing that we keep the scala file in the mima excludes and not worry about the binary incompatibility for that.
On the java side, we will be breaking api compatibility, and we can add a method there that is overloaded. This will allow us to not break compatibility. There's already 2 `treeAggregate` methods in the javaRdd, so roughly speaking we will just add a 3rd one. Something like this:
```
/**
* Aggregates the elements of this RDD in a multi-level tree pattern.
*
* @param depth suggested depth of the tree
* @see [[org.apache.spark.api.java.JavaRDDLike#aggregate]]
*/
def treeAggregate[U](
zeroValue: U,
seqOp: JFunction2[U, T, U],
combOp: JFunction2[U, U, U],
depth: Int): U = {
rdd.treeAggregate(zeroValue)(seqOp, combOp, depth)(fakeClassTag[U])
}
/**
* `org.apache.spark.api.java.JavaRDDLike.treeAggregate` with suggested depth 2.
*/
def treeAggregate[U](
zeroValue: U,
seqOp: JFunction2[U, T, U],
combOp: JFunction2[U, U, U]): U = {
treeAggregate(zeroValue, seqOp, combOp, 2)
}
/**
* Aggregates the elements of this RDD in a multi-level tree pattern and provides the option of doing
* the final aggregation on an executor
*
* @param depth suggested depth of the tree
* @see [[org.apache.spark.api.java.JavaRDDLike#aggregate]]
*/
def treeAggregate[U](
zeroValue: U,
seqOp: JFunction2[U, T, U],
combOp: JFunction2[U, U, U],
depth: Int,
finalAggregateOnExecutor: Boolean): U = {
rdd.treeAggregate(zeroValue)(seqOp, combOp, depth, finalAggregateOnExecutor)(fakeClassTag[U])
}
```
Then we can remove the mima exludes for the javaRdd class. what do you think?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] akpatnam25 commented on pull request #33644: [SPARK-36419][CORE]: Optionally move final aggregation in RDD.treeAggregate to executor
Posted by GitBox <gi...@apache.org>.
akpatnam25 commented on pull request #33644:
URL: https://github.com/apache/spark/pull/33644#issuecomment-895487510
I have no problem with adding it in as a parameter defaulted to false (and removing it as a config), if @venkata91 and @mridulm are fine with it. I'll wait for some of the others to chime in before I go ahead and make the change to the pr.
Thanks for reviewing @HyukjinKwon .
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] srowen commented on pull request #33644: [SPARK-36419][CORE]: Optionally move final aggregation in RDD.treeAggregate to executor
Posted by GitBox <gi...@apache.org>.
srowen commented on pull request #33644:
URL: https://github.com/apache/spark/pull/33644#issuecomment-894839816
I end up neutral on this - that the driver may not have enough memory doesn't seem quite like justification here, but possibly the argument about better aggregation on the executor (really?). I am concerned there is no quantified argument about overhead yet here. I wouldn't veto it if others feel strongly but might ask for a third (fourth?) opinion to be cautious
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] srowen commented on a change in pull request #33644: [SPARK-36419][CORE]: Optionally move final aggregation in RDD.treeAggregate to executor
Posted by GitBox <gi...@apache.org>.
srowen commented on a change in pull request #33644:
URL: https://github.com/apache/spark/pull/33644#discussion_r683851357
##########
File path: core/src/main/scala/org/apache/spark/rdd/RDD.scala
##########
@@ -1233,6 +1233,21 @@ abstract class RDD[T: ClassTag](
(i, iter) => iter.map((i % curNumPartitions, _))
}.foldByKey(zeroValue, new HashPartitioner(curNumPartitions))(cleanCombOp).values
}
+ if (conf.get(ENABLE_EXECUTOR_TREE_AGGREGATE) && partiallyAggregated.partitions.length > 1) {
+ // define a new partitioner that results in only 1 partition
+ val constantPartitioner = new Partitioner {
+ override def numPartitions: Int = 1
+
+ override def getPartition(key: Any): Int = 0
+ }
+ // map the partially aggregated rdd into a key-value rdd
+ // do the computation in the single executor with one partition
+ // get the new RDD[U]
+ partiallyAggregated = partiallyAggregated
Review comment:
Why would this be any less memory intensive after this change? there is still the same last N-to-1 aggregation _somewhere_.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] venkata91 commented on a change in pull request #33644: [SPARK-36419][CORE] Optionally move final aggregation in RDD.treeAggregate to executor
Posted by GitBox <gi...@apache.org>.
venkata91 commented on a change in pull request #33644:
URL: https://github.com/apache/spark/pull/33644#discussion_r687280002
##########
File path: core/src/test/java/test/org/apache/spark/JavaAPISuite.java
##########
@@ -541,7 +541,7 @@ public void treeAggregate() {
JavaRDD<Integer> rdd = sc.parallelize(Arrays.asList(-5, -4, -3, -2, -1, 1, 2, 3, 4), 10);
Function2<Integer, Integer, Integer> add = (a, b) -> a + b;
for (int depth = 1; depth <= 10; depth++) {
- int sum = rdd.treeAggregate(0, add, add, depth);
+ int sum = rdd.treeAggregate(0, add, add, depth, false);
Review comment:
The other one is not overloaded right, it has additional param `depth`. This seems to be java <-> scala interoperability issue which is why even with default param, the value needs to be specified I think. Not exactly the same case but similar problem discussed here - https://stackoverflow.com/questions/13059528/instantiate-a-scala-class-from-java-and-use-the-default-parameters-of-the-const
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] akpatnam25 commented on pull request #33644: [SPARK-36419][CORE]: Optionally move final aggregation in RDD.treeAggregate to executor
Posted by GitBox <gi...@apache.org>.
akpatnam25 commented on pull request #33644:
URL: https://github.com/apache/spark/pull/33644#issuecomment-893594625
retest this please
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #33644: [SPARK-36419][CORE] Optionally move final aggregation in RDD.treeAggregate to executor
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #33644:
URL: https://github.com/apache/spark/pull/33644#issuecomment-898724156
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/46956/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] akpatnam25 commented on pull request #33644: [SPARK-36419][CORE] Optionally move final aggregation in RDD.treeAggregate to executor
Posted by GitBox <gi...@apache.org>.
akpatnam25 commented on pull request #33644:
URL: https://github.com/apache/spark/pull/33644#issuecomment-898629721
i just pushed up a new change that reverts back to the older config approach. I also changed the config as per HyukjinKwon's guidance on the naming.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] akpatnam25 commented on a change in pull request #33644: [SPARK-36419][CORE] Optionally move final aggregation in RDD.treeAggregate to executor
Posted by GitBox <gi...@apache.org>.
akpatnam25 commented on a change in pull request #33644:
URL: https://github.com/apache/spark/pull/33644#discussion_r690888342
##########
File path: core/src/main/scala/org/apache/spark/rdd/RDD.scala
##########
@@ -1233,6 +1233,22 @@ abstract class RDD[T: ClassTag](
(i, iter) => iter.map((i % curNumPartitions, _))
Review comment:
yes, we can add a new method with signature that looks like this:
```
def treeAggregate[U](
zeroValue: U,
seqOp: JFunction2[U, T, U],
combOp: JFunction2[U, U, U],
depth: Int,
finalAggregateOnExecutor: Boolean): U = {
...
```
you're right, we dont need to worry about Mima excludes then.
but that is not the problem. the problem is that this method will need to be overloaded by the other treeAggregate methods. for example, the other treeAggregate methods will have to call this version of treeAggregate. this is an issue because of zeroValue:U is only inspected by the compiler to resolve the overloaded methods. Due to that the compiler throws ambiguous reference to overloaded definition, and this cannot be resolved..
there are only 2 workarounds:
1. create a new method with a different name but same signature with the new parameter. something like `treeAggregateExecutor` and have somewhat duplicate code in that method. me and @venkata91 thought this approach was kind of ugly
2. use config approach which is the current implementation
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #33644: [SPARK-36419][CORE] Optionally move final aggregation in RDD.treeAggregate to executor
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #33644:
URL: https://github.com/apache/spark/pull/33644#issuecomment-897220044
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/142347/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] venkata91 commented on a change in pull request #33644: [SPARK-36419][CORE] Optionally move final aggregation in RDD.treeAggregate to executor
Posted by GitBox <gi...@apache.org>.
venkata91 commented on a change in pull request #33644:
URL: https://github.com/apache/spark/pull/33644#discussion_r687122613
##########
File path: project/MimaExcludes.scala
##########
@@ -104,6 +104,10 @@ object MimaExcludes {
ProblemFilters.exclude[DirectMissingMethodProblem]("org.apache.spark.shuffle.api.SingleSpillShuffleMapOutputWriter.transferMapSpillFile"),
ProblemFilters.exclude[ReversedMissingMethodProblem]("org.apache.spark.shuffle.api.SingleSpillShuffleMapOutputWriter.transferMapSpillFile"),
ProblemFilters.exclude[ReversedMissingMethodProblem]("org.apache.spark.shuffle.api.ShuffleMapOutputWriter.commitAllPartitions")
+
+ // [SPARK-36419][CORE] Optionally move final aggregation in RDD.treeAggregate to executor
+ ProblemFilters.exclude[ReversedMissingMethodProblem]("org.apache.spark.api.java.JavaRDDLike.treeAggregate")
Review comment:
I think we need to add
`ProblemFilters.exclude[DirectMissingMethodProblem("org.apache.spark.api.java.JavaRDDLike.treeAggregate")`
`ProblemFilters.exclude[DirectMissingMethodProblem]("org.apache.spark.rdd.RDD.treeAggregate")`
##########
File path: core/src/test/java/test/org/apache/spark/JavaAPISuite.java
##########
@@ -541,7 +541,7 @@ public void treeAggregate() {
JavaRDD<Integer> rdd = sc.parallelize(Arrays.asList(-5, -4, -3, -2, -1, 1, 2, 3, 4), 10);
Function2<Integer, Integer, Integer> add = (a, b) -> a + b;
for (int depth = 1; depth <= 10; depth++) {
- int sum = rdd.treeAggregate(0, add, add, depth);
+ int sum = rdd.treeAggregate(0, add, add, depth, false);
Review comment:
nit: already the default value is `false` right?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #33644: [SPARK-36419][CORE] Optionally move final aggregation in RDD.treeAggregate to executor
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #33644:
URL: https://github.com/apache/spark/pull/33644#issuecomment-898698104
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/46954/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33644: [SPARK-36419][CORE] Optionally move final aggregation in RDD.treeAggregate to executor
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33644:
URL: https://github.com/apache/spark/pull/33644#issuecomment-901421457
**[Test build #142621 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/142621/testReport)** for PR 33644 at commit [`d50601d`](https://github.com/apache/spark/commit/d50601d26b8347255d9890d0c8671a0e2c89ae89).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] akpatnam25 commented on pull request #33644: [SPARK-36419][CORE] Optionally move final aggregation in RDD.treeAggregate to executor
Posted by GitBox <gi...@apache.org>.
akpatnam25 commented on pull request #33644:
URL: https://github.com/apache/spark/pull/33644#issuecomment-899676449
can you take one more look at this pr so that we can merge? @HyukjinKwon @cloud-fan @srowen
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #33644: [SPARK-36419][CORE] Optionally move final aggregation in RDD.treeAggregate to executor
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #33644:
URL: https://github.com/apache/spark/pull/33644#issuecomment-902149355
**[Test build #142650 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/142650/testReport)** for PR 33644 at commit [`3d49c1d`](https://github.com/apache/spark/commit/3d49c1d4a28e28856953ee5ec492e3a5249965da).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] akpatnam25 commented on a change in pull request #33644: [SPARK-36419][CORE]: Optionally move final aggregation in RDD.treeAggregate to executor
Posted by GitBox <gi...@apache.org>.
akpatnam25 commented on a change in pull request #33644:
URL: https://github.com/apache/spark/pull/33644#discussion_r683031421
##########
File path: core/src/main/scala/org/apache/spark/rdd/RDD.scala
##########
@@ -1233,6 +1233,21 @@ abstract class RDD[T: ClassTag](
(i, iter) => iter.map((i % curNumPartitions, _))
}.foldByKey(zeroValue, new HashPartitioner(curNumPartitions))(cleanCombOp).values
}
+ if (conf.get(ENABLE_EXECUTOR_TREE_AGGREGATE) && partiallyAggregated.partitions.length > 1) {
+ // define a new partitioner that results in only 1 partition
+ val constantPartitioner = new Partitioner {
+ override def numPartitions: Int = 1
+
+ override def getPartition(key: Any): Int = 0
+ }
+ // map the partially aggregated rdd into a key-value rdd
+ // do the computation in the single executor with one partition
+ // get the new RDD[U]
+ partiallyAggregated = partiallyAggregated
+ .map(v => (0.toByte, v))
+ .foldByKey(zeroValue, constantPartitioner)(cleanCombOp)
+ .values
+ }
val copiedZeroValue = Utils.clone(zeroValue, sc.env.closureSerializer.newInstance())
Review comment:
no it would not submit another job, because there is only one partition. spark would do the fold internal the partition (we only have 1 in this case), and return.
As described on Apache Spark's website the fold definition states:
```
"Aggregate the elements of each partition, and then the results for all the partitions, using a given associative function and a neutral "zero value"."
```
Thus fold operates by folding first each element of partition and then the result of partition
it is partition dependent
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #33644: [SPARK-36419][CORE] Optionally move final aggregation in RDD.treeAggregate to executor
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #33644:
URL: https://github.com/apache/spark/pull/33644#issuecomment-898650330
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/142450/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #33644: [SPARK-36419][CORE] Optionally move final aggregation in RDD.treeAggregate to executor
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #33644:
URL: https://github.com/apache/spark/pull/33644#issuecomment-898669051
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/46951/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #33644: [SPARK-36419][CORE] Optionally move final aggregation in RDD.treeAggregate to executor
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #33644:
URL: https://github.com/apache/spark/pull/33644#issuecomment-898698104
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/46954/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #33644: [SPARK-36419][CORE] Optionally move final aggregation in RDD.treeAggregate to executor
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #33644:
URL: https://github.com/apache/spark/pull/33644#issuecomment-898641048
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/142445/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #33644: [SPARK-36419][CORE] Optionally move final aggregation in RDD.treeAggregate to executor
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #33644:
URL: https://github.com/apache/spark/pull/33644#issuecomment-898729809
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/142451/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] akpatnam25 commented on a change in pull request #33644: [SPARK-36419][CORE] Optionally move final aggregation in RDD.treeAggregate to executor
Posted by GitBox <gi...@apache.org>.
akpatnam25 commented on a change in pull request #33644:
URL: https://github.com/apache/spark/pull/33644#discussion_r690888342
##########
File path: core/src/main/scala/org/apache/spark/rdd/RDD.scala
##########
@@ -1233,6 +1233,22 @@ abstract class RDD[T: ClassTag](
(i, iter) => iter.map((i % curNumPartitions, _))
Review comment:
yes, we can add a new method with signature that looks like this:
```
def treeAggregate[U](
zeroValue: U,
seqOp: JFunction2[U, T, U],
combOp: JFunction2[U, U, U],
depth: Int,
finalAggregateOnExecutor: Boolean): U = {
...
```
you're right, we dont need to worry about Mima excludes then.
but that is not the problem. the problem is that this method will need to be overloaded by the other treeAggregate methods. for example, the other treeAggregate methods will have to call this version of treeAggregate. this is an issue because of zeroValue:U is only inspected by the compiler to resolve the overloaded methods. Due to that the compiler throws ambiguous reference to overloaded definition, and this cannot be resolved..
there are only 2 workarounds:
1. create a new method with a different name but same signature with the new parameter. something like `treeAggregateExecutor` and have somewhat duplicate code in that method. me and @venkata91 thought this approach was kind of ugly
2. use config approach
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33644: [SPARK-36419][CORE] Optionally move final aggregation in RDD.treeAggregate to executor
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33644:
URL: https://github.com/apache/spark/pull/33644#issuecomment-898720433
Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/46956/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on pull request #33644: [SPARK-36419][CORE] Optionally move final aggregation in RDD.treeAggregate to executor
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on pull request #33644:
URL: https://github.com/apache/spark/pull/33644#issuecomment-899908732
I am okay w this change. Seems like @cloud-fan and @srowen are fine either.
I think we agreed upon having it as a parameter instead of a configuration which seems not addressed yet.
Otherwise, I would defer to @mridulm
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #33644: [SPARK-36419][CORE] Optionally move final aggregation in RDD.treeAggregate to executor
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #33644:
URL: https://github.com/apache/spark/pull/33644#issuecomment-898646177
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on a change in pull request #33644: [SPARK-36419][CORE] Optionally move final aggregation in RDD.treeAggregate to executor
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on a change in pull request #33644:
URL: https://github.com/apache/spark/pull/33644#discussion_r686418122
##########
File path: core/src/main/scala/org/apache/spark/internal/config/package.scala
##########
@@ -315,6 +315,12 @@ package object config {
.bytesConf(ByteUnit.MiB)
.createOptional
+ private[spark] val ENABLE_EXECUTOR_TREE_AGGREGATE = ConfigBuilder("spark.executor.treeAggregate")
Review comment:
BTW, let's don't forget to add `since` version too
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #33644: [SPARK-36419][CORE] Optionally move final aggregation in RDD.treeAggregate to executor
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #33644:
URL: https://github.com/apache/spark/pull/33644#issuecomment-897120340
**[Test build #142347 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/142347/testReport)** for PR 33644 at commit [`811eb3a`](https://github.com/apache/spark/commit/811eb3a0acc62b3cf881999f24be8cd2a37998f3).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33644: [SPARK-36419][CORE] Optionally move final aggregation in RDD.treeAggregate to executor
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33644:
URL: https://github.com/apache/spark/pull/33644#issuecomment-898646177
**[Test build #142448 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/142448/testReport)** for PR 33644 at commit [`3542781`](https://github.com/apache/spark/commit/3542781e0cb377ae40406c466b7fe20d08630d96).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33644: [SPARK-36419][CORE] Optionally move final aggregation in RDD.treeAggregate to executor
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33644:
URL: https://github.com/apache/spark/pull/33644#issuecomment-898650308
**[Test build #142450 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/142450/testReport)** for PR 33644 at commit [`2ab08c4`](https://github.com/apache/spark/commit/2ab08c4e054b922c0fe450761fbaf7407294ef33).
* This patch **fails Scala style tests**.
* This patch merges cleanly.
* This patch adds no public classes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon removed a comment on pull request #33644: [SPARK-36419][CORE] Optionally move final aggregation in RDD.treeAggregate to executor
Posted by GitBox <gi...@apache.org>.
HyukjinKwon removed a comment on pull request #33644:
URL: https://github.com/apache/spark/pull/33644#issuecomment-899908732
I am okay w this change. Seems like @cloud-fan and @srowen are fine either.
I think we agreed upon having it as a parameter instead of a configuration which seems not addressed yet.
Otherwise, I would defer to @mridulm
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] akpatnam25 edited a comment on pull request #33644: [SPARK-36419][CORE] Optionally move final aggregation in RDD.treeAggregate to executor
Posted by GitBox <gi...@apache.org>.
akpatnam25 edited a comment on pull request #33644:
URL: https://github.com/apache/spark/pull/33644#issuecomment-897069946
I've modified the code to include `finalAggregateOnExecutor` as a parameter being passed into treeAggregate. I've also updated the JavaRDDLike and the relevant unit tests. Let me know if we want to expand the scope of the change to any other api's to be updated as well where treeAggregate is used.
@mridulm @HyukjinKwon
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] akpatnam25 commented on pull request #33644: [SPARK-36419][CORE] Optionally move final aggregation in RDD.treeAggregate to executor
Posted by GitBox <gi...@apache.org>.
akpatnam25 commented on pull request #33644:
URL: https://github.com/apache/spark/pull/33644#issuecomment-900921583
@HyukjinKwon can we merge this then now? seems like we have consensus from everyone.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33644: [SPARK-36419][CORE] Optionally move final aggregation in RDD.treeAggregate to executor
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33644:
URL: https://github.com/apache/spark/pull/33644#issuecomment-897154494
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/46855/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] srowen commented on a change in pull request #33644: [SPARK-36419][CORE] Optionally move final aggregation in RDD.treeAggregate to executor
Posted by GitBox <gi...@apache.org>.
srowen commented on a change in pull request #33644:
URL: https://github.com/apache/spark/pull/33644#discussion_r688113678
##########
File path: project/MimaExcludes.scala
##########
@@ -104,6 +104,10 @@ object MimaExcludes {
ProblemFilters.exclude[DirectMissingMethodProblem]("org.apache.spark.shuffle.api.SingleSpillShuffleMapOutputWriter.transferMapSpillFile"),
ProblemFilters.exclude[ReversedMissingMethodProblem]("org.apache.spark.shuffle.api.SingleSpillShuffleMapOutputWriter.transferMapSpillFile"),
ProblemFilters.exclude[ReversedMissingMethodProblem]("org.apache.spark.shuffle.api.ShuffleMapOutputWriter.commitAllPartitions")
+
+ // [SPARK-36419][CORE] Optionally move final aggregation in RDD.treeAggregate to executor
+ ProblemFilters.exclude[ReversedMissingMethodProblem]("org.apache.spark.api.java.JavaRDDLike.treeAggregate")
Review comment:
No, it may be source compatible, but not binary compatible in Scala, and this isn't worth breaking an API over.
Yes you're right there is a similar consideration for Java-specific methods.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on pull request #33644: [SPARK-36419][CORE] Optionally move final aggregation in RDD.treeAggregate to executor
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on pull request #33644:
URL: https://github.com/apache/spark/pull/33644#issuecomment-901013878
We can - smems like the test failure is by linter which is unrelated. But I haven't taken a close look for the codes. TBH. I'll leave it to @mridulm
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33644: [SPARK-36419][CORE] Optionally move final aggregation in RDD.treeAggregate to executor
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33644:
URL: https://github.com/apache/spark/pull/33644#issuecomment-897209263
**[Test build #142347 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/142347/testReport)** for PR 33644 at commit [`811eb3a`](https://github.com/apache/spark/commit/811eb3a0acc62b3cf881999f24be8cd2a37998f3).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #33644: [SPARK-36419][CORE] Optionally move final aggregation in RDD.treeAggregate to executor
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #33644:
URL: https://github.com/apache/spark/pull/33644#issuecomment-898647249
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/142448/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] akpatnam25 commented on pull request #33644: [SPARK-36419][CORE] Optionally move final aggregation in RDD.treeAggregate to executor
Posted by GitBox <gi...@apache.org>.
akpatnam25 commented on pull request #33644:
URL: https://github.com/apache/spark/pull/33644#issuecomment-897069946
I've modified the code to include `finalAggregateOnExecutor` as a parameter being passed into treeAggregate. I've also updated the JavaRDDLike and the relevant unit tests. Let me know if we want any other api's to be updated as well where treeAggregate is used.
@mridulm @HyukjinKwon
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on pull request #33644: [SPARK-36419][CORE]: Optionally move final aggregation in RDD.treeAggregate to executor
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on pull request #33644:
URL: https://github.com/apache/spark/pull/33644#issuecomment-896413705
This is a new behaviour and this change won't be beneficial always given the discussion above so I would think it might be better for users to selectively choose the behaviour. But I really don't have a strong preference. I see no objection on this change so I would defer to @mridulm on how to proceed further - he has a much better insight in core than me.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33644: [SPARK-36419][CORE] Optionally move final aggregation in RDD.treeAggregate to executor
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33644:
URL: https://github.com/apache/spark/pull/33644#issuecomment-902242060
Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/47152/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #33644: [SPARK-36419][CORE] Optionally move final aggregation in RDD.treeAggregate to executor
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #33644:
URL: https://github.com/apache/spark/pull/33644#issuecomment-902262779
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/142650/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] akpatnam25 commented on a change in pull request #33644: [SPARK-36419][CORE] Optionally move final aggregation in RDD.treeAggregate to executor
Posted by GitBox <gi...@apache.org>.
akpatnam25 commented on a change in pull request #33644:
URL: https://github.com/apache/spark/pull/33644#discussion_r688253279
##########
File path: project/MimaExcludes.scala
##########
@@ -104,6 +104,10 @@ object MimaExcludes {
ProblemFilters.exclude[DirectMissingMethodProblem]("org.apache.spark.shuffle.api.SingleSpillShuffleMapOutputWriter.transferMapSpillFile"),
ProblemFilters.exclude[ReversedMissingMethodProblem]("org.apache.spark.shuffle.api.SingleSpillShuffleMapOutputWriter.transferMapSpillFile"),
ProblemFilters.exclude[ReversedMissingMethodProblem]("org.apache.spark.shuffle.api.ShuffleMapOutputWriter.commitAllPartitions")
+
+ // [SPARK-36419][CORE] Optionally move final aggregation in RDD.treeAggregate to executor
+ ProblemFilters.exclude[ReversedMissingMethodProblem]("org.apache.spark.api.java.JavaRDDLike.treeAggregate")
Review comment:
^^ do we have consensus on that? @mridulm @venkata91 @srowen
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #33644: [SPARK-36419][CORE] Optionally move final aggregation in RDD.treeAggregate to executor
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #33644:
URL: https://github.com/apache/spark/pull/33644#issuecomment-898729809
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/142451/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] mridulm commented on pull request #33644: [SPARK-36419][CORE] Optionally move final aggregation in RDD.treeAggregate to executor
Posted by GitBox <gi...@apache.org>.
mridulm commented on pull request #33644:
URL: https://github.com/apache/spark/pull/33644#issuecomment-897100468
Ok to test
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] akpatnam25 commented on pull request #33644: [SPARK-36419][CORE]: Optionally move final aggregation in RDD.treeAggregate to executor
Posted by GitBox <gi...@apache.org>.
akpatnam25 commented on pull request #33644:
URL: https://github.com/apache/spark/pull/33644#issuecomment-894694073
for example, below is a snippet of a case that we would want to safeguard against:
```
def createValue() = new Array[Byte](180 * 1024 * 1024)
def createRdd(n: Int) = sc.parallelize(0 until n, n).map(_ => createValue())
val rdd = createRdd(13)
rdd.treeAggregate(createValue())((v1: Array[Byte], v2: Array[Byte]) => v1, (v1: Array[Byte], v2: Array[Byte]) => v1, 4)
```
For the above snippet, we are generating synthetic data to maximize the partitions being pulled into the driver. This is just to illustrate an example of what "bad behavior" might look like. With optimal resources on the driver side, this succeeds, but does not succeed when the driver memory is not set high enough. The applications that have encountered this issue have obviously dealt with this by increasing driver memory, etc.
Given that, the overhead for most normal sized cases would just be the extra stage. In our above synthetic workload, there was no noticeable difference in time to complete despite the number of partitions we are pulling into the last stage. This last reduce step would be happening on the driver side anyways, so the only real overhead is the shuffle and the scheduling.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on pull request #33644: [SPARK-36419][CORE] Optionally move final aggregation in RDD.treeAggregate to executor
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on pull request #33644:
URL: https://github.com/apache/spark/pull/33644#issuecomment-904693739
thanks, merging to master!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] akpatnam25 commented on a change in pull request #33644: [SPARK-36419][CORE] Optionally move final aggregation in RDD.treeAggregate to executor
Posted by GitBox <gi...@apache.org>.
akpatnam25 commented on a change in pull request #33644:
URL: https://github.com/apache/spark/pull/33644#discussion_r690888342
##########
File path: core/src/main/scala/org/apache/spark/rdd/RDD.scala
##########
@@ -1233,6 +1233,22 @@ abstract class RDD[T: ClassTag](
(i, iter) => iter.map((i % curNumPartitions, _))
Review comment:
yes, we can add a new method with signature that looks like this:
```
def treeAggregate[U](
zeroValue: U,
seqOp: JFunction2[U, T, U],
combOp: JFunction2[U, U, U],
depth: Int,
finalAggregateOnExecutor: Boolean): U = {
...
```
you're right, we dont need to worry about Mima excludes then.
but that is not the problem. the problem is that this method will need to be overloaded by the other treeAggregate methods. for example, the other treeAggregate methods will have to call this version of treeAggregate. this is an issue because of zeroValue:U is only inspected by the compiler to resolve the overloaded methods. Due to that the compiler throws ambiguous reference to overloaded definition, and this cannot be resolved..
there are only 2 workarounds:
1. create a new method with a different name but same signature with the new parameter. something like `treeAggregateExecutor` and have somewhat duplicate code in that method
2. use config approach
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] mridulm commented on pull request #33644: [SPARK-36419][CORE]: Optionally move final aggregation in RDD.treeAggregate to executor
Posted by GitBox <gi...@apache.org>.
mridulm commented on pull request #33644:
URL: https://github.com/apache/spark/pull/33644#issuecomment-894871776
Makes sense @srowen, thanks for taking a look !
+CC @mengxr, @HyukjinKwon, @cloud-fan, @dongjoon-hyun - would be great if you can take a look (Tagging based on history).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on a change in pull request #33644: [SPARK-36419][CORE] Optionally move final aggregation in RDD.treeAggregate to executor
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on a change in pull request #33644:
URL: https://github.com/apache/spark/pull/33644#discussion_r689951539
##########
File path: core/src/main/scala/org/apache/spark/rdd/RDD.scala
##########
@@ -1233,6 +1233,22 @@ abstract class RDD[T: ClassTag](
(i, iter) => iter.map((i % curNumPartitions, _))
Review comment:
I think exposing this signature alone is fine:
```scala
def treeAggregate[U](
zeroValue: U,
seqOp: JFunction2[U, T, U],
combOp: JFunction2[U, U, U],
depth: Int,
finalAggregateOnExecutor: Boolean): U = {
...
```
To use `finalAggregateOnExecutor`, all parameters have to be fully specified. In this way, none of things will be broken.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #33644: [SPARK-36419][CORE] Optionally move final aggregation in RDD.treeAggregate to executor
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #33644:
URL: https://github.com/apache/spark/pull/33644#issuecomment-901302037
**[Test build #142621 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/142621/testReport)** for PR 33644 at commit [`d50601d`](https://github.com/apache/spark/commit/d50601d26b8347255d9890d0c8671a0e2c89ae89).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #33644: [SPARK-36419][CORE] Optionally move final aggregation in RDD.treeAggregate to executor
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #33644:
URL: https://github.com/apache/spark/pull/33644#issuecomment-902423682
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/142651/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on pull request #33644: [SPARK-36419][CORE] Optionally move final aggregation in RDD.treeAggregate to executor
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on pull request #33644:
URL: https://github.com/apache/spark/pull/33644#issuecomment-900811204
I am good with conf approach (see https://github.com/apache/spark/pull/33644#discussion_r690888342)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on a change in pull request #33644: [SPARK-36419][CORE] Optionally move final aggregation in RDD.treeAggregate to executor
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on a change in pull request #33644:
URL: https://github.com/apache/spark/pull/33644#discussion_r690836547
##########
File path: core/src/main/scala/org/apache/spark/rdd/RDD.scala
##########
@@ -1233,6 +1233,22 @@ abstract class RDD[T: ClassTag](
(i, iter) => iter.map((i % curNumPartitions, _))
Review comment:
Does this signature alone requires the changes in MiMa exclusion? This is a completely new separate signature that shouldn't affect other signatures apparently. I think the exclusion was added for other signatures you added like:
```
def treeAggregate[U](
zeroValue: U,
seqOp: JFunction2[U, T, U],
combOp: JFunction2[U, U, U],
depth: Int): U = {
...
```
?
##########
File path: core/src/main/scala/org/apache/spark/rdd/RDD.scala
##########
@@ -1233,6 +1233,22 @@ abstract class RDD[T: ClassTag](
(i, iter) => iter.map((i % curNumPartitions, _))
Review comment:
Okie, got the problem now. Configuration SGTM
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] srowen commented on a change in pull request #33644: [SPARK-36419][CORE] Optionally move final aggregation in RDD.treeAggregate to executor
Posted by GitBox <gi...@apache.org>.
srowen commented on a change in pull request #33644:
URL: https://github.com/apache/spark/pull/33644#discussion_r687289195
##########
File path: project/MimaExcludes.scala
##########
@@ -104,6 +104,10 @@ object MimaExcludes {
ProblemFilters.exclude[DirectMissingMethodProblem]("org.apache.spark.shuffle.api.SingleSpillShuffleMapOutputWriter.transferMapSpillFile"),
ProblemFilters.exclude[ReversedMissingMethodProblem]("org.apache.spark.shuffle.api.SingleSpillShuffleMapOutputWriter.transferMapSpillFile"),
ProblemFilters.exclude[ReversedMissingMethodProblem]("org.apache.spark.shuffle.api.ShuffleMapOutputWriter.commitAllPartitions")
+
+ // [SPARK-36419][CORE] Optionally move final aggregation in RDD.treeAggregate to executor
+ ProblemFilters.exclude[ReversedMissingMethodProblem]("org.apache.spark.api.java.JavaRDDLike.treeAggregate")
Review comment:
Actually, no you don't want to break this. You'll have to write the overloads manually in Scala to avoid removing the existing method
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33644: [SPARK-36419][CORE] Optionally move final aggregation in RDD.treeAggregate to executor
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33644:
URL: https://github.com/apache/spark/pull/33644#issuecomment-902149355
**[Test build #142650 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/142650/testReport)** for PR 33644 at commit [`3d49c1d`](https://github.com/apache/spark/commit/3d49c1d4a28e28856953ee5ec492e3a5249965da).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33644: [SPARK-36419][CORE] Optionally move final aggregation in RDD.treeAggregate to executor
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33644:
URL: https://github.com/apache/spark/pull/33644#issuecomment-898647198
**[Test build #142448 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/142448/testReport)** for PR 33644 at commit [`3542781`](https://github.com/apache/spark/commit/3542781e0cb377ae40406c466b7fe20d08630d96).
* This patch **fails Scala style tests**.
* This patch merges cleanly.
* This patch adds no public classes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33644: [SPARK-36419][CORE] Optionally move final aggregation in RDD.treeAggregate to executor
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33644:
URL: https://github.com/apache/spark/pull/33644#issuecomment-898702266
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/46956/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33644: [SPARK-36419][CORE] Optionally move final aggregation in RDD.treeAggregate to executor
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33644:
URL: https://github.com/apache/spark/pull/33644#issuecomment-902220325
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/47152/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] srowen commented on pull request #33644: [SPARK-36419][CORE] Optionally move final aggregation in RDD.treeAggregate to executor
Posted by GitBox <gi...@apache.org>.
srowen commented on pull request #33644:
URL: https://github.com/apache/spark/pull/33644#issuecomment-897610589
OK I find that argument convincing BTW, thanks
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33644: [SPARK-36419][CORE] Optionally move final aggregation in RDD.treeAggregate to executor
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33644:
URL: https://github.com/apache/spark/pull/33644#issuecomment-898639826
**[Test build #142445 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/142445/testReport)** for PR 33644 at commit [`722cbed`](https://github.com/apache/spark/commit/722cbede07f72644092e906530bce1213b0e1901).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] akpatnam25 removed a comment on pull request #33644: [SPARK-36419][CORE]: Optionally move final aggregation in RDD.treeAggregate to executor
Posted by GitBox <gi...@apache.org>.
akpatnam25 removed a comment on pull request #33644:
URL: https://github.com/apache/spark/pull/33644#issuecomment-894693640
for example, below is a snippet of a case that we would want to safeguard against:
```
def createValue() = new Array[Byte](180 * 1024 * 1024)
def createRdd(n: Int) = sc.parallelize(0 until n, n).map(_ => createValue())
val rdd = createRdd(13)
rdd.treeAggregate(createValue())((v1: Array[Byte], v2: Array[Byte]) => v1, (v1: Array[Byte], v2: Array[Byte]) => v1, 4)
```
For the above snippet, we are generating synthetic data to maximize the partitions being pulled into the driver. This is just to illustrate an example of what "bad behavior" might look like. With optimal resources on the driver side, this succeeds, but does not succeed when the driver memory is not set high enough. The applications that have encountered this issue have obviously dealt with this by increasing driver memory, etc.
Given that, the overhead for most normal sized cases would just be the extra stage.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] venkata91 commented on a change in pull request #33644: [SPARK-36419][CORE]: Optionally move final aggregation in RDD.treeAggregate to executor
Posted by GitBox <gi...@apache.org>.
venkata91 commented on a change in pull request #33644:
URL: https://github.com/apache/spark/pull/33644#discussion_r683848434
##########
File path: core/src/main/scala/org/apache/spark/rdd/RDD.scala
##########
@@ -1233,6 +1233,21 @@ abstract class RDD[T: ClassTag](
(i, iter) => iter.map((i % curNumPartitions, _))
}.foldByKey(zeroValue, new HashPartitioner(curNumPartitions))(cleanCombOp).values
}
+ if (conf.get(ENABLE_EXECUTOR_TREE_AGGREGATE) && partiallyAggregated.partitions.length > 1) {
+ // define a new partitioner that results in only 1 partition
+ val constantPartitioner = new Partitioner {
+ override def numPartitions: Int = 1
+
+ override def getPartition(key: Any): Int = 0
+ }
+ // map the partially aggregated rdd into a key-value rdd
+ // do the computation in the single executor with one partition
+ // get the new RDD[U]
+ partiallyAggregated = partiallyAggregated
Review comment:
We have seen internally in some cases the last fold causes OOM due to the number of values that gets folded. With this approach, the last fold would just be only 1 value.
There is a slight overhead of adding one more hop but overall this should help cases where driver memory is a bottleneck for the last fold operation.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] venkata91 commented on pull request #33644: [SPARK-36419][CORE]: Optionally move final aggregation in RDD.treeAggregate to executor
Posted by GitBox <gi...@apache.org>.
venkata91 commented on pull request #33644:
URL: https://github.com/apache/spark/pull/33644#issuecomment-893000200
cc @mridulm @xkrogen @Ngone51 to take a look.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] akpatnam25 commented on a change in pull request #33644: [SPARK-36419][CORE] Optionally move final aggregation in RDD.treeAggregate to executor
Posted by GitBox <gi...@apache.org>.
akpatnam25 commented on a change in pull request #33644:
URL: https://github.com/apache/spark/pull/33644#discussion_r688119953
##########
File path: project/MimaExcludes.scala
##########
@@ -104,6 +104,10 @@ object MimaExcludes {
ProblemFilters.exclude[DirectMissingMethodProblem]("org.apache.spark.shuffle.api.SingleSpillShuffleMapOutputWriter.transferMapSpillFile"),
ProblemFilters.exclude[ReversedMissingMethodProblem]("org.apache.spark.shuffle.api.SingleSpillShuffleMapOutputWriter.transferMapSpillFile"),
ProblemFilters.exclude[ReversedMissingMethodProblem]("org.apache.spark.shuffle.api.ShuffleMapOutputWriter.commitAllPartitions")
+
+ // [SPARK-36419][CORE] Optionally move final aggregation in RDD.treeAggregate to executor
+ ProblemFilters.exclude[ReversedMissingMethodProblem]("org.apache.spark.api.java.JavaRDDLike.treeAggregate")
Review comment:
okay, in that case i think the best option we have now is to revert back to the earlier version that used a config. not sure if making a new method is worth it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] venkata91 commented on a change in pull request #33644: [SPARK-36419][CORE]: Optionally move final aggregation in RDD.treeAggregate to executor
Posted by GitBox <gi...@apache.org>.
venkata91 commented on a change in pull request #33644:
URL: https://github.com/apache/spark/pull/33644#discussion_r683848900
##########
File path: core/src/main/scala/org/apache/spark/internal/config/package.scala
##########
@@ -315,6 +315,12 @@ package object config {
.bytesConf(ByteUnit.MiB)
.createOptional
+ private[spark] val ENABLE_EXECUTOR_TREE_AGGREGATE = ConfigBuilder("spark.executor.treeAggregate")
Review comment:
For simpler treeAggregate operations, this might add an additional overhead (not sure if it is worth it or not) if the default behavior is to spawn a spark job for the last fold operation.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] srowen commented on a change in pull request #33644: [SPARK-36419][CORE] Optionally move final aggregation in RDD.treeAggregate to executor
Posted by GitBox <gi...@apache.org>.
srowen commented on a change in pull request #33644:
URL: https://github.com/apache/spark/pull/33644#discussion_r688039722
##########
File path: project/MimaExcludes.scala
##########
@@ -104,6 +104,10 @@ object MimaExcludes {
ProblemFilters.exclude[DirectMissingMethodProblem]("org.apache.spark.shuffle.api.SingleSpillShuffleMapOutputWriter.transferMapSpillFile"),
ProblemFilters.exclude[ReversedMissingMethodProblem]("org.apache.spark.shuffle.api.SingleSpillShuffleMapOutputWriter.transferMapSpillFile"),
ProblemFilters.exclude[ReversedMissingMethodProblem]("org.apache.spark.shuffle.api.ShuffleMapOutputWriter.commitAllPartitions")
+
+ // [SPARK-36419][CORE] Optionally move final aggregation in RDD.treeAggregate to executor
+ ProblemFilters.exclude[ReversedMissingMethodProblem]("org.apache.spark.api.java.JavaRDDLike.treeAggregate")
Review comment:
You need to declare two methods, and not use optional args
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] akpatnam25 commented on a change in pull request #33644: [SPARK-36419][CORE] Optionally move final aggregation in RDD.treeAggregate to executor
Posted by GitBox <gi...@apache.org>.
akpatnam25 commented on a change in pull request #33644:
URL: https://github.com/apache/spark/pull/33644#discussion_r690532188
##########
File path: core/src/main/scala/org/apache/spark/rdd/RDD.scala
##########
@@ -1233,6 +1233,22 @@ abstract class RDD[T: ClassTag](
(i, iter) => iter.map((i % curNumPartitions, _))
Review comment:
i don't think we can do that as we did not want to cause binary incompabilities. that was the whole reason i had to revert back to the config approach. refer to this thread: https://github.com/apache/spark/pull/33644#discussion_r687122613
##########
File path: core/src/main/scala/org/apache/spark/rdd/RDD.scala
##########
@@ -1233,6 +1233,22 @@ abstract class RDD[T: ClassTag](
(i, iter) => iter.map((i % curNumPartitions, _))
Review comment:
Overloading also does not work due to the reasons given in the above thread
##########
File path: core/src/main/scala/org/apache/spark/rdd/RDD.scala
##########
@@ -1233,6 +1233,22 @@ abstract class RDD[T: ClassTag](
(i, iter) => iter.map((i % curNumPartitions, _))
Review comment:
yes, we can add a new method with signature that looks like this:
```
def treeAggregate[U](
zeroValue: U,
seqOp: JFunction2[U, T, U],
combOp: JFunction2[U, U, U],
depth: Int,
finalAggregateOnExecutor: Boolean): U = {
...
```
you're right, we dont need to worry about Mima excludes then.
but that is not the problem. the problem is that this method will need to be overloaded by the other treeAggregate methods. for example, the other treeAggregate methods will have to call this version of treeAggregate. this is an issue because of zeroValue:U is only inspected by the compiler to resolve the overloaded methods. Due to that the compiler throws ambiguous reference to overloaded definition, and this cannot be resolved..
there are only 2 workarounds:
1. create a new method with a different name but same signature with the new parameter. something like `treeAggregateExecutor` and have somewhat duplicate code in that method
2. use config approach
##########
File path: core/src/main/scala/org/apache/spark/rdd/RDD.scala
##########
@@ -1233,6 +1233,22 @@ abstract class RDD[T: ClassTag](
(i, iter) => iter.map((i % curNumPartitions, _))
Review comment:
yes, we can add a new method with signature that looks like this:
```
def treeAggregate[U](
zeroValue: U,
seqOp: JFunction2[U, T, U],
combOp: JFunction2[U, U, U],
depth: Int,
finalAggregateOnExecutor: Boolean): U = {
...
```
you're right, we dont need to worry about Mima excludes then.
but that is not the problem. the problem is that this method will need to be overloaded by the other treeAggregate methods. for example, the other treeAggregate methods will have to call this version of treeAggregate. this is an issue because of zeroValue:U is only inspected by the compiler to resolve the overloaded methods. Due to that the compiler throws ambiguous reference to overloaded definition, and this cannot be resolved..
there are only 2 workarounds:
1. create a new method with a different name but same signature with the new parameter. something like `treeAggregateExecutor` and have somewhat duplicate code in that method. me and @venkata91 thought this approach was kind of ugly
2. use config approach
##########
File path: core/src/main/scala/org/apache/spark/rdd/RDD.scala
##########
@@ -1233,6 +1233,22 @@ abstract class RDD[T: ClassTag](
(i, iter) => iter.map((i % curNumPartitions, _))
Review comment:
yes, we can add a new method with signature that looks like this:
```
def treeAggregate[U](
zeroValue: U,
seqOp: JFunction2[U, T, U],
combOp: JFunction2[U, U, U],
depth: Int,
finalAggregateOnExecutor: Boolean): U = {
...
```
you're right, we dont need to worry about Mima excludes then.
but that is not the problem. the problem is that this method will need to be overloaded by the other treeAggregate methods. for example, the other treeAggregate methods will have to call this version of treeAggregate. this is an issue because of zeroValue:U is only inspected by the compiler to resolve the overloaded methods. Due to that the compiler throws ambiguous reference to overloaded definition, and this cannot be resolved..
there are only 2 workarounds:
1. create a new method with a different name but same signature with the new parameter. something like `treeAggregateExecutor` and have somewhat duplicate code in that method. me and @venkata91 thought this approach was kind of ugly
2. use config approach which is the current implementation
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #33644: [SPARK-36419][CORE] Optionally move final aggregation in RDD.treeAggregate to executor
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #33644:
URL: https://github.com/apache/spark/pull/33644#issuecomment-898639826
**[Test build #142445 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/142445/testReport)** for PR 33644 at commit [`722cbed`](https://github.com/apache/spark/commit/722cbede07f72644092e906530bce1213b0e1901).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] akpatnam25 commented on a change in pull request #33644: [SPARK-36419][CORE] Optionally move final aggregation in RDD.treeAggregate to executor
Posted by GitBox <gi...@apache.org>.
akpatnam25 commented on a change in pull request #33644:
URL: https://github.com/apache/spark/pull/33644#discussion_r687124982
##########
File path: project/MimaExcludes.scala
##########
@@ -104,6 +104,10 @@ object MimaExcludes {
ProblemFilters.exclude[DirectMissingMethodProblem]("org.apache.spark.shuffle.api.SingleSpillShuffleMapOutputWriter.transferMapSpillFile"),
ProblemFilters.exclude[ReversedMissingMethodProblem]("org.apache.spark.shuffle.api.SingleSpillShuffleMapOutputWriter.transferMapSpillFile"),
ProblemFilters.exclude[ReversedMissingMethodProblem]("org.apache.spark.shuffle.api.ShuffleMapOutputWriter.commitAllPartitions")
+
+ // [SPARK-36419][CORE] Optionally move final aggregation in RDD.treeAggregate to executor
+ ProblemFilters.exclude[ReversedMissingMethodProblem]("org.apache.spark.api.java.JavaRDDLike.treeAggregate")
Review comment:
yes
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on a change in pull request #33644: [SPARK-36419][CORE] Optionally move final aggregation in RDD.treeAggregate to executor
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on a change in pull request #33644:
URL: https://github.com/apache/spark/pull/33644#discussion_r689951539
##########
File path: core/src/main/scala/org/apache/spark/rdd/RDD.scala
##########
@@ -1233,6 +1233,22 @@ abstract class RDD[T: ClassTag](
(i, iter) => iter.map((i % curNumPartitions, _))
Review comment:
I think this option is fine:
```scala
def treeAggregate[U](
zeroValue: U,
seqOp: JFunction2[U, T, U],
combOp: JFunction2[U, U, U],
depth: Int,
finalAggregateOnExecutor: Boolean): U = {
...
```
To use `finalAggregateOnExecutor`, all parameters have to be fully specified. In this way, none of things will be broken.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33644: [SPARK-36419][CORE] Optionally move final aggregation in RDD.treeAggregate to executor
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33644:
URL: https://github.com/apache/spark/pull/33644#issuecomment-897120340
**[Test build #142347 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/142347/testReport)** for PR 33644 at commit [`811eb3a`](https://github.com/apache/spark/commit/811eb3a0acc62b3cf881999f24be8cd2a37998f3).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] srowen commented on a change in pull request #33644: [SPARK-36419][CORE]: Optionally move final aggregation in RDD.treeAggregate to executor
Posted by GitBox <gi...@apache.org>.
srowen commented on a change in pull request #33644:
URL: https://github.com/apache/spark/pull/33644#discussion_r683803934
##########
File path: core/src/main/scala/org/apache/spark/internal/config/package.scala
##########
@@ -315,6 +315,12 @@ package object config {
.bytesConf(ByteUnit.MiB)
.createOptional
+ private[spark] val ENABLE_EXECUTOR_TREE_AGGREGATE = ConfigBuilder("spark.executor.treeAggregate")
Review comment:
I don't think I'd add another config for this - it becomes yet another of 100 things that 99% of users won't know about. I'd imagine there is a generally better or worse behavior we can agree on
##########
File path: core/src/main/scala/org/apache/spark/rdd/RDD.scala
##########
@@ -1233,6 +1233,21 @@ abstract class RDD[T: ClassTag](
(i, iter) => iter.map((i % curNumPartitions, _))
}.foldByKey(zeroValue, new HashPartitioner(curNumPartitions))(cleanCombOp).values
}
+ if (conf.get(ENABLE_EXECUTOR_TREE_AGGREGATE) && partiallyAggregated.partitions.length > 1) {
+ // define a new partitioner that results in only 1 partition
+ val constantPartitioner = new Partitioner {
+ override def numPartitions: Int = 1
+
+ override def getPartition(key: Any): Int = 0
+ }
+ // map the partially aggregated rdd into a key-value rdd
+ // do the computation in the single executor with one partition
+ // get the new RDD[U]
+ partiallyAggregated = partiallyAggregated
Review comment:
I sort of get the argument about this being more robust on the executor side, but, this does strictly more work - there is always an additional hop from the last agg to the driver now. The amount of work done is no less. I am not convinced this is generally helping, but, it is mostly a guess based on the previous sentence - it's more work
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #33644: [SPARK-36419][CORE] Optionally move final aggregation in RDD.treeAggregate to executor
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #33644:
URL: https://github.com/apache/spark/pull/33644#issuecomment-902242086
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/47152/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #33644: [SPARK-36419][CORE] Optionally move final aggregation in RDD.treeAggregate to executor
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #33644:
URL: https://github.com/apache/spark/pull/33644#issuecomment-901422581
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/142621/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] venkata91 commented on a change in pull request #33644: [SPARK-36419][CORE] Optionally move final aggregation in RDD.treeAggregate to executor
Posted by GitBox <gi...@apache.org>.
venkata91 commented on a change in pull request #33644:
URL: https://github.com/apache/spark/pull/33644#discussion_r688669489
##########
File path: project/MimaExcludes.scala
##########
@@ -104,6 +104,10 @@ object MimaExcludes {
ProblemFilters.exclude[DirectMissingMethodProblem]("org.apache.spark.shuffle.api.SingleSpillShuffleMapOutputWriter.transferMapSpillFile"),
ProblemFilters.exclude[ReversedMissingMethodProblem]("org.apache.spark.shuffle.api.SingleSpillShuffleMapOutputWriter.transferMapSpillFile"),
ProblemFilters.exclude[ReversedMissingMethodProblem]("org.apache.spark.shuffle.api.ShuffleMapOutputWriter.commitAllPartitions")
+
+ // [SPARK-36419][CORE] Optionally move final aggregation in RDD.treeAggregate to executor
+ ProblemFilters.exclude[ReversedMissingMethodProblem]("org.apache.spark.api.java.JavaRDDLike.treeAggregate")
Review comment:
Given the situation, +1 on adding this change behind a config unless there is a strong reason not to do so.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #33644: [SPARK-36419][CORE] Optionally move final aggregation in RDD.treeAggregate to executor
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #33644:
URL: https://github.com/apache/spark/pull/33644#issuecomment-902186886
**[Test build #142651 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/142651/testReport)** for PR 33644 at commit [`f728a02`](https://github.com/apache/spark/commit/f728a02407c31faf4a13665c4763c13344e4036d).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #33644: [SPARK-36419][CORE] Optionally move final aggregation in RDD.treeAggregate to executor
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #33644:
URL: https://github.com/apache/spark/pull/33644#issuecomment-892967870
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] akpatnam25 commented on a change in pull request #33644: [SPARK-36419][CORE] Optionally move final aggregation in RDD.treeAggregate to executor
Posted by GitBox <gi...@apache.org>.
akpatnam25 commented on a change in pull request #33644:
URL: https://github.com/apache/spark/pull/33644#discussion_r688081717
##########
File path: project/MimaExcludes.scala
##########
@@ -104,6 +104,10 @@ object MimaExcludes {
ProblemFilters.exclude[DirectMissingMethodProblem]("org.apache.spark.shuffle.api.SingleSpillShuffleMapOutputWriter.transferMapSpillFile"),
ProblemFilters.exclude[ReversedMissingMethodProblem]("org.apache.spark.shuffle.api.SingleSpillShuffleMapOutputWriter.transferMapSpillFile"),
ProblemFilters.exclude[ReversedMissingMethodProblem]("org.apache.spark.shuffle.api.ShuffleMapOutputWriter.commitAllPartitions")
+
+ // [SPARK-36419][CORE] Optionally move final aggregation in RDD.treeAggregate to executor
+ ProblemFilters.exclude[ReversedMissingMethodProblem]("org.apache.spark.api.java.JavaRDDLike.treeAggregate")
Review comment:
or we can revert back to the older version with configs.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33644: [SPARK-36419][CORE] Optionally move final aggregation in RDD.treeAggregate to executor
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33644:
URL: https://github.com/apache/spark/pull/33644#issuecomment-902251811
**[Test build #142650 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/142650/testReport)** for PR 33644 at commit [`3d49c1d`](https://github.com/apache/spark/commit/3d49c1d4a28e28856953ee5ec492e3a5249965da).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33644: [SPARK-36419][CORE] Optionally move final aggregation in RDD.treeAggregate to executor
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33644:
URL: https://github.com/apache/spark/pull/33644#issuecomment-898694659
Kubernetes integration test unable to build dist.
exiting with code: 1
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/46954/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] akpatnam25 commented on pull request #33644: [SPARK-36419][CORE] Optionally move final aggregation in RDD.treeAggregate to executor
Posted by GitBox <gi...@apache.org>.
akpatnam25 commented on pull request #33644:
URL: https://github.com/apache/spark/pull/33644#issuecomment-901283489
updated from latest changes from master
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] akpatnam25 commented on a change in pull request #33644: [SPARK-36419][CORE]: Optionally move final aggregation in RDD.treeAggregate to executor
Posted by GitBox <gi...@apache.org>.
akpatnam25 commented on a change in pull request #33644:
URL: https://github.com/apache/spark/pull/33644#discussion_r683031421
##########
File path: core/src/main/scala/org/apache/spark/rdd/RDD.scala
##########
@@ -1233,6 +1233,21 @@ abstract class RDD[T: ClassTag](
(i, iter) => iter.map((i % curNumPartitions, _))
}.foldByKey(zeroValue, new HashPartitioner(curNumPartitions))(cleanCombOp).values
}
+ if (conf.get(ENABLE_EXECUTOR_TREE_AGGREGATE) && partiallyAggregated.partitions.length > 1) {
+ // define a new partitioner that results in only 1 partition
+ val constantPartitioner = new Partitioner {
+ override def numPartitions: Int = 1
+
+ override def getPartition(key: Any): Int = 0
+ }
+ // map the partially aggregated rdd into a key-value rdd
+ // do the computation in the single executor with one partition
+ // get the new RDD[U]
+ partiallyAggregated = partiallyAggregated
+ .map(v => (0.toByte, v))
+ .foldByKey(zeroValue, constantPartitioner)(cleanCombOp)
+ .values
+ }
val copiedZeroValue = Utils.clone(zeroValue, sc.env.closureSerializer.newInstance())
Review comment:
no it would not submit another job, because there is only one partition. spark would do the fold internal the partition (we only have 1 in this case), and return.
As described on Apache Spark's website the fold definition states:
```
"Aggregate the elements of each partition, and then the results for all the partitions, using a given associative function and a neutral "zero value"."
```
Thus fold operates by folding first each element of partition and then the result of partition
it is partition dependent
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] akpatnam25 commented on a change in pull request #33644: [SPARK-36419][CORE] Optionally move final aggregation in RDD.treeAggregate to executor
Posted by GitBox <gi...@apache.org>.
akpatnam25 commented on a change in pull request #33644:
URL: https://github.com/apache/spark/pull/33644#discussion_r687125305
##########
File path: core/src/test/java/test/org/apache/spark/JavaAPISuite.java
##########
@@ -541,7 +541,7 @@ public void treeAggregate() {
JavaRDD<Integer> rdd = sc.parallelize(Arrays.asList(-5, -4, -3, -2, -1, 1, 2, 3, 4), 10);
Function2<Integer, Integer, Integer> add = (a, b) -> a + b;
for (int depth = 1; depth <= 10; depth++) {
- int sum = rdd.treeAggregate(0, add, add, depth);
+ int sum = rdd.treeAggregate(0, add, add, depth, false);
Review comment:
theres an overloaded version, so we have to pass in `false` or else it does not compile
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] venkata91 commented on pull request #33644: [SPARK-36419][CORE]: Optionally move final aggregation in RDD.treeAggregate to executor
Posted by GitBox <gi...@apache.org>.
venkata91 commented on pull request #33644:
URL: https://github.com/apache/spark/pull/33644#issuecomment-893000200
cc @mridulm @xkrogen @Ngone51 to take a look.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] akpatnam25 commented on a change in pull request #33644: [SPARK-36419][CORE] Optionally move final aggregation in RDD.treeAggregate to executor
Posted by GitBox <gi...@apache.org>.
akpatnam25 commented on a change in pull request #33644:
URL: https://github.com/apache/spark/pull/33644#discussion_r688029062
##########
File path: project/MimaExcludes.scala
##########
@@ -104,6 +104,10 @@ object MimaExcludes {
ProblemFilters.exclude[DirectMissingMethodProblem]("org.apache.spark.shuffle.api.SingleSpillShuffleMapOutputWriter.transferMapSpillFile"),
ProblemFilters.exclude[ReversedMissingMethodProblem]("org.apache.spark.shuffle.api.SingleSpillShuffleMapOutputWriter.transferMapSpillFile"),
ProblemFilters.exclude[ReversedMissingMethodProblem]("org.apache.spark.shuffle.api.ShuffleMapOutputWriter.commitAllPartitions")
+
+ // [SPARK-36419][CORE] Optionally move final aggregation in RDD.treeAggregate to executor
+ ProblemFilters.exclude[ReversedMissingMethodProblem]("org.apache.spark.api.java.JavaRDDLike.treeAggregate")
Review comment:
@venkata91 and I tried doing this and ran into a couple issues while trying to write the overloaded function. Seems like the issue is scala while resolving overloaded methods, it looks at the first argument and if the type is same on both the methods then that becomes ambiguous. Here is the version of the overloaded method we were trying with (along with several other variations of this :) ):
```
def treeAggregate[U: ClassTag](zeroValue: U)(
seqOp: (U, T) => U,
combOp: (U, U) => U,
depth: Int = 2): U = withScope {
treeAggregate(zeroValue)(seqOp, combOp, depth, false)
}
```
any suggestions? @srowen
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on a change in pull request #33644: [SPARK-36419][CORE] Optionally move final aggregation in RDD.treeAggregate to executor
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on a change in pull request #33644:
URL: https://github.com/apache/spark/pull/33644#discussion_r690836547
##########
File path: core/src/main/scala/org/apache/spark/rdd/RDD.scala
##########
@@ -1233,6 +1233,22 @@ abstract class RDD[T: ClassTag](
(i, iter) => iter.map((i % curNumPartitions, _))
Review comment:
Does this signature alone requires the changes in MiMa exclusion? This is a completely new separate signature that shouldn't affect other signatures apparently. I think the exclusion was added for other signatures you added like:
```
def treeAggregate[U](
zeroValue: U,
seqOp: JFunction2[U, T, U],
combOp: JFunction2[U, U, U],
depth: Int): U = {
...
```
?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] mridulm commented on pull request #33644: [SPARK-36419][CORE]: Optionally move final aggregation in RDD.treeAggregate to executor
Posted by GitBox <gi...@apache.org>.
mridulm commented on pull request #33644:
URL: https://github.com/apache/spark/pull/33644#issuecomment-894377429
@srowen That is controlled by scale - which is unfortunately not customizable (`val scale = math.max(math.ceil(math.pow(numPartitions, 1.0 / depth)).toInt, 2)`).
The potential for increased cost was the reason I guess @akpatnam25 had the config (in my impl from quite a while back, I always did the final fold at executors when partitions > 1 :-) ).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] venkata91 commented on pull request #33644: [SPARK-36419][CORE] Optionally move final aggregation in RDD.treeAggregate to executor
Posted by GitBox <gi...@apache.org>.
venkata91 commented on pull request #33644:
URL: https://github.com/apache/spark/pull/33644#issuecomment-902132796
New approach looks good to me. I think I didn't pay attention to @HyukjinKwon's suggestion earlier 😞 .
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #33644: [SPARK-36419][CORE] Optionally move final aggregation in RDD.treeAggregate to executor
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #33644:
URL: https://github.com/apache/spark/pull/33644#issuecomment-902200052
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/47151/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] akpatnam25 commented on pull request #33644: [SPARK-36419][CORE]: Optionally move final aggregation in RDD.treeAggregate to executor
Posted by GitBox <gi...@apache.org>.
akpatnam25 commented on pull request #33644:
URL: https://github.com/apache/spark/pull/33644#issuecomment-894693640
for example, below is a snippet of a case that we would want to safeguard against:
```
def createValue() = new Array[Byte](180 * 1024 * 1024)
def createRdd(n: Int) = sc.parallelize(0 until n, n).map(_ => createValue())
val rdd = createRdd(13)
rdd.treeAggregate(createValue())((v1: Array[Byte], v2: Array[Byte]) => v1, (v1: Array[Byte], v2: Array[Byte]) => v1, 4)
```
For the above snippet, we are generating synthetic data to maximize the partitions being pulled into the driver. This is just to illustrate an example of what "bad behavior" might look like. With optimal resources on the driver side, this succeeds, but does not succeed when the driver memory is not set high enough. The applications that have encountered this issue have obviously dealt with this by increasing driver memory, etc.
Given that, the overhead for most normal sized cases would just be the extra stage.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] mridulm commented on pull request #33644: [SPARK-36419][CORE]: Optionally move final aggregation in RDD.treeAggregate to executor
Posted by GitBox <gi...@apache.org>.
mridulm commented on pull request #33644:
URL: https://github.com/apache/spark/pull/33644#issuecomment-894320564
@srowen You are right that the total computation is the same (it is slightly higher actually given the additional fold with zero value at driver and ofcourse the extra shuffle).
The main issue is what @venkata91 mentioned - at executor, it is done in context of shuffle, where memory is better managed (not necessarily because executor has more memory).
At driver, it is essentially fetching all partitions (single value per partition) as part of the `fold`'s `runJob` : for nontrivial values, this causes OOM at driver, failing the application (memory usage for io + N serde in addition to the fold).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org