You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by GitBox <gi...@apache.org> on 2021/07/13 00:25:21 UTC
[GitHub] [spark] michaelzhang-db opened a new pull request #33310: [WIP][SPARK-36105][SQL] OptimizeLocalShuffleReader support reading data of multiple mappers in one task
michaelzhang-db opened a new pull request #33310:
URL: https://github.com/apache/spark/pull/33310
### What changes were proposed in this pull request?
Added another partition spec to allow OptimizeLocalShuffleReader rule to read data from multiple mappers if the parallelism is less than the number of mappers.
### Why are the changes needed?
Optimization to the OptimizeLocalShuffleReader rule
### Does this PR introduce _any_ user-facing change?
No
### How was this patch tested?
Unit tests
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] ulysses-you commented on a change in pull request #33310: [SPARK-36105][SQL] OptimizeLocalShuffleReader support reading data of multiple mappers in one task
Posted by GitBox <gi...@apache.org>.
ulysses-you commented on a change in pull request #33310:
URL: https://github.com/apache/spark/pull/33310#discussion_r673612589
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/ShuffledRowRDD.scala
##########
@@ -181,6 +187,9 @@ class ShuffledRowRDD(
case PartialMapperPartitionSpec(mapIndex, _, _) =>
tracker.getMapLocation(dependency, mapIndex, mapIndex + 1)
+
+ case CoalescedMapperPartitionSpec(startMapIndex, endMapIndex, numReducers) =>
+ tracker.getMapLocation(dependency, startMapIndex, endMapIndex)
Review comment:
I see this can reduce the partition number, but I'm not sure this approach has benefits of perf. The origin idea of `OptimizeLocalShuffleReader` is make reducer task run at the same executor of target mapper so that it can reduce some network IO.
I think order by partition size only solve the issue partially. If we want to coalesce mappers, shall we check the coalesced mappers are at the same executor or node ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #33310: [SPARK-36105][SQL] OptimizeLocalShuffleReader support reading data of multiple mappers in one task
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #33310:
URL: https://github.com/apache/spark/pull/33310#issuecomment-884529365
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/141428/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33310: [SPARK-36105][SQL] OptimizeLocalShuffleReader support reading data of multiple mappers in one task
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33310:
URL: https://github.com/apache/spark/pull/33310#issuecomment-882874758
Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/45791/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #33310: [SPARK-36105][SQL] OptimizeLocalShuffleReader support reading data of multiple mappers in one task
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on a change in pull request #33310:
URL: https://github.com/apache/spark/pull/33310#discussion_r673151153
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/ShuffledRowRDD.scala
##########
@@ -58,6 +58,12 @@ case class PartialMapperPartitionSpec(
startReducerIndex: Int,
endReducerIndex: Int) extends ShufflePartitionSpec
+// Follow-up: consider using map location for coalescing mappers
Review comment:
This is different from what I asked, but seems more reasonable. It should be better to read the entire mapper in one task if we need small parallelism, to avoid network data transfer. Except for location, we should also consider the shuffle block size.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #33310: [SPARK-36105][SQL] OptimizeLocalShuffleReader support reading data of multiple mappers in one task
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on a change in pull request #33310:
URL: https://github.com/apache/spark/pull/33310#discussion_r673952383
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/adaptive/OptimizeLocalShuffleReader.scala
##########
@@ -72,10 +72,24 @@ object OptimizeLocalShuffleReader extends CustomShuffleReaderRule {
assert(numMappers > 0)
val numReducers = shuffleStage.shuffle.numPartitions
val expectedParallelism = advisoryParallelism.getOrElse(numReducers)
- val splitPoints = equallyDivide(numReducers, math.max(1, expectedParallelism / numMappers))
- (0 until numMappers).flatMap { mapIndex =>
- (splitPoints :+ numReducers).sliding(2).map {
- case Seq(start, end) => PartialMapperPartitionSpec(mapIndex, start, end)
+ val splitPoints = if (numMappers == 0) {
Review comment:
good point!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33310: [SPARK-36105][SQL] OptimizeLocalShuffleReader support reading data of multiple mappers in one task
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33310:
URL: https://github.com/apache/spark/pull/33310#issuecomment-882852549
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/45791/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #33310: [SPARK-36105][SQL] OptimizeLocalShuffleReader support reading data of multiple mappers in one task
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #33310:
URL: https://github.com/apache/spark/pull/33310#issuecomment-883758947
**[Test build #141365 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/141365/testReport)** for PR 33310 at commit [`113f0c8`](https://github.com/apache/spark/commit/113f0c8f7490a24c4de3b90577af59c1aa840605).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #33310: [WIP][SPARK-36105][SQL] OptimizeLocalShuffleReader support reading data of multiple mappers in one task
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #33310:
URL: https://github.com/apache/spark/pull/33310#issuecomment-878685125
Can one of the admins verify this patch?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on a change in pull request #33310: [SPARK-36105][SQL] OptimizeLocalShuffleReader support reading data of multiple mappers in one task
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on a change in pull request #33310:
URL: https://github.com/apache/spark/pull/33310#discussion_r672720220
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/ShuffledRowRDD.scala
##########
@@ -58,6 +58,12 @@ case class PartialMapperPartitionSpec(
startReducerIndex: Int,
endReducerIndex: Int) extends ShufflePartitionSpec
+// Follow-up: consider using map location for coalescing mappers
Review comment:
I guess @cloud-fan wants to address https://github.com/apache/spark/pull/33310/files#r669399284 in this PR though (?). cc @maryannxue FYI too.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #33310: [WIP][SPARK-36105][SQL] OptimizeLocalShuffleReader support reading data of multiple mappers in one task
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #33310:
URL: https://github.com/apache/spark/pull/33310#issuecomment-878845915
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/45469/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #33310: [SPARK-36105][SQL] OptimizeLocalShuffleReader support reading data of multiple mappers in one task
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on a change in pull request #33310:
URL: https://github.com/apache/spark/pull/33310#discussion_r669399284
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/adaptive/OptimizeLocalShuffleReader.scala
##########
@@ -85,12 +85,24 @@ object OptimizeLocalShuffleReader extends CustomShuffleReaderRule {
val expectedParallelism = advisoryParallelism.getOrElse(numReducers)
val splitPoints = if (numMappers == 0) {
Seq.empty
- } else {
- equallyDivide(numReducers, math.max(1, expectedParallelism / numMappers))
+ } else if (expectedParallelism >= numMappers) {
+ equallyDivide(numReducers, expectedParallelism / numMappers)
+ }
+ else {
+ equallyDivide(numMappers, expectedParallelism)
+ }
+ if (expectedParallelism >= numMappers) {
+ (0 until numMappers).flatMap { mapIndex =>
+ (splitPoints :+ numReducers).sliding(2).map {
+ case Seq(start, end) => PartialMapperPartitionSpec(mapIndex, start, end)
+ }
+ }
}
- (0 until numMappers).flatMap { mapIndex =>
- (splitPoints :+ numReducers).sliding(2).map {
- case Seq(start, end) => PartialMapperPartitionSpec(mapIndex, start, end)
+ else {
+ (0 until 1).flatMap { _ =>
+ (splitPoints :+ numMappers).sliding(2).map {
+ case Seq(start, end) => CoalescedMapperPartitionSpec(start, end, numReducers)
Review comment:
I'm wondering that if we should have a more meticulous algorithm.
Let's say that there are 3 mappers and 2 reducers, so 6 shuffle blocks in total: `(M0, R0), (M0, R1), (M1, R0), (M1, R1), (M2, R0), (M2, R1)`. If the expected parallelism is 2, I think each task should read 3 blocks:
task 0: `(M0, R0), (M0, R1), (M1, R0)`
task1: `(M1, R1), (M2, R0), (M2, R1)`
So one task can read some entire mappers and part of one mapper.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on a change in pull request #33310: [SPARK-36105][SQL] OptimizeLocalShuffleReader support reading data of multiple mappers in one task
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on a change in pull request #33310:
URL: https://github.com/apache/spark/pull/33310#discussion_r672720057
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/adaptive/OptimizeLocalShuffleReader.scala
##########
@@ -72,12 +72,23 @@ object OptimizeLocalShuffleReader extends CustomShuffleReaderRule {
val expectedParallelism = advisoryParallelism.getOrElse(numReducers)
val splitPoints = if (numMappers == 0) {
Seq.empty
+ } else if (expectedParallelism >= numMappers) {
+ equallyDivide(numReducers, expectedParallelism / numMappers)
} else {
- equallyDivide(numReducers, math.max(1, expectedParallelism / numMappers))
+ equallyDivide(numMappers, expectedParallelism)
}
- (0 until numMappers).flatMap { mapIndex =>
- (splitPoints :+ numReducers).sliding(2).map {
- case Seq(start, end) => PartialMapperPartitionSpec(mapIndex, start, end)
+ if (expectedParallelism >= numMappers) {
+ (0 until numMappers).flatMap { mapIndex =>
+ (splitPoints :+ numReducers).sliding(2).map {
+ case Seq(start, end) => PartialMapperPartitionSpec(mapIndex, start, end)
+ }
+ }
+ }
+ else {
Review comment:
```suggestion
} else {
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #33310: [WIP][SPARK-36105][SQL] OptimizeLocalShuffleReader support reading data of multiple mappers in one task
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #33310:
URL: https://github.com/apache/spark/pull/33310#issuecomment-878856721
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/140951/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33310: [SPARK-36105][SQL] OptimizeLocalShuffleReader support reading data of multiple mappers in one task
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33310:
URL: https://github.com/apache/spark/pull/33310#issuecomment-882759735
Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/45781/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #33310: [WIP][SPARK-36105][SQL] OptimizeLocalShuffleReader support reading data of multiple mappers in one task
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #33310:
URL: https://github.com/apache/spark/pull/33310#issuecomment-878845915
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/45469/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #33310: [SPARK-36105][SQL] OptimizeLocalShuffleReader support reading data of multiple mappers in one task
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #33310:
URL: https://github.com/apache/spark/pull/33310#issuecomment-882777961
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/45781/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33310: [SPARK-36105][SQL] OptimizeLocalShuffleReader support reading data of multiple mappers in one task
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33310:
URL: https://github.com/apache/spark/pull/33310#issuecomment-884343180
**[Test build #141428 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/141428/testReport)** for PR 33310 at commit [`b4b6b15`](https://github.com/apache/spark/commit/b4b6b15b8841bc6e9e2452861db74440e5dcbcd2).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #33310: [SPARK-36105][SQL] OptimizeLocalShuffleReader support reading data of multiple mappers in one task
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #33310:
URL: https://github.com/apache/spark/pull/33310#issuecomment-882885003
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #33310: [WIP][SPARK-36105][SQL] OptimizeLocalShuffleReader support reading data of multiple mappers in one task
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #33310:
URL: https://github.com/apache/spark/pull/33310#issuecomment-878790553
**[Test build #140955 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/140955/testReport)** for PR 33310 at commit [`e9255b9`](https://github.com/apache/spark/commit/e9255b90cc2ce49a72f8cb147eb99de8f9988f01).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #33310: [SPARK-36105][SQL] OptimizeLocalShuffleReader support reading data of multiple mappers in one task
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #33310:
URL: https://github.com/apache/spark/pull/33310#issuecomment-879272271
Can one of the admins verify this patch?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #33310: [SPARK-36105][SQL] OptimizeLocalShuffleReader support reading data of multiple mappers in one task
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #33310:
URL: https://github.com/apache/spark/pull/33310#issuecomment-884529365
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/141428/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #33310: [SPARK-36105][SQL] OptimizeLocalShuffleReader support reading data of multiple mappers in one task
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #33310:
URL: https://github.com/apache/spark/pull/33310#issuecomment-882703043
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #33310: [SPARK-36105][SQL] OptimizeLocalShuffleReader support reading data of multiple mappers in one task
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #33310:
URL: https://github.com/apache/spark/pull/33310#issuecomment-879272271
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33310: [SPARK-36105][SQL] OptimizeLocalShuffleReader support reading data of multiple mappers in one task
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33310:
URL: https://github.com/apache/spark/pull/33310#issuecomment-882703043
**[Test build #141267 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/141267/testReport)** for PR 33310 at commit [`59dc049`](https://github.com/apache/spark/commit/59dc04954ae8c7f4b4be88530f68a8c570a651e9).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33310: [SPARK-36105][SQL] OptimizeLocalShuffleReader support reading data of multiple mappers in one task
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33310:
URL: https://github.com/apache/spark/pull/33310#issuecomment-883758947
**[Test build #141365 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/141365/testReport)** for PR 33310 at commit [`113f0c8`](https://github.com/apache/spark/commit/113f0c8f7490a24c4de3b90577af59c1aa840605).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #33310: [SPARK-36105][SQL] OptimizeLocalShuffleReader support reading data of multiple mappers in one task
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #33310:
URL: https://github.com/apache/spark/pull/33310#issuecomment-882885002
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33310: [SPARK-36105][SQL] OptimizeLocalShuffleReader support reading data of multiple mappers in one task
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33310:
URL: https://github.com/apache/spark/pull/33310#issuecomment-883909935
**[Test build #141365 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/141365/testReport)** for PR 33310 at commit [`113f0c8`](https://github.com/apache/spark/commit/113f0c8f7490a24c4de3b90577af59c1aa840605).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #33310: [SPARK-36105][SQL] OptimizeLocalShuffleReader support reading data of multiple mappers in one task
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #33310:
URL: https://github.com/apache/spark/pull/33310#issuecomment-882703043
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #33310: [SPARK-36105][SQL] OptimizeLocalShuffleReader support reading data of multiple mappers in one task
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #33310:
URL: https://github.com/apache/spark/pull/33310#issuecomment-882961111
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/141277/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #33310: [WIP][SPARK-36105][SQL] OptimizeLocalShuffleReader support reading data of multiple mappers in one task
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #33310:
URL: https://github.com/apache/spark/pull/33310#issuecomment-878969668
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/140955/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #33310: [SPARK-36105][SQL] OptimizeLocalShuffleReader support reading data of multiple mappers in one task
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on a change in pull request #33310:
URL: https://github.com/apache/spark/pull/33310#discussion_r669399284
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/adaptive/OptimizeLocalShuffleReader.scala
##########
@@ -85,12 +85,24 @@ object OptimizeLocalShuffleReader extends CustomShuffleReaderRule {
val expectedParallelism = advisoryParallelism.getOrElse(numReducers)
val splitPoints = if (numMappers == 0) {
Seq.empty
- } else {
- equallyDivide(numReducers, math.max(1, expectedParallelism / numMappers))
+ } else if (expectedParallelism >= numMappers) {
+ equallyDivide(numReducers, expectedParallelism / numMappers)
+ }
+ else {
+ equallyDivide(numMappers, expectedParallelism)
+ }
+ if (expectedParallelism >= numMappers) {
+ (0 until numMappers).flatMap { mapIndex =>
+ (splitPoints :+ numReducers).sliding(2).map {
+ case Seq(start, end) => PartialMapperPartitionSpec(mapIndex, start, end)
+ }
+ }
}
- (0 until numMappers).flatMap { mapIndex =>
- (splitPoints :+ numReducers).sliding(2).map {
- case Seq(start, end) => PartialMapperPartitionSpec(mapIndex, start, end)
+ else {
+ (0 until 1).flatMap { _ =>
+ (splitPoints :+ numMappers).sliding(2).map {
+ case Seq(start, end) => CoalescedMapperPartitionSpec(start, end, numReducers)
Review comment:
I'm wondering that if we should have a more meticulous algorithm.
Let's say that there are 3 mappers and 2 reducers, so 6 shuffle blocks in total: `(M0, R0), (M0, R1), (M1, R0), (M1, R1), (M2, R0), (M2, R1)`. If the expected parallelism is 2, I think each task should read 3 blocks:
task 0: `(M0, R0), (M0, R1), (M1, R0)`
task1: `(M1, R1), (M2, R0), (M2, R1)`
So one task can read some entire mappers and some partial mapper.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] ulysses-you commented on a change in pull request #33310: [SPARK-36105][SQL] OptimizeLocalShuffleReader support reading data of multiple mappers in one task
Posted by GitBox <gi...@apache.org>.
ulysses-you commented on a change in pull request #33310:
URL: https://github.com/apache/spark/pull/33310#discussion_r673783808
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/ShuffledRowRDD.scala
##########
@@ -181,6 +187,9 @@ class ShuffledRowRDD(
case PartialMapperPartitionSpec(mapIndex, _, _) =>
tracker.getMapLocation(dependency, mapIndex, mapIndex + 1)
+
+ case CoalescedMapperPartitionSpec(startMapIndex, endMapIndex, numReducers) =>
+ tracker.getMapLocation(dependency, startMapIndex, endMapIndex)
Review comment:
OK got it
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #33310: [SPARK-36105][SQL] OptimizeLocalShuffleReader support reading data of multiple mappers in one task
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #33310:
URL: https://github.com/apache/spark/pull/33310#issuecomment-879272271
Can one of the admins verify this patch?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #33310: [WIP][SPARK-36105][SQL] OptimizeLocalShuffleReader support reading data of multiple mappers in one task
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #33310:
URL: https://github.com/apache/spark/pull/33310#issuecomment-878749275
**[Test build #140951 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/140951/testReport)** for PR 33310 at commit [`f004267`](https://github.com/apache/spark/commit/f004267282c600e719d4a67a79618c525e27ec6c).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33310: [WIP][SPARK-36105][SQL] OptimizeLocalShuffleReader support reading data of multiple mappers in one task
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33310:
URL: https://github.com/apache/spark/pull/33310#issuecomment-878855529
**[Test build #140951 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/140951/testReport)** for PR 33310 at commit [`f004267`](https://github.com/apache/spark/commit/f004267282c600e719d4a67a79618c525e27ec6c).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33310: [SPARK-36105][SQL] OptimizeLocalShuffleReader support reading data of multiple mappers in one task
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33310:
URL: https://github.com/apache/spark/pull/33310#issuecomment-882738522
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/45781/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33310: [SPARK-36105][SQL] OptimizeLocalShuffleReader support reading data of multiple mappers in one task
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33310:
URL: https://github.com/apache/spark/pull/33310#issuecomment-882703043
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #33310: [SPARK-36105][SQL] OptimizeLocalShuffleReader support reading data of multiple mappers in one task
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on a change in pull request #33310:
URL: https://github.com/apache/spark/pull/33310#discussion_r673682713
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/ShuffledRowRDD.scala
##########
@@ -181,6 +187,9 @@ class ShuffledRowRDD(
case PartialMapperPartitionSpec(mapIndex, _, _) =>
tracker.getMapLocation(dependency, mapIndex, mapIndex + 1)
+
+ case CoalescedMapperPartitionSpec(startMapIndex, endMapIndex, numReducers) =>
+ tracker.getMapLocation(dependency, startMapIndex, endMapIndex)
Review comment:
This PR has benefits when all the mappers have very small shuffle output and it's better to coalesce them.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #33310: [SPARK-36105][SQL] OptimizeLocalShuffleReader support reading data of multiple mappers in one task
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #33310:
URL: https://github.com/apache/spark/pull/33310#issuecomment-883912480
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/141365/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #33310: [SPARK-36105][SQL] OptimizeLocalShuffleReader support reading data of multiple mappers in one task
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #33310:
URL: https://github.com/apache/spark/pull/33310#issuecomment-884343180
**[Test build #141428 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/141428/testReport)** for PR 33310 at commit [`b4b6b15`](https://github.com/apache/spark/commit/b4b6b15b8841bc6e9e2452861db74440e5dcbcd2).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] michaelzhang-db removed a comment on pull request #33310: [WIP][SPARK-36105][SQL] OptimizeLocalShuffleReader support reading data of multiple mappers in one task
Posted by GitBox <gi...@apache.org>.
michaelzhang-db removed a comment on pull request #33310:
URL: https://github.com/apache/spark/pull/33310#issuecomment-878780180
ok to test
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33310: [WIP][SPARK-36105][SQL] OptimizeLocalShuffleReader support reading data of multiple mappers in one task
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33310:
URL: https://github.com/apache/spark/pull/33310#issuecomment-878761359
Kubernetes integration test unable to build dist.
exiting with code: 1
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/45465/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] michaelzhang-db commented on pull request #33310: [SPARK-36105][SQL] OptimizeLocalShuffleReader support reading data of multiple mappers in one task
Posted by GitBox <gi...@apache.org>.
michaelzhang-db commented on pull request #33310:
URL: https://github.com/apache/spark/pull/33310#issuecomment-879214835
@cloud-fan @maryannxue Can you both quickly review this?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #33310: [SPARK-36105][SQL] OptimizeLocalShuffleReader support reading data of multiple mappers in one task
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on a change in pull request #33310:
URL: https://github.com/apache/spark/pull/33310#discussion_r673680582
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/ShuffledRowRDD.scala
##########
@@ -181,6 +187,9 @@ class ShuffledRowRDD(
case PartialMapperPartitionSpec(mapIndex, _, _) =>
tracker.getMapLocation(dependency, mapIndex, mapIndex + 1)
+
+ case CoalescedMapperPartitionSpec(startMapIndex, endMapIndex, numReducers) =>
+ tracker.getMapLocation(dependency, startMapIndex, endMapIndex)
Review comment:
There is a TODO for it: https://github.com/apache/spark/pull/33310/files#diff-9df2adce46395944a9245289a01eb85e797ec460aaef725a8108ec1054e2772aR61
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on pull request #33310: [SPARK-36105][SQL] OptimizeLocalShuffleReader support reading data of multiple mappers in one task
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on pull request #33310:
URL: https://github.com/apache/spark/pull/33310#issuecomment-886555280
thanks, merging to master/3.2 (since AQE is turned on by default in 3.2)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on a change in pull request #33310: [SPARK-36105][SQL] OptimizeLocalShuffleReader support reading data of multiple mappers in one task
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on a change in pull request #33310:
URL: https://github.com/apache/spark/pull/33310#discussion_r672302356
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/adaptive/OptimizeLocalShuffleReader.scala
##########
@@ -85,12 +85,24 @@ object OptimizeLocalShuffleReader extends CustomShuffleReaderRule {
val expectedParallelism = advisoryParallelism.getOrElse(numReducers)
val splitPoints = if (numMappers == 0) {
Seq.empty
- } else {
- equallyDivide(numReducers, math.max(1, expectedParallelism / numMappers))
+ } else if (expectedParallelism >= numMappers) {
+ equallyDivide(numReducers, expectedParallelism / numMappers)
+ }
+ else {
Review comment:
```suggestion
} else {
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33310: [SPARK-36105][SQL] OptimizeLocalShuffleReader support reading data of multiple mappers in one task
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33310:
URL: https://github.com/apache/spark/pull/33310#issuecomment-884492211
Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/45950/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33310: [SPARK-36105][SQL] OptimizeLocalShuffleReader support reading data of multiple mappers in one task
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33310:
URL: https://github.com/apache/spark/pull/33310#issuecomment-882703043
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #33310: [WIP][SPARK-36105][SQL] OptimizeLocalShuffleReader support reading data of multiple mappers in one task
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #33310:
URL: https://github.com/apache/spark/pull/33310#issuecomment-878856721
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/140951/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #33310: [SPARK-36105][SQL] OptimizeLocalShuffleReader support reading data of multiple mappers in one task
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #33310:
URL: https://github.com/apache/spark/pull/33310#issuecomment-883807581
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/45881/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #33310: [SPARK-36105][SQL] OptimizeLocalShuffleReader support reading data of multiple mappers in one task
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #33310:
URL: https://github.com/apache/spark/pull/33310#issuecomment-883912480
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/141365/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #33310: [SPARK-36105][SQL] OptimizeLocalShuffleReader support reading data of multiple mappers in one task
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #33310:
URL: https://github.com/apache/spark/pull/33310#issuecomment-882777961
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #33310: [SPARK-36105][SQL] OptimizeLocalShuffleReader support reading data of multiple mappers in one task
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #33310:
URL: https://github.com/apache/spark/pull/33310#issuecomment-882703043
**[Test build #141267 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/141267/testReport)** for PR 33310 at commit [`59dc049`](https://github.com/apache/spark/commit/59dc04954ae8c7f4b4be88530f68a8c570a651e9).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #33310: [WIP][SPARK-36105][SQL] OptimizeLocalShuffleReader support reading data of multiple mappers in one task
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #33310:
URL: https://github.com/apache/spark/pull/33310#issuecomment-878767004
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/45465/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33310: [WIP][SPARK-36105][SQL] OptimizeLocalShuffleReader support reading data of multiple mappers in one task
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33310:
URL: https://github.com/apache/spark/pull/33310#issuecomment-878830685
Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/45469/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #33310: [WIP][SPARK-36105][SQL] OptimizeLocalShuffleReader support reading data of multiple mappers in one task
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #33310:
URL: https://github.com/apache/spark/pull/33310#issuecomment-878767004
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/45465/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #33310: [SPARK-36105][SQL] OptimizeLocalShuffleReader support reading data of multiple mappers in one task
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #33310:
URL: https://github.com/apache/spark/pull/33310#issuecomment-879272271
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33310: [SPARK-36105][SQL] OptimizeLocalShuffleReader support reading data of multiple mappers in one task
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33310:
URL: https://github.com/apache/spark/pull/33310#issuecomment-882952638
**[Test build #141277 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/141277/testReport)** for PR 33310 at commit [`8c18a6f`](https://github.com/apache/spark/commit/8c18a6fb6d17c2603f07fcfdb473a20e231d0a15).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds the following public classes _(experimental)_:
* `public class ShuffleChecksumHelper `
* `class MutableCheckedOutputStream(out: OutputStream) extends OutputStream `
* `case class ShuffleChecksumBlockId(shuffleId: Int, mapId: Long, reduceId: Int) extends BlockId `
* `case class SessionWindow(timeColumn: Expression, gapDuration: Long) extends UnaryExpression`
* `case class LocalTimestamp(timeZoneId: Option[String] = None) extends LeafExpression`
* `case class ParseToTimestampLTZ(`
* `case class DomainJoin(`
* `protected abstract class ConnectionProviderBase extends Logging `
* `class MergingSortWithSessionWindowStateIterator(`
* `sealed trait StreamingSessionWindowStateManager extends Serializable `
* `class StreamingSessionWindowStateManagerImplV1(`
* `class StreamingSessionWindowHelper(sessionExpression: Attribute, inputSchema: Seq[Attribute]) `
* `case class SessionWindowStateStoreRestoreExec(`
* `case class SessionWindowStateStoreSaveExec(`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on a change in pull request #33310: [SPARK-36105][SQL] OptimizeLocalShuffleReader support reading data of multiple mappers in one task
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on a change in pull request #33310:
URL: https://github.com/apache/spark/pull/33310#discussion_r672302356
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/adaptive/OptimizeLocalShuffleReader.scala
##########
@@ -85,12 +85,24 @@ object OptimizeLocalShuffleReader extends CustomShuffleReaderRule {
val expectedParallelism = advisoryParallelism.getOrElse(numReducers)
val splitPoints = if (numMappers == 0) {
Seq.empty
- } else {
- equallyDivide(numReducers, math.max(1, expectedParallelism / numMappers))
+ } else if (expectedParallelism >= numMappers) {
+ equallyDivide(numReducers, expectedParallelism / numMappers)
+ }
+ else {
Review comment:
```suggestion
} else {
```
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/ShuffledRowRDD.scala
##########
@@ -58,6 +58,12 @@ case class PartialMapperPartitionSpec(
startReducerIndex: Int,
endReducerIndex: Int) extends ShufflePartitionSpec
+// Follow-up: consider using map location for coalescing mappers
Review comment:
If we plan to leave it as a followup, we should file a JIRA, and fix the comment as, for example, `// TODO(SPARK-XXXXX): blah blah blah`
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/adaptive/OptimizeLocalShuffleReader.scala
##########
@@ -72,12 +72,23 @@ object OptimizeLocalShuffleReader extends CustomShuffleReaderRule {
val expectedParallelism = advisoryParallelism.getOrElse(numReducers)
val splitPoints = if (numMappers == 0) {
Seq.empty
+ } else if (expectedParallelism >= numMappers) {
+ equallyDivide(numReducers, expectedParallelism / numMappers)
} else {
- equallyDivide(numReducers, math.max(1, expectedParallelism / numMappers))
+ equallyDivide(numMappers, expectedParallelism)
}
- (0 until numMappers).flatMap { mapIndex =>
- (splitPoints :+ numReducers).sliding(2).map {
- case Seq(start, end) => PartialMapperPartitionSpec(mapIndex, start, end)
+ if (expectedParallelism >= numMappers) {
+ (0 until numMappers).flatMap { mapIndex =>
+ (splitPoints :+ numReducers).sliding(2).map {
+ case Seq(start, end) => PartialMapperPartitionSpec(mapIndex, start, end)
+ }
+ }
+ }
+ else {
Review comment:
```suggestion
} else {
```
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/ShuffledRowRDD.scala
##########
@@ -58,6 +58,12 @@ case class PartialMapperPartitionSpec(
startReducerIndex: Int,
endReducerIndex: Int) extends ShufflePartitionSpec
+// Follow-up: consider using map location for coalescing mappers
Review comment:
I guess @cloud-fan wants to address https://github.com/apache/spark/pull/33310/files#r669399284 in this PR though (?). cc @maryannxue FYI too.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan closed pull request #33310: [SPARK-36105][SQL] OptimizeLocalShuffleReader support reading data of multiple mappers in one task
Posted by GitBox <gi...@apache.org>.
cloud-fan closed pull request #33310:
URL: https://github.com/apache/spark/pull/33310
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] ulysses-you commented on a change in pull request #33310: [SPARK-36105][SQL] OptimizeLocalShuffleReader support reading data of multiple mappers in one task
Posted by GitBox <gi...@apache.org>.
ulysses-you commented on a change in pull request #33310:
URL: https://github.com/apache/spark/pull/33310#discussion_r673784952
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/adaptive/OptimizeLocalShuffleReader.scala
##########
@@ -72,10 +72,24 @@ object OptimizeLocalShuffleReader extends CustomShuffleReaderRule {
assert(numMappers > 0)
val numReducers = shuffleStage.shuffle.numPartitions
val expectedParallelism = advisoryParallelism.getOrElse(numReducers)
- val splitPoints = equallyDivide(numReducers, math.max(1, expectedParallelism / numMappers))
- (0 until numMappers).flatMap { mapIndex =>
- (splitPoints :+ numReducers).sliding(2).map {
- case Seq(start, end) => PartialMapperPartitionSpec(mapIndex, start, end)
+ val splitPoints = if (numMappers == 0) {
Review comment:
we don't need this check anymore, see line 72.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33310: [SPARK-36105][SQL] OptimizeLocalShuffleReader support reading data of multiple mappers in one task
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33310:
URL: https://github.com/apache/spark/pull/33310#issuecomment-882823807
**[Test build #141277 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/141277/testReport)** for PR 33310 at commit [`8c18a6f`](https://github.com/apache/spark/commit/8c18a6fb6d17c2603f07fcfdb473a20e231d0a15).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33310: [WIP][SPARK-36105][SQL] OptimizeLocalShuffleReader support reading data of multiple mappers in one task
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33310:
URL: https://github.com/apache/spark/pull/33310#issuecomment-878790553
**[Test build #140955 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/140955/testReport)** for PR 33310 at commit [`e9255b9`](https://github.com/apache/spark/commit/e9255b90cc2ce49a72f8cb147eb99de8f9988f01).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33310: [WIP][SPARK-36105][SQL] OptimizeLocalShuffleReader support reading data of multiple mappers in one task
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33310:
URL: https://github.com/apache/spark/pull/33310#issuecomment-878940939
**[Test build #140955 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/140955/testReport)** for PR 33310 at commit [`e9255b9`](https://github.com/apache/spark/commit/e9255b90cc2ce49a72f8cb147eb99de8f9988f01).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #33310: [SPARK-36105][SQL] OptimizeLocalShuffleReader support reading data of multiple mappers in one task
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #33310:
URL: https://github.com/apache/spark/pull/33310#issuecomment-882823807
**[Test build #141277 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/141277/testReport)** for PR 33310 at commit [`8c18a6f`](https://github.com/apache/spark/commit/8c18a6fb6d17c2603f07fcfdb473a20e231d0a15).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33310: [SPARK-36105][SQL] OptimizeLocalShuffleReader support reading data of multiple mappers in one task
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33310:
URL: https://github.com/apache/spark/pull/33310#issuecomment-882864115
**[Test build #141267 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/141267/testReport)** for PR 33310 at commit [`59dc049`](https://github.com/apache/spark/commit/59dc04954ae8c7f4b4be88530f68a8c570a651e9).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #33310: [SPARK-36105][SQL] OptimizeLocalShuffleReader support reading data of multiple mappers in one task
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #33310:
URL: https://github.com/apache/spark/pull/33310#issuecomment-882961111
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/141277/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33310: [SPARK-36105][SQL] OptimizeLocalShuffleReader support reading data of multiple mappers in one task
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33310:
URL: https://github.com/apache/spark/pull/33310#issuecomment-883791422
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/45881/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #33310: [SPARK-36105][SQL] OptimizeLocalShuffleReader support reading data of multiple mappers in one task
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #33310:
URL: https://github.com/apache/spark/pull/33310#issuecomment-882777961
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/45781/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on a change in pull request #33310: [WIP][SPARK-36105][SQL] OptimizeLocalShuffleReader support reading data of multiple mappers in one task
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on a change in pull request #33310:
URL: https://github.com/apache/spark/pull/33310#discussion_r668394926
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/ShuffledRowRDD.scala
##########
@@ -58,6 +58,11 @@ case class PartialMapperPartitionSpec(
startReducerIndex: Int,
endReducerIndex: Int) extends ShufflePartitionSpec
+case class CoalescedMapperPartitionSpec(
+ startMapIndex: Int,
+ endMapIndex: Int,
+ numReducers: Int) extends ShufflePartitionSpec
Review comment:
```suggestion
startMapIndex: Int,
endMapIndex: Int,
numReducers: Int) extends ShufflePartitionSpec
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33310: [SPARK-36105][SQL] OptimizeLocalShuffleReader support reading data of multiple mappers in one task
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33310:
URL: https://github.com/apache/spark/pull/33310#issuecomment-883807367
Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/45881/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #33310: [SPARK-36105][SQL] OptimizeLocalShuffleReader support reading data of multiple mappers in one task
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #33310:
URL: https://github.com/apache/spark/pull/33310#issuecomment-884495195
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/45950/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #33310: [SPARK-36105][SQL] OptimizeLocalShuffleReader support reading data of multiple mappers in one task
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #33310:
URL: https://github.com/apache/spark/pull/33310#issuecomment-884495195
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/45950/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #33310: [WIP][SPARK-36105][SQL] OptimizeLocalShuffleReader support reading data of multiple mappers in one task
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #33310:
URL: https://github.com/apache/spark/pull/33310#issuecomment-878789756
Can one of the admins verify this patch?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #33310: [WIP][SPARK-36105][SQL] OptimizeLocalShuffleReader support reading data of multiple mappers in one task
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #33310:
URL: https://github.com/apache/spark/pull/33310#issuecomment-878969668
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/140955/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #33310: [SPARK-36105][SQL] OptimizeLocalShuffleReader support reading data of multiple mappers in one task
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #33310:
URL: https://github.com/apache/spark/pull/33310#issuecomment-882777961
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on a change in pull request #33310: [SPARK-36105][SQL] OptimizeLocalShuffleReader support reading data of multiple mappers in one task
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on a change in pull request #33310:
URL: https://github.com/apache/spark/pull/33310#discussion_r672719894
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/ShuffledRowRDD.scala
##########
@@ -58,6 +58,12 @@ case class PartialMapperPartitionSpec(
startReducerIndex: Int,
endReducerIndex: Int) extends ShufflePartitionSpec
+// Follow-up: consider using map location for coalescing mappers
Review comment:
If we plan to leave it as a followup, we should file a JIRA, and fix the comment as, for example, `// TODO(SPARK-XXXXX): blah blah blah`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on pull request #33310: [WIP][SPARK-36105][SQL] OptimizeLocalShuffleReader support reading data of multiple mappers in one task
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on pull request #33310:
URL: https://github.com/apache/spark/pull/33310#issuecomment-878743794
ok to test
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #33310: [WIP][SPARK-36105][SQL] OptimizeLocalShuffleReader support reading data of multiple mappers in one task
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #33310:
URL: https://github.com/apache/spark/pull/33310#issuecomment-878789756
Can one of the admins verify this patch?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33310: [WIP][SPARK-36105][SQL] OptimizeLocalShuffleReader support reading data of multiple mappers in one task
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33310:
URL: https://github.com/apache/spark/pull/33310#issuecomment-878749275
**[Test build #140951 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/140951/testReport)** for PR 33310 at commit [`f004267`](https://github.com/apache/spark/commit/f004267282c600e719d4a67a79618c525e27ec6c).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33310: [SPARK-36105][SQL] OptimizeLocalShuffleReader support reading data of multiple mappers in one task
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33310:
URL: https://github.com/apache/spark/pull/33310#issuecomment-884464269
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/45950/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #33310: [SPARK-36105][SQL] OptimizeLocalShuffleReader support reading data of multiple mappers in one task
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #33310:
URL: https://github.com/apache/spark/pull/33310#issuecomment-883807581
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/45881/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33310: [SPARK-36105][SQL] OptimizeLocalShuffleReader support reading data of multiple mappers in one task
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33310:
URL: https://github.com/apache/spark/pull/33310#issuecomment-884528107
**[Test build #141428 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/141428/testReport)** for PR 33310 at commit [`b4b6b15`](https://github.com/apache/spark/commit/b4b6b15b8841bc6e9e2452861db74440e5dcbcd2).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #33310: [WIP][SPARK-36105][SQL] OptimizeLocalShuffleReader support reading data of multiple mappers in one task
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #33310:
URL: https://github.com/apache/spark/pull/33310#issuecomment-878685125
Can one of the admins verify this patch?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on a change in pull request #33310: [SPARK-36105][SQL] OptimizeLocalShuffleReader support reading data of multiple mappers in one task
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on a change in pull request #33310:
URL: https://github.com/apache/spark/pull/33310#discussion_r672302356
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/adaptive/OptimizeLocalShuffleReader.scala
##########
@@ -85,12 +85,24 @@ object OptimizeLocalShuffleReader extends CustomShuffleReaderRule {
val expectedParallelism = advisoryParallelism.getOrElse(numReducers)
val splitPoints = if (numMappers == 0) {
Seq.empty
- } else {
- equallyDivide(numReducers, math.max(1, expectedParallelism / numMappers))
+ } else if (expectedParallelism >= numMappers) {
+ equallyDivide(numReducers, expectedParallelism / numMappers)
+ }
+ else {
Review comment:
```suggestion
} else {
```
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/ShuffledRowRDD.scala
##########
@@ -58,6 +58,12 @@ case class PartialMapperPartitionSpec(
startReducerIndex: Int,
endReducerIndex: Int) extends ShufflePartitionSpec
+// Follow-up: consider using map location for coalescing mappers
Review comment:
If we plan to leave it as a followup, we should file a JIRA, and fix the comment as, for example, `// TODO(SPARK-XXXXX): blah blah blah`
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/adaptive/OptimizeLocalShuffleReader.scala
##########
@@ -72,12 +72,23 @@ object OptimizeLocalShuffleReader extends CustomShuffleReaderRule {
val expectedParallelism = advisoryParallelism.getOrElse(numReducers)
val splitPoints = if (numMappers == 0) {
Seq.empty
+ } else if (expectedParallelism >= numMappers) {
+ equallyDivide(numReducers, expectedParallelism / numMappers)
} else {
- equallyDivide(numReducers, math.max(1, expectedParallelism / numMappers))
+ equallyDivide(numMappers, expectedParallelism)
}
- (0 until numMappers).flatMap { mapIndex =>
- (splitPoints :+ numReducers).sliding(2).map {
- case Seq(start, end) => PartialMapperPartitionSpec(mapIndex, start, end)
+ if (expectedParallelism >= numMappers) {
+ (0 until numMappers).flatMap { mapIndex =>
+ (splitPoints :+ numReducers).sliding(2).map {
+ case Seq(start, end) => PartialMapperPartitionSpec(mapIndex, start, end)
+ }
+ }
+ }
+ else {
Review comment:
```suggestion
} else {
```
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/ShuffledRowRDD.scala
##########
@@ -58,6 +58,12 @@ case class PartialMapperPartitionSpec(
startReducerIndex: Int,
endReducerIndex: Int) extends ShufflePartitionSpec
+// Follow-up: consider using map location for coalescing mappers
Review comment:
I guess @cloud-fan wants to address https://github.com/apache/spark/pull/33310/files#r669399284 in this PR though (?). cc @maryannxue FYI too.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #33310: [SPARK-36105][SQL] OptimizeLocalShuffleReader support reading data of multiple mappers in one task
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on a change in pull request #33310:
URL: https://github.com/apache/spark/pull/33310#discussion_r673149116
##########
File path: sql/core/src/test/scala/org/apache/spark/sql/execution/adaptive/AdaptiveQueryExecSuite.scala
##########
@@ -203,7 +203,7 @@ class AdaptiveQueryExecSuite
// the final parallelism is
// math.max(1, advisoryParallelism / numMappers): math.max(1, 1/2) = 1
// and the partitions length is 1 * numMappers = 2
Review comment:
can we update the code comment as well?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33310: [WIP][SPARK-36105][SQL] OptimizeLocalShuffleReader support reading data of multiple mappers in one task
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33310:
URL: https://github.com/apache/spark/pull/33310#issuecomment-878812570
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/45469/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] michaelzhang-db commented on pull request #33310: [WIP][SPARK-36105][SQL] OptimizeLocalShuffleReader support reading data of multiple mappers in one task
Posted by GitBox <gi...@apache.org>.
michaelzhang-db commented on pull request #33310:
URL: https://github.com/apache/spark/pull/33310#issuecomment-878780180
ok to test
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on a change in pull request #33310: [SPARK-36105][SQL] OptimizeLocalShuffleReader support reading data of multiple mappers in one task
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on a change in pull request #33310:
URL: https://github.com/apache/spark/pull/33310#discussion_r673595867
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/ShuffledRowRDD.scala
##########
@@ -58,6 +58,12 @@ case class PartialMapperPartitionSpec(
startReducerIndex: Int,
endReducerIndex: Int) extends ShufflePartitionSpec
+// Follow-up: consider using map location for coalescing mappers
Review comment:
👌
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org