You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by GitBox <gi...@apache.org> on 2022/01/25 23:54:32 UTC
[GitHub] [spark] otterc opened a new pull request #35325: [WIP][SPARK-37675][SPARK-37793] Prevent overwriting of push shuffle merged files once the shuffle is finalized
otterc opened a new pull request #35325:
URL: https://github.com/apache/spark/pull/35325
### What changes were proposed in this pull request?
This is fixes the bugs that were reported in SPARK-37675 and SPARK-37793.
- Empty merged partitions were reported by the shuffle server to the driver.
- The push merged files were getting overwritten after a shuffle merge is finalized.
It is WIP because we need to validate that the fix will not result in increased memory on the shuffle server.
### Why are the changes needed?
Changes are need to fix the bug.
### Does this PR introduce _any_ user-facing change?
No
### How was this patch tested?
Have added unit test.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] pan3793 commented on pull request #35325: [SPARK-37675][SPARK-37793] Prevent overwriting of push shuffle merged files once the shuffle is finalized
Posted by GitBox <gi...@apache.org>.
pan3793 commented on pull request #35325:
URL: https://github.com/apache/spark/pull/35325#issuecomment-1022959475
Thanks @otterc and @mridulm, I verified this patch with 2 rounds of 1TB TPC-DS query, the issue is gone.
One minor question, can we suppress this warning message? I think it's a little bit noisy for users.
```
01-27 16:18:35 WARN [shuffle-client-8-5] scheduler.DAGScheduler@94: Exception encountered when trying to finalize shuffle merge on beta-spark2 for shuffle 189
java.lang.RuntimeException: java.lang.RuntimeException: Shuffle merge finalize request for shuffle 189 with shuffleMergeId 0 is stale shuffle finalize request as shuffle blocks of a higher shuffleMergeId for the shuffle is already being pushed
at org.apache.spark.network.shuffle.RemoteBlockPushResolver.lambda$finalizeShuffleMerge$11(RemoteBlockPushResolver.java:523)
at java.util.concurrent.ConcurrentHashMap.compute(ConcurrentHashMap.java:1853)
at org.apache.spark.network.shuffle.RemoteBlockPushResolver.finalizeShuffleMerge(RemoteBlockPushResolver.java:519)
at org.apache.spark.network.shuffle.ExternalBlockHandler.handleMessage(ExternalBlockHandler.java:218)
at org.apache.spark.network.shuffle.ExternalBlockHandler.receive(ExternalBlockHandler.java:122)
at org.apache.spark.network.server.TransportRequestHandler.processRpcRequest(TransportRequestHandler.java:163)
at org.apache.spark.network.server.TransportRequestHandler.handle(TransportRequestHandler.java:109)
at org.apache.spark.network.server.TransportChannelHandler.channelRead0(TransportChannelHandler.java:140)
at org.apache.spark.network.server.TransportChannelHandler.channelRead0(TransportChannelHandler.java:53)
at org.sparkproject.io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99)
at org.sparkproject.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
at org.sparkproject.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
at org.sparkproject.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
at org.sparkproject.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286)
at org.sparkproject.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
at org.sparkproject.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
at org.sparkproject.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
at org.sparkproject.io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103)
at org.sparkproject.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
at org.sparkproject.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
at org.sparkproject.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
at org.apache.spark.network.util.TransportFrameDecoder.channelRead(TransportFrameDecoder.java:102)
at org.sparkproject.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
at org.sparkproject.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
at org.sparkproject.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
at org.sparkproject.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
at org.sparkproject.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
at org.sparkproject.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
at org.sparkproject.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
at org.sparkproject.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166)
at org.sparkproject.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:722)
at org.sparkproject.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:658)
at org.sparkproject.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:584)
at org.sparkproject.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:496)
at org.sparkproject.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:986)
at org.sparkproject.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at org.sparkproject.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
at java.lang.Thread.run(Thread.java:748)
at org.apache.spark.network.client.TransportResponseHandler.handle(TransportResponseHandler.java:209) [spark-network-common_2.12-3.3.0-SNAPSHOT.jar:3.3.0-SNAPSHOT]
at org.apache.spark.network.server.TransportChannelHandler.channelRead0(TransportChannelHandler.java:142) [spark-network-common_2.12-3.3.0-SNAPSHOT.jar:3.3.0-SNAPSHOT]
at org.apache.spark.network.server.TransportChannelHandler.channelRead0(TransportChannelHandler.java:53) [spark-network-common_2.12-3.3.0-SNAPSHOT.jar:3.3.0-SNAPSHOT]
at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) [netty-transport-4.1.72.Final.jar:4.1.72.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.72.Final.jar:4.1.72.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.72.Final.jar:4.1.72.Final]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) [netty-transport-4.1.72.Final.jar:4.1.72.Final]
at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) [netty-handler-4.1.73.Final.jar:4.1.73.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.72.Final.jar:4.1.72.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.72.Final.jar:4.1.72.Final]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) [netty-transport-4.1.72.Final.jar:4.1.72.Final]
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103) [netty-codec-4.1.72.Final.jar:4.1.72.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.72.Final.jar:4.1.72.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.72.Final.jar:4.1.72.Final]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) [netty-transport-4.1.72.Final.jar:4.1.72.Final]
at org.apache.spark.network.util.TransportFrameDecoder.channelRead(TransportFrameDecoder.java:102) [spark-network-common_2.12-3.3.0-SNAPSHOT.jar:3.3.0-SNAPSHOT]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.72.Final.jar:4.1.72.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.72.Final.jar:4.1.72.Final]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) [netty-transport-4.1.72.Final.jar:4.1.72.Final]
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) [netty-transport-4.1.72.Final.jar:4.1.72.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.72.Final.jar:4.1.72.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.72.Final.jar:4.1.72.Final]
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) [netty-transport-4.1.72.Final.jar:4.1.72.Final]
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) [netty-transport-4.1.72.Final.jar:4.1.72.Final]
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:722) [netty-transport-4.1.72.Final.jar:4.1.72.Final]
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:658) [netty-transport-4.1.72.Final.jar:4.1.72.Final]
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:584) [netty-transport-4.1.72.Final.jar:4.1.72.Final]
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:496) [netty-transport-4.1.72.Final.jar:4.1.72.Final]
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:986) [netty-common-4.1.73.Final.jar:4.1.73.Final]
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) [netty-common-4.1.73.Final.jar:4.1.73.Final]
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) [netty-common-4.1.73.Final.jar:4.1.73.Final]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_312]
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] otterc commented on pull request #35325: [SPARK-37675][SPARK-37793] Prevent overwriting of push shuffle merged files once the shuffle is finalized
Posted by GitBox <gi...@apache.org>.
otterc commented on pull request #35325:
URL: https://github.com/apache/spark/pull/35325#issuecomment-1023594956
- The test failures seem to be unrelated to this PR
- @mridulm @pan3793 I have updated the description of SPARK-37675
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] mridulm edited a comment on pull request #35325: [WIP][SPARK-37675][SPARK-37793] Prevent overwriting of push shuffle merged files once the shuffle is finalized
Posted by GitBox <gi...@apache.org>.
mridulm edited a comment on pull request #35325:
URL: https://github.com/apache/spark/pull/35325#issuecomment-1022312145
@pan3793 Functionally, @otterc's patch looks good (except for testing enhancements, renames, etc).
Can you also confirm if this works for you, given that your env has been great with producing this issue consistently ? Thanks !
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] mridulm edited a comment on pull request #35325: [SPARK-37675][SPARK-37793] Prevent overwriting of push shuffle merged files once the shuffle is finalized
Posted by GitBox <gi...@apache.org>.
mridulm edited a comment on pull request #35325:
URL: https://github.com/apache/spark/pull/35325#issuecomment-1024818604
@otterc Can you take a look at the test failure ? It is for `YarnShuffleIntegrationSuite`: external shuffle service
Can you retrigger the tests in case it is not related ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] mridulm commented on pull request #35325: [SPARK-37675][SPARK-37793] Prevent overwriting of push shuffle merged files once the shuffle is finalized
Posted by GitBox <gi...@apache.org>.
mridulm commented on pull request #35325:
URL: https://github.com/apache/spark/pull/35325#issuecomment-1024820506
Merged to master, branch-3.2
Thanks for working on this @otterc
Thanks for reporting and helping test this @pan3793 !
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] mridulm commented on pull request #35325: [WIP][SPARK-37675][SPARK-37793] Prevent overwriting of push shuffle merged files once the shuffle is finalized
Posted by GitBox <gi...@apache.org>.
mridulm commented on pull request #35325:
URL: https://github.com/apache/spark/pull/35325#issuecomment-1022312145
@pan3793 Functionally, @otterc's patch looks good (except for testing enhancements, renames, etc).
Can you also confirm if this works for you ? Thanks !
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] otterc commented on a change in pull request #35325: [WIP][SPARK-37675][SPARK-37793] Prevent overwriting of push shuffle merged files once the shuffle is finalized
Posted by GitBox <gi...@apache.org>.
otterc commented on a change in pull request #35325:
URL: https://github.com/apache/spark/pull/35325#discussion_r792982698
##########
File path: common/network-shuffle/src/test/java/org/apache/spark/network/shuffle/RemoteBlockPushResolverSuite.java
##########
@@ -1287,6 +1287,52 @@ void closeAndDeletePartitionFiles(Map<Integer, AppShufflePartitionInfo> partitio
+ " up", appShuffleInfo.getMergedShuffleDataFile(0, 4, 0).exists());
}
+ // Test for SPARK-37675 and SPARK-37793
+ @Test
+ public void testAllBlocksAreRejectedWhenReceivedAfterFinalization() throws IOException {
Review comment:
Added the test.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] pan3793 commented on pull request #35325: [WIP][SPARK-37675][SPARK-37793] Prevent overwriting of push shuffle merged files once the shuffle is finalized
Posted by GitBox <gi...@apache.org>.
pan3793 commented on pull request #35325:
URL: https://github.com/apache/spark/pull/35325#issuecomment-1021889173
I tested your first commit 388559f, it does not solve the problem.

--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] pan3793 edited a comment on pull request #35325: [WIP][SPARK-37675][SPARK-37793] Prevent overwriting of push shuffle merged files once the shuffle is finalized
Posted by GitBox <gi...@apache.org>.
pan3793 edited a comment on pull request #35325:
URL: https://github.com/apache/spark/pull/35325#issuecomment-1021889173
~~I tested your first commit 388559f, it does not solve the problem.~~

--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] asfgit closed pull request #35325: [SPARK-37675][SPARK-37793] Prevent overwriting of push shuffle merged files once the shuffle is finalized
Posted by GitBox <gi...@apache.org>.
asfgit closed pull request #35325:
URL: https://github.com/apache/spark/pull/35325
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] mridulm commented on a change in pull request #35325: [WIP][SPARK-37675][SPARK-37793] Prevent overwriting of push shuffle merged files once the shuffle is finalized
Posted by GitBox <gi...@apache.org>.
mridulm commented on a change in pull request #35325:
URL: https://github.com/apache/spark/pull/35325#discussion_r793202348
##########
File path: common/network-shuffle/src/main/java/org/apache/spark/network/shuffle/RemoteBlockPushResolver.java
##########
@@ -534,35 +515,33 @@ public MergeStatuses finalizeShuffleMerge(FinalizeShuffleMerge msg) {
}
AtomicReference<Map<Integer, AppShufflePartitionInfo>> shuffleMergePartitionsRef =
new AtomicReference<>(null);
- // Metadata of the determinate stage shuffle can be safely removed as part of finalizing
- // shuffle merge. Currently once the shuffle is finalized for a determinate stages, retry
- // stages of the same shuffle will have shuffle push disabled.
- if (msg.shuffleMergeId == DETERMINATE_SHUFFLE_MERGE_ID) {
- AppShuffleMergePartitionsInfo appShuffleMergePartitionsInfo =
- appShuffleInfo.shuffles.remove(msg.shuffleId);
- if (appShuffleMergePartitionsInfo != null) {
- shuffleMergePartitionsRef.set(appShuffleMergePartitionsInfo.shuffleMergePartitions);
+ appShuffleInfo.shuffles.compute(msg.shuffleId, (shuffleId, mergePartitionsInfo) -> {
+ if (null == mergePartitionsInfo) {
+ //If the mergePartitions was never created then it means that there weren't any push
+ //blocks that were ever received for this shuffle. This could be the case when the driver
+ //doesn't wait for enough time to start the stage which reads this shuffle data.
Review comment:
nit: space between `//` and text.
##########
File path: common/network-shuffle/src/main/java/org/apache/spark/network/shuffle/RemoteBlockPushResolver.java
##########
@@ -534,35 +515,33 @@ public MergeStatuses finalizeShuffleMerge(FinalizeShuffleMerge msg) {
}
AtomicReference<Map<Integer, AppShufflePartitionInfo>> shuffleMergePartitionsRef =
new AtomicReference<>(null);
- // Metadata of the determinate stage shuffle can be safely removed as part of finalizing
- // shuffle merge. Currently once the shuffle is finalized for a determinate stages, retry
- // stages of the same shuffle will have shuffle push disabled.
- if (msg.shuffleMergeId == DETERMINATE_SHUFFLE_MERGE_ID) {
- AppShuffleMergePartitionsInfo appShuffleMergePartitionsInfo =
- appShuffleInfo.shuffles.remove(msg.shuffleId);
- if (appShuffleMergePartitionsInfo != null) {
- shuffleMergePartitionsRef.set(appShuffleMergePartitionsInfo.shuffleMergePartitions);
+ appShuffleInfo.shuffles.compute(msg.shuffleId, (shuffleId, mergePartitionsInfo) -> {
+ if (null == mergePartitionsInfo) {
+ //If the mergePartitions was never created then it means that there weren't any push
+ //blocks that were ever received for this shuffle. This could be the case when the driver
+ //doesn't wait for enough time to start the stage which reads this shuffle data.
+ return new AppShuffleMergePartitionsInfo(msg.shuffleMergeId, true);
+ } else if (msg.shuffleMergeId < mergePartitionsInfo.shuffleMergeId
+ || mergePartitionsInfo.isFinalized()) {
+ throw new RuntimeException(
+ String.format("Shuffle merge finalize request for shuffle %s with" +
+ " shuffleMergeId %s is %s", msg.shuffleId, msg.shuffleMergeId,
+ ErrorHandler.BlockPushErrorHandler.STALE_SHUFFLE_FINALIZE_SUFFIX));
+ } else if (msg.shuffleMergeId > mergePartitionsInfo.shuffleMergeId) {
+ // If no blocks pushed for the finalizeShuffleMerge shuffleMergeId then return
+ // empty MergeStatuses but cleanup the older shuffleMergeId files.
+ mergedShuffleCleaner.execute(() ->
+ closeAndDeletePartitionFiles(mergePartitionsInfo.shuffleMergePartitions));
+ return new AppShuffleMergePartitionsInfo(msg.shuffleMergeId, true);
+ } else {
+ // This block covers:
+ // 1. finalization of determinate stage
+ // 2. finalization of indeterminate stage if the shuffleMergeId related to it is the one
+ // for which the message is received.
+ shuffleMergePartitionsRef.set(mergePartitionsInfo.shuffleMergePartitions);
+ return new AppShuffleMergePartitionsInfo(msg.shuffleMergeId, true);
Review comment:
nit: All paths in this lambda, which are not throwing exceptions, are doing the same thing - `new AppShuffleMergePartitionsInfo(msg.shuffleMergeId, true);`
Make that common ?
##########
File path: common/network-shuffle/src/main/java/org/apache/spark/network/shuffle/RemoteBlockPushResolver.java
##########
@@ -576,14 +555,25 @@ public MergeStatuses finalizeShuffleMerge(FinalizeShuffleMerge msg) {
for (AppShufflePartitionInfo partition: shuffleMergePartitions.values()) {
synchronized (partition) {
try {
+ logger.debug("{} attempt {} shuffle {} shuffleMerge {}: finalizing shuffle partition "
+ + "{} ", msg.appId, msg.appAttemptId, msg.shuffleId,
+ msg.shuffleMergeId, partition.reduceId);
// This can throw IOException which will marks this shuffle partition as not merged.
partition.finalizePartition();
- bitmaps.add(partition.mapTracker);
- reduceIds.add(partition.reduceId);
- sizes.add(partition.getLastChunkOffset());
+ if (partition.mapTracker.getCardinality() > 0) {
+ bitmaps.add(partition.mapTracker);
+ reduceIds.add(partition.reduceId);
+ sizes.add(partition.getLastChunkOffset());
+ logger.debug("{} attempt {} shuffle {} shuffleMerge {}: finalization results added"
+ + " for partition {} data size {} index size {} meta size {}",
Review comment:
nit: move the `+` to prev line (here and elsewhere in string concat for log msgs)
##########
File path: common/network-shuffle/src/main/java/org/apache/spark/network/shuffle/RemoteBlockPushResolver.java
##########
@@ -534,35 +515,33 @@ public MergeStatuses finalizeShuffleMerge(FinalizeShuffleMerge msg) {
}
AtomicReference<Map<Integer, AppShufflePartitionInfo>> shuffleMergePartitionsRef =
new AtomicReference<>(null);
- // Metadata of the determinate stage shuffle can be safely removed as part of finalizing
- // shuffle merge. Currently once the shuffle is finalized for a determinate stages, retry
- // stages of the same shuffle will have shuffle push disabled.
- if (msg.shuffleMergeId == DETERMINATE_SHUFFLE_MERGE_ID) {
- AppShuffleMergePartitionsInfo appShuffleMergePartitionsInfo =
- appShuffleInfo.shuffles.remove(msg.shuffleId);
- if (appShuffleMergePartitionsInfo != null) {
- shuffleMergePartitionsRef.set(appShuffleMergePartitionsInfo.shuffleMergePartitions);
+ appShuffleInfo.shuffles.compute(msg.shuffleId, (shuffleId, mergePartitionsInfo) -> {
+ if (null == mergePartitionsInfo) {
+ //If the mergePartitions was never created then it means that there weren't any push
+ //blocks that were ever received for this shuffle. This could be the case when the driver
+ //doesn't wait for enough time to start the stage which reads this shuffle data.
+ return new AppShuffleMergePartitionsInfo(msg.shuffleMergeId, true);
+ } else if (msg.shuffleMergeId < mergePartitionsInfo.shuffleMergeId
+ || mergePartitionsInfo.isFinalized()) {
Review comment:
nit: move `||` to prev line
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] mridulm commented on pull request #35325: [SPARK-37675][SPARK-37793] Prevent overwriting of push shuffle merged files once the shuffle is finalized
Posted by GitBox <gi...@apache.org>.
mridulm commented on pull request #35325:
URL: https://github.com/apache/spark/pull/35325#issuecomment-1023455246
@otterc, can you please update the jira with details of root cause ? That should help @pan3793 and future investigations understand the issue fixed here better. Thanks !
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] pan3793 commented on pull request #35325: [WIP][SPARK-37675][SPARK-37793] Prevent overwriting of push shuffle merged files once the shuffle is finalized
Posted by GitBox <gi...@apache.org>.
pan3793 commented on pull request #35325:
URL: https://github.com/apache/spark/pull/35325#issuecomment-1021915320
@otterc Oops, sorry, I forgot to restart the node manager so the patch does not take effect, let me retry again.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] mridulm commented on pull request #35325: [SPARK-37675][SPARK-37793] Prevent overwriting of push shuffle merged files once the shuffle is finalized
Posted by GitBox <gi...@apache.org>.
mridulm commented on pull request #35325:
URL: https://github.com/apache/spark/pull/35325#issuecomment-1024818604
@otterc Can you take a look at the test failure ? Thx
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] mridulm commented on a change in pull request #35325: [WIP][SPARK-37675][SPARK-37793] Prevent overwriting of push shuffle merged files once the shuffle is finalized
Posted by GitBox <gi...@apache.org>.
mridulm commented on a change in pull request #35325:
URL: https://github.com/apache/spark/pull/35325#discussion_r792754171
##########
File path: common/network-shuffle/src/main/java/org/apache/spark/network/shuffle/RemoteBlockPushResolver.java
##########
@@ -1008,19 +995,28 @@ AppShufflePartitionInfo getPartitionInfo() {
* required for the shuffles of indeterminate stages.
*/
public static class AppShuffleMergePartitionsInfo {
+ // ConcurrentHashMap doesn't allow null for keys or values which is why this is required.
+ // Marker to identify finalized indeterminate shuffle partitions in the case of indeterminate
+ // stage retries.
+ @VisibleForTesting
+ public static final Map<Integer, AppShufflePartitionInfo> SHUFFLE_FINALIZED_MARKER =
+ Collections.emptyMap();
private final int shuffleMergeId;
- private final Map<Integer, AppShufflePartitionInfo> shuffleMergePartitions;
+ private final Map<Integer, AppShufflePartitionInfo> mergePartitions;
Review comment:
nit: avoid rename here ?
##########
File path: common/network-shuffle/src/test/java/org/apache/spark/network/shuffle/RemoteBlockPushResolverSuite.java
##########
@@ -1287,6 +1287,52 @@ void closeAndDeletePartitionFiles(Map<Integer, AppShufflePartitionInfo> partitio
+ " up", appShuffleInfo.getMergedShuffleDataFile(0, 4, 0).exists());
}
+ // Test for SPARK-37675 and SPARK-37793
+ @Test
+ public void testAllBlocksAreRejectedWhenReceivedAfterFinalization() throws IOException {
Review comment:
Also add a test for the cardinality change ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #35325: [WIP][SPARK-37675][SPARK-37793] Prevent overwriting of push shuffle merged files once the shuffle is finalized
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #35325:
URL: https://github.com/apache/spark/pull/35325#issuecomment-1022152268
Can one of the admins verify this patch?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] mridulm edited a comment on pull request #35325: [SPARK-37675][SPARK-37793] Prevent overwriting of push shuffle merged files once the shuffle is finalized
Posted by GitBox <gi...@apache.org>.
mridulm edited a comment on pull request #35325:
URL: https://github.com/apache/spark/pull/35325#issuecomment-1024818604
~~@otterc Can you take a look at the test failure ? It is for `YarnShuffleIntegrationSuite`: external shuffle service Can you retrigger the tests in case it is not related ?~~
It looks unrelated, based on logs.
Merging to master and branch-3.2.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] otterc commented on a change in pull request #35325: [WIP][SPARK-37675][SPARK-37793] Prevent overwriting of push shuffle merged files once the shuffle is finalized
Posted by GitBox <gi...@apache.org>.
otterc commented on a change in pull request #35325:
URL: https://github.com/apache/spark/pull/35325#discussion_r792983163
##########
File path: common/network-shuffle/src/main/java/org/apache/spark/network/shuffle/RemoteBlockPushResolver.java
##########
@@ -1008,19 +995,28 @@ AppShufflePartitionInfo getPartitionInfo() {
* required for the shuffles of indeterminate stages.
*/
public static class AppShuffleMergePartitionsInfo {
+ // ConcurrentHashMap doesn't allow null for keys or values which is why this is required.
+ // Marker to identify finalized indeterminate shuffle partitions in the case of indeterminate
+ // stage retries.
+ @VisibleForTesting
+ public static final Map<Integer, AppShufflePartitionInfo> SHUFFLE_FINALIZED_MARKER =
+ Collections.emptyMap();
private final int shuffleMergeId;
- private final Map<Integer, AppShufflePartitionInfo> shuffleMergePartitions;
+ private final Map<Integer, AppShufflePartitionInfo> mergePartitions;
Review comment:
reverted the rename
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] otterc commented on pull request #35325: [SPARK-37675][SPARK-37793] Prevent overwriting of push shuffle merged files once the shuffle is finalized
Posted by GitBox <gi...@apache.org>.
otterc commented on pull request #35325:
URL: https://github.com/apache/spark/pull/35325#issuecomment-1023302640
> Thanks @otterc and @mridulm, I verified this patch with 2 rounds of 1TB TPC-DS query, the issue is gone.
>
> One minor question, can we suppress this warning message? I think it's a little bit noisy for users.
>
> ```
> 01-27 16:18:35 WARN [shuffle-client-8-5] scheduler.DAGScheduler@94: Exception encountered when trying to finalize shuffle merge on beta-spark2 for shuffle 189
> java.lang.RuntimeException: java.lang.RuntimeException: Shuffle merge finalize request for shuffle 189 with shuffleMergeId 0 is stale shuffle finalize request as shuffle blocks of a higher shuffleMergeId for the shuffle is already being pushed
> ```
Thanks @pan3793 for verifying this change. I saw those logs on the shuffle sever as well and the commit `c606185` fixes that already. This will now be logged for a true stale finalize message. Have added a UT for it as well.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] mridulm commented on pull request #35325: [SPARK-37675][SPARK-37793] Prevent overwriting of push shuffle merged files once the shuffle is finalized
Posted by GitBox <gi...@apache.org>.
mridulm commented on pull request #35325:
URL: https://github.com/apache/spark/pull/35325#issuecomment-1023454443
Thanks for verifying @pan3793 !
I will leave the PR open until tomorrow in case there are other comments/reviews, before merging.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] otterc commented on pull request #35325: [WIP][SPARK-37675][SPARK-37793] Prevent overwriting of push shuffle merged files once the shuffle is finalized
Posted by GitBox <gi...@apache.org>.
otterc commented on pull request #35325:
URL: https://github.com/apache/spark/pull/35325#issuecomment-1021914080
@pan3793 Thanks for testing this. So with this patch are you still seeing that the executor is requesting for merged blocks which either had no data or were overwritten by the shuffle service?
Will you be able to share the logs with me again? It will be great if you can turn the debug level logging on for the `RemoteBlockPushResolver`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] pan3793 commented on pull request #35325: [SPARK-37675][SPARK-37793] Prevent overwriting of push shuffle merged files once the shuffle is finalized
Posted by GitBox <gi...@apache.org>.
pan3793 commented on pull request #35325:
URL: https://github.com/apache/spark/pull/35325#issuecomment-1023312819
Thanks @otterc, would you please add some description of how the corner case caused the file to get overwritten to help others understand how the patch works?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] otterc commented on a change in pull request #35325: [SPARK-37675][SPARK-37793] Prevent overwriting of push shuffle merged files once the shuffle is finalized
Posted by GitBox <gi...@apache.org>.
otterc commented on a change in pull request #35325:
URL: https://github.com/apache/spark/pull/35325#discussion_r793258742
##########
File path: common/network-shuffle/src/test/java/org/apache/spark/network/shuffle/RemoteBlockPushResolverSuite.java
##########
@@ -1161,8 +1161,8 @@ public void testFinalizeOfDeterminateShuffle() throws IOException {
RemoteBlockPushResolver.AppShuffleInfo appShuffleInfo =
pushResolver.validateAndGetAppShuffleInfo(TEST_APP);
- assertTrue("Metadata of determinate shuffle should be removed after finalize shuffle"
- + " merge", appShuffleInfo.getShuffles().get(0) == null);
+ assertTrue("Determinate shuffle should be marked finalized"
+ + " merge", appShuffleInfo.getShuffles().get(0).isFinalized());
Review comment:
Nit: remove ` merge`
##########
File path: common/network-shuffle/src/main/java/org/apache/spark/network/shuffle/RemoteBlockPushResolver.java
##########
@@ -1008,20 +996,28 @@ AppShufflePartitionInfo getPartitionInfo() {
* required for the shuffles of indeterminate stages.
*/
public static class AppShuffleMergePartitionsInfo {
+ // ConcurrentHashMap doesn't allow null for keys or values which is why this is required.
+ // Marker to identify finalized indeterminate shuffle partitions in the case of indeterminate
Review comment:
Nit: remove indeterminate
##########
File path: common/network-shuffle/src/test/java/org/apache/spark/network/shuffle/RemoteBlockPushResolverSuite.java
##########
@@ -1287,6 +1287,79 @@ void closeAndDeletePartitionFiles(Map<Integer, AppShufflePartitionInfo> partitio
+ " up", appShuffleInfo.getMergedShuffleDataFile(0, 4, 0).exists());
}
+ @Test
+ public void testFinalizationResultIsEmptyWhenTheServerDidNotReceiveAnyBlocks() {
+ //shuffle 1 0 is finalized even though the server didn't receive any blocks for it.
+ MergeStatuses statuses = pushResolver.finalizeShuffleMerge(
+ new FinalizeShuffleMerge(TEST_APP, NO_ATTEMPT_ID, 1, 0));
+ assertEquals("no partitions were merged", 0, statuses.reduceIds.length);
+ RemoteBlockPushResolver.AppShuffleInfo appShuffleInfo =
+ pushResolver.validateAndGetAppShuffleInfo(TEST_APP);
+ assertTrue("Determinate shuffle should be marked finalized merge",
Review comment:
Nit: fix the msg
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] mridulm commented on pull request #35325: [WIP][SPARK-37675][SPARK-37793] Prevent overwriting of push shuffle merged files once the shuffle is finalized
Posted by GitBox <gi...@apache.org>.
mridulm commented on pull request #35325:
URL: https://github.com/apache/spark/pull/35325#issuecomment-1022786290
@otterc Can you remove the WIP tag, given the PR should be complete now ? Thx
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org