You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by GitBox <gi...@apache.org> on 2020/11/19 20:08:29 UTC
[GitHub] [spark] otterc opened a new pull request #30433: [SPARK-32916][SHUFFLE][test-maven][test-hadoop2.7] Ensure the number of chunks in meta file and index file are equal
otterc opened a new pull request #30433:
URL: https://github.com/apache/spark/pull/30433
### What changes were proposed in this pull request?
This change contains fixs for bugs in `RemoteBlockPushResolver` where the number of chunks in meta file and index file are inconsistent due to exceptions while writing to either index file or meta file. This java class was introduced in https://github.com/apache/spark/pull/30062.
1. If the writing to index file fails, the position of meta file is not reset. This means that the number of chunks in meta file is inconsistent with index file.
2. We changed to using `RandomAccessFile` for meta/index files. However, during the exception handling while writing to these file, we just set the pointer to the start position. If the files are closed just after this then it doesn't get rid of any the extra bytes written to it.
### Why are the changes needed?
This fix is needed for the bugs mentioned above.
1. Moved writing to meta file after index file. This fixes the issue because if there is an exception writing to meta file, then the index file position is already reset. With this change, if there is an exception writing to index file, then none of the files are effectively updated and same is true vice-versa.
2. Setting the lengths of meta/index files when the files are closed.
### Does this PR introduce _any_ user-facing change?
No
### How was this patch tested?
I have added unit tests for both of these bugs.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #30433: [SPARK-32916][SHUFFLE][test-maven][test-hadoop2.7] Ensure the number of chunks in meta file and index file are equal
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #30433:
URL: https://github.com/apache/spark/pull/30433#issuecomment-741569139
**[Test build #132471 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/132471/testReport)** for PR 30433 at commit [`0bce631`](https://github.com/apache/spark/commit/0bce63198e2e5304dd31aa39602f16d9d7e90af4).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #30433: [SPARK-32916][SHUFFLE][test-maven][test-hadoop2.7] Ensure the number of chunks in meta file and index file are equal
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #30433:
URL: https://github.com/apache/spark/pull/30433#issuecomment-738555062
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/132158/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #30433: [SPARK-32916][SHUFFLE][test-maven][test-hadoop2.7] Ensure the number of chunks in meta file and index file are equal
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #30433:
URL: https://github.com/apache/spark/pull/30433#issuecomment-732672158
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] Victsm commented on a change in pull request #30433: [SPARK-32916][SHUFFLE][test-maven][test-hadoop2.7] Ensure the number of chunks in meta file and index file are equal
Posted by GitBox <gi...@apache.org>.
Victsm commented on a change in pull request #30433:
URL: https://github.com/apache/spark/pull/30433#discussion_r539086277
##########
File path: common/network-shuffle/src/main/java/org/apache/spark/network/shuffle/RemoteBlockPushResolver.java
##########
@@ -586,6 +592,7 @@ public void onData(String streamId, ByteBuffer buf) throws IOException {
deferredBufs = null;
return;
}
+ abortIfNecessary();
Review comment:
Purpose is to only do this check once per block instead of once per buf and avoid throwing unnecessary exceptions from onData which leads to channel close.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #30433: [SPARK-32916][SHUFFLE][test-maven][test-hadoop2.7] Ensure the number of chunks in meta file and index file are equal
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #30433:
URL: https://github.com/apache/spark/pull/30433#issuecomment-741828632
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/132471/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] mridulm commented on a change in pull request #30433: [SPARK-32916][SHUFFLE][test-maven][test-hadoop2.7] Ensure the number of chunks in meta file and index file are equal
Posted by GitBox <gi...@apache.org>.
mridulm commented on a change in pull request #30433:
URL: https://github.com/apache/spark/pull/30433#discussion_r533068912
##########
File path: common/network-shuffle/src/main/java/org/apache/spark/network/shuffle/RemoteBlockPushResolver.java
##########
@@ -827,13 +833,16 @@ void resetChunkTracker() {
void updateChunkInfo(long chunkOffset, int mapIndex) throws IOException {
long idxStartPos = -1;
try {
- // update the chunk tracker to meta file before index file
- writeChunkTracker(mapIndex);
idxStartPos = indexFile.getFilePointer();
logger.trace("{} shuffleId {} reduceId {} updated index current {} updated {}",
appShuffleId.appId, appShuffleId.shuffleId, reduceId, this.lastChunkOffset,
chunkOffset);
- indexFile.writeLong(chunkOffset);
+ indexFile.write(Longs.toByteArray(chunkOffset));
+ // Chunk bitmap should be written to the meta file after the index file because if there are
+ // any exceptions during writing the offset to the index file, meta file should not be
+ // updated. If the update to the index file is successful but the update to meta file isn't
+ // then the index file position is reset in the catch clause.
+ writeChunkTracker(mapIndex);
Review comment:
Thanks for confirming @otterc, this means we should not have seek throwing exception in almost all cases for us.
We have two paths here:
a) Ignore seek exception as unexpected.
b) Abort merge, assuming EBADF/etc - as it is unrecoverable error.
I am partial towards (b) as we cannot recover with seek exception. Thoughts ?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] mridulm commented on a change in pull request #30433: [SPARK-32916][SHUFFLE][test-maven][test-hadoop2.7] Ensure the number of chunks in meta file and index file are equal
Posted by GitBox <gi...@apache.org>.
mridulm commented on a change in pull request #30433:
URL: https://github.com/apache/spark/pull/30433#discussion_r528175717
##########
File path: common/network-shuffle/src/main/java/org/apache/spark/network/shuffle/RemoteBlockPushResolver.java
##########
@@ -877,7 +886,7 @@ void closeAllFiles() {
}
if (metaFile != null) {
try {
- // if the stream is closed, channel get's closed as well.
+ metaFile.setLength(metaFile.getFilePointer());
metaFile.close();
Review comment:
Move close to finally
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] mridulm commented on a change in pull request #30433: [SPARK-32916][SHUFFLE][test-maven][test-hadoop2.7] Ensure the number of chunks in meta file and index file are equal
Posted by GitBox <gi...@apache.org>.
mridulm commented on a change in pull request #30433:
URL: https://github.com/apache/spark/pull/30433#discussion_r536438019
##########
File path: common/network-shuffle/src/main/java/org/apache/spark/network/shuffle/RemoteBlockPushResolver.java
##########
@@ -825,45 +881,79 @@ void resetChunkTracker() {
* @param mapIndex the map index to be added to chunk tracker.
*/
void updateChunkInfo(long chunkOffset, int mapIndex) throws IOException {
- long idxStartPos = -1;
try {
- // update the chunk tracker to meta file before index file
- writeChunkTracker(mapIndex);
- idxStartPos = indexFile.getFilePointer();
logger.trace("{} shuffleId {} reduceId {} updated index current {} updated {}",
- appShuffleId.appId, appShuffleId.shuffleId, reduceId, this.lastChunkOffset,
- chunkOffset);
- indexFile.writeLong(chunkOffset);
- } catch (IOException ioe) {
- if (idxStartPos != -1) {
- // reset the position to avoid corrupting index files during exception.
- logger.warn("{} shuffleId {} reduceId {} reset index to position {}",
- appShuffleId.appId, appShuffleId.shuffleId, reduceId, idxStartPos);
- indexFile.seek(idxStartPos);
+ appShuffleId.appId, appShuffleId.shuffleId, reduceId, this.lastChunkOffset, chunkOffset);
+ if (indexMetaUpdateFailed) {
+ indexFile.getChannel().position(indexFile.getPos());
}
+ indexFile.getDos().writeLong(chunkOffset);
+ // Chunk bitmap should be written to the meta file after the index file because if there are
+ // any exceptions during writing the offset to the index file, meta file should not be
+ // updated. If the update to the index file is successful but the update to meta file isn't
+ // then the index file position is reset in the catch clause.
+ writeChunkTracker(mapIndex);
+ indexFile.updatePos(8);
+ this.lastChunkOffset = chunkOffset;
+ indexMetaUpdateFailed = false;
+ } catch (IOException ioe) {
+ logger.warn("{} shuffleId {} reduceId {} update to index/meta failed", appShuffleId.appId,
+ appShuffleId.shuffleId, reduceId);
+ indexMetaUpdateFailed = true;
+ // Any exception here is propagated to the caller and the caller can decide whether to
+ // abort or not.
throw ioe;
}
- this.lastChunkOffset = chunkOffset;
}
private void writeChunkTracker(int mapIndex) throws IOException {
if (mapIndex == -1) {
return;
}
chunkTracker.add(mapIndex);
- long metaStartPos = metaFile.getFilePointer();
try {
logger.trace("{} shuffleId {} reduceId {} mapIndex {} write chunk to meta file",
appShuffleId.appId, appShuffleId.shuffleId, reduceId, mapIndex);
- chunkTracker.serialize(metaFile);
+ if (indexMetaUpdateFailed) {
+ metaFile.getChannel().position(metaFile.getPos());
+ }
+ int chunkTrackerSize = chunkTracker.serializedSizeInBytes();
+ chunkTracker.serialize(metaFile.getDos());
+ metaFile.updatePos(chunkTrackerSize);
} catch (IOException ioe) {
- logger.warn("{} shuffleId {} reduceId {} mapIndex {} reset position of meta file to {}",
- appShuffleId.appId, appShuffleId.shuffleId, reduceId, mapIndex, metaStartPos);
- metaFile.seek(metaStartPos);
+ logger.warn("{} shuffleId {} reduceId {} mapIndex {} update to meta file failed",
+ appShuffleId.appId, appShuffleId.shuffleId, reduceId, mapIndex);
+ // Any exceptions while writing to meta file is propagated to updateChunkInfo so that
+ // the index file position can be reset as well.
throw ioe;
}
}
+ private void incrementIOExceptions() {
+ numIOExceptions++;
+ }
+
+ private boolean shouldAbort(int ioExceptionsThresholdDuringMerge) {
+ return numIOExceptions > ioExceptionsThresholdDuringMerge;
+ }
+
+ private void finalizePartition() throws IOException {
+ if (dataFilePos != lastChunkOffset) {
+ try {
+ updateChunkInfo(dataFilePos, lastMergedMapIndex);
+ } catch (IOException ioe) {
+ // Any exceptions here while updating the meta files can be ignored. If the files
+ // aren't successfully updated they will be truncated.
+ }
+ }
+ // Get rid of any partial block data at the end of the file. This could either
+ // be due to failure, or a request still being processed when the shuffle
+ // merge gets finalized, or any exceptions while updating index/meta files.
+ dataChannel.truncate(lastChunkOffset);
+ indexFile.getChannel().truncate(indexFile.getPos());
+ metaFile.getChannel().truncate(metaFile.getPos());
Review comment:
Had offline discussion with @otterc ... ignoring truncate exceptions here will require more changes to how we compute and manage chunks, keep appShufflePartitionInfo around, etc ...
Given this, I am fine with handling the exception and not ignoring it.
Thanks for details @otterc
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] Victsm commented on a change in pull request #30433: [SPARK-32916][SHUFFLE][test-maven][test-hadoop2.7] Ensure the number of chunks in meta file and index file are equal
Posted by GitBox <gi...@apache.org>.
Victsm commented on a change in pull request #30433:
URL: https://github.com/apache/spark/pull/30433#discussion_r539614385
##########
File path: common/network-shuffle/src/main/java/org/apache/spark/network/shuffle/RemoteBlockPushResolver.java
##########
@@ -586,6 +592,7 @@ public void onData(String streamId, ByteBuffer buf) throws IOException {
deferredBufs = null;
return;
}
+ abortIfNecessary();
Review comment:
The way I see the issue mentioned in that comment is that we are not preventing new blocks to be merged when the IOException threshold is reached.
To do that, we only need to invoke `abortIfNecessary` inside `onComplete`, whether we still have any deferredBuf to write at that point.
This way, for normal case without IOException, we are only invoking `abortIfNecessary` once per block.
By invoking it here, we would invoke it once per buf for normal case.
Of course, if we only check inside `onComplete`, we would delay rejection of these pending blocks until we reach their stream's end.
I think this is a reasonable tradeoff to make, considering that majority of the time the code is executing for normal case instead of the exception case.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #30433: [SPARK-32916][SHUFFLE][test-maven][test-hadoop2.7] Ensure the number of chunks in meta file and index file are equal
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #30433:
URL: https://github.com/apache/spark/pull/30433#issuecomment-741092475
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/37050/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] mridulm commented on a change in pull request #30433: [SPARK-32916][SHUFFLE][test-maven][test-hadoop2.7] Ensure the number of chunks in meta file and index file are equal
Posted by GitBox <gi...@apache.org>.
mridulm commented on a change in pull request #30433:
URL: https://github.com/apache/spark/pull/30433#discussion_r533068912
##########
File path: common/network-shuffle/src/main/java/org/apache/spark/network/shuffle/RemoteBlockPushResolver.java
##########
@@ -827,13 +833,16 @@ void resetChunkTracker() {
void updateChunkInfo(long chunkOffset, int mapIndex) throws IOException {
long idxStartPos = -1;
try {
- // update the chunk tracker to meta file before index file
- writeChunkTracker(mapIndex);
idxStartPos = indexFile.getFilePointer();
logger.trace("{} shuffleId {} reduceId {} updated index current {} updated {}",
appShuffleId.appId, appShuffleId.shuffleId, reduceId, this.lastChunkOffset,
chunkOffset);
- indexFile.writeLong(chunkOffset);
+ indexFile.write(Longs.toByteArray(chunkOffset));
+ // Chunk bitmap should be written to the meta file after the index file because if there are
+ // any exceptions during writing the offset to the index file, meta file should not be
+ // updated. If the update to the index file is successful but the update to meta file isn't
+ // then the index file position is reset in the catch clause.
+ writeChunkTracker(mapIndex);
Review comment:
Thanks for confirming @otterc.
We have two paths here:
a) Ignore seek exception as unexpected.
b) Abort merge, assuming EBADF/etc - as it is unrecoverable error.
I am partial towards (b) as we cannot recover with seek exception. Thoughts ?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #30433: [SPARK-32916][SHUFFLE][test-maven][test-hadoop2.7] Ensure the number of chunks in meta file and index file are equal
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #30433:
URL: https://github.com/apache/spark/pull/30433#issuecomment-737698947
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #30433: [SPARK-32916][SHUFFLE][test-maven][test-hadoop2.7] Ensure the number of chunks in meta file and index file are equal
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #30433:
URL: https://github.com/apache/spark/pull/30433#issuecomment-740830147
**[Test build #132444 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/132444/testReport)** for PR 30433 at commit [`22c7a32`](https://github.com/apache/spark/commit/22c7a328c48bf64de758fa7fa97f8b34333bc641).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #30433: [SPARK-32916][SHUFFLE][test-maven][test-hadoop2.7] Ensure the number of chunks in meta file and index file are equal
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #30433:
URL: https://github.com/apache/spark/pull/30433#issuecomment-737721310
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] Ngone51 commented on a change in pull request #30433: [SPARK-32916][SHUFFLE][test-maven][test-hadoop2.7] Ensure the number of chunks in meta file and index file are equal
Posted by GitBox <gi...@apache.org>.
Ngone51 commented on a change in pull request #30433:
URL: https://github.com/apache/spark/pull/30433#discussion_r529611998
##########
File path: common/network-shuffle/src/main/java/org/apache/spark/network/shuffle/RemoteBlockPushResolver.java
##########
@@ -827,13 +833,16 @@ void resetChunkTracker() {
void updateChunkInfo(long chunkOffset, int mapIndex) throws IOException {
long idxStartPos = -1;
try {
- // update the chunk tracker to meta file before index file
- writeChunkTracker(mapIndex);
idxStartPos = indexFile.getFilePointer();
logger.trace("{} shuffleId {} reduceId {} updated index current {} updated {}",
appShuffleId.appId, appShuffleId.shuffleId, reduceId, this.lastChunkOffset,
chunkOffset);
- indexFile.writeLong(chunkOffset);
+ indexFile.write(Longs.toByteArray(chunkOffset));
+ // Chunk bitmap should be written to the meta file after the index file because if there are
+ // any exceptions during writing the offset to the index file, meta file should not be
+ // updated. If the update to the index file is successful but the update to meta file isn't
+ // then the index file position is reset in the catch clause.
+ writeChunkTracker(mapIndex);
Review comment:
Tracking IO exceptions sounds a little bit complex to me...I'd prefer to stop merging in case of IO exception for the seek.
Besides, I'm thinking it would be good if we could dynamically change the merger location when such IO exception happens.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] otterc commented on a change in pull request #30433: [SPARK-32916][SHUFFLE][test-maven][test-hadoop2.7] Ensure the number of chunks in meta file and index file are equal
Posted by GitBox <gi...@apache.org>.
otterc commented on a change in pull request #30433:
URL: https://github.com/apache/spark/pull/30433#discussion_r533785143
##########
File path: common/network-shuffle/src/main/java/org/apache/spark/network/shuffle/RemoteBlockPushResolver.java
##########
@@ -877,7 +886,7 @@ void closeAllFiles() {
}
Review comment:
Also we truncate the data file before `closeAllFiles` is invoked in `finalizeShuffleMerge`
```
partition.dataChannel.truncate(partition.getPosition());
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #30433: [SPARK-32916][SHUFFLE][test-maven][test-hadoop2.7] Ensure the number of chunks in meta file and index file are equal
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #30433:
URL: https://github.com/apache/spark/pull/30433#issuecomment-740902480
**[Test build #132444 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/132444/testReport)** for PR 30433 at commit [`22c7a32`](https://github.com/apache/spark/commit/22c7a328c48bf64de758fa7fa97f8b34333bc641).
* This patch **fails Spark unit tests**.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #30433: [SPARK-32916][SHUFFLE][test-maven][test-hadoop2.7] Ensure the number of chunks in meta file and index file are equal
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #30433:
URL: https://github.com/apache/spark/pull/30433#issuecomment-749835868
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/133233/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] otterc commented on a change in pull request #30433: [SPARK-32916][SHUFFLE][test-maven][test-hadoop2.7] Ensure the number of chunks in meta file and index file are equal
Posted by GitBox <gi...@apache.org>.
otterc commented on a change in pull request #30433:
URL: https://github.com/apache/spark/pull/30433#discussion_r542886903
##########
File path: common/network-shuffle/src/main/java/org/apache/spark/network/shuffle/RemoteBlockPushResolver.java
##########
@@ -586,6 +592,7 @@ public void onData(String streamId, ByteBuffer buf) throws IOException {
deferredBufs = null;
return;
}
+ abortIfNecessary();
Review comment:
The client does receive the exception thrown from `onData`. I simulated an exception from `onData` at the server for a particular push block.
Below are the logs of the client.
Note: I am running an older version of magnet which still uses `ShuffleBlockId` for shuffle push and some other classes are old.
```
20/12/11 19:14:48 INFO TransportClientFactory: Successfully created connection to ltx1-hcl3213.grid.linkedin.com/10.150.24.33:7337 after 12 ms (5 ms spent in bootstraps)
20/12/11 19:14:56 ERROR RetryingBlockFetcher: Failed to fetch block shuffle_1_7_7, and will not retry (1 retries)
org.apache.spark.network.shuffle.BlockPushException: ^H^@^@^@^_application_1602506816280_53624^@^@^@
shuffle_1_7_7^@^@^@^Gjava.io.IOException: Destination failed while reading stream
at org.apache.spark.network.server.TransportRequestHandler$3.onFailure(TransportRequestHandler.java:244)
at org.apache.spark.network.client.StreamInterceptor.exceptionCaught(StreamInterceptor.java:58)
at org.apache.spark.network.util.TransportFrameDecoder.exceptionCaught(TransportFrameDecoder.java:188)
at org.spark_project.io.netty.channel.AbstractChannelHandlerContext.invokeExceptionCaught(AbstractChannelHandlerContext.java:285)
at org.spark_project.io.netty.channel.AbstractChannelHandlerContext.notifyHandlerException(AbstractChannelHandlerContext.java:850)
at org.spark_project.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:364)
at org.spark_project.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
at org.spark_project.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
at org.spark_project.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1359)
at org.spark_project.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
at org.spark_project.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
at org.spark_project.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:935)
at org.spark_project.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:138)
at org.spark_project.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:645)
at org.spark_project.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:580)
at org.spark_project.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:497)
at org.spark_project.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:459)
at org.spark_project.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858)
at org.spark_project.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.RuntimeException: FAILING this stream
```
After this, the client logs show that the server has terminated the connection.
```
20/12/11 19:14:57 WARN TransportChannelHandler: Exception in connection from ltx1-hcl3412.grid.linkedin.com/10.150.55.78:7337
java.io.IOException: Connection reset by peer
at sun.nio.ch.FileDispatcherImpl.read0(Native Method)
at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:39)
at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:223)
at sun.nio.ch.IOUtil.read(IOUtil.java:192)
at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:380)
at io.netty.buffer.PooledUnsafeDirectByteBuf.setBytes(PooledUnsafeDirectByteBuf.java:288)
at io.netty.buffer.AbstractByteBuf.writeBytes(AbstractByteBuf.java:1106)
at io.netty.channel.socket.nio.NioSocketChannel.doReadBytes(NioSocketChannel.java:343)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:123)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:645)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:580)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:497)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:459)
at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858)
at io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138)
at java.lang.Thread.run(Thread.java:748)
20/12/11 19:14:57 ERROR TransportResponseHandler: Still have 38 requests outstanding when connection from ltx1-hcl3412.grid.linkedin.com/10.150.55.78:7337 is closed
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] otterc commented on a change in pull request #30433: [SPARK-32916][SHUFFLE][test-maven][test-hadoop2.7] Ensure the number of chunks in meta file and index file are equal
Posted by GitBox <gi...@apache.org>.
otterc commented on a change in pull request #30433:
URL: https://github.com/apache/spark/pull/30433#discussion_r528266788
##########
File path: common/network-shuffle/src/main/java/org/apache/spark/network/shuffle/RemoteBlockPushResolver.java
##########
@@ -827,13 +833,16 @@ void resetChunkTracker() {
void updateChunkInfo(long chunkOffset, int mapIndex) throws IOException {
long idxStartPos = -1;
try {
- // update the chunk tracker to meta file before index file
- writeChunkTracker(mapIndex);
idxStartPos = indexFile.getFilePointer();
logger.trace("{} shuffleId {} reduceId {} updated index current {} updated {}",
appShuffleId.appId, appShuffleId.shuffleId, reduceId, this.lastChunkOffset,
chunkOffset);
- indexFile.writeLong(chunkOffset);
+ indexFile.write(Longs.toByteArray(chunkOffset));
+ // Chunk bitmap should be written to the meta file after the index file because if there are
+ // any exceptions during writing the offset to the index file, meta file should not be
+ // updated. If the update to the index file is successful but the update to meta file isn't
+ // then the index file position is reset in the catch clause.
+ writeChunkTracker(mapIndex);
Review comment:
On a second thought, if there was an exception writing and then again an exception while seek, is there a possibility that any further updates to these files will succeed?
Usually these exceptions wouldn't indicate any transient error with the FileSystem.
Wouldn't it be better to just stop merging any more blocks of this shuffle partition?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] otterc commented on a change in pull request #30433: [SPARK-32916][SHUFFLE][test-maven][test-hadoop2.7] Ensure the number of chunks in meta file and index file are equal
Posted by GitBox <gi...@apache.org>.
otterc commented on a change in pull request #30433:
URL: https://github.com/apache/spark/pull/30433#discussion_r533694433
##########
File path: common/network-shuffle/src/main/java/org/apache/spark/network/shuffle/RemoteBlockPushResolver.java
##########
@@ -827,13 +833,16 @@ void resetChunkTracker() {
void updateChunkInfo(long chunkOffset, int mapIndex) throws IOException {
long idxStartPos = -1;
try {
- // update the chunk tracker to meta file before index file
- writeChunkTracker(mapIndex);
idxStartPos = indexFile.getFilePointer();
logger.trace("{} shuffleId {} reduceId {} updated index current {} updated {}",
appShuffleId.appId, appShuffleId.shuffleId, reduceId, this.lastChunkOffset,
chunkOffset);
- indexFile.writeLong(chunkOffset);
+ indexFile.write(Longs.toByteArray(chunkOffset));
+ // Chunk bitmap should be written to the meta file after the index file because if there are
+ // any exceptions during writing the offset to the index file, meta file should not be
+ // updated. If the update to the index file is successful but the update to meta file isn't
+ // then the index file position is reset in the catch clause.
+ writeChunkTracker(mapIndex);
Review comment:
Just want to reiterate the change I am working on:
- Since seek is not recoverable, we let the clients know to stop pushing for a particular shuffle partition.
- Have a threshold on number of IOExceptions from writes and when this threshold is reached for a single partition, inform the client to stop pushing and stop merging this partition.
- When the update to metadata fails, not propagate this exception back to client so that they push the block again. The size of the current chunk may grow but with (2) in place it will still be of a manageable size.
These changes will impact https://github.com/apache/spark/pull/30312. Right now the client stops pushing for the entire shuffle not a particular shuffle partition. The support needs to be extended to stop pushing for a particular shuffle partition.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #30433: [SPARK-32916][SHUFFLE][test-maven][test-hadoop2.7] Ensure the number of chunks in meta file and index file are equal
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #30433:
URL: https://github.com/apache/spark/pull/30433#issuecomment-741521841
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/132448/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] Ngone51 commented on a change in pull request #30433: [SPARK-32916][SHUFFLE][test-maven][test-hadoop2.7] Ensure the number of chunks in meta file and index file are equal
Posted by GitBox <gi...@apache.org>.
Ngone51 commented on a change in pull request #30433:
URL: https://github.com/apache/spark/pull/30433#discussion_r539855837
##########
File path: common/network-shuffle/src/main/java/org/apache/spark/network/shuffle/RemoteBlockPushResolver.java
##########
@@ -586,6 +592,7 @@ public void onData(String streamId, ByteBuffer buf) throws IOException {
deferredBufs = null;
return;
}
+ abortIfNecessary();
Review comment:
It's OK if the further writes throw IOExceptions. I just worry, can we handle it correctly if writes succeed? Or it may lead to some undefined situation which could affect later shuffle fetch in turn.
So I'd like to avoid any writes in the first place (`onData`) so we ensure that we don't go into any undefined situation. And I think, maybe, we can update it a little, like this - we still receive data in `onData` but just not write when the partition already reach IO threshold and only call `abortIfNecessary` inside `onCompelte`.
WDYT? @Victsm @otterc
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #30433: [SPARK-32916][SHUFFLE][test-maven][test-hadoop2.7] Ensure the number of chunks in meta file and index file are equal
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #30433:
URL: https://github.com/apache/spark/pull/30433#issuecomment-732475491
Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/36176/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] Victsm commented on a change in pull request #30433: [SPARK-32916][SHUFFLE][test-maven][test-hadoop2.7] Ensure the number of chunks in meta file and index file are equal
Posted by GitBox <gi...@apache.org>.
Victsm commented on a change in pull request #30433:
URL: https://github.com/apache/spark/pull/30433#discussion_r528512614
##########
File path: common/network-shuffle/src/main/java/org/apache/spark/network/shuffle/RemoteBlockPushResolver.java
##########
@@ -827,13 +833,16 @@ void resetChunkTracker() {
void updateChunkInfo(long chunkOffset, int mapIndex) throws IOException {
long idxStartPos = -1;
try {
- // update the chunk tracker to meta file before index file
- writeChunkTracker(mapIndex);
idxStartPos = indexFile.getFilePointer();
logger.trace("{} shuffleId {} reduceId {} updated index current {} updated {}",
appShuffleId.appId, appShuffleId.shuffleId, reduceId, this.lastChunkOffset,
chunkOffset);
- indexFile.writeLong(chunkOffset);
+ indexFile.write(Longs.toByteArray(chunkOffset));
+ // Chunk bitmap should be written to the meta file after the index file because if there are
+ // any exceptions during writing the offset to the index file, meta file should not be
+ // updated. If the update to the index file is successful but the update to meta file isn't
+ // then the index file position is reset in the catch clause.
+ writeChunkTracker(mapIndex);
Review comment:
Stopping merging blocks only after 2 consecutive IOExceptions seems to be giving up too early.
If there's a need to stop early due to too many disk failures, I think that logic should be on the client side inside `ErrorHandler`.
For the server side, it should just handle the received merge requests as best as it can.
Also, index/meta file writing failure should be recoverable just like how we recover data file writing failures.
We just need to handle the data file and index/meta file IOExceptions separately.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] mridulm commented on a change in pull request #30433: [SPARK-32916][SHUFFLE][test-maven][test-hadoop2.7] Ensure the number of chunks in meta file and index file are equal
Posted by GitBox <gi...@apache.org>.
mridulm commented on a change in pull request #30433:
URL: https://github.com/apache/spark/pull/30433#discussion_r528175717
##########
File path: common/network-shuffle/src/main/java/org/apache/spark/network/shuffle/RemoteBlockPushResolver.java
##########
@@ -877,7 +886,7 @@ void closeAllFiles() {
}
if (metaFile != null) {
try {
- // if the stream is closed, channel get's closed as well.
+ metaFile.setLength(metaFile.getFilePointer());
metaFile.close();
Review comment:
Move close to finally (here and below)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] otterc commented on a change in pull request #30433: [SPARK-32916][SHUFFLE][test-maven][test-hadoop2.7] Ensure the number of chunks in meta file and index file are equal
Posted by GitBox <gi...@apache.org>.
otterc commented on a change in pull request #30433:
URL: https://github.com/apache/spark/pull/30433#discussion_r535507691
##########
File path: common/network-shuffle/src/main/java/org/apache/spark/network/shuffle/RemoteBlockPushResolver.java
##########
@@ -825,45 +881,79 @@ void resetChunkTracker() {
* @param mapIndex the map index to be added to chunk tracker.
*/
void updateChunkInfo(long chunkOffset, int mapIndex) throws IOException {
- long idxStartPos = -1;
try {
- // update the chunk tracker to meta file before index file
- writeChunkTracker(mapIndex);
- idxStartPos = indexFile.getFilePointer();
logger.trace("{} shuffleId {} reduceId {} updated index current {} updated {}",
- appShuffleId.appId, appShuffleId.shuffleId, reduceId, this.lastChunkOffset,
- chunkOffset);
- indexFile.writeLong(chunkOffset);
- } catch (IOException ioe) {
- if (idxStartPos != -1) {
- // reset the position to avoid corrupting index files during exception.
- logger.warn("{} shuffleId {} reduceId {} reset index to position {}",
- appShuffleId.appId, appShuffleId.shuffleId, reduceId, idxStartPos);
- indexFile.seek(idxStartPos);
+ appShuffleId.appId, appShuffleId.shuffleId, reduceId, this.lastChunkOffset, chunkOffset);
+ if (indexMetaUpdateFailed) {
+ indexFile.getChannel().position(indexFile.getPos());
}
+ indexFile.getDos().writeLong(chunkOffset);
+ // Chunk bitmap should be written to the meta file after the index file because if there are
+ // any exceptions during writing the offset to the index file, meta file should not be
+ // updated. If the update to the index file is successful but the update to meta file isn't
+ // then the index file position is reset in the catch clause.
+ writeChunkTracker(mapIndex);
+ indexFile.updatePos(8);
+ this.lastChunkOffset = chunkOffset;
+ indexMetaUpdateFailed = false;
+ } catch (IOException ioe) {
+ logger.warn("{} shuffleId {} reduceId {} update to index/meta failed", appShuffleId.appId,
+ appShuffleId.shuffleId, reduceId);
+ indexMetaUpdateFailed = true;
+ // Any exception here is propagated to the caller and the caller can decide whether to
+ // abort or not.
throw ioe;
}
- this.lastChunkOffset = chunkOffset;
}
private void writeChunkTracker(int mapIndex) throws IOException {
if (mapIndex == -1) {
return;
}
chunkTracker.add(mapIndex);
- long metaStartPos = metaFile.getFilePointer();
try {
logger.trace("{} shuffleId {} reduceId {} mapIndex {} write chunk to meta file",
appShuffleId.appId, appShuffleId.shuffleId, reduceId, mapIndex);
- chunkTracker.serialize(metaFile);
+ if (indexMetaUpdateFailed) {
+ metaFile.getChannel().position(metaFile.getPos());
+ }
+ int chunkTrackerSize = chunkTracker.serializedSizeInBytes();
+ chunkTracker.serialize(metaFile.getDos());
+ metaFile.updatePos(chunkTrackerSize);
} catch (IOException ioe) {
- logger.warn("{} shuffleId {} reduceId {} mapIndex {} reset position of meta file to {}",
- appShuffleId.appId, appShuffleId.shuffleId, reduceId, mapIndex, metaStartPos);
- metaFile.seek(metaStartPos);
+ logger.warn("{} shuffleId {} reduceId {} mapIndex {} update to meta file failed",
+ appShuffleId.appId, appShuffleId.shuffleId, reduceId, mapIndex);
+ // Any exceptions while writing to meta file is propagated to updateChunkInfo so that
+ // the index file position can be reset as well.
throw ioe;
}
}
+ private void incrementIOExceptions() {
+ numIOExceptions++;
+ }
+
+ private boolean shouldAbort(int ioExceptionsThresholdDuringMerge) {
+ return numIOExceptions > ioExceptionsThresholdDuringMerge;
+ }
+
+ private void finalizePartition() throws IOException {
+ if (dataFilePos != lastChunkOffset) {
+ try {
+ updateChunkInfo(dataFilePos, lastMergedMapIndex);
+ } catch (IOException ioe) {
+ // Any exceptions here while updating the meta files can be ignored. If the files
+ // aren't successfully updated they will be truncated.
+ }
+ }
+ // Get rid of any partial block data at the end of the file. This could either
+ // be due to failure, or a request still being processed when the shuffle
+ // merge gets finalized, or any exceptions while updating index/meta files.
+ dataChannel.truncate(lastChunkOffset);
+ indexFile.getChannel().truncate(indexFile.getPos());
+ metaFile.getChannel().truncate(metaFile.getPos());
Review comment:
We are not ignoring the exceptions for truncate. If any of these truncate fail, the finalize of this shuffle partition fails and the shuffle partition is considered not to be merged. So clients will fetch original un-merged data.
If we want, we can ignore exceptions during truncation of data file. However, I didn't do that because the existing code doesn't ignore that.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] otterc commented on a change in pull request #30433: [SPARK-32916][SHUFFLE][test-maven][test-hadoop2.7] Ensure the number of chunks in meta file and index file are equal
Posted by GitBox <gi...@apache.org>.
otterc commented on a change in pull request #30433:
URL: https://github.com/apache/spark/pull/30433#discussion_r538952136
##########
File path: common/network-shuffle/src/test/java/org/apache/spark/network/shuffle/RemoteBlockPushResolverSuite.java
##########
@@ -411,6 +415,269 @@ void deleteExecutorDirs(Path[] dirs) {
}
}
+ @Test
+ public void testRecoverIndexFileAfterIOExceptions() throws IOException {
+ useTestFiles(true, false);
+ RemoteBlockPushResolver.PushBlockStreamCallback callback1 =
+ (RemoteBlockPushResolver.PushBlockStreamCallback) pushResolver.receiveBlockDataAsStream(
+ new PushBlockStream(TEST_APP, 0, 0, 0, 0));
+ callback1.onData(callback1.getID(), ByteBuffer.wrap(new byte[4]));
+ callback1.onComplete(callback1.getID());
+ RemoteBlockPushResolver.AppShufflePartitionInfo partitionInfo = callback1.getPartitionInfo();
+ // Close the index stream so it throws IOException
+ TestMergeShuffleFile testIndexFile = (TestMergeShuffleFile) partitionInfo.getIndexFile();
+ testIndexFile.close();
+ StreamCallbackWithID callback2 = pushResolver.receiveBlockDataAsStream(
+ new PushBlockStream(TEST_APP, 0, 1, 0, 0));
+ callback2.onData(callback2.getID(), ByteBuffer.wrap(new byte[5]));
+ // This will complete without any IOExceptions because number of IOExceptions are less than
+ // the threshold but the update to index file will be unsuccessful.
+ callback2.onComplete(callback2.getID());
+ assertEquals("index position", 16, testIndexFile.getPos());
+ // Restore the index stream so it can write successfully again.
+ testIndexFile.restore();
+ StreamCallbackWithID callback3 = pushResolver.receiveBlockDataAsStream(
+ new PushBlockStream(TEST_APP, 0, 2, 0, 0));
+ callback3.onData(callback3.getID(), ByteBuffer.wrap(new byte[2]));
+ callback3.onComplete(callback3.getID());
+ assertEquals("index position", 24, testIndexFile.getPos());
+ MergeStatuses statuses = pushResolver.finalizeShuffleMerge(
+ new FinalizeShuffleMerge(TEST_APP, 0));
+ validateMergeStatuses(statuses, new int[] {0}, new long[] {11});
+ MergedBlockMeta blockMeta = pushResolver.getMergedBlockMeta(TEST_APP, 0, 0);
+ validateChunks(TEST_APP, 0, 0, blockMeta, new int[] {4, 7}, new int[][] {{0}, {1, 2}});
+ }
+
+ @Test
+ public void testRecoverIndexFileAfterIOExceptionsInFinalize() throws IOException {
+ useTestFiles(true, false);
+ registerExecutor(TEST_APP, prepareLocalDirs(localDirs));
+ RemoteBlockPushResolver.PushBlockStreamCallback callback1 =
+ (RemoteBlockPushResolver.PushBlockStreamCallback) pushResolver.receiveBlockDataAsStream(
+ new PushBlockStream(TEST_APP, 0, 0, 0, 0));
+ callback1.onData(callback1.getID(), ByteBuffer.wrap(new byte[4]));
+ callback1.onComplete(callback1.getID());
+ RemoteBlockPushResolver.AppShufflePartitionInfo partitionInfo = callback1.getPartitionInfo();
+ // Close the index stream so it throws IOException
+ TestMergeShuffleFile testIndexFile = (TestMergeShuffleFile) partitionInfo.getIndexFile();
+ testIndexFile.close();
+ StreamCallbackWithID callback2 = pushResolver.receiveBlockDataAsStream(
+ new PushBlockStream(TEST_APP, 0, 1, 0, 0));
+ callback2.onData(callback2.getID(), ByteBuffer.wrap(new byte[5]));
+ // This will complete without any IOExceptions because number of IOExceptions are less than
+ // the threshold but the update to index file will be unsuccessful.
+ callback2.onComplete(callback2.getID());
+ assertEquals("index position", 16, testIndexFile.getPos());
+ // The last update to index was unsuccessful however any further updates will be successful.
+ // Restore the index stream so it can write successfully again.
+ testIndexFile.restore();
+ MergeStatuses statuses = pushResolver.finalizeShuffleMerge(
+ new FinalizeShuffleMerge(TEST_APP, 0));
+ assertEquals("index position", 24, testIndexFile.getPos());
+ validateMergeStatuses(statuses, new int[] {0}, new long[] {9});
+ MergedBlockMeta blockMeta = pushResolver.getMergedBlockMeta(TEST_APP, 0, 0);
+ validateChunks(TEST_APP, 0, 0, blockMeta, new int[] {4, 5}, new int[][] {{0}, {1}});
+ }
+
+ @Test
+ public void testRecoverMetaFileAfterIOExceptions() throws IOException {
+ useTestFiles(false, true);
+ RemoteBlockPushResolver.PushBlockStreamCallback callback1 =
+ (RemoteBlockPushResolver.PushBlockStreamCallback) pushResolver.receiveBlockDataAsStream(
+ new PushBlockStream(TEST_APP, 0, 0, 0, 0));
+ callback1.onData(callback1.getID(), ByteBuffer.wrap(new byte[4]));
+ callback1.onComplete(callback1.getID());
+ RemoteBlockPushResolver.AppShufflePartitionInfo partitionInfo = callback1.getPartitionInfo();
+ // Close the meta stream so it throws IOException
+ TestMergeShuffleFile testMetaFile = (TestMergeShuffleFile) partitionInfo.getMetaFile();
+ long metaPosBeforeClose = testMetaFile.getPos();
+ testMetaFile.close();
+ StreamCallbackWithID callback2 = pushResolver.receiveBlockDataAsStream(
+ new PushBlockStream(TEST_APP, 0, 1, 0, 0));
+ callback2.onData(callback2.getID(), ByteBuffer.wrap(new byte[5]));
+ // This will complete without any IOExceptions because number of IOExceptions are less than
+ // the threshold but the update to index and meta file will be unsuccessful.
+ callback2.onComplete(callback2.getID());
+ assertEquals("index position", 16, partitionInfo.getIndexFile().getPos());
+ assertEquals("meta position", metaPosBeforeClose, testMetaFile.getPos());
+ // Restore the meta stream so it can write successfully again.
+ testMetaFile.restore();
+ StreamCallbackWithID callback3 = pushResolver.receiveBlockDataAsStream(
+ new PushBlockStream(TEST_APP, 0, 2, 0, 0));
+ callback3.onData(callback3.getID(), ByteBuffer.wrap(new byte[2]));
+ callback3.onComplete(callback3.getID());
+ assertEquals("index position", 24, partitionInfo.getIndexFile().getPos());
+ assertTrue("meta position", testMetaFile.getPos() > metaPosBeforeClose);
+ MergeStatuses statuses = pushResolver.finalizeShuffleMerge(
+ new FinalizeShuffleMerge(TEST_APP, 0));
+ validateMergeStatuses(statuses, new int[] {0}, new long[] {11});
+ MergedBlockMeta blockMeta = pushResolver.getMergedBlockMeta(TEST_APP, 0, 0);
+ validateChunks(TEST_APP, 0, 0, blockMeta, new int[] {4, 7}, new int[][] {{0}, {1, 2}});
+ }
+
+ @Test
+ public void testRecoverMetaFileAfterIOExceptionsInFinalize() throws IOException {
+ useTestFiles(false, true);
+ RemoteBlockPushResolver.PushBlockStreamCallback callback1 =
+ (RemoteBlockPushResolver.PushBlockStreamCallback) pushResolver.receiveBlockDataAsStream(
+ new PushBlockStream(TEST_APP, 0, 0, 0, 0));
+ callback1.onData(callback1.getID(), ByteBuffer.wrap(new byte[4]));
+ callback1.onComplete(callback1.getID());
+ RemoteBlockPushResolver.AppShufflePartitionInfo partitionInfo = callback1.getPartitionInfo();
+ // Close the meta stream so it throws IOException
+ TestMergeShuffleFile testMetaFile = (TestMergeShuffleFile) partitionInfo.getMetaFile();
+ long metaPosBeforeClose = testMetaFile.getPos();
+ testMetaFile.close();
+ StreamCallbackWithID callback2 = pushResolver.receiveBlockDataAsStream(
+ new PushBlockStream(TEST_APP, 0, 1, 0, 0));
+ callback2.onData(callback2.getID(), ByteBuffer.wrap(new byte[5]));
+ // This will complete without any IOExceptions because number of IOExceptions are less than
+ // the threshold but the update to index and meta file will be unsuccessful.
+ callback2.onComplete(callback2.getID());
+ MergeShuffleFile indexFile = partitionInfo.getIndexFile();
+ assertEquals("index position", 16, indexFile.getPos());
+ assertEquals("meta position", metaPosBeforeClose, testMetaFile.getPos());
+ // Restore the meta stream so it can write successfully again.
+ testMetaFile.restore();
+ MergeStatuses statuses = pushResolver.finalizeShuffleMerge(
+ new FinalizeShuffleMerge(TEST_APP, 0));
+ assertEquals("index position", 24, indexFile.getPos());
+ assertTrue("meta position", testMetaFile.getPos() > metaPosBeforeClose);
+ validateMergeStatuses(statuses, new int[] {0}, new long[] {9});
+ MergedBlockMeta blockMeta = pushResolver.getMergedBlockMeta(TEST_APP, 0, 0);
+ validateChunks(TEST_APP, 0, 0, blockMeta, new int[] {4, 5}, new int[][] {{0}, {1}});
+ }
+
+ @Test (expected = RuntimeException.class)
+ public void testIOExceptionsExceededThreshold() throws IOException {
+ RemoteBlockPushResolver.PushBlockStreamCallback callback =
+ (RemoteBlockPushResolver.PushBlockStreamCallback) pushResolver.receiveBlockDataAsStream(
+ new PushBlockStream(TEST_APP, 0, 0, 0, 0));
+ RemoteBlockPushResolver.AppShufflePartitionInfo partitionInfo = callback.getPartitionInfo();
+ callback.onData(callback.getID(), ByteBuffer.wrap(new byte[4]));
+ callback.onComplete(callback.getID());
+ // Close the data stream so it throws continuous IOException
+ partitionInfo.getDataChannel().close();
+ for (int i = 1; i < 5; i++) {
+ RemoteBlockPushResolver.PushBlockStreamCallback callback1 =
+ (RemoteBlockPushResolver.PushBlockStreamCallback) pushResolver.receiveBlockDataAsStream(
+ new PushBlockStream(TEST_APP, 0, i, 0, 0));
+ try {
+ callback1.onData(callback1.getID(), ByteBuffer.wrap(new byte[2]));
+ } catch (IOException ioe) {
+ // this will throw IOException so the client can retry.
+ callback1.onFailure(callback1.getID(), ioe);
+ }
+ }
+ assertEquals(4, partitionInfo.getNumIOExceptions());
+ // After 4 IOException, the server will respond with IOExceptions exceeded threshold
+ try {
+ RemoteBlockPushResolver.PushBlockStreamCallback callback2 =
+ (RemoteBlockPushResolver.PushBlockStreamCallback) pushResolver.receiveBlockDataAsStream(
+ new PushBlockStream(TEST_APP, 0, 5, 0, 0));
+ callback2.onData(callback.getID(), ByteBuffer.wrap(new byte[1]));
+ } catch (Throwable t) {
+ assertEquals("IOExceptions exceeded the threshold when merging shufflePush_0_5_0",
+ t.getMessage());
+ throw t;
+ }
Review comment:
> I actually mean those blocks which are waiting for the partition lock at the same time.
I see. That's a good point. The server would attempt to merge them. If they don't see any which IOException, which is unlikely they will be merged. Do you think we should check before writing them?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] otterc commented on a change in pull request #30433: [SPARK-32916][SHUFFLE][test-maven][test-hadoop2.7] Ensure the number of chunks in meta file and index file are equal
Posted by GitBox <gi...@apache.org>.
otterc commented on a change in pull request #30433:
URL: https://github.com/apache/spark/pull/30433#discussion_r538952136
##########
File path: common/network-shuffle/src/test/java/org/apache/spark/network/shuffle/RemoteBlockPushResolverSuite.java
##########
@@ -411,6 +415,269 @@ void deleteExecutorDirs(Path[] dirs) {
}
}
+ @Test
+ public void testRecoverIndexFileAfterIOExceptions() throws IOException {
+ useTestFiles(true, false);
+ RemoteBlockPushResolver.PushBlockStreamCallback callback1 =
+ (RemoteBlockPushResolver.PushBlockStreamCallback) pushResolver.receiveBlockDataAsStream(
+ new PushBlockStream(TEST_APP, 0, 0, 0, 0));
+ callback1.onData(callback1.getID(), ByteBuffer.wrap(new byte[4]));
+ callback1.onComplete(callback1.getID());
+ RemoteBlockPushResolver.AppShufflePartitionInfo partitionInfo = callback1.getPartitionInfo();
+ // Close the index stream so it throws IOException
+ TestMergeShuffleFile testIndexFile = (TestMergeShuffleFile) partitionInfo.getIndexFile();
+ testIndexFile.close();
+ StreamCallbackWithID callback2 = pushResolver.receiveBlockDataAsStream(
+ new PushBlockStream(TEST_APP, 0, 1, 0, 0));
+ callback2.onData(callback2.getID(), ByteBuffer.wrap(new byte[5]));
+ // This will complete without any IOExceptions because number of IOExceptions are less than
+ // the threshold but the update to index file will be unsuccessful.
+ callback2.onComplete(callback2.getID());
+ assertEquals("index position", 16, testIndexFile.getPos());
+ // Restore the index stream so it can write successfully again.
+ testIndexFile.restore();
+ StreamCallbackWithID callback3 = pushResolver.receiveBlockDataAsStream(
+ new PushBlockStream(TEST_APP, 0, 2, 0, 0));
+ callback3.onData(callback3.getID(), ByteBuffer.wrap(new byte[2]));
+ callback3.onComplete(callback3.getID());
+ assertEquals("index position", 24, testIndexFile.getPos());
+ MergeStatuses statuses = pushResolver.finalizeShuffleMerge(
+ new FinalizeShuffleMerge(TEST_APP, 0));
+ validateMergeStatuses(statuses, new int[] {0}, new long[] {11});
+ MergedBlockMeta blockMeta = pushResolver.getMergedBlockMeta(TEST_APP, 0, 0);
+ validateChunks(TEST_APP, 0, 0, blockMeta, new int[] {4, 7}, new int[][] {{0}, {1, 2}});
+ }
+
+ @Test
+ public void testRecoverIndexFileAfterIOExceptionsInFinalize() throws IOException {
+ useTestFiles(true, false);
+ registerExecutor(TEST_APP, prepareLocalDirs(localDirs));
+ RemoteBlockPushResolver.PushBlockStreamCallback callback1 =
+ (RemoteBlockPushResolver.PushBlockStreamCallback) pushResolver.receiveBlockDataAsStream(
+ new PushBlockStream(TEST_APP, 0, 0, 0, 0));
+ callback1.onData(callback1.getID(), ByteBuffer.wrap(new byte[4]));
+ callback1.onComplete(callback1.getID());
+ RemoteBlockPushResolver.AppShufflePartitionInfo partitionInfo = callback1.getPartitionInfo();
+ // Close the index stream so it throws IOException
+ TestMergeShuffleFile testIndexFile = (TestMergeShuffleFile) partitionInfo.getIndexFile();
+ testIndexFile.close();
+ StreamCallbackWithID callback2 = pushResolver.receiveBlockDataAsStream(
+ new PushBlockStream(TEST_APP, 0, 1, 0, 0));
+ callback2.onData(callback2.getID(), ByteBuffer.wrap(new byte[5]));
+ // This will complete without any IOExceptions because number of IOExceptions are less than
+ // the threshold but the update to index file will be unsuccessful.
+ callback2.onComplete(callback2.getID());
+ assertEquals("index position", 16, testIndexFile.getPos());
+ // The last update to index was unsuccessful however any further updates will be successful.
+ // Restore the index stream so it can write successfully again.
+ testIndexFile.restore();
+ MergeStatuses statuses = pushResolver.finalizeShuffleMerge(
+ new FinalizeShuffleMerge(TEST_APP, 0));
+ assertEquals("index position", 24, testIndexFile.getPos());
+ validateMergeStatuses(statuses, new int[] {0}, new long[] {9});
+ MergedBlockMeta blockMeta = pushResolver.getMergedBlockMeta(TEST_APP, 0, 0);
+ validateChunks(TEST_APP, 0, 0, blockMeta, new int[] {4, 5}, new int[][] {{0}, {1}});
+ }
+
+ @Test
+ public void testRecoverMetaFileAfterIOExceptions() throws IOException {
+ useTestFiles(false, true);
+ RemoteBlockPushResolver.PushBlockStreamCallback callback1 =
+ (RemoteBlockPushResolver.PushBlockStreamCallback) pushResolver.receiveBlockDataAsStream(
+ new PushBlockStream(TEST_APP, 0, 0, 0, 0));
+ callback1.onData(callback1.getID(), ByteBuffer.wrap(new byte[4]));
+ callback1.onComplete(callback1.getID());
+ RemoteBlockPushResolver.AppShufflePartitionInfo partitionInfo = callback1.getPartitionInfo();
+ // Close the meta stream so it throws IOException
+ TestMergeShuffleFile testMetaFile = (TestMergeShuffleFile) partitionInfo.getMetaFile();
+ long metaPosBeforeClose = testMetaFile.getPos();
+ testMetaFile.close();
+ StreamCallbackWithID callback2 = pushResolver.receiveBlockDataAsStream(
+ new PushBlockStream(TEST_APP, 0, 1, 0, 0));
+ callback2.onData(callback2.getID(), ByteBuffer.wrap(new byte[5]));
+ // This will complete without any IOExceptions because number of IOExceptions are less than
+ // the threshold but the update to index and meta file will be unsuccessful.
+ callback2.onComplete(callback2.getID());
+ assertEquals("index position", 16, partitionInfo.getIndexFile().getPos());
+ assertEquals("meta position", metaPosBeforeClose, testMetaFile.getPos());
+ // Restore the meta stream so it can write successfully again.
+ testMetaFile.restore();
+ StreamCallbackWithID callback3 = pushResolver.receiveBlockDataAsStream(
+ new PushBlockStream(TEST_APP, 0, 2, 0, 0));
+ callback3.onData(callback3.getID(), ByteBuffer.wrap(new byte[2]));
+ callback3.onComplete(callback3.getID());
+ assertEquals("index position", 24, partitionInfo.getIndexFile().getPos());
+ assertTrue("meta position", testMetaFile.getPos() > metaPosBeforeClose);
+ MergeStatuses statuses = pushResolver.finalizeShuffleMerge(
+ new FinalizeShuffleMerge(TEST_APP, 0));
+ validateMergeStatuses(statuses, new int[] {0}, new long[] {11});
+ MergedBlockMeta blockMeta = pushResolver.getMergedBlockMeta(TEST_APP, 0, 0);
+ validateChunks(TEST_APP, 0, 0, blockMeta, new int[] {4, 7}, new int[][] {{0}, {1, 2}});
+ }
+
+ @Test
+ public void testRecoverMetaFileAfterIOExceptionsInFinalize() throws IOException {
+ useTestFiles(false, true);
+ RemoteBlockPushResolver.PushBlockStreamCallback callback1 =
+ (RemoteBlockPushResolver.PushBlockStreamCallback) pushResolver.receiveBlockDataAsStream(
+ new PushBlockStream(TEST_APP, 0, 0, 0, 0));
+ callback1.onData(callback1.getID(), ByteBuffer.wrap(new byte[4]));
+ callback1.onComplete(callback1.getID());
+ RemoteBlockPushResolver.AppShufflePartitionInfo partitionInfo = callback1.getPartitionInfo();
+ // Close the meta stream so it throws IOException
+ TestMergeShuffleFile testMetaFile = (TestMergeShuffleFile) partitionInfo.getMetaFile();
+ long metaPosBeforeClose = testMetaFile.getPos();
+ testMetaFile.close();
+ StreamCallbackWithID callback2 = pushResolver.receiveBlockDataAsStream(
+ new PushBlockStream(TEST_APP, 0, 1, 0, 0));
+ callback2.onData(callback2.getID(), ByteBuffer.wrap(new byte[5]));
+ // This will complete without any IOExceptions because number of IOExceptions are less than
+ // the threshold but the update to index and meta file will be unsuccessful.
+ callback2.onComplete(callback2.getID());
+ MergeShuffleFile indexFile = partitionInfo.getIndexFile();
+ assertEquals("index position", 16, indexFile.getPos());
+ assertEquals("meta position", metaPosBeforeClose, testMetaFile.getPos());
+ // Restore the meta stream so it can write successfully again.
+ testMetaFile.restore();
+ MergeStatuses statuses = pushResolver.finalizeShuffleMerge(
+ new FinalizeShuffleMerge(TEST_APP, 0));
+ assertEquals("index position", 24, indexFile.getPos());
+ assertTrue("meta position", testMetaFile.getPos() > metaPosBeforeClose);
+ validateMergeStatuses(statuses, new int[] {0}, new long[] {9});
+ MergedBlockMeta blockMeta = pushResolver.getMergedBlockMeta(TEST_APP, 0, 0);
+ validateChunks(TEST_APP, 0, 0, blockMeta, new int[] {4, 5}, new int[][] {{0}, {1}});
+ }
+
+ @Test (expected = RuntimeException.class)
+ public void testIOExceptionsExceededThreshold() throws IOException {
+ RemoteBlockPushResolver.PushBlockStreamCallback callback =
+ (RemoteBlockPushResolver.PushBlockStreamCallback) pushResolver.receiveBlockDataAsStream(
+ new PushBlockStream(TEST_APP, 0, 0, 0, 0));
+ RemoteBlockPushResolver.AppShufflePartitionInfo partitionInfo = callback.getPartitionInfo();
+ callback.onData(callback.getID(), ByteBuffer.wrap(new byte[4]));
+ callback.onComplete(callback.getID());
+ // Close the data stream so it throws continuous IOException
+ partitionInfo.getDataChannel().close();
+ for (int i = 1; i < 5; i++) {
+ RemoteBlockPushResolver.PushBlockStreamCallback callback1 =
+ (RemoteBlockPushResolver.PushBlockStreamCallback) pushResolver.receiveBlockDataAsStream(
+ new PushBlockStream(TEST_APP, 0, i, 0, 0));
+ try {
+ callback1.onData(callback1.getID(), ByteBuffer.wrap(new byte[2]));
+ } catch (IOException ioe) {
+ // this will throw IOException so the client can retry.
+ callback1.onFailure(callback1.getID(), ioe);
+ }
+ }
+ assertEquals(4, partitionInfo.getNumIOExceptions());
+ // After 4 IOException, the server will respond with IOExceptions exceeded threshold
+ try {
+ RemoteBlockPushResolver.PushBlockStreamCallback callback2 =
+ (RemoteBlockPushResolver.PushBlockStreamCallback) pushResolver.receiveBlockDataAsStream(
+ new PushBlockStream(TEST_APP, 0, 5, 0, 0));
+ callback2.onData(callback.getID(), ByteBuffer.wrap(new byte[1]));
+ } catch (Throwable t) {
+ assertEquals("IOExceptions exceeded the threshold when merging shufflePush_0_5_0",
+ t.getMessage());
+ throw t;
+ }
Review comment:
> I actually mean those blocks which are waiting for the partition lock at the same time.
I see. That's a good point. The server would attempt to merge them. If they don't see any which is unlikely they will be merged. Do you think we should check before writing them?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #30433: [SPARK-32916][SHUFFLE][test-maven][test-hadoop2.7] Ensure the number of chunks in meta file and index file are equal
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #30433:
URL: https://github.com/apache/spark/pull/30433#issuecomment-732447775
**[Test build #131575 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/131575/testReport)** for PR 30433 at commit [`30c8f83`](https://github.com/apache/spark/commit/30c8f83b7ba3e39724c90865fd1318b41876f927).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] otterc commented on a change in pull request #30433: [SPARK-32916][SHUFFLE][test-maven][test-hadoop2.7] Ensure the number of chunks in meta file and index file are equal
Posted by GitBox <gi...@apache.org>.
otterc commented on a change in pull request #30433:
URL: https://github.com/apache/spark/pull/30433#discussion_r528507583
##########
File path: common/network-shuffle/src/main/java/org/apache/spark/network/shuffle/RemoteBlockPushResolver.java
##########
@@ -827,13 +833,16 @@ void resetChunkTracker() {
void updateChunkInfo(long chunkOffset, int mapIndex) throws IOException {
long idxStartPos = -1;
try {
- // update the chunk tracker to meta file before index file
- writeChunkTracker(mapIndex);
idxStartPos = indexFile.getFilePointer();
logger.trace("{} shuffleId {} reduceId {} updated index current {} updated {}",
appShuffleId.appId, appShuffleId.shuffleId, reduceId, this.lastChunkOffset,
chunkOffset);
- indexFile.writeLong(chunkOffset);
+ indexFile.write(Longs.toByteArray(chunkOffset));
+ // Chunk bitmap should be written to the meta file after the index file because if there are
+ // any exceptions during writing the offset to the index file, meta file should not be
+ // updated. If the update to the index file is successful but the update to meta file isn't
+ // then the index file position is reset in the catch clause.
Review comment:
We don't need to update the `lastChunkOffset`. Since the current chunk offset was not successfully written to the index file, the `lastChunkOffset` would stay the same. The last chunk offset indicates the successfully written offset of the last chunk.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] Victsm commented on a change in pull request #30433: [SPARK-32916][SHUFFLE][test-maven][test-hadoop2.7] Ensure the number of chunks in meta file and index file are equal
Posted by GitBox <gi...@apache.org>.
Victsm commented on a change in pull request #30433:
URL: https://github.com/apache/spark/pull/30433#discussion_r530564551
##########
File path: common/network-shuffle/src/main/java/org/apache/spark/network/shuffle/RemoteBlockPushResolver.java
##########
@@ -827,13 +833,16 @@ void resetChunkTracker() {
void updateChunkInfo(long chunkOffset, int mapIndex) throws IOException {
long idxStartPos = -1;
try {
- // update the chunk tracker to meta file before index file
- writeChunkTracker(mapIndex);
idxStartPos = indexFile.getFilePointer();
logger.trace("{} shuffleId {} reduceId {} updated index current {} updated {}",
appShuffleId.appId, appShuffleId.shuffleId, reduceId, this.lastChunkOffset,
chunkOffset);
- indexFile.writeLong(chunkOffset);
+ indexFile.write(Longs.toByteArray(chunkOffset));
+ // Chunk bitmap should be written to the meta file after the index file because if there are
+ // any exceptions during writing the offset to the index file, meta file should not be
+ // updated. If the update to the index file is successful but the update to meta file isn't
+ // then the index file position is reset in the catch clause.
+ writeChunkTracker(mapIndex);
Review comment:
For dynamically changing merger locations for the current push, if the non-recoverable IOException happen after majority of the blocks have already been merged, we won't gain much by pushing the remaining blocks in this shuffle targeted to the failed shuffle service to a new location. Probably need to check the merge ratio before deciding whether to do so or not. The current framework is designed around the best-effort nature of the block push operation, which simplifies the handling of these failure scenarios.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] otterc commented on pull request #30433: [SPARK-32916][SHUFFLE][test-maven][test-hadoop2.7] Ensure the number of chunks in meta file and index file are equal
Posted by GitBox <gi...@apache.org>.
otterc commented on pull request #30433:
URL: https://github.com/apache/spark/pull/30433#issuecomment-748521321
> The changes look good to me. Is there anything else pending here @otterc ?
No, I have addressed all the review comments so far.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] Ngone51 commented on a change in pull request #30433: [SPARK-32916][SHUFFLE][test-maven][test-hadoop2.7] Ensure the number of chunks in meta file and index file are equal
Posted by GitBox <gi...@apache.org>.
Ngone51 commented on a change in pull request #30433:
URL: https://github.com/apache/spark/pull/30433#discussion_r528466840
##########
File path: common/network-shuffle/src/main/java/org/apache/spark/network/shuffle/RemoteBlockPushResolver.java
##########
@@ -827,13 +833,16 @@ void resetChunkTracker() {
void updateChunkInfo(long chunkOffset, int mapIndex) throws IOException {
long idxStartPos = -1;
try {
- // update the chunk tracker to meta file before index file
- writeChunkTracker(mapIndex);
idxStartPos = indexFile.getFilePointer();
logger.trace("{} shuffleId {} reduceId {} updated index current {} updated {}",
appShuffleId.appId, appShuffleId.shuffleId, reduceId, this.lastChunkOffset,
chunkOffset);
- indexFile.writeLong(chunkOffset);
+ indexFile.write(Longs.toByteArray(chunkOffset));
+ // Chunk bitmap should be written to the meta file after the index file because if there are
+ // any exceptions during writing the offset to the index file, meta file should not be
+ // updated. If the update to the index file is successful but the update to meta file isn't
+ // then the index file position is reset in the catch clause.
Review comment:
What about L855 (`this.lastChunkOffset = chunkOffset;`)? Do we need to update `lastChunkOffset` when the exception throws during writing the offset to the index file?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #30433: [SPARK-32916][SHUFFLE][test-maven][test-hadoop2.7] Ensure the number of chunks in meta file and index file are equal
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #30433:
URL: https://github.com/apache/spark/pull/30433#issuecomment-732504512
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] Victsm commented on a change in pull request #30433: [SPARK-32916][SHUFFLE][test-maven][test-hadoop2.7] Ensure the number of chunks in meta file and index file are equal
Posted by GitBox <gi...@apache.org>.
Victsm commented on a change in pull request #30433:
URL: https://github.com/apache/spark/pull/30433#discussion_r539086277
##########
File path: common/network-shuffle/src/main/java/org/apache/spark/network/shuffle/RemoteBlockPushResolver.java
##########
@@ -586,6 +592,7 @@ public void onData(String streamId, ByteBuffer buf) throws IOException {
deferredBufs = null;
return;
}
+ abortIfNecessary();
Review comment:
Purpose is to only do this check once per block instead of once per buf.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #30433: [SPARK-32916][SHUFFLE][test-maven][test-hadoop2.7] Ensure the number of chunks in meta file and index file are equal
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #30433:
URL: https://github.com/apache/spark/pull/30433#issuecomment-738314413
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/36759/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #30433: [SPARK-32916][SHUFFLE][test-maven][test-hadoop2.7] Ensure the number of chunks in meta file and index file are equal
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #30433:
URL: https://github.com/apache/spark/pull/30433#issuecomment-738284550
**[Test build #132158 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/132158/testReport)** for PR 30433 at commit [`72b5976`](https://github.com/apache/spark/commit/72b5976579aa959689aaa78f1b8a841d6c9a09e1).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #30433: [SPARK-32916][SHUFFLE][test-maven][test-hadoop2.7] Ensure the number of chunks in meta file and index file are equal
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #30433:
URL: https://github.com/apache/spark/pull/30433#issuecomment-738336357
Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/36759/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] mridulm edited a comment on pull request #30433: [SPARK-32916][SHUFFLE][test-maven][test-hadoop2.7] Ensure the number of chunks in meta file and index file are equal
Posted by GitBox <gi...@apache.org>.
mridulm edited a comment on pull request #30433:
URL: https://github.com/apache/spark/pull/30433#issuecomment-731555135
> Any particular reason for this change ? writeLong should be able to do it without the intermediate byte[] creation ?
Scratch that, missed it in the description
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #30433: [SPARK-32916][SHUFFLE][test-maven][test-hadoop2.7] Ensure the number of chunks in meta file and index file are equal
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #30433:
URL: https://github.com/apache/spark/pull/30433#issuecomment-741092475
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/37050/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] Ngone51 commented on a change in pull request #30433: [SPARK-32916][SHUFFLE][test-maven][test-hadoop2.7] Ensure the number of chunks in meta file and index file are equal
Posted by GitBox <gi...@apache.org>.
Ngone51 commented on a change in pull request #30433:
URL: https://github.com/apache/spark/pull/30433#discussion_r538991760
##########
File path: common/network-shuffle/src/test/java/org/apache/spark/network/shuffle/RemoteBlockPushResolverSuite.java
##########
@@ -411,6 +415,269 @@ void deleteExecutorDirs(Path[] dirs) {
}
}
+ @Test
+ public void testRecoverIndexFileAfterIOExceptions() throws IOException {
+ useTestFiles(true, false);
+ RemoteBlockPushResolver.PushBlockStreamCallback callback1 =
+ (RemoteBlockPushResolver.PushBlockStreamCallback) pushResolver.receiveBlockDataAsStream(
+ new PushBlockStream(TEST_APP, 0, 0, 0, 0));
+ callback1.onData(callback1.getID(), ByteBuffer.wrap(new byte[4]));
+ callback1.onComplete(callback1.getID());
+ RemoteBlockPushResolver.AppShufflePartitionInfo partitionInfo = callback1.getPartitionInfo();
+ // Close the index stream so it throws IOException
+ TestMergeShuffleFile testIndexFile = (TestMergeShuffleFile) partitionInfo.getIndexFile();
+ testIndexFile.close();
+ StreamCallbackWithID callback2 = pushResolver.receiveBlockDataAsStream(
+ new PushBlockStream(TEST_APP, 0, 1, 0, 0));
+ callback2.onData(callback2.getID(), ByteBuffer.wrap(new byte[5]));
+ // This will complete without any IOExceptions because number of IOExceptions are less than
+ // the threshold but the update to index file will be unsuccessful.
+ callback2.onComplete(callback2.getID());
+ assertEquals("index position", 16, testIndexFile.getPos());
+ // Restore the index stream so it can write successfully again.
+ testIndexFile.restore();
+ StreamCallbackWithID callback3 = pushResolver.receiveBlockDataAsStream(
+ new PushBlockStream(TEST_APP, 0, 2, 0, 0));
+ callback3.onData(callback3.getID(), ByteBuffer.wrap(new byte[2]));
+ callback3.onComplete(callback3.getID());
+ assertEquals("index position", 24, testIndexFile.getPos());
+ MergeStatuses statuses = pushResolver.finalizeShuffleMerge(
+ new FinalizeShuffleMerge(TEST_APP, 0));
+ validateMergeStatuses(statuses, new int[] {0}, new long[] {11});
+ MergedBlockMeta blockMeta = pushResolver.getMergedBlockMeta(TEST_APP, 0, 0);
+ validateChunks(TEST_APP, 0, 0, blockMeta, new int[] {4, 7}, new int[][] {{0}, {1, 2}});
+ }
+
+ @Test
+ public void testRecoverIndexFileAfterIOExceptionsInFinalize() throws IOException {
+ useTestFiles(true, false);
+ registerExecutor(TEST_APP, prepareLocalDirs(localDirs));
+ RemoteBlockPushResolver.PushBlockStreamCallback callback1 =
+ (RemoteBlockPushResolver.PushBlockStreamCallback) pushResolver.receiveBlockDataAsStream(
+ new PushBlockStream(TEST_APP, 0, 0, 0, 0));
+ callback1.onData(callback1.getID(), ByteBuffer.wrap(new byte[4]));
+ callback1.onComplete(callback1.getID());
+ RemoteBlockPushResolver.AppShufflePartitionInfo partitionInfo = callback1.getPartitionInfo();
+ // Close the index stream so it throws IOException
+ TestMergeShuffleFile testIndexFile = (TestMergeShuffleFile) partitionInfo.getIndexFile();
+ testIndexFile.close();
+ StreamCallbackWithID callback2 = pushResolver.receiveBlockDataAsStream(
+ new PushBlockStream(TEST_APP, 0, 1, 0, 0));
+ callback2.onData(callback2.getID(), ByteBuffer.wrap(new byte[5]));
+ // This will complete without any IOExceptions because number of IOExceptions are less than
+ // the threshold but the update to index file will be unsuccessful.
+ callback2.onComplete(callback2.getID());
+ assertEquals("index position", 16, testIndexFile.getPos());
+ // The last update to index was unsuccessful however any further updates will be successful.
+ // Restore the index stream so it can write successfully again.
+ testIndexFile.restore();
+ MergeStatuses statuses = pushResolver.finalizeShuffleMerge(
+ new FinalizeShuffleMerge(TEST_APP, 0));
+ assertEquals("index position", 24, testIndexFile.getPos());
+ validateMergeStatuses(statuses, new int[] {0}, new long[] {9});
+ MergedBlockMeta blockMeta = pushResolver.getMergedBlockMeta(TEST_APP, 0, 0);
+ validateChunks(TEST_APP, 0, 0, blockMeta, new int[] {4, 5}, new int[][] {{0}, {1}});
+ }
+
+ @Test
+ public void testRecoverMetaFileAfterIOExceptions() throws IOException {
+ useTestFiles(false, true);
+ RemoteBlockPushResolver.PushBlockStreamCallback callback1 =
+ (RemoteBlockPushResolver.PushBlockStreamCallback) pushResolver.receiveBlockDataAsStream(
+ new PushBlockStream(TEST_APP, 0, 0, 0, 0));
+ callback1.onData(callback1.getID(), ByteBuffer.wrap(new byte[4]));
+ callback1.onComplete(callback1.getID());
+ RemoteBlockPushResolver.AppShufflePartitionInfo partitionInfo = callback1.getPartitionInfo();
+ // Close the meta stream so it throws IOException
+ TestMergeShuffleFile testMetaFile = (TestMergeShuffleFile) partitionInfo.getMetaFile();
+ long metaPosBeforeClose = testMetaFile.getPos();
+ testMetaFile.close();
+ StreamCallbackWithID callback2 = pushResolver.receiveBlockDataAsStream(
+ new PushBlockStream(TEST_APP, 0, 1, 0, 0));
+ callback2.onData(callback2.getID(), ByteBuffer.wrap(new byte[5]));
+ // This will complete without any IOExceptions because number of IOExceptions are less than
+ // the threshold but the update to index and meta file will be unsuccessful.
+ callback2.onComplete(callback2.getID());
+ assertEquals("index position", 16, partitionInfo.getIndexFile().getPos());
+ assertEquals("meta position", metaPosBeforeClose, testMetaFile.getPos());
+ // Restore the meta stream so it can write successfully again.
+ testMetaFile.restore();
+ StreamCallbackWithID callback3 = pushResolver.receiveBlockDataAsStream(
+ new PushBlockStream(TEST_APP, 0, 2, 0, 0));
+ callback3.onData(callback3.getID(), ByteBuffer.wrap(new byte[2]));
+ callback3.onComplete(callback3.getID());
+ assertEquals("index position", 24, partitionInfo.getIndexFile().getPos());
+ assertTrue("meta position", testMetaFile.getPos() > metaPosBeforeClose);
+ MergeStatuses statuses = pushResolver.finalizeShuffleMerge(
+ new FinalizeShuffleMerge(TEST_APP, 0));
+ validateMergeStatuses(statuses, new int[] {0}, new long[] {11});
+ MergedBlockMeta blockMeta = pushResolver.getMergedBlockMeta(TEST_APP, 0, 0);
+ validateChunks(TEST_APP, 0, 0, blockMeta, new int[] {4, 7}, new int[][] {{0}, {1, 2}});
+ }
+
+ @Test
+ public void testRecoverMetaFileAfterIOExceptionsInFinalize() throws IOException {
+ useTestFiles(false, true);
+ RemoteBlockPushResolver.PushBlockStreamCallback callback1 =
+ (RemoteBlockPushResolver.PushBlockStreamCallback) pushResolver.receiveBlockDataAsStream(
+ new PushBlockStream(TEST_APP, 0, 0, 0, 0));
+ callback1.onData(callback1.getID(), ByteBuffer.wrap(new byte[4]));
+ callback1.onComplete(callback1.getID());
+ RemoteBlockPushResolver.AppShufflePartitionInfo partitionInfo = callback1.getPartitionInfo();
+ // Close the meta stream so it throws IOException
+ TestMergeShuffleFile testMetaFile = (TestMergeShuffleFile) partitionInfo.getMetaFile();
+ long metaPosBeforeClose = testMetaFile.getPos();
+ testMetaFile.close();
+ StreamCallbackWithID callback2 = pushResolver.receiveBlockDataAsStream(
+ new PushBlockStream(TEST_APP, 0, 1, 0, 0));
+ callback2.onData(callback2.getID(), ByteBuffer.wrap(new byte[5]));
+ // This will complete without any IOExceptions because number of IOExceptions are less than
+ // the threshold but the update to index and meta file will be unsuccessful.
+ callback2.onComplete(callback2.getID());
+ MergeShuffleFile indexFile = partitionInfo.getIndexFile();
+ assertEquals("index position", 16, indexFile.getPos());
+ assertEquals("meta position", metaPosBeforeClose, testMetaFile.getPos());
+ // Restore the meta stream so it can write successfully again.
+ testMetaFile.restore();
+ MergeStatuses statuses = pushResolver.finalizeShuffleMerge(
+ new FinalizeShuffleMerge(TEST_APP, 0));
+ assertEquals("index position", 24, indexFile.getPos());
+ assertTrue("meta position", testMetaFile.getPos() > metaPosBeforeClose);
+ validateMergeStatuses(statuses, new int[] {0}, new long[] {9});
+ MergedBlockMeta blockMeta = pushResolver.getMergedBlockMeta(TEST_APP, 0, 0);
+ validateChunks(TEST_APP, 0, 0, blockMeta, new int[] {4, 5}, new int[][] {{0}, {1}});
+ }
+
+ @Test (expected = RuntimeException.class)
+ public void testIOExceptionsExceededThreshold() throws IOException {
+ RemoteBlockPushResolver.PushBlockStreamCallback callback =
+ (RemoteBlockPushResolver.PushBlockStreamCallback) pushResolver.receiveBlockDataAsStream(
+ new PushBlockStream(TEST_APP, 0, 0, 0, 0));
+ RemoteBlockPushResolver.AppShufflePartitionInfo partitionInfo = callback.getPartitionInfo();
+ callback.onData(callback.getID(), ByteBuffer.wrap(new byte[4]));
+ callback.onComplete(callback.getID());
+ // Close the data stream so it throws continuous IOException
+ partitionInfo.getDataChannel().close();
+ for (int i = 1; i < 5; i++) {
+ RemoteBlockPushResolver.PushBlockStreamCallback callback1 =
+ (RemoteBlockPushResolver.PushBlockStreamCallback) pushResolver.receiveBlockDataAsStream(
+ new PushBlockStream(TEST_APP, 0, i, 0, 0));
+ try {
+ callback1.onData(callback1.getID(), ByteBuffer.wrap(new byte[2]));
+ } catch (IOException ioe) {
+ // this will throw IOException so the client can retry.
+ callback1.onFailure(callback1.getID(), ioe);
+ }
+ }
+ assertEquals(4, partitionInfo.getNumIOExceptions());
+ // After 4 IOException, the server will respond with IOExceptions exceeded threshold
+ try {
+ RemoteBlockPushResolver.PushBlockStreamCallback callback2 =
+ (RemoteBlockPushResolver.PushBlockStreamCallback) pushResolver.receiveBlockDataAsStream(
+ new PushBlockStream(TEST_APP, 0, 5, 0, 0));
+ callback2.onData(callback.getID(), ByteBuffer.wrap(new byte[1]));
+ } catch (Throwable t) {
+ assertEquals("IOExceptions exceeded the threshold when merging shufflePush_0_5_0",
+ t.getMessage());
+ throw t;
+ }
Review comment:
> Do you think we should check before writing them?
I think so. And stop writing when the partition already hits IO exceptions threshold.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #30433: [SPARK-32916][SHUFFLE][test-maven][test-hadoop2.7] Ensure the number of chunks in meta file and index file are equal
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #30433:
URL: https://github.com/apache/spark/pull/30433#issuecomment-737721310
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #30433: [SPARK-32916][SHUFFLE][test-maven][test-hadoop2.7] Ensure the number of chunks in meta file and index file are equal
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #30433:
URL: https://github.com/apache/spark/pull/30433#issuecomment-732475511
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] mridulm commented on pull request #30433: [SPARK-32916][SHUFFLE][test-maven][test-hadoop2.7] Ensure the number of chunks in meta file and index file are equal
Posted by GitBox <gi...@apache.org>.
mridulm commented on pull request #30433:
URL: https://github.com/apache/spark/pull/30433#issuecomment-750426404
Thanks for fixing this @otterc !
And thanks for the reviews @Ngone51, @Victsm :-)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #30433: [SPARK-32916][SHUFFLE][test-maven][test-hadoop2.7] Ensure the number of chunks in meta file and index file are equal
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #30433:
URL: https://github.com/apache/spark/pull/30433#issuecomment-741995704
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/37109/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] otterc commented on a change in pull request #30433: [SPARK-32916][SHUFFLE][test-maven][test-hadoop2.7] Ensure the number of chunks in meta file and index file are equal
Posted by GitBox <gi...@apache.org>.
otterc commented on a change in pull request #30433:
URL: https://github.com/apache/spark/pull/30433#discussion_r537778170
##########
File path: common/network-shuffle/src/main/java/org/apache/spark/network/shuffle/RemoteBlockPushResolver.java
##########
@@ -17,6 +17,7 @@
package org.apache.spark.network.shuffle;
+import com.google.common.primitives.Longs;
Review comment:
done
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #30433: [SPARK-32916][SHUFFLE][test-maven][test-hadoop2.7] Ensure the number of chunks in meta file and index file are equal
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #30433:
URL: https://github.com/apache/spark/pull/30433#issuecomment-742198864
**[Test build #132507 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/132507/testReport)** for PR 30433 at commit [`f48d52f`](https://github.com/apache/spark/commit/f48d52f68e52b63742e7cce5dbb4fbcc89845402).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] otterc commented on a change in pull request #30433: [SPARK-32916][SHUFFLE][test-maven][test-hadoop2.7] Ensure the number of chunks in meta file and index file are equal
Posted by GitBox <gi...@apache.org>.
otterc commented on a change in pull request #30433:
URL: https://github.com/apache/spark/pull/30433#discussion_r528519186
##########
File path: common/network-shuffle/src/main/java/org/apache/spark/network/shuffle/RemoteBlockPushResolver.java
##########
@@ -827,13 +833,16 @@ void resetChunkTracker() {
void updateChunkInfo(long chunkOffset, int mapIndex) throws IOException {
long idxStartPos = -1;
try {
- // update the chunk tracker to meta file before index file
- writeChunkTracker(mapIndex);
idxStartPos = indexFile.getFilePointer();
logger.trace("{} shuffleId {} reduceId {} updated index current {} updated {}",
appShuffleId.appId, appShuffleId.shuffleId, reduceId, this.lastChunkOffset,
chunkOffset);
- indexFile.writeLong(chunkOffset);
+ indexFile.write(Longs.toByteArray(chunkOffset));
+ // Chunk bitmap should be written to the meta file after the index file because if there are
+ // any exceptions during writing the offset to the index file, meta file should not be
+ // updated. If the update to the index file is successful but the update to meta file isn't
+ // then the index file position is reset in the catch clause.
+ writeChunkTracker(mapIndex);
Review comment:
> I think we should catch the chunk level IOExceptions inside onComplete, setting certain flags to make the onFailure handling logic know about whether we encountered a block level failure or a chunk level failure.
For a chunk level failure, we shouldn't overwrite the previous block, we should effectively only delay the closure of the chunk.
The closure of chunk write now depends on the minimum size of the chunk. If we keep growing the chunk then eventually it will be very large and the executors will get killed fetching them because they will exceed the physical memory limit. This was the main reason to serve data in manageable-sized chunks.
> Stopping merging blocks only after 2 consecutive IOExceptions seems to be giving up too early.
With local file systems the IOExceptions that I can think of would be due to disk failures, file corruption, permission and these will not go away when retrying to update these files. Are there any other scenarios that I am missing?
> If there's a need to stop early due to too many disk failures, I think that logic should be on the client side inside ErrorHandler.
Sure, however the server here can respond with a similar runtime exception that it can't update merge file like the `TooLate` runtime exception and the error handle for the push side can stop pushing the data.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #30433: [SPARK-32916][SHUFFLE][test-maven][test-hadoop2.7] Ensure the number of chunks in meta file and index file are equal
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #30433:
URL: https://github.com/apache/spark/pull/30433#issuecomment-740998644
**[Test build #132448 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/132448/testReport)** for PR 30433 at commit [`5422f10`](https://github.com/apache/spark/commit/5422f100a197d96f646864c75fc1a0beb2d41af6).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] Victsm commented on a change in pull request #30433: [SPARK-32916][SHUFFLE][test-maven][test-hadoop2.7] Ensure the number of chunks in meta file and index file are equal
Posted by GitBox <gi...@apache.org>.
Victsm commented on a change in pull request #30433:
URL: https://github.com/apache/spark/pull/30433#discussion_r530817151
##########
File path: common/network-shuffle/src/main/java/org/apache/spark/network/shuffle/RemoteBlockPushResolver.java
##########
@@ -877,7 +886,7 @@ void closeAllFiles() {
}
Review comment:
Do we want to truncate data file as well to the length recorded by the last offset in the index file?
https://github.com/apache/spark/blob/dfa3978d9191e02eabf65d1829c970644d25d57e/common/network-shuffle/src/main/java/org/apache/spark/network/shuffle/RemoteBlockPushResolver.java#L869-L871
BTW, just realized this limitation in GitHub (https://github.com/isaacs/github/issues/284).
People know any workaround?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] Ngone51 commented on a change in pull request #30433: [SPARK-32916][SHUFFLE][test-maven][test-hadoop2.7] Ensure the number of chunks in meta file and index file are equal
Posted by GitBox <gi...@apache.org>.
Ngone51 commented on a change in pull request #30433:
URL: https://github.com/apache/spark/pull/30433#discussion_r540978417
##########
File path: common/network-shuffle/src/main/java/org/apache/spark/network/shuffle/RemoteBlockPushResolver.java
##########
@@ -586,6 +592,7 @@ public void onData(String streamId, ByteBuffer buf) throws IOException {
deferredBufs = null;
return;
}
+ abortIfNecessary();
Review comment:
I'm wondering does the server closes the channel or client would stop streaming the remaining data when we throw the exception in `onData`. Otherwise, we'd receive the following data from the client and throw the exception for multiple times.
(I didn't find anywhere we close the channel or the client may have a chance to stop streaming. I may miss it somewhere.)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] otterc commented on a change in pull request #30433: [SPARK-32916][SHUFFLE][test-maven][test-hadoop2.7] Ensure the number of chunks in meta file and index file are equal
Posted by GitBox <gi...@apache.org>.
otterc commented on a change in pull request #30433:
URL: https://github.com/apache/spark/pull/30433#discussion_r539630414
##########
File path: common/network-shuffle/src/main/java/org/apache/spark/network/shuffle/RemoteBlockPushResolver.java
##########
@@ -586,6 +592,7 @@ public void onData(String streamId, ByteBuffer buf) throws IOException {
deferredBufs = null;
return;
}
+ abortIfNecessary();
Review comment:
hmmm. I think the intention is to not have the server attempt writing if the threshold is reached for the partition. Probably this check here makes this behavior more accurate.
However, I don't have a strong opinion on this since the assumption is that if these number of IOExceptions have already reached the threshold, any further writes will result in an IOException as well. With that assumption, the write in `onData` after this threshold is met will very likely throw `IOException` as well and since the threshold is already met, the server will instead throw `IOExceptions exceeded threshold`.
I can remove it from `onData`. @Ngone51 What do you think?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] mridulm commented on a change in pull request #30433: [SPARK-32916][SHUFFLE][test-maven][test-hadoop2.7] Ensure the number of chunks in meta file and index file are equal
Posted by GitBox <gi...@apache.org>.
mridulm commented on a change in pull request #30433:
URL: https://github.com/apache/spark/pull/30433#discussion_r528175549
##########
File path: common/network-shuffle/src/main/java/org/apache/spark/network/shuffle/RemoteBlockPushResolver.java
##########
@@ -827,13 +833,16 @@ void resetChunkTracker() {
void updateChunkInfo(long chunkOffset, int mapIndex) throws IOException {
long idxStartPos = -1;
try {
- // update the chunk tracker to meta file before index file
- writeChunkTracker(mapIndex);
idxStartPos = indexFile.getFilePointer();
logger.trace("{} shuffleId {} reduceId {} updated index current {} updated {}",
appShuffleId.appId, appShuffleId.shuffleId, reduceId, this.lastChunkOffset,
chunkOffset);
- indexFile.writeLong(chunkOffset);
+ indexFile.write(Longs.toByteArray(chunkOffset));
+ // Chunk bitmap should be written to the meta file after the index file because if there are
+ // any exceptions during writing the offset to the index file, meta file should not be
+ // updated. If the update to the index file is successful but the update to meta file isn't
+ // then the index file position is reset in the catch clause.
+ writeChunkTracker(mapIndex);
Review comment:
Given the focus here is error handling, how are we handling a `seek` failure in `catch` block ? (Here and elsewhere in this PR)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] Ngone51 commented on a change in pull request #30433: [SPARK-32916][SHUFFLE][test-maven][test-hadoop2.7] Ensure the number of chunks in meta file and index file are equal
Posted by GitBox <gi...@apache.org>.
Ngone51 commented on a change in pull request #30433:
URL: https://github.com/apache/spark/pull/30433#discussion_r539855837
##########
File path: common/network-shuffle/src/main/java/org/apache/spark/network/shuffle/RemoteBlockPushResolver.java
##########
@@ -586,6 +592,7 @@ public void onData(String streamId, ByteBuffer buf) throws IOException {
deferredBufs = null;
return;
}
+ abortIfNecessary();
Review comment:
It's OK if the further writes throw IOExceptions. I just worry, can we handle it correctly if writes succeed? Or it may lead to some undefined situation which could avoid later shuffle fetch in turn.
So I'd like to avoid any writes in the first place (`onData`) so we ensure that we don't go into any undefined situation. And I think, maybe, we can update it a little, like this - we still receive data in `onData` but just not write when the partition already reach IO threshold and only call `abortIfNecessary` inside `onCompelte`.
WDYT? @Victsm @otterc
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] otterc commented on a change in pull request #30433: [SPARK-32916][SHUFFLE][test-maven][test-hadoop2.7] Ensure the number of chunks in meta file and index file are equal
Posted by GitBox <gi...@apache.org>.
otterc commented on a change in pull request #30433:
URL: https://github.com/apache/spark/pull/30433#discussion_r534352437
##########
File path: common/network-shuffle/src/main/java/org/apache/spark/network/shuffle/RemoteBlockPushResolver.java
##########
@@ -877,7 +886,7 @@ void closeAllFiles() {
}
Review comment:
Yes, but I think truncating the data file twice while finalizing is unnecessary. Also we can't simply ignore the exceptions while truncating all these 3 files during close. I think truncation of these files should be separated from `closeAllFiles`.
This is the change I have made:
```
public MergeStatuses finalizeShuffleMerge(FinalizeShuffleMerge msg) throws IOException {
...
while (partitionsIter.hasNext()) {
AppShufflePartitionInfo partition = partitionsIter.next();
synchronized (partition) {
try {
partition.finalizePartition();
bitmaps[idx] = partition.mapTracker;
reduceIds[idx] = partition.reduceId;
sizes[idx++] = partition.getPosition();
} catch (IOException ioe) {
logger.warn("Exception while finalizing shuffle partition {} {} {}", msg.appId,
msg.shuffleId, partition.reduceId, ioe);
} finally {
partition.closeAllFiles();
// The partition should be removed after the files are written so that any new stream
// for the same reduce partition will see that the data file exists.
partitionsIter.remove();
}
}
}
mergeStatuses = new MergeStatuses(msg.shuffleId, bitmaps, reduceIds, sizes);
}
partitions.remove(appShuffleId);
logger.info("Finalized shuffle {} from Application {}.", msg.shuffleId, msg.appId);
return mergeStatuses;
}
private void finalizePartition() throws IOException {
if (position != lastChunkOffset) {
try {
updateChunkInfo(position, lastMergedMapIndex);
} catch (IOException ioe) {
if (seekFailed) {
// If seek has failed then fail finalize of this partition by propagating the exception.
throw ioe;
}
// If seek hasn't failed then ignore this exception.
}
}
// Get rid of any partial block data at the end of the file. This could either
// be due to failure or a request still being processed when the shuffle
// merge gets finalized.
dataChannel.truncate(lastChunkOffset);
metaFile.setLength(metaFile.getFilePointer());
indexFile.setLength(indexFile.getFilePointer());
}
```
Any exception while truncating data/meta/index file during finalize will propagate the exception to the caller and that partition will not be considered merged.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] otterc commented on a change in pull request #30433: [SPARK-32916][SHUFFLE][test-maven][test-hadoop2.7] Ensure the number of chunks in meta file and index file are equal
Posted by GitBox <gi...@apache.org>.
otterc commented on a change in pull request #30433:
URL: https://github.com/apache/spark/pull/30433#discussion_r539504808
##########
File path: common/network-shuffle/src/main/java/org/apache/spark/network/shuffle/RemoteBlockPushResolver.java
##########
@@ -674,7 +683,10 @@ public void onComplete(String streamId) throws IOException {
}
if (partitionInfo.getCurrentMapIndex() < 0) {
try {
- writeAnyDeferredBufs();
+ if (deferredBufs != null && !deferredBufs.isEmpty()) {
+ abortIfNecessary();
Review comment:
I was thinking that if a stream is completing but doesn't need to write any deferred bufs then we should let it complete without failures.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #30433: [SPARK-32916][SHUFFLE][test-maven][test-hadoop2.7] Ensure the number of chunks in meta file and index file are equal
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #30433:
URL: https://github.com/apache/spark/pull/30433#issuecomment-741521841
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/132448/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] otterc commented on a change in pull request #30433: [SPARK-32916][SHUFFLE][test-maven][test-hadoop2.7] Ensure the number of chunks in meta file and index file are equal
Posted by GitBox <gi...@apache.org>.
otterc commented on a change in pull request #30433:
URL: https://github.com/apache/spark/pull/30433#discussion_r528229428
##########
File path: common/network-shuffle/src/main/java/org/apache/spark/network/shuffle/RemoteBlockPushResolver.java
##########
@@ -827,13 +833,16 @@ void resetChunkTracker() {
void updateChunkInfo(long chunkOffset, int mapIndex) throws IOException {
long idxStartPos = -1;
try {
- // update the chunk tracker to meta file before index file
- writeChunkTracker(mapIndex);
idxStartPos = indexFile.getFilePointer();
logger.trace("{} shuffleId {} reduceId {} updated index current {} updated {}",
appShuffleId.appId, appShuffleId.shuffleId, reduceId, this.lastChunkOffset,
chunkOffset);
- indexFile.writeLong(chunkOffset);
+ indexFile.write(Longs.toByteArray(chunkOffset));
+ // Chunk bitmap should be written to the meta file after the index file because if there are
+ // any exceptions during writing the offset to the index file, meta file should not be
+ // updated. If the update to the index file is successful but the update to meta file isn't
+ // then the index file position is reset in the catch clause.
+ writeChunkTracker(mapIndex);
Review comment:
This is a good point. I have overlooked IOExceptions from the seeks. Any IOException from the seek would not guarantee that the position is updated. So any more updates to the file may not overwrite the corrupt data which means that the files are corrupted.
One way of handling this I think is maintain the expected length of both index and meta files in AppShufflePartitionInfo instead of relying on the FilePointer. Updates to these files would be similar to writing to the merged data file as below:
```
if (partitionInfo.isEncounteredFailure()) {
long updatedPos = partitionInfo.getPosition() + length;
logger.debug(
"{} shuffleId {} reduceId {} encountered failure current pos {} updated pos {}",
partitionInfo.appShuffleId.appId, partitionInfo.appShuffleId.shuffleId,
partitionInfo.reduceId, partitionInfo.getPosition(), updatedPos);
length += partitionInfo.dataChannel.write(buf, updatedPos);
} else {
length += partitionInfo.dataChannel.write(buf);
}
```
Please let me know if there are any other suggestions?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] Victsm commented on a change in pull request #30433: [SPARK-32916][SHUFFLE][test-maven][test-hadoop2.7] Ensure the number of chunks in meta file and index file are equal
Posted by GitBox <gi...@apache.org>.
Victsm commented on a change in pull request #30433:
URL: https://github.com/apache/spark/pull/30433#discussion_r542684269
##########
File path: common/network-shuffle/src/main/java/org/apache/spark/network/shuffle/RemoteBlockPushResolver.java
##########
@@ -586,6 +592,7 @@ public void onData(String streamId, ByteBuffer buf) throws IOException {
deferredBufs = null;
return;
}
+ abortIfNecessary();
Review comment:
@Ngone51 I think that's an existing behavior in Spark with how it uses `StreamInterceptor` to stream data coming from a RPC message that's out of the frame.
This is also documented with SPARK-6237 in #21346:
https://github.com/apache/spark/blob/82aca7eb8f2501dceaf610f1aaa86082153ef5ee/common/network-common/src/main/java/org/apache/spark/network/server/RpcHandler.java#L62-L66
It is the server which closes the channel if exception is thrown from `onData`.
Once an exception gets thrown from `onData` while `StreamInterceptor` hasn't finished processing all the out of frame bytes for a given RPC message, the `TransportFrameDecoder` will no longer be able to successfully decode following RPC messages from this channel.
Thus, the server needs to close the channel at this point.
Once the channel gets closed, the client will no longer be able to transfer any more data to the server using the same channel.
The connection needs to be reestablished at this time, resetting state on the client side.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #30433: [SPARK-32916][SHUFFLE][test-maven][test-hadoop2.7] Ensure the number of chunks in meta file and index file are equal
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #30433:
URL: https://github.com/apache/spark/pull/30433#issuecomment-741622882
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/37073/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #30433: [SPARK-32916][SHUFFLE][test-maven][test-hadoop2.7] Ensure the number of chunks in meta file and index file are equal
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #30433:
URL: https://github.com/apache/spark/pull/30433#issuecomment-740936463
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/37046/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] otterc commented on a change in pull request #30433: [SPARK-32916][SHUFFLE][test-maven][test-hadoop2.7] Ensure the number of chunks in meta file and index file are equal
Posted by GitBox <gi...@apache.org>.
otterc commented on a change in pull request #30433:
URL: https://github.com/apache/spark/pull/30433#discussion_r536347251
##########
File path: common/network-shuffle/src/main/java/org/apache/spark/network/shuffle/RemoteBlockPushResolver.java
##########
@@ -369,27 +388,19 @@ public MergeStatuses finalizeShuffleMerge(FinalizeShuffleMerge msg) throws IOExc
mergeStatuses =
new MergeStatuses(msg.shuffleId, new RoaringBitmap[0], new int[0], new long[0]);
} else {
- Collection<AppShufflePartitionInfo> partitionsToFinalize = shufflePartitions.values();
- int totalPartitions = partitionsToFinalize.size();
- RoaringBitmap[] bitmaps = new RoaringBitmap[totalPartitions];
- int[] reduceIds = new int[totalPartitions];
- long[] sizes = new long[totalPartitions];
- Iterator<AppShufflePartitionInfo> partitionsIter = partitionsToFinalize.iterator();
- int idx = 0;
+ List<RoaringBitmap> bitmaps = new LinkedList<>();
+ List<Integer> reduceIds = new LinkedList<>();
+ List<Long> sizes = new LinkedList<>();
Review comment:
done
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] otterc commented on a change in pull request #30433: [SPARK-32916][SHUFFLE][test-maven][test-hadoop2.7] Ensure the number of chunks in meta file and index file are equal
Posted by GitBox <gi...@apache.org>.
otterc commented on a change in pull request #30433:
URL: https://github.com/apache/spark/pull/30433#discussion_r538952136
##########
File path: common/network-shuffle/src/test/java/org/apache/spark/network/shuffle/RemoteBlockPushResolverSuite.java
##########
@@ -411,6 +415,269 @@ void deleteExecutorDirs(Path[] dirs) {
}
}
+ @Test
+ public void testRecoverIndexFileAfterIOExceptions() throws IOException {
+ useTestFiles(true, false);
+ RemoteBlockPushResolver.PushBlockStreamCallback callback1 =
+ (RemoteBlockPushResolver.PushBlockStreamCallback) pushResolver.receiveBlockDataAsStream(
+ new PushBlockStream(TEST_APP, 0, 0, 0, 0));
+ callback1.onData(callback1.getID(), ByteBuffer.wrap(new byte[4]));
+ callback1.onComplete(callback1.getID());
+ RemoteBlockPushResolver.AppShufflePartitionInfo partitionInfo = callback1.getPartitionInfo();
+ // Close the index stream so it throws IOException
+ TestMergeShuffleFile testIndexFile = (TestMergeShuffleFile) partitionInfo.getIndexFile();
+ testIndexFile.close();
+ StreamCallbackWithID callback2 = pushResolver.receiveBlockDataAsStream(
+ new PushBlockStream(TEST_APP, 0, 1, 0, 0));
+ callback2.onData(callback2.getID(), ByteBuffer.wrap(new byte[5]));
+ // This will complete without any IOExceptions because number of IOExceptions are less than
+ // the threshold but the update to index file will be unsuccessful.
+ callback2.onComplete(callback2.getID());
+ assertEquals("index position", 16, testIndexFile.getPos());
+ // Restore the index stream so it can write successfully again.
+ testIndexFile.restore();
+ StreamCallbackWithID callback3 = pushResolver.receiveBlockDataAsStream(
+ new PushBlockStream(TEST_APP, 0, 2, 0, 0));
+ callback3.onData(callback3.getID(), ByteBuffer.wrap(new byte[2]));
+ callback3.onComplete(callback3.getID());
+ assertEquals("index position", 24, testIndexFile.getPos());
+ MergeStatuses statuses = pushResolver.finalizeShuffleMerge(
+ new FinalizeShuffleMerge(TEST_APP, 0));
+ validateMergeStatuses(statuses, new int[] {0}, new long[] {11});
+ MergedBlockMeta blockMeta = pushResolver.getMergedBlockMeta(TEST_APP, 0, 0);
+ validateChunks(TEST_APP, 0, 0, blockMeta, new int[] {4, 7}, new int[][] {{0}, {1, 2}});
+ }
+
+ @Test
+ public void testRecoverIndexFileAfterIOExceptionsInFinalize() throws IOException {
+ useTestFiles(true, false);
+ registerExecutor(TEST_APP, prepareLocalDirs(localDirs));
+ RemoteBlockPushResolver.PushBlockStreamCallback callback1 =
+ (RemoteBlockPushResolver.PushBlockStreamCallback) pushResolver.receiveBlockDataAsStream(
+ new PushBlockStream(TEST_APP, 0, 0, 0, 0));
+ callback1.onData(callback1.getID(), ByteBuffer.wrap(new byte[4]));
+ callback1.onComplete(callback1.getID());
+ RemoteBlockPushResolver.AppShufflePartitionInfo partitionInfo = callback1.getPartitionInfo();
+ // Close the index stream so it throws IOException
+ TestMergeShuffleFile testIndexFile = (TestMergeShuffleFile) partitionInfo.getIndexFile();
+ testIndexFile.close();
+ StreamCallbackWithID callback2 = pushResolver.receiveBlockDataAsStream(
+ new PushBlockStream(TEST_APP, 0, 1, 0, 0));
+ callback2.onData(callback2.getID(), ByteBuffer.wrap(new byte[5]));
+ // This will complete without any IOExceptions because number of IOExceptions are less than
+ // the threshold but the update to index file will be unsuccessful.
+ callback2.onComplete(callback2.getID());
+ assertEquals("index position", 16, testIndexFile.getPos());
+ // The last update to index was unsuccessful however any further updates will be successful.
+ // Restore the index stream so it can write successfully again.
+ testIndexFile.restore();
+ MergeStatuses statuses = pushResolver.finalizeShuffleMerge(
+ new FinalizeShuffleMerge(TEST_APP, 0));
+ assertEquals("index position", 24, testIndexFile.getPos());
+ validateMergeStatuses(statuses, new int[] {0}, new long[] {9});
+ MergedBlockMeta blockMeta = pushResolver.getMergedBlockMeta(TEST_APP, 0, 0);
+ validateChunks(TEST_APP, 0, 0, blockMeta, new int[] {4, 5}, new int[][] {{0}, {1}});
+ }
+
+ @Test
+ public void testRecoverMetaFileAfterIOExceptions() throws IOException {
+ useTestFiles(false, true);
+ RemoteBlockPushResolver.PushBlockStreamCallback callback1 =
+ (RemoteBlockPushResolver.PushBlockStreamCallback) pushResolver.receiveBlockDataAsStream(
+ new PushBlockStream(TEST_APP, 0, 0, 0, 0));
+ callback1.onData(callback1.getID(), ByteBuffer.wrap(new byte[4]));
+ callback1.onComplete(callback1.getID());
+ RemoteBlockPushResolver.AppShufflePartitionInfo partitionInfo = callback1.getPartitionInfo();
+ // Close the meta stream so it throws IOException
+ TestMergeShuffleFile testMetaFile = (TestMergeShuffleFile) partitionInfo.getMetaFile();
+ long metaPosBeforeClose = testMetaFile.getPos();
+ testMetaFile.close();
+ StreamCallbackWithID callback2 = pushResolver.receiveBlockDataAsStream(
+ new PushBlockStream(TEST_APP, 0, 1, 0, 0));
+ callback2.onData(callback2.getID(), ByteBuffer.wrap(new byte[5]));
+ // This will complete without any IOExceptions because number of IOExceptions are less than
+ // the threshold but the update to index and meta file will be unsuccessful.
+ callback2.onComplete(callback2.getID());
+ assertEquals("index position", 16, partitionInfo.getIndexFile().getPos());
+ assertEquals("meta position", metaPosBeforeClose, testMetaFile.getPos());
+ // Restore the meta stream so it can write successfully again.
+ testMetaFile.restore();
+ StreamCallbackWithID callback3 = pushResolver.receiveBlockDataAsStream(
+ new PushBlockStream(TEST_APP, 0, 2, 0, 0));
+ callback3.onData(callback3.getID(), ByteBuffer.wrap(new byte[2]));
+ callback3.onComplete(callback3.getID());
+ assertEquals("index position", 24, partitionInfo.getIndexFile().getPos());
+ assertTrue("meta position", testMetaFile.getPos() > metaPosBeforeClose);
+ MergeStatuses statuses = pushResolver.finalizeShuffleMerge(
+ new FinalizeShuffleMerge(TEST_APP, 0));
+ validateMergeStatuses(statuses, new int[] {0}, new long[] {11});
+ MergedBlockMeta blockMeta = pushResolver.getMergedBlockMeta(TEST_APP, 0, 0);
+ validateChunks(TEST_APP, 0, 0, blockMeta, new int[] {4, 7}, new int[][] {{0}, {1, 2}});
+ }
+
+ @Test
+ public void testRecoverMetaFileAfterIOExceptionsInFinalize() throws IOException {
+ useTestFiles(false, true);
+ RemoteBlockPushResolver.PushBlockStreamCallback callback1 =
+ (RemoteBlockPushResolver.PushBlockStreamCallback) pushResolver.receiveBlockDataAsStream(
+ new PushBlockStream(TEST_APP, 0, 0, 0, 0));
+ callback1.onData(callback1.getID(), ByteBuffer.wrap(new byte[4]));
+ callback1.onComplete(callback1.getID());
+ RemoteBlockPushResolver.AppShufflePartitionInfo partitionInfo = callback1.getPartitionInfo();
+ // Close the meta stream so it throws IOException
+ TestMergeShuffleFile testMetaFile = (TestMergeShuffleFile) partitionInfo.getMetaFile();
+ long metaPosBeforeClose = testMetaFile.getPos();
+ testMetaFile.close();
+ StreamCallbackWithID callback2 = pushResolver.receiveBlockDataAsStream(
+ new PushBlockStream(TEST_APP, 0, 1, 0, 0));
+ callback2.onData(callback2.getID(), ByteBuffer.wrap(new byte[5]));
+ // This will complete without any IOExceptions because number of IOExceptions are less than
+ // the threshold but the update to index and meta file will be unsuccessful.
+ callback2.onComplete(callback2.getID());
+ MergeShuffleFile indexFile = partitionInfo.getIndexFile();
+ assertEquals("index position", 16, indexFile.getPos());
+ assertEquals("meta position", metaPosBeforeClose, testMetaFile.getPos());
+ // Restore the meta stream so it can write successfully again.
+ testMetaFile.restore();
+ MergeStatuses statuses = pushResolver.finalizeShuffleMerge(
+ new FinalizeShuffleMerge(TEST_APP, 0));
+ assertEquals("index position", 24, indexFile.getPos());
+ assertTrue("meta position", testMetaFile.getPos() > metaPosBeforeClose);
+ validateMergeStatuses(statuses, new int[] {0}, new long[] {9});
+ MergedBlockMeta blockMeta = pushResolver.getMergedBlockMeta(TEST_APP, 0, 0);
+ validateChunks(TEST_APP, 0, 0, blockMeta, new int[] {4, 5}, new int[][] {{0}, {1}});
+ }
+
+ @Test (expected = RuntimeException.class)
+ public void testIOExceptionsExceededThreshold() throws IOException {
+ RemoteBlockPushResolver.PushBlockStreamCallback callback =
+ (RemoteBlockPushResolver.PushBlockStreamCallback) pushResolver.receiveBlockDataAsStream(
+ new PushBlockStream(TEST_APP, 0, 0, 0, 0));
+ RemoteBlockPushResolver.AppShufflePartitionInfo partitionInfo = callback.getPartitionInfo();
+ callback.onData(callback.getID(), ByteBuffer.wrap(new byte[4]));
+ callback.onComplete(callback.getID());
+ // Close the data stream so it throws continuous IOException
+ partitionInfo.getDataChannel().close();
+ for (int i = 1; i < 5; i++) {
+ RemoteBlockPushResolver.PushBlockStreamCallback callback1 =
+ (RemoteBlockPushResolver.PushBlockStreamCallback) pushResolver.receiveBlockDataAsStream(
+ new PushBlockStream(TEST_APP, 0, i, 0, 0));
+ try {
+ callback1.onData(callback1.getID(), ByteBuffer.wrap(new byte[2]));
+ } catch (IOException ioe) {
+ // this will throw IOException so the client can retry.
+ callback1.onFailure(callback1.getID(), ioe);
+ }
+ }
+ assertEquals(4, partitionInfo.getNumIOExceptions());
+ // After 4 IOException, the server will respond with IOExceptions exceeded threshold
+ try {
+ RemoteBlockPushResolver.PushBlockStreamCallback callback2 =
+ (RemoteBlockPushResolver.PushBlockStreamCallback) pushResolver.receiveBlockDataAsStream(
+ new PushBlockStream(TEST_APP, 0, 5, 0, 0));
+ callback2.onData(callback.getID(), ByteBuffer.wrap(new byte[1]));
+ } catch (Throwable t) {
+ assertEquals("IOExceptions exceeded the threshold when merging shufflePush_0_5_0",
+ t.getMessage());
+ throw t;
+ }
Review comment:
> I actually mean those blocks which are waiting for the partition lock at the same time.
I see. That's a good point. The server would attempt to merge them. If they don't see any IOExceptions, which is unlikely they will be merged. Do you think we should check before writing them?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] otterc commented on a change in pull request #30433: [SPARK-32916][SHUFFLE][test-maven][test-hadoop2.7] Ensure the number of chunks in meta file and index file are equal
Posted by GitBox <gi...@apache.org>.
otterc commented on a change in pull request #30433:
URL: https://github.com/apache/spark/pull/30433#discussion_r529837473
##########
File path: common/network-shuffle/src/main/java/org/apache/spark/network/shuffle/RemoteBlockPushResolver.java
##########
@@ -827,13 +833,16 @@ void resetChunkTracker() {
void updateChunkInfo(long chunkOffset, int mapIndex) throws IOException {
long idxStartPos = -1;
try {
- // update the chunk tracker to meta file before index file
- writeChunkTracker(mapIndex);
idxStartPos = indexFile.getFilePointer();
logger.trace("{} shuffleId {} reduceId {} updated index current {} updated {}",
appShuffleId.appId, appShuffleId.shuffleId, reduceId, this.lastChunkOffset,
chunkOffset);
- indexFile.writeLong(chunkOffset);
+ indexFile.write(Longs.toByteArray(chunkOffset));
+ // Chunk bitmap should be written to the meta file after the index file because if there are
+ // any exceptions during writing the offset to the index file, meta file should not be
+ // updated. If the update to the index file is successful but the update to meta file isn't
+ // then the index file position is reset in the catch clause.
+ writeChunkTracker(mapIndex);
Review comment:
> Besides, I'm thinking it would be good if we could dynamically change the merger location when such IO exception happens.
Which one of these do you mean?
1. The executor should start pushing to a different server for the current shuffle that is going on.
2. Do you mean that for a future shuffle, the driver should decide the merger locations considering these IOExceptions.
(1) is currently difficult. The driver decides the specific merger locations before a shuffle stage and all the executors work on the assumption that these are all the shuffle servers that they push the data to. This list of shuffle servers should be consistent across the shuffle servers because if it's not then they will be push the shuffle data belonging to a reducer to different shuffle servers. This is the code I am talking about- [code](https://github.com/linkedin/spark/blob/002cb69e8ddf49bbe114744b84c46e0fd452d852/core/src/main/scala/org/apache/spark/shuffle/ShuffleBlockPusher.scala#L360).
(2) This could be a future improvement where we also include the metrics for merge reported back to driver when the driver triggers the finalize. The driver could use these metrics when deciding which shuffle server to use for the next push.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] mridulm commented on a change in pull request #30433: [SPARK-32916][SHUFFLE][test-maven][test-hadoop2.7] Ensure the number of chunks in meta file and index file are equal
Posted by GitBox <gi...@apache.org>.
mridulm commented on a change in pull request #30433:
URL: https://github.com/apache/spark/pull/30433#discussion_r540262870
##########
File path: common/network-shuffle/src/main/java/org/apache/spark/network/shuffle/RemoteBlockPushResolver.java
##########
@@ -586,6 +592,7 @@ public void onData(String streamId, ByteBuffer buf) throws IOException {
deferredBufs = null;
return;
}
+ abortIfNecessary();
Review comment:
I agree with @Ngone51 - if we failed on a previous write, there should not be any subsequent writes: it will lead to inconsistent state (for example, if previous write failed due to lack of space and subsequent write can go through due to some async cleanup).
It is better to avoid these states rather than reason about them based on current code state - as things evolve, it makes it more brittle.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #30433: [SPARK-32916][SHUFFLE][test-maven][test-hadoop2.7] Ensure the number of chunks in meta file and index file are equal
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #30433:
URL: https://github.com/apache/spark/pull/30433#issuecomment-730611561
Can one of the admins verify this patch?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] Victsm commented on a change in pull request #30433: [SPARK-32916][SHUFFLE][test-maven][test-hadoop2.7] Ensure the number of chunks in meta file and index file are equal
Posted by GitBox <gi...@apache.org>.
Victsm commented on a change in pull request #30433:
URL: https://github.com/apache/spark/pull/30433#discussion_r540452532
##########
File path: common/network-shuffle/src/main/java/org/apache/spark/network/shuffle/RemoteBlockPushResolver.java
##########
@@ -586,6 +592,7 @@ public void onData(String streamId, ByteBuffer buf) throws IOException {
deferredBufs = null;
return;
}
+ abortIfNecessary();
Review comment:
If we want consistency, i.e., no more write after reaching IOException threshold, then we should alway invoke `abortIfNecessary` in `onComplete` to check, and if that's what we want, only invoking `abortIfNecessary` in `onComplete` should be sufficient.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] mridulm commented on pull request #30433: [SPARK-32916][SHUFFLE][test-maven][test-hadoop2.7] Ensure the number of chunks in meta file and index file are equal
Posted by GitBox <gi...@apache.org>.
mridulm commented on pull request #30433:
URL: https://github.com/apache/spark/pull/30433#issuecomment-748435251
The changes look good to me. Is there anything else pending here @otterc ?
If not, would be great if you can take a pass over it @Ngone51, @Victsm, @tgravescs before I merge it.
Will wait until Tuesday given holidays.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #30433: [SPARK-32916][SHUFFLE][test-maven][test-hadoop2.7] Ensure the number of chunks in meta file and index file are equal
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #30433:
URL: https://github.com/apache/spark/pull/30433#issuecomment-749708838
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] Ngone51 commented on pull request #30433: [SPARK-32916][SHUFFLE][test-maven][test-hadoop2.7] Ensure the number of chunks in meta file and index file are equal
Posted by GitBox <gi...@apache.org>.
Ngone51 commented on pull request #30433:
URL: https://github.com/apache/spark/pull/30433#issuecomment-740434969
Please also update the PR description.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #30433: [SPARK-32916][SHUFFLE][test-maven][test-hadoop2.7] Ensure the number of chunks in meta file and index file are equal
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #30433:
URL: https://github.com/apache/spark/pull/30433#issuecomment-732672158
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #30433: [SPARK-32916][SHUFFLE][test-maven][test-hadoop2.7] Ensure the number of chunks in meta file and index file are equal
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #30433:
URL: https://github.com/apache/spark/pull/30433#issuecomment-738336375
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #30433: [SPARK-32916][SHUFFLE][test-maven][test-hadoop2.7] Ensure the number of chunks in meta file and index file are equal
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #30433:
URL: https://github.com/apache/spark/pull/30433#issuecomment-749708838
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/37832/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #30433: [SPARK-32916][SHUFFLE][test-maven][test-hadoop2.7] Ensure the number of chunks in meta file and index file are equal
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #30433:
URL: https://github.com/apache/spark/pull/30433#issuecomment-737671937
**[Test build #132091 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/132091/testReport)** for PR 30433 at commit [`d3eb3e7`](https://github.com/apache/spark/commit/d3eb3e76c48b7dd2addc70b2309d0831e6e19e73).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] otterc commented on a change in pull request #30433: [SPARK-32916][SHUFFLE][test-maven][test-hadoop2.7] Ensure the number of chunks in meta file and index file are equal
Posted by GitBox <gi...@apache.org>.
otterc commented on a change in pull request #30433:
URL: https://github.com/apache/spark/pull/30433#discussion_r538672700
##########
File path: common/network-shuffle/src/test/java/org/apache/spark/network/shuffle/RemoteBlockPushResolverSuite.java
##########
@@ -411,6 +415,269 @@ void deleteExecutorDirs(Path[] dirs) {
}
}
+ @Test
+ public void testRecoverIndexFileAfterIOExceptions() throws IOException {
+ useTestFiles(true, false);
+ RemoteBlockPushResolver.PushBlockStreamCallback callback1 =
+ (RemoteBlockPushResolver.PushBlockStreamCallback) pushResolver.receiveBlockDataAsStream(
+ new PushBlockStream(TEST_APP, 0, 0, 0, 0));
+ callback1.onData(callback1.getID(), ByteBuffer.wrap(new byte[4]));
+ callback1.onComplete(callback1.getID());
+ RemoteBlockPushResolver.AppShufflePartitionInfo partitionInfo = callback1.getPartitionInfo();
+ // Close the index stream so it throws IOException
+ TestMergeShuffleFile testIndexFile = (TestMergeShuffleFile) partitionInfo.getIndexFile();
+ testIndexFile.close();
+ StreamCallbackWithID callback2 = pushResolver.receiveBlockDataAsStream(
+ new PushBlockStream(TEST_APP, 0, 1, 0, 0));
+ callback2.onData(callback2.getID(), ByteBuffer.wrap(new byte[5]));
+ // This will complete without any IOExceptions because number of IOExceptions are less than
+ // the threshold but the update to index file will be unsuccessful.
+ callback2.onComplete(callback2.getID());
+ assertEquals("index position", 16, testIndexFile.getPos());
+ // Restore the index stream so it can write successfully again.
+ testIndexFile.restore();
+ StreamCallbackWithID callback3 = pushResolver.receiveBlockDataAsStream(
+ new PushBlockStream(TEST_APP, 0, 2, 0, 0));
+ callback3.onData(callback3.getID(), ByteBuffer.wrap(new byte[2]));
+ callback3.onComplete(callback3.getID());
+ assertEquals("index position", 24, testIndexFile.getPos());
+ MergeStatuses statuses = pushResolver.finalizeShuffleMerge(
+ new FinalizeShuffleMerge(TEST_APP, 0));
+ validateMergeStatuses(statuses, new int[] {0}, new long[] {11});
+ MergedBlockMeta blockMeta = pushResolver.getMergedBlockMeta(TEST_APP, 0, 0);
+ validateChunks(TEST_APP, 0, 0, blockMeta, new int[] {4, 7}, new int[][] {{0}, {1, 2}});
+ }
+
+ @Test
+ public void testRecoverIndexFileAfterIOExceptionsInFinalize() throws IOException {
+ useTestFiles(true, false);
+ registerExecutor(TEST_APP, prepareLocalDirs(localDirs));
+ RemoteBlockPushResolver.PushBlockStreamCallback callback1 =
+ (RemoteBlockPushResolver.PushBlockStreamCallback) pushResolver.receiveBlockDataAsStream(
+ new PushBlockStream(TEST_APP, 0, 0, 0, 0));
+ callback1.onData(callback1.getID(), ByteBuffer.wrap(new byte[4]));
+ callback1.onComplete(callback1.getID());
+ RemoteBlockPushResolver.AppShufflePartitionInfo partitionInfo = callback1.getPartitionInfo();
+ // Close the index stream so it throws IOException
+ TestMergeShuffleFile testIndexFile = (TestMergeShuffleFile) partitionInfo.getIndexFile();
+ testIndexFile.close();
+ StreamCallbackWithID callback2 = pushResolver.receiveBlockDataAsStream(
+ new PushBlockStream(TEST_APP, 0, 1, 0, 0));
+ callback2.onData(callback2.getID(), ByteBuffer.wrap(new byte[5]));
+ // This will complete without any IOExceptions because number of IOExceptions are less than
+ // the threshold but the update to index file will be unsuccessful.
+ callback2.onComplete(callback2.getID());
+ assertEquals("index position", 16, testIndexFile.getPos());
+ // The last update to index was unsuccessful however any further updates will be successful.
+ // Restore the index stream so it can write successfully again.
+ testIndexFile.restore();
+ MergeStatuses statuses = pushResolver.finalizeShuffleMerge(
+ new FinalizeShuffleMerge(TEST_APP, 0));
+ assertEquals("index position", 24, testIndexFile.getPos());
+ validateMergeStatuses(statuses, new int[] {0}, new long[] {9});
+ MergedBlockMeta blockMeta = pushResolver.getMergedBlockMeta(TEST_APP, 0, 0);
+ validateChunks(TEST_APP, 0, 0, blockMeta, new int[] {4, 5}, new int[][] {{0}, {1}});
+ }
+
+ @Test
+ public void testRecoverMetaFileAfterIOExceptions() throws IOException {
+ useTestFiles(false, true);
+ RemoteBlockPushResolver.PushBlockStreamCallback callback1 =
+ (RemoteBlockPushResolver.PushBlockStreamCallback) pushResolver.receiveBlockDataAsStream(
+ new PushBlockStream(TEST_APP, 0, 0, 0, 0));
+ callback1.onData(callback1.getID(), ByteBuffer.wrap(new byte[4]));
+ callback1.onComplete(callback1.getID());
+ RemoteBlockPushResolver.AppShufflePartitionInfo partitionInfo = callback1.getPartitionInfo();
+ // Close the meta stream so it throws IOException
+ TestMergeShuffleFile testMetaFile = (TestMergeShuffleFile) partitionInfo.getMetaFile();
+ long metaPosBeforeClose = testMetaFile.getPos();
+ testMetaFile.close();
+ StreamCallbackWithID callback2 = pushResolver.receiveBlockDataAsStream(
+ new PushBlockStream(TEST_APP, 0, 1, 0, 0));
+ callback2.onData(callback2.getID(), ByteBuffer.wrap(new byte[5]));
+ // This will complete without any IOExceptions because number of IOExceptions are less than
+ // the threshold but the update to index and meta file will be unsuccessful.
+ callback2.onComplete(callback2.getID());
+ assertEquals("index position", 16, partitionInfo.getIndexFile().getPos());
+ assertEquals("meta position", metaPosBeforeClose, testMetaFile.getPos());
+ // Restore the meta stream so it can write successfully again.
+ testMetaFile.restore();
+ StreamCallbackWithID callback3 = pushResolver.receiveBlockDataAsStream(
+ new PushBlockStream(TEST_APP, 0, 2, 0, 0));
+ callback3.onData(callback3.getID(), ByteBuffer.wrap(new byte[2]));
+ callback3.onComplete(callback3.getID());
+ assertEquals("index position", 24, partitionInfo.getIndexFile().getPos());
+ assertTrue("meta position", testMetaFile.getPos() > metaPosBeforeClose);
+ MergeStatuses statuses = pushResolver.finalizeShuffleMerge(
+ new FinalizeShuffleMerge(TEST_APP, 0));
+ validateMergeStatuses(statuses, new int[] {0}, new long[] {11});
+ MergedBlockMeta blockMeta = pushResolver.getMergedBlockMeta(TEST_APP, 0, 0);
+ validateChunks(TEST_APP, 0, 0, blockMeta, new int[] {4, 7}, new int[][] {{0}, {1, 2}});
+ }
+
+ @Test
+ public void testRecoverMetaFileAfterIOExceptionsInFinalize() throws IOException {
+ useTestFiles(false, true);
+ RemoteBlockPushResolver.PushBlockStreamCallback callback1 =
+ (RemoteBlockPushResolver.PushBlockStreamCallback) pushResolver.receiveBlockDataAsStream(
+ new PushBlockStream(TEST_APP, 0, 0, 0, 0));
+ callback1.onData(callback1.getID(), ByteBuffer.wrap(new byte[4]));
+ callback1.onComplete(callback1.getID());
+ RemoteBlockPushResolver.AppShufflePartitionInfo partitionInfo = callback1.getPartitionInfo();
+ // Close the meta stream so it throws IOException
+ TestMergeShuffleFile testMetaFile = (TestMergeShuffleFile) partitionInfo.getMetaFile();
+ long metaPosBeforeClose = testMetaFile.getPos();
+ testMetaFile.close();
+ StreamCallbackWithID callback2 = pushResolver.receiveBlockDataAsStream(
+ new PushBlockStream(TEST_APP, 0, 1, 0, 0));
+ callback2.onData(callback2.getID(), ByteBuffer.wrap(new byte[5]));
+ // This will complete without any IOExceptions because number of IOExceptions are less than
+ // the threshold but the update to index and meta file will be unsuccessful.
+ callback2.onComplete(callback2.getID());
+ MergeShuffleFile indexFile = partitionInfo.getIndexFile();
+ assertEquals("index position", 16, indexFile.getPos());
+ assertEquals("meta position", metaPosBeforeClose, testMetaFile.getPos());
+ // Restore the meta stream so it can write successfully again.
+ testMetaFile.restore();
+ MergeStatuses statuses = pushResolver.finalizeShuffleMerge(
+ new FinalizeShuffleMerge(TEST_APP, 0));
+ assertEquals("index position", 24, indexFile.getPos());
+ assertTrue("meta position", testMetaFile.getPos() > metaPosBeforeClose);
+ validateMergeStatuses(statuses, new int[] {0}, new long[] {9});
+ MergedBlockMeta blockMeta = pushResolver.getMergedBlockMeta(TEST_APP, 0, 0);
+ validateChunks(TEST_APP, 0, 0, blockMeta, new int[] {4, 5}, new int[][] {{0}, {1}});
+ }
+
+ @Test (expected = RuntimeException.class)
+ public void testIOExceptionsExceededThreshold() throws IOException {
+ RemoteBlockPushResolver.PushBlockStreamCallback callback =
+ (RemoteBlockPushResolver.PushBlockStreamCallback) pushResolver.receiveBlockDataAsStream(
+ new PushBlockStream(TEST_APP, 0, 0, 0, 0));
+ RemoteBlockPushResolver.AppShufflePartitionInfo partitionInfo = callback.getPartitionInfo();
+ callback.onData(callback.getID(), ByteBuffer.wrap(new byte[4]));
+ callback.onComplete(callback.getID());
+ // Close the data stream so it throws continuous IOException
+ partitionInfo.getDataChannel().close();
+ for (int i = 1; i < 5; i++) {
+ RemoteBlockPushResolver.PushBlockStreamCallback callback1 =
+ (RemoteBlockPushResolver.PushBlockStreamCallback) pushResolver.receiveBlockDataAsStream(
+ new PushBlockStream(TEST_APP, 0, i, 0, 0));
+ try {
+ callback1.onData(callback1.getID(), ByteBuffer.wrap(new byte[2]));
+ } catch (IOException ioe) {
+ // this will throw IOException so the client can retry.
+ callback1.onFailure(callback1.getID(), ioe);
+ }
+ }
+ assertEquals(4, partitionInfo.getNumIOExceptions());
+ // After 4 IOException, the server will respond with IOExceptions exceeded threshold
+ try {
+ RemoteBlockPushResolver.PushBlockStreamCallback callback2 =
+ (RemoteBlockPushResolver.PushBlockStreamCallback) pushResolver.receiveBlockDataAsStream(
+ new PushBlockStream(TEST_APP, 0, 5, 0, 0));
+ callback2.onData(callback.getID(), ByteBuffer.wrap(new byte[1]));
+ } catch (Throwable t) {
+ assertEquals("IOExceptions exceeded the threshold when merging shufflePush_0_5_0",
+ t.getMessage());
+ throw t;
+ }
Review comment:
The deferred blocks are just dereferenced. This is same as what happens when the server responds with the `TooLate` response.
```
private void incrementIOExceptionsAndAbortIfNecessary() {
// Update the count of IOExceptions
partitionInfo.incrementIOExceptions();
if (partitionInfo.shouldAbort(mergeManager.ioExceptionsThresholdDuringMerge)) {
deferredBufs = null;
throw new RuntimeException(String.format("%s when merging %s",
ErrorHandler.BlockPushErrorHandler.IOEXCEPTIONS_EXCEEDED_THRESHOLD_PREFIX,
streamId));
}
}
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] otterc commented on a change in pull request #30433: [SPARK-32916][SHUFFLE][test-maven][test-hadoop2.7] Ensure the number of chunks in meta file and index file are equal
Posted by GitBox <gi...@apache.org>.
otterc commented on a change in pull request #30433:
URL: https://github.com/apache/spark/pull/30433#discussion_r540379906
##########
File path: common/network-shuffle/src/main/java/org/apache/spark/network/shuffle/RemoteBlockPushResolver.java
##########
@@ -586,6 +592,7 @@ public void onData(String streamId, ByteBuffer buf) throws IOException {
deferredBufs = null;
return;
}
+ abortIfNecessary();
Review comment:
Ok. Then I will not remove the `abortIfNecessary()` from `onData`.
cc. @mridulm @Ngone51 @Victsm
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] otterc commented on a change in pull request #30433: [SPARK-32916][SHUFFLE][test-maven][test-hadoop2.7] Ensure the number of chunks in meta file and index file are equal
Posted by GitBox <gi...@apache.org>.
otterc commented on a change in pull request #30433:
URL: https://github.com/apache/spark/pull/30433#discussion_r532947313
##########
File path: common/network-shuffle/src/main/java/org/apache/spark/network/shuffle/RemoteBlockPushResolver.java
##########
@@ -827,13 +833,16 @@ void resetChunkTracker() {
void updateChunkInfo(long chunkOffset, int mapIndex) throws IOException {
long idxStartPos = -1;
try {
- // update the chunk tracker to meta file before index file
- writeChunkTracker(mapIndex);
idxStartPos = indexFile.getFilePointer();
logger.trace("{} shuffleId {} reduceId {} updated index current {} updated {}",
appShuffleId.appId, appShuffleId.shuffleId, reduceId, this.lastChunkOffset,
chunkOffset);
- indexFile.writeLong(chunkOffset);
+ indexFile.write(Longs.toByteArray(chunkOffset));
+ // Chunk bitmap should be written to the meta file after the index file because if there are
+ // any exceptions during writing the offset to the index file, meta file should not be
+ // updated. If the update to the index file is successful but the update to meta file isn't
+ // then the index file position is reset in the catch clause.
+ writeChunkTracker(mapIndex);
Review comment:
Openjdk `seek` seems to use `lseek64`
```
// move file pointer to the specified offset
jlong os::seek_to_file_offset(int fd, jlong offset) {
return (jlong)::lseek64(fd, (off64_t)offset, SEEK_SET);
}
```
`lseek64` is just a variation of `lseek` that uses 64 bit file offset.
Below are the errors from https://linux.die.net/man/2/lseek
```
EBADF
fd is not an open file descriptor.
EINVAL
whence is not valid. Or: the resulting file offset would be negative, or beyond the end of a seekable device.
EOVERFLOW
The resulting file offset cannot be represented in an off_t.
ESPIPE
fd is associated with a pipe, socket, or FIFO.
ENXIO
whence is SEEK_DATA or SEEK_HOLE, and the current file offset is beyond the end of the file.
```
In our case, we always seek to a previous point in the file and none of these errors would be recoverable in that case.
So any exceptions during seek should not be recoverable.
@mridulm @Victsm
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #30433: [SPARK-32916][SHUFFLE][test-maven][test-hadoop2.7] Ensure the number of chunks in meta file and index file are equal
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #30433:
URL: https://github.com/apache/spark/pull/30433#issuecomment-742011292
Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/37109/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #30433: [SPARK-32916][SHUFFLE][test-maven][test-hadoop2.7] Ensure the number of chunks in meta file and index file are equal
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #30433:
URL: https://github.com/apache/spark/pull/30433#issuecomment-749641378
**[Test build #133233 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/133233/testReport)** for PR 30433 at commit [`f48d52f`](https://github.com/apache/spark/commit/f48d52f68e52b63742e7cce5dbb4fbcc89845402).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #30433: [SPARK-32916][SHUFFLE][test-maven][test-hadoop2.7] Ensure the number of chunks in meta file and index file are equal
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #30433:
URL: https://github.com/apache/spark/pull/30433#issuecomment-742011336
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/37109/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] Ngone51 commented on a change in pull request #30433: [SPARK-32916][SHUFFLE][test-maven][test-hadoop2.7] Ensure the number of chunks in meta file and index file are equal
Posted by GitBox <gi...@apache.org>.
Ngone51 commented on a change in pull request #30433:
URL: https://github.com/apache/spark/pull/30433#discussion_r537982682
##########
File path: common/network-shuffle/src/main/java/org/apache/spark/network/shuffle/RemoteBlockPushResolver.java
##########
@@ -825,43 +884,68 @@ void resetChunkTracker() {
* @param mapIndex the map index to be added to chunk tracker.
*/
void updateChunkInfo(long chunkOffset, int mapIndex) throws IOException {
- long idxStartPos = -1;
try {
- // update the chunk tracker to meta file before index file
- writeChunkTracker(mapIndex);
- idxStartPos = indexFile.getFilePointer();
logger.trace("{} shuffleId {} reduceId {} updated index current {} updated {}",
- appShuffleId.appId, appShuffleId.shuffleId, reduceId, this.lastChunkOffset,
- chunkOffset);
- indexFile.writeLong(chunkOffset);
- } catch (IOException ioe) {
- if (idxStartPos != -1) {
- // reset the position to avoid corrupting index files during exception.
- logger.warn("{} shuffleId {} reduceId {} reset index to position {}",
- appShuffleId.appId, appShuffleId.shuffleId, reduceId, idxStartPos);
- indexFile.seek(idxStartPos);
+ appShuffleId.appId, appShuffleId.shuffleId, reduceId, this.lastChunkOffset, chunkOffset);
+ if (indexMetaUpdateFailed) {
+ indexFile.getChannel().position(indexFile.getPos());
}
+ indexFile.getDos().writeLong(chunkOffset);
+ // Chunk bitmap should be written to the meta file after the index file because if there are
+ // any exceptions during writing the offset to the index file, meta file should not be
+ // updated. If the update to the index file is successful but the update to meta file isn't
+ // then the index file position is reset in the catch clause.
Review comment:
The comment looks out of date. The index file position doesn't reset in the catch clause now.
##########
File path: common/network-shuffle/src/test/java/org/apache/spark/network/shuffle/RemoteBlockPushResolverSuite.java
##########
@@ -411,6 +415,269 @@ void deleteExecutorDirs(Path[] dirs) {
}
}
+ @Test
+ public void testRecoverIndexFileAfterIOExceptions() throws IOException {
+ useTestFiles(true, false);
+ RemoteBlockPushResolver.PushBlockStreamCallback callback1 =
+ (RemoteBlockPushResolver.PushBlockStreamCallback) pushResolver.receiveBlockDataAsStream(
+ new PushBlockStream(TEST_APP, 0, 0, 0, 0));
+ callback1.onData(callback1.getID(), ByteBuffer.wrap(new byte[4]));
+ callback1.onComplete(callback1.getID());
+ RemoteBlockPushResolver.AppShufflePartitionInfo partitionInfo = callback1.getPartitionInfo();
+ // Close the index stream so it throws IOException
+ TestMergeShuffleFile testIndexFile = (TestMergeShuffleFile) partitionInfo.getIndexFile();
+ testIndexFile.close();
+ StreamCallbackWithID callback2 = pushResolver.receiveBlockDataAsStream(
+ new PushBlockStream(TEST_APP, 0, 1, 0, 0));
+ callback2.onData(callback2.getID(), ByteBuffer.wrap(new byte[5]));
+ // This will complete without any IOExceptions because number of IOExceptions are less than
+ // the threshold but the update to index file will be unsuccessful.
+ callback2.onComplete(callback2.getID());
+ assertEquals("index position", 16, testIndexFile.getPos());
+ // Restore the index stream so it can write successfully again.
+ testIndexFile.restore();
+ StreamCallbackWithID callback3 = pushResolver.receiveBlockDataAsStream(
+ new PushBlockStream(TEST_APP, 0, 2, 0, 0));
+ callback3.onData(callback3.getID(), ByteBuffer.wrap(new byte[2]));
+ callback3.onComplete(callback3.getID());
+ assertEquals("index position", 24, testIndexFile.getPos());
+ MergeStatuses statuses = pushResolver.finalizeShuffleMerge(
+ new FinalizeShuffleMerge(TEST_APP, 0));
+ validateMergeStatuses(statuses, new int[] {0}, new long[] {11});
+ MergedBlockMeta blockMeta = pushResolver.getMergedBlockMeta(TEST_APP, 0, 0);
+ validateChunks(TEST_APP, 0, 0, blockMeta, new int[] {4, 7}, new int[][] {{0}, {1, 2}});
+ }
+
+ @Test
+ public void testRecoverIndexFileAfterIOExceptionsInFinalize() throws IOException {
+ useTestFiles(true, false);
+ registerExecutor(TEST_APP, prepareLocalDirs(localDirs));
+ RemoteBlockPushResolver.PushBlockStreamCallback callback1 =
+ (RemoteBlockPushResolver.PushBlockStreamCallback) pushResolver.receiveBlockDataAsStream(
+ new PushBlockStream(TEST_APP, 0, 0, 0, 0));
+ callback1.onData(callback1.getID(), ByteBuffer.wrap(new byte[4]));
+ callback1.onComplete(callback1.getID());
+ RemoteBlockPushResolver.AppShufflePartitionInfo partitionInfo = callback1.getPartitionInfo();
+ // Close the index stream so it throws IOException
+ TestMergeShuffleFile testIndexFile = (TestMergeShuffleFile) partitionInfo.getIndexFile();
+ testIndexFile.close();
+ StreamCallbackWithID callback2 = pushResolver.receiveBlockDataAsStream(
+ new PushBlockStream(TEST_APP, 0, 1, 0, 0));
+ callback2.onData(callback2.getID(), ByteBuffer.wrap(new byte[5]));
+ // This will complete without any IOExceptions because number of IOExceptions are less than
+ // the threshold but the update to index file will be unsuccessful.
+ callback2.onComplete(callback2.getID());
+ assertEquals("index position", 16, testIndexFile.getPos());
+ // The last update to index was unsuccessful however any further updates will be successful.
+ // Restore the index stream so it can write successfully again.
+ testIndexFile.restore();
+ MergeStatuses statuses = pushResolver.finalizeShuffleMerge(
+ new FinalizeShuffleMerge(TEST_APP, 0));
+ assertEquals("index position", 24, testIndexFile.getPos());
+ validateMergeStatuses(statuses, new int[] {0}, new long[] {9});
+ MergedBlockMeta blockMeta = pushResolver.getMergedBlockMeta(TEST_APP, 0, 0);
+ validateChunks(TEST_APP, 0, 0, blockMeta, new int[] {4, 5}, new int[][] {{0}, {1}});
+ }
+
+ @Test
+ public void testRecoverMetaFileAfterIOExceptions() throws IOException {
+ useTestFiles(false, true);
+ RemoteBlockPushResolver.PushBlockStreamCallback callback1 =
+ (RemoteBlockPushResolver.PushBlockStreamCallback) pushResolver.receiveBlockDataAsStream(
+ new PushBlockStream(TEST_APP, 0, 0, 0, 0));
+ callback1.onData(callback1.getID(), ByteBuffer.wrap(new byte[4]));
+ callback1.onComplete(callback1.getID());
+ RemoteBlockPushResolver.AppShufflePartitionInfo partitionInfo = callback1.getPartitionInfo();
+ // Close the meta stream so it throws IOException
+ TestMergeShuffleFile testMetaFile = (TestMergeShuffleFile) partitionInfo.getMetaFile();
+ long metaPosBeforeClose = testMetaFile.getPos();
+ testMetaFile.close();
+ StreamCallbackWithID callback2 = pushResolver.receiveBlockDataAsStream(
+ new PushBlockStream(TEST_APP, 0, 1, 0, 0));
+ callback2.onData(callback2.getID(), ByteBuffer.wrap(new byte[5]));
+ // This will complete without any IOExceptions because number of IOExceptions are less than
+ // the threshold but the update to index and meta file will be unsuccessful.
+ callback2.onComplete(callback2.getID());
+ assertEquals("index position", 16, partitionInfo.getIndexFile().getPos());
+ assertEquals("meta position", metaPosBeforeClose, testMetaFile.getPos());
+ // Restore the meta stream so it can write successfully again.
+ testMetaFile.restore();
+ StreamCallbackWithID callback3 = pushResolver.receiveBlockDataAsStream(
+ new PushBlockStream(TEST_APP, 0, 2, 0, 0));
+ callback3.onData(callback3.getID(), ByteBuffer.wrap(new byte[2]));
+ callback3.onComplete(callback3.getID());
+ assertEquals("index position", 24, partitionInfo.getIndexFile().getPos());
+ assertTrue("meta position", testMetaFile.getPos() > metaPosBeforeClose);
+ MergeStatuses statuses = pushResolver.finalizeShuffleMerge(
+ new FinalizeShuffleMerge(TEST_APP, 0));
+ validateMergeStatuses(statuses, new int[] {0}, new long[] {11});
+ MergedBlockMeta blockMeta = pushResolver.getMergedBlockMeta(TEST_APP, 0, 0);
+ validateChunks(TEST_APP, 0, 0, blockMeta, new int[] {4, 7}, new int[][] {{0}, {1, 2}});
+ }
+
+ @Test
+ public void testRecoverMetaFileAfterIOExceptionsInFinalize() throws IOException {
+ useTestFiles(false, true);
+ RemoteBlockPushResolver.PushBlockStreamCallback callback1 =
+ (RemoteBlockPushResolver.PushBlockStreamCallback) pushResolver.receiveBlockDataAsStream(
+ new PushBlockStream(TEST_APP, 0, 0, 0, 0));
+ callback1.onData(callback1.getID(), ByteBuffer.wrap(new byte[4]));
+ callback1.onComplete(callback1.getID());
+ RemoteBlockPushResolver.AppShufflePartitionInfo partitionInfo = callback1.getPartitionInfo();
+ // Close the meta stream so it throws IOException
+ TestMergeShuffleFile testMetaFile = (TestMergeShuffleFile) partitionInfo.getMetaFile();
+ long metaPosBeforeClose = testMetaFile.getPos();
+ testMetaFile.close();
+ StreamCallbackWithID callback2 = pushResolver.receiveBlockDataAsStream(
+ new PushBlockStream(TEST_APP, 0, 1, 0, 0));
+ callback2.onData(callback2.getID(), ByteBuffer.wrap(new byte[5]));
+ // This will complete without any IOExceptions because number of IOExceptions are less than
+ // the threshold but the update to index and meta file will be unsuccessful.
+ callback2.onComplete(callback2.getID());
+ MergeShuffleFile indexFile = partitionInfo.getIndexFile();
+ assertEquals("index position", 16, indexFile.getPos());
+ assertEquals("meta position", metaPosBeforeClose, testMetaFile.getPos());
+ // Restore the meta stream so it can write successfully again.
+ testMetaFile.restore();
+ MergeStatuses statuses = pushResolver.finalizeShuffleMerge(
+ new FinalizeShuffleMerge(TEST_APP, 0));
+ assertEquals("index position", 24, indexFile.getPos());
+ assertTrue("meta position", testMetaFile.getPos() > metaPosBeforeClose);
+ validateMergeStatuses(statuses, new int[] {0}, new long[] {9});
+ MergedBlockMeta blockMeta = pushResolver.getMergedBlockMeta(TEST_APP, 0, 0);
+ validateChunks(TEST_APP, 0, 0, blockMeta, new int[] {4, 5}, new int[][] {{0}, {1}});
+ }
+
+ @Test (expected = RuntimeException.class)
+ public void testIOExceptionsExceededThreshold() throws IOException {
+ RemoteBlockPushResolver.PushBlockStreamCallback callback =
+ (RemoteBlockPushResolver.PushBlockStreamCallback) pushResolver.receiveBlockDataAsStream(
+ new PushBlockStream(TEST_APP, 0, 0, 0, 0));
+ RemoteBlockPushResolver.AppShufflePartitionInfo partitionInfo = callback.getPartitionInfo();
+ callback.onData(callback.getID(), ByteBuffer.wrap(new byte[4]));
+ callback.onComplete(callback.getID());
+ // Close the data stream so it throws continuous IOException
+ partitionInfo.getDataChannel().close();
+ for (int i = 1; i < 5; i++) {
+ RemoteBlockPushResolver.PushBlockStreamCallback callback1 =
+ (RemoteBlockPushResolver.PushBlockStreamCallback) pushResolver.receiveBlockDataAsStream(
+ new PushBlockStream(TEST_APP, 0, i, 0, 0));
+ try {
+ callback1.onData(callback1.getID(), ByteBuffer.wrap(new byte[2]));
+ } catch (IOException ioe) {
+ // this will throw IOException so the client can retry.
+ callback1.onFailure(callback1.getID(), ioe);
+ }
+ }
+ assertEquals(4, partitionInfo.getNumIOExceptions());
+ // After 4 IOException, the server will respond with IOExceptions exceeded threshold
+ try {
+ RemoteBlockPushResolver.PushBlockStreamCallback callback2 =
+ (RemoteBlockPushResolver.PushBlockStreamCallback) pushResolver.receiveBlockDataAsStream(
+ new PushBlockStream(TEST_APP, 0, 5, 0, 0));
+ callback2.onData(callback.getID(), ByteBuffer.wrap(new byte[1]));
+ } catch (Throwable t) {
+ assertEquals("IOExceptions exceeded the threshold when merging shufflePush_0_5_0",
+ t.getMessage());
+ throw t;
+ }
Review comment:
What's the expected behavior for those deferred blocks after exceeding the IO exception threshold? Seem like we don't any strategy for those blocks yet.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #30433: [SPARK-32916][SHUFFLE][test-maven][test-hadoop2.7] Ensure the number of chunks in meta file and index file are equal
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #30433:
URL: https://github.com/apache/spark/pull/30433#issuecomment-732504512
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] otterc commented on a change in pull request #30433: [SPARK-32916][SHUFFLE][test-maven][test-hadoop2.7] Ensure the number of chunks in meta file and index file are equal
Posted by GitBox <gi...@apache.org>.
otterc commented on a change in pull request #30433:
URL: https://github.com/apache/spark/pull/30433#discussion_r528229564
##########
File path: common/network-shuffle/src/main/java/org/apache/spark/network/shuffle/RemoteBlockPushResolver.java
##########
@@ -827,13 +833,16 @@ void resetChunkTracker() {
void updateChunkInfo(long chunkOffset, int mapIndex) throws IOException {
long idxStartPos = -1;
try {
- // update the chunk tracker to meta file before index file
- writeChunkTracker(mapIndex);
idxStartPos = indexFile.getFilePointer();
logger.trace("{} shuffleId {} reduceId {} updated index current {} updated {}",
appShuffleId.appId, appShuffleId.shuffleId, reduceId, this.lastChunkOffset,
chunkOffset);
- indexFile.writeLong(chunkOffset);
+ indexFile.write(Longs.toByteArray(chunkOffset));
Review comment:
I vaguely remember there is a utility for this somewhere either in Spark code or Hadoop code. I tried searching for it in Spark but couldn't find it.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #30433: [SPARK-32916][SHUFFLE][test-maven][test-hadoop2.7] Ensure the number of chunks in meta file and index file are equal
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #30433:
URL: https://github.com/apache/spark/pull/30433#issuecomment-738284550
**[Test build #132158 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/132158/testReport)** for PR 30433 at commit [`72b5976`](https://github.com/apache/spark/commit/72b5976579aa959689aaa78f1b8a841d6c9a09e1).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #30433: [SPARK-32916][SHUFFLE][test-maven][test-hadoop2.7] Ensure the number of chunks in meta file and index file are equal
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #30433:
URL: https://github.com/apache/spark/pull/30433#issuecomment-732503804
**[Test build #131594 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/131594/testReport)** for PR 30433 at commit [`30c8f83`](https://github.com/apache/spark/commit/30c8f83b7ba3e39724c90865fd1318b41876f927).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #30433: [SPARK-32916][SHUFFLE][test-maven][test-hadoop2.7] Ensure the number of chunks in meta file and index file are equal
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #30433:
URL: https://github.com/apache/spark/pull/30433#issuecomment-741593649
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/37073/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] Victsm commented on a change in pull request #30433: [SPARK-32916][SHUFFLE][test-maven][test-hadoop2.7] Ensure the number of chunks in meta file and index file are equal
Posted by GitBox <gi...@apache.org>.
Victsm commented on a change in pull request #30433:
URL: https://github.com/apache/spark/pull/30433#discussion_r528850602
##########
File path: common/network-shuffle/src/main/java/org/apache/spark/network/shuffle/RemoteBlockPushResolver.java
##########
@@ -827,13 +833,16 @@ void resetChunkTracker() {
void updateChunkInfo(long chunkOffset, int mapIndex) throws IOException {
long idxStartPos = -1;
try {
- // update the chunk tracker to meta file before index file
- writeChunkTracker(mapIndex);
idxStartPos = indexFile.getFilePointer();
logger.trace("{} shuffleId {} reduceId {} updated index current {} updated {}",
appShuffleId.appId, appShuffleId.shuffleId, reduceId, this.lastChunkOffset,
chunkOffset);
- indexFile.writeLong(chunkOffset);
+ indexFile.write(Longs.toByteArray(chunkOffset));
+ // Chunk bitmap should be written to the meta file after the index file because if there are
+ // any exceptions during writing the offset to the index file, meta file should not be
+ // updated. If the update to the index file is successful but the update to meta file isn't
+ // then the index file position is reset in the catch clause.
+ writeChunkTracker(mapIndex);
Review comment:
There are temporary issues as well.
For example, a user's job might be filling up the yarn local dir with its temporary data, which would then lead to IOExceptions for merge writes.
When this issue occurs, it could trigger a clean up script which purges the temporary data from that job to recover the disk.
In this scenario, the IOException could be temporary.
For the too large chunk issue, it would only be too large if we have many consecutive IOExceptions.
Once recovered, the following chunks would go back to the normal size.
Only 2 consecutive IOExceptions within the same chunk is far from causing the potential too large chunk issue.
If we want to have a reasonable policy to stop early due to too many IOExceptions, I think that policy should require more than 2 consecutive IOExceptions during chunk metadata write to trigger.
Note that, the same issue could affect writing data file as well.
If we want to address this issue holistically, it would require some state tracking.
If so, I'd rather track it on the client side instead of the server side.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] otterc commented on a change in pull request #30433: [SPARK-32916][SHUFFLE][test-maven][test-hadoop2.7] Ensure the number of chunks in meta file and index file are equal
Posted by GitBox <gi...@apache.org>.
otterc commented on a change in pull request #30433:
URL: https://github.com/apache/spark/pull/30433#discussion_r538944996
##########
File path: common/network-shuffle/src/main/java/org/apache/spark/network/shuffle/RemoteBlockPushResolver.java
##########
@@ -370,26 +391,19 @@ public MergeStatuses finalizeShuffleMerge(FinalizeShuffleMerge msg) throws IOExc
new MergeStatuses(msg.shuffleId, new RoaringBitmap[0], new int[0], new long[0]);
} else {
Collection<AppShufflePartitionInfo> partitionsToFinalize = shufflePartitions.values();
- int totalPartitions = partitionsToFinalize.size();
- RoaringBitmap[] bitmaps = new RoaringBitmap[totalPartitions];
- int[] reduceIds = new int[totalPartitions];
- long[] sizes = new long[totalPartitions];
+ List<RoaringBitmap> bitmaps = new ArrayList<>(partitionsToFinalize.size());
+ List<Integer> reduceIds = new ArrayList<>(partitionsToFinalize.size());
+ List<Long> sizes = new ArrayList<>(partitionsToFinalize.size());
Review comment:
I would prefer not sending null bitmaps. It seems awkward to do that. It is an API method so we also then have to document the API that it can send `nulls`. And what is the performance benefit? This happens once in finalize and the list will at most be few thousand elements big. Making the code optimize for this seems premature.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #30433: [SPARK-32916][SHUFFLE][test-maven][test-hadoop2.7] Ensure the number of chunks in meta file and index file are equal
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #30433:
URL: https://github.com/apache/spark/pull/30433#issuecomment-740915871
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/132444/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] otterc commented on a change in pull request #30433: [SPARK-32916][SHUFFLE][test-maven][test-hadoop2.7] Ensure the number of chunks in meta file and index file are equal
Posted by GitBox <gi...@apache.org>.
otterc commented on a change in pull request #30433:
URL: https://github.com/apache/spark/pull/30433#discussion_r528907429
##########
File path: common/network-shuffle/src/main/java/org/apache/spark/network/shuffle/RemoteBlockPushResolver.java
##########
@@ -827,13 +833,16 @@ void resetChunkTracker() {
void updateChunkInfo(long chunkOffset, int mapIndex) throws IOException {
long idxStartPos = -1;
try {
- // update the chunk tracker to meta file before index file
- writeChunkTracker(mapIndex);
idxStartPos = indexFile.getFilePointer();
logger.trace("{} shuffleId {} reduceId {} updated index current {} updated {}",
appShuffleId.appId, appShuffleId.shuffleId, reduceId, this.lastChunkOffset,
chunkOffset);
- indexFile.writeLong(chunkOffset);
+ indexFile.write(Longs.toByteArray(chunkOffset));
+ // Chunk bitmap should be written to the meta file after the index file because if there are
+ // any exceptions during writing the offset to the index file, meta file should not be
+ // updated. If the update to the index file is successful but the update to meta file isn't
+ // then the index file position is reset in the catch clause.
+ writeChunkTracker(mapIndex);
Review comment:
@Victsm The case that you described how often does it happen? How does this cleanup script differentiates shuffle data from temporary user data and how quickly does it remove this data. This has an assumption that cluster is setup in a certain way.
Would you agree that the more common cases are the non-recoverable disk failures, permission issues, file corruption etc?
If yes, then there will be continuous IOExceptions while updating these index/meta file. So if we just let the chunk grow when the index/meta file couldn't be updated, in most of the cases the executor will fail to fetch these.
In the non-push shuffle mechanism, if writing to index file fails does the executor try multiple times to write to the same file locally? I understand that in the push-mechanism, we are updating the index/meta in a continuous manner but these are also just local files.
> I think that policy should require more than 2 consecutive IOExceptions during chunk metadata write to trigger.
Note that, the same issue could affect writing data file as well.
Just a single IOException while updating the index/meta file should be enough to indicate the client to stop pushing. Yes, it affects writing the data file as well. The reason I am not bringing that up because in this change we are focusing on the exceptions while updating metadata. If others think that we should address the failures during writes to data file as well in this followup fix, we can address it.
> If we want to address this issue holistically, it would require some state tracking.
If so, I'd rather track it on the client side instead of the server side.
We are already tracking the state of shuffle partition in `AppShufflePartitionInfo`. Adding few other flags to it shouldn't be an issue.
Would like to know the thoughts of other folks as well?
@mridulm @tgravescs @attilapiros @Ngone51
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #30433: [SPARK-32916][SHUFFLE][test-maven][test-hadoop2.7] Ensure the number of chunks in meta file and index file are equal
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #30433:
URL: https://github.com/apache/spark/pull/30433#issuecomment-737998185
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #30433: [SPARK-32916][SHUFFLE][test-maven][test-hadoop2.7] Ensure the number of chunks in meta file and index file are equal
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #30433:
URL: https://github.com/apache/spark/pull/30433#issuecomment-740936434
Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/37046/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] Victsm commented on a change in pull request #30433: [SPARK-32916][SHUFFLE][test-maven][test-hadoop2.7] Ensure the number of chunks in meta file and index file are equal
Posted by GitBox <gi...@apache.org>.
Victsm commented on a change in pull request #30433:
URL: https://github.com/apache/spark/pull/30433#discussion_r540455676
##########
File path: common/network-shuffle/src/main/java/org/apache/spark/network/shuffle/RemoteBlockPushResolver.java
##########
@@ -586,6 +592,7 @@ public void onData(String streamId, ByteBuffer buf) throws IOException {
deferredBufs = null;
return;
}
+ abortIfNecessary();
Review comment:
@Ngone51 @mridulm , by throwing exception inside `onComplete`, we are still effectively rejecting any new blocks from being successfully appended to the file.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] Ngone51 commented on a change in pull request #30433: [SPARK-32916][SHUFFLE][test-maven][test-hadoop2.7] Ensure the number of chunks in meta file and index file are equal
Posted by GitBox <gi...@apache.org>.
Ngone51 commented on a change in pull request #30433:
URL: https://github.com/apache/spark/pull/30433#discussion_r529604208
##########
File path: common/network-shuffle/src/main/java/org/apache/spark/network/shuffle/RemoteBlockPushResolver.java
##########
@@ -827,13 +833,16 @@ void resetChunkTracker() {
void updateChunkInfo(long chunkOffset, int mapIndex) throws IOException {
long idxStartPos = -1;
try {
- // update the chunk tracker to meta file before index file
- writeChunkTracker(mapIndex);
idxStartPos = indexFile.getFilePointer();
logger.trace("{} shuffleId {} reduceId {} updated index current {} updated {}",
appShuffleId.appId, appShuffleId.shuffleId, reduceId, this.lastChunkOffset,
chunkOffset);
- indexFile.writeLong(chunkOffset);
+ indexFile.write(Longs.toByteArray(chunkOffset));
+ // Chunk bitmap should be written to the meta file after the index file because if there are
+ // any exceptions during writing the offset to the index file, meta file should not be
+ // updated. If the update to the index file is successful but the update to meta file isn't
+ // then the index file position is reset in the catch clause.
Review comment:
Yeah..but the current implementation seems not correct. It still update the `lastChunkOffset` when exception happens, no?
https://github.com/apache/spark/blob/8113c88542ee282b510c7e046d64df1761a85d14/common/network-shuffle/src/main/java/org/apache/spark/network/shuffle/RemoteBlockPushResolver.java#L846
##########
File path: common/network-shuffle/src/main/java/org/apache/spark/network/shuffle/RemoteBlockPushResolver.java
##########
@@ -827,13 +833,16 @@ void resetChunkTracker() {
void updateChunkInfo(long chunkOffset, int mapIndex) throws IOException {
long idxStartPos = -1;
try {
- // update the chunk tracker to meta file before index file
- writeChunkTracker(mapIndex);
idxStartPos = indexFile.getFilePointer();
logger.trace("{} shuffleId {} reduceId {} updated index current {} updated {}",
appShuffleId.appId, appShuffleId.shuffleId, reduceId, this.lastChunkOffset,
chunkOffset);
- indexFile.writeLong(chunkOffset);
+ indexFile.write(Longs.toByteArray(chunkOffset));
+ // Chunk bitmap should be written to the meta file after the index file because if there are
+ // any exceptions during writing the offset to the index file, meta file should not be
+ // updated. If the update to the index file is successful but the update to meta file isn't
+ // then the index file position is reset in the catch clause.
Review comment:
Yeah..I mean the current implementation seems not correct. It still update the `lastChunkOffset` when exception happens, no?
https://github.com/apache/spark/blob/8113c88542ee282b510c7e046d64df1761a85d14/common/network-shuffle/src/main/java/org/apache/spark/network/shuffle/RemoteBlockPushResolver.java#L846
##########
File path: common/network-shuffle/src/main/java/org/apache/spark/network/shuffle/RemoteBlockPushResolver.java
##########
@@ -827,13 +833,16 @@ void resetChunkTracker() {
void updateChunkInfo(long chunkOffset, int mapIndex) throws IOException {
long idxStartPos = -1;
try {
- // update the chunk tracker to meta file before index file
- writeChunkTracker(mapIndex);
idxStartPos = indexFile.getFilePointer();
logger.trace("{} shuffleId {} reduceId {} updated index current {} updated {}",
appShuffleId.appId, appShuffleId.shuffleId, reduceId, this.lastChunkOffset,
chunkOffset);
- indexFile.writeLong(chunkOffset);
+ indexFile.write(Longs.toByteArray(chunkOffset));
+ // Chunk bitmap should be written to the meta file after the index file because if there are
+ // any exceptions during writing the offset to the index file, meta file should not be
+ // updated. If the update to the index file is successful but the update to meta file isn't
+ // then the index file position is reset in the catch clause.
Review comment:
Yeah..I mean the current implementation seems not correct. It still updates the `lastChunkOffset` when exception happens, no?
https://github.com/apache/spark/blob/8113c88542ee282b510c7e046d64df1761a85d14/common/network-shuffle/src/main/java/org/apache/spark/network/shuffle/RemoteBlockPushResolver.java#L846
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #30433: [SPARK-32916][SHUFFLE][test-maven][test-hadoop2.7] Ensure the number of chunks in meta file and index file are equal
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #30433:
URL: https://github.com/apache/spark/pull/30433#issuecomment-737692908
**[Test build #132098 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/132098/testReport)** for PR 30433 at commit [`f5783a9`](https://github.com/apache/spark/commit/f5783a9e3f83d91308572230a7d050e000c49fa7).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] otterc commented on a change in pull request #30433: [SPARK-32916][SHUFFLE][test-maven][test-hadoop2.7] Ensure the number of chunks in meta file and index file are equal
Posted by GitBox <gi...@apache.org>.
otterc commented on a change in pull request #30433:
URL: https://github.com/apache/spark/pull/30433#discussion_r539500739
##########
File path: common/network-shuffle/src/main/java/org/apache/spark/network/shuffle/RemoteBlockPushResolver.java
##########
@@ -586,6 +592,7 @@ public void onData(String streamId, ByteBuffer buf) throws IOException {
deferredBufs = null;
return;
}
+ abortIfNecessary();
Review comment:
@Victsm This change is to address this comment:
https://github.com/apache/spark/pull/30433#discussion_r538947833
The scenario is that there can be pending streams which are waiting on the lock for the `partitionInfo` and meanwhile the exception threshold has met. When the pending stream acquires the lock it will attempt to write to the data file even though the exception threshold is reached.
I have added the UT `testPendingBlockIsAbortedImmediately` to verify this.
> Purpose is to only do this check once per block instead of once per buf and avoid throwing unnecessary exceptions from onData which leads to channel close.
We already throw IOExceptions when write to data file fails. I don't see how throwing `IOException exceeded threshold` makes it any worse.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] Ngone51 commented on a change in pull request #30433: [SPARK-32916][SHUFFLE][test-maven][test-hadoop2.7] Ensure the number of chunks in meta file and index file are equal
Posted by GitBox <gi...@apache.org>.
Ngone51 commented on a change in pull request #30433:
URL: https://github.com/apache/spark/pull/30433#discussion_r538947833
##########
File path: common/network-shuffle/src/test/java/org/apache/spark/network/shuffle/RemoteBlockPushResolverSuite.java
##########
@@ -411,6 +415,269 @@ void deleteExecutorDirs(Path[] dirs) {
}
}
+ @Test
+ public void testRecoverIndexFileAfterIOExceptions() throws IOException {
+ useTestFiles(true, false);
+ RemoteBlockPushResolver.PushBlockStreamCallback callback1 =
+ (RemoteBlockPushResolver.PushBlockStreamCallback) pushResolver.receiveBlockDataAsStream(
+ new PushBlockStream(TEST_APP, 0, 0, 0, 0));
+ callback1.onData(callback1.getID(), ByteBuffer.wrap(new byte[4]));
+ callback1.onComplete(callback1.getID());
+ RemoteBlockPushResolver.AppShufflePartitionInfo partitionInfo = callback1.getPartitionInfo();
+ // Close the index stream so it throws IOException
+ TestMergeShuffleFile testIndexFile = (TestMergeShuffleFile) partitionInfo.getIndexFile();
+ testIndexFile.close();
+ StreamCallbackWithID callback2 = pushResolver.receiveBlockDataAsStream(
+ new PushBlockStream(TEST_APP, 0, 1, 0, 0));
+ callback2.onData(callback2.getID(), ByteBuffer.wrap(new byte[5]));
+ // This will complete without any IOExceptions because number of IOExceptions are less than
+ // the threshold but the update to index file will be unsuccessful.
+ callback2.onComplete(callback2.getID());
+ assertEquals("index position", 16, testIndexFile.getPos());
+ // Restore the index stream so it can write successfully again.
+ testIndexFile.restore();
+ StreamCallbackWithID callback3 = pushResolver.receiveBlockDataAsStream(
+ new PushBlockStream(TEST_APP, 0, 2, 0, 0));
+ callback3.onData(callback3.getID(), ByteBuffer.wrap(new byte[2]));
+ callback3.onComplete(callback3.getID());
+ assertEquals("index position", 24, testIndexFile.getPos());
+ MergeStatuses statuses = pushResolver.finalizeShuffleMerge(
+ new FinalizeShuffleMerge(TEST_APP, 0));
+ validateMergeStatuses(statuses, new int[] {0}, new long[] {11});
+ MergedBlockMeta blockMeta = pushResolver.getMergedBlockMeta(TEST_APP, 0, 0);
+ validateChunks(TEST_APP, 0, 0, blockMeta, new int[] {4, 7}, new int[][] {{0}, {1, 2}});
+ }
+
+ @Test
+ public void testRecoverIndexFileAfterIOExceptionsInFinalize() throws IOException {
+ useTestFiles(true, false);
+ registerExecutor(TEST_APP, prepareLocalDirs(localDirs));
+ RemoteBlockPushResolver.PushBlockStreamCallback callback1 =
+ (RemoteBlockPushResolver.PushBlockStreamCallback) pushResolver.receiveBlockDataAsStream(
+ new PushBlockStream(TEST_APP, 0, 0, 0, 0));
+ callback1.onData(callback1.getID(), ByteBuffer.wrap(new byte[4]));
+ callback1.onComplete(callback1.getID());
+ RemoteBlockPushResolver.AppShufflePartitionInfo partitionInfo = callback1.getPartitionInfo();
+ // Close the index stream so it throws IOException
+ TestMergeShuffleFile testIndexFile = (TestMergeShuffleFile) partitionInfo.getIndexFile();
+ testIndexFile.close();
+ StreamCallbackWithID callback2 = pushResolver.receiveBlockDataAsStream(
+ new PushBlockStream(TEST_APP, 0, 1, 0, 0));
+ callback2.onData(callback2.getID(), ByteBuffer.wrap(new byte[5]));
+ // This will complete without any IOExceptions because number of IOExceptions are less than
+ // the threshold but the update to index file will be unsuccessful.
+ callback2.onComplete(callback2.getID());
+ assertEquals("index position", 16, testIndexFile.getPos());
+ // The last update to index was unsuccessful however any further updates will be successful.
+ // Restore the index stream so it can write successfully again.
+ testIndexFile.restore();
+ MergeStatuses statuses = pushResolver.finalizeShuffleMerge(
+ new FinalizeShuffleMerge(TEST_APP, 0));
+ assertEquals("index position", 24, testIndexFile.getPos());
+ validateMergeStatuses(statuses, new int[] {0}, new long[] {9});
+ MergedBlockMeta blockMeta = pushResolver.getMergedBlockMeta(TEST_APP, 0, 0);
+ validateChunks(TEST_APP, 0, 0, blockMeta, new int[] {4, 5}, new int[][] {{0}, {1}});
+ }
+
+ @Test
+ public void testRecoverMetaFileAfterIOExceptions() throws IOException {
+ useTestFiles(false, true);
+ RemoteBlockPushResolver.PushBlockStreamCallback callback1 =
+ (RemoteBlockPushResolver.PushBlockStreamCallback) pushResolver.receiveBlockDataAsStream(
+ new PushBlockStream(TEST_APP, 0, 0, 0, 0));
+ callback1.onData(callback1.getID(), ByteBuffer.wrap(new byte[4]));
+ callback1.onComplete(callback1.getID());
+ RemoteBlockPushResolver.AppShufflePartitionInfo partitionInfo = callback1.getPartitionInfo();
+ // Close the meta stream so it throws IOException
+ TestMergeShuffleFile testMetaFile = (TestMergeShuffleFile) partitionInfo.getMetaFile();
+ long metaPosBeforeClose = testMetaFile.getPos();
+ testMetaFile.close();
+ StreamCallbackWithID callback2 = pushResolver.receiveBlockDataAsStream(
+ new PushBlockStream(TEST_APP, 0, 1, 0, 0));
+ callback2.onData(callback2.getID(), ByteBuffer.wrap(new byte[5]));
+ // This will complete without any IOExceptions because number of IOExceptions are less than
+ // the threshold but the update to index and meta file will be unsuccessful.
+ callback2.onComplete(callback2.getID());
+ assertEquals("index position", 16, partitionInfo.getIndexFile().getPos());
+ assertEquals("meta position", metaPosBeforeClose, testMetaFile.getPos());
+ // Restore the meta stream so it can write successfully again.
+ testMetaFile.restore();
+ StreamCallbackWithID callback3 = pushResolver.receiveBlockDataAsStream(
+ new PushBlockStream(TEST_APP, 0, 2, 0, 0));
+ callback3.onData(callback3.getID(), ByteBuffer.wrap(new byte[2]));
+ callback3.onComplete(callback3.getID());
+ assertEquals("index position", 24, partitionInfo.getIndexFile().getPos());
+ assertTrue("meta position", testMetaFile.getPos() > metaPosBeforeClose);
+ MergeStatuses statuses = pushResolver.finalizeShuffleMerge(
+ new FinalizeShuffleMerge(TEST_APP, 0));
+ validateMergeStatuses(statuses, new int[] {0}, new long[] {11});
+ MergedBlockMeta blockMeta = pushResolver.getMergedBlockMeta(TEST_APP, 0, 0);
+ validateChunks(TEST_APP, 0, 0, blockMeta, new int[] {4, 7}, new int[][] {{0}, {1, 2}});
+ }
+
+ @Test
+ public void testRecoverMetaFileAfterIOExceptionsInFinalize() throws IOException {
+ useTestFiles(false, true);
+ RemoteBlockPushResolver.PushBlockStreamCallback callback1 =
+ (RemoteBlockPushResolver.PushBlockStreamCallback) pushResolver.receiveBlockDataAsStream(
+ new PushBlockStream(TEST_APP, 0, 0, 0, 0));
+ callback1.onData(callback1.getID(), ByteBuffer.wrap(new byte[4]));
+ callback1.onComplete(callback1.getID());
+ RemoteBlockPushResolver.AppShufflePartitionInfo partitionInfo = callback1.getPartitionInfo();
+ // Close the meta stream so it throws IOException
+ TestMergeShuffleFile testMetaFile = (TestMergeShuffleFile) partitionInfo.getMetaFile();
+ long metaPosBeforeClose = testMetaFile.getPos();
+ testMetaFile.close();
+ StreamCallbackWithID callback2 = pushResolver.receiveBlockDataAsStream(
+ new PushBlockStream(TEST_APP, 0, 1, 0, 0));
+ callback2.onData(callback2.getID(), ByteBuffer.wrap(new byte[5]));
+ // This will complete without any IOExceptions because number of IOExceptions are less than
+ // the threshold but the update to index and meta file will be unsuccessful.
+ callback2.onComplete(callback2.getID());
+ MergeShuffleFile indexFile = partitionInfo.getIndexFile();
+ assertEquals("index position", 16, indexFile.getPos());
+ assertEquals("meta position", metaPosBeforeClose, testMetaFile.getPos());
+ // Restore the meta stream so it can write successfully again.
+ testMetaFile.restore();
+ MergeStatuses statuses = pushResolver.finalizeShuffleMerge(
+ new FinalizeShuffleMerge(TEST_APP, 0));
+ assertEquals("index position", 24, indexFile.getPos());
+ assertTrue("meta position", testMetaFile.getPos() > metaPosBeforeClose);
+ validateMergeStatuses(statuses, new int[] {0}, new long[] {9});
+ MergedBlockMeta blockMeta = pushResolver.getMergedBlockMeta(TEST_APP, 0, 0);
+ validateChunks(TEST_APP, 0, 0, blockMeta, new int[] {4, 5}, new int[][] {{0}, {1}});
+ }
+
+ @Test (expected = RuntimeException.class)
+ public void testIOExceptionsExceededThreshold() throws IOException {
+ RemoteBlockPushResolver.PushBlockStreamCallback callback =
+ (RemoteBlockPushResolver.PushBlockStreamCallback) pushResolver.receiveBlockDataAsStream(
+ new PushBlockStream(TEST_APP, 0, 0, 0, 0));
+ RemoteBlockPushResolver.AppShufflePartitionInfo partitionInfo = callback.getPartitionInfo();
+ callback.onData(callback.getID(), ByteBuffer.wrap(new byte[4]));
+ callback.onComplete(callback.getID());
+ // Close the data stream so it throws continuous IOException
+ partitionInfo.getDataChannel().close();
+ for (int i = 1; i < 5; i++) {
+ RemoteBlockPushResolver.PushBlockStreamCallback callback1 =
+ (RemoteBlockPushResolver.PushBlockStreamCallback) pushResolver.receiveBlockDataAsStream(
+ new PushBlockStream(TEST_APP, 0, i, 0, 0));
+ try {
+ callback1.onData(callback1.getID(), ByteBuffer.wrap(new byte[2]));
+ } catch (IOException ioe) {
+ // this will throw IOException so the client can retry.
+ callback1.onFailure(callback1.getID(), ioe);
+ }
+ }
+ assertEquals(4, partitionInfo.getNumIOExceptions());
+ // After 4 IOException, the server will respond with IOExceptions exceeded threshold
+ try {
+ RemoteBlockPushResolver.PushBlockStreamCallback callback2 =
+ (RemoteBlockPushResolver.PushBlockStreamCallback) pushResolver.receiveBlockDataAsStream(
+ new PushBlockStream(TEST_APP, 0, 5, 0, 0));
+ callback2.onData(callback.getID(), ByteBuffer.wrap(new byte[1]));
+ } catch (Throwable t) {
+ assertEquals("IOExceptions exceeded the threshold when merging shufflePush_0_5_0",
+ t.getMessage());
+ throw t;
+ }
Review comment:
It's `deferredBufs` rather than deferred blocks, right? I actually mean those blocks which are waiting for the partition lock at the same time.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] otterc commented on a change in pull request #30433: [SPARK-32916][SHUFFLE][test-maven][test-hadoop2.7] Ensure the number of chunks in meta file and index file are equal
Posted by GitBox <gi...@apache.org>.
otterc commented on a change in pull request #30433:
URL: https://github.com/apache/spark/pull/30433#discussion_r538950538
##########
File path: common/network-shuffle/src/test/java/org/apache/spark/network/shuffle/RemoteBlockPushResolverSuite.java
##########
@@ -411,6 +415,269 @@ void deleteExecutorDirs(Path[] dirs) {
}
}
+ @Test
+ public void testRecoverIndexFileAfterIOExceptions() throws IOException {
+ useTestFiles(true, false);
+ RemoteBlockPushResolver.PushBlockStreamCallback callback1 =
+ (RemoteBlockPushResolver.PushBlockStreamCallback) pushResolver.receiveBlockDataAsStream(
+ new PushBlockStream(TEST_APP, 0, 0, 0, 0));
+ callback1.onData(callback1.getID(), ByteBuffer.wrap(new byte[4]));
+ callback1.onComplete(callback1.getID());
+ RemoteBlockPushResolver.AppShufflePartitionInfo partitionInfo = callback1.getPartitionInfo();
+ // Close the index stream so it throws IOException
+ TestMergeShuffleFile testIndexFile = (TestMergeShuffleFile) partitionInfo.getIndexFile();
+ testIndexFile.close();
+ StreamCallbackWithID callback2 = pushResolver.receiveBlockDataAsStream(
+ new PushBlockStream(TEST_APP, 0, 1, 0, 0));
+ callback2.onData(callback2.getID(), ByteBuffer.wrap(new byte[5]));
+ // This will complete without any IOExceptions because number of IOExceptions are less than
+ // the threshold but the update to index file will be unsuccessful.
+ callback2.onComplete(callback2.getID());
+ assertEquals("index position", 16, testIndexFile.getPos());
+ // Restore the index stream so it can write successfully again.
+ testIndexFile.restore();
+ StreamCallbackWithID callback3 = pushResolver.receiveBlockDataAsStream(
+ new PushBlockStream(TEST_APP, 0, 2, 0, 0));
+ callback3.onData(callback3.getID(), ByteBuffer.wrap(new byte[2]));
+ callback3.onComplete(callback3.getID());
+ assertEquals("index position", 24, testIndexFile.getPos());
+ MergeStatuses statuses = pushResolver.finalizeShuffleMerge(
+ new FinalizeShuffleMerge(TEST_APP, 0));
+ validateMergeStatuses(statuses, new int[] {0}, new long[] {11});
+ MergedBlockMeta blockMeta = pushResolver.getMergedBlockMeta(TEST_APP, 0, 0);
+ validateChunks(TEST_APP, 0, 0, blockMeta, new int[] {4, 7}, new int[][] {{0}, {1, 2}});
+ }
+
+ @Test
+ public void testRecoverIndexFileAfterIOExceptionsInFinalize() throws IOException {
+ useTestFiles(true, false);
+ registerExecutor(TEST_APP, prepareLocalDirs(localDirs));
+ RemoteBlockPushResolver.PushBlockStreamCallback callback1 =
+ (RemoteBlockPushResolver.PushBlockStreamCallback) pushResolver.receiveBlockDataAsStream(
+ new PushBlockStream(TEST_APP, 0, 0, 0, 0));
+ callback1.onData(callback1.getID(), ByteBuffer.wrap(new byte[4]));
+ callback1.onComplete(callback1.getID());
+ RemoteBlockPushResolver.AppShufflePartitionInfo partitionInfo = callback1.getPartitionInfo();
+ // Close the index stream so it throws IOException
+ TestMergeShuffleFile testIndexFile = (TestMergeShuffleFile) partitionInfo.getIndexFile();
+ testIndexFile.close();
+ StreamCallbackWithID callback2 = pushResolver.receiveBlockDataAsStream(
+ new PushBlockStream(TEST_APP, 0, 1, 0, 0));
+ callback2.onData(callback2.getID(), ByteBuffer.wrap(new byte[5]));
+ // This will complete without any IOExceptions because number of IOExceptions are less than
+ // the threshold but the update to index file will be unsuccessful.
+ callback2.onComplete(callback2.getID());
+ assertEquals("index position", 16, testIndexFile.getPos());
+ // The last update to index was unsuccessful however any further updates will be successful.
+ // Restore the index stream so it can write successfully again.
+ testIndexFile.restore();
+ MergeStatuses statuses = pushResolver.finalizeShuffleMerge(
+ new FinalizeShuffleMerge(TEST_APP, 0));
+ assertEquals("index position", 24, testIndexFile.getPos());
+ validateMergeStatuses(statuses, new int[] {0}, new long[] {9});
+ MergedBlockMeta blockMeta = pushResolver.getMergedBlockMeta(TEST_APP, 0, 0);
+ validateChunks(TEST_APP, 0, 0, blockMeta, new int[] {4, 5}, new int[][] {{0}, {1}});
+ }
+
+ @Test
+ public void testRecoverMetaFileAfterIOExceptions() throws IOException {
+ useTestFiles(false, true);
+ RemoteBlockPushResolver.PushBlockStreamCallback callback1 =
+ (RemoteBlockPushResolver.PushBlockStreamCallback) pushResolver.receiveBlockDataAsStream(
+ new PushBlockStream(TEST_APP, 0, 0, 0, 0));
+ callback1.onData(callback1.getID(), ByteBuffer.wrap(new byte[4]));
+ callback1.onComplete(callback1.getID());
+ RemoteBlockPushResolver.AppShufflePartitionInfo partitionInfo = callback1.getPartitionInfo();
+ // Close the meta stream so it throws IOException
+ TestMergeShuffleFile testMetaFile = (TestMergeShuffleFile) partitionInfo.getMetaFile();
+ long metaPosBeforeClose = testMetaFile.getPos();
+ testMetaFile.close();
+ StreamCallbackWithID callback2 = pushResolver.receiveBlockDataAsStream(
+ new PushBlockStream(TEST_APP, 0, 1, 0, 0));
+ callback2.onData(callback2.getID(), ByteBuffer.wrap(new byte[5]));
+ // This will complete without any IOExceptions because number of IOExceptions are less than
+ // the threshold but the update to index and meta file will be unsuccessful.
+ callback2.onComplete(callback2.getID());
+ assertEquals("index position", 16, partitionInfo.getIndexFile().getPos());
+ assertEquals("meta position", metaPosBeforeClose, testMetaFile.getPos());
+ // Restore the meta stream so it can write successfully again.
+ testMetaFile.restore();
+ StreamCallbackWithID callback3 = pushResolver.receiveBlockDataAsStream(
+ new PushBlockStream(TEST_APP, 0, 2, 0, 0));
+ callback3.onData(callback3.getID(), ByteBuffer.wrap(new byte[2]));
+ callback3.onComplete(callback3.getID());
+ assertEquals("index position", 24, partitionInfo.getIndexFile().getPos());
+ assertTrue("meta position", testMetaFile.getPos() > metaPosBeforeClose);
+ MergeStatuses statuses = pushResolver.finalizeShuffleMerge(
+ new FinalizeShuffleMerge(TEST_APP, 0));
+ validateMergeStatuses(statuses, new int[] {0}, new long[] {11});
+ MergedBlockMeta blockMeta = pushResolver.getMergedBlockMeta(TEST_APP, 0, 0);
+ validateChunks(TEST_APP, 0, 0, blockMeta, new int[] {4, 7}, new int[][] {{0}, {1, 2}});
+ }
+
+ @Test
+ public void testRecoverMetaFileAfterIOExceptionsInFinalize() throws IOException {
+ useTestFiles(false, true);
+ RemoteBlockPushResolver.PushBlockStreamCallback callback1 =
+ (RemoteBlockPushResolver.PushBlockStreamCallback) pushResolver.receiveBlockDataAsStream(
+ new PushBlockStream(TEST_APP, 0, 0, 0, 0));
+ callback1.onData(callback1.getID(), ByteBuffer.wrap(new byte[4]));
+ callback1.onComplete(callback1.getID());
+ RemoteBlockPushResolver.AppShufflePartitionInfo partitionInfo = callback1.getPartitionInfo();
+ // Close the meta stream so it throws IOException
+ TestMergeShuffleFile testMetaFile = (TestMergeShuffleFile) partitionInfo.getMetaFile();
+ long metaPosBeforeClose = testMetaFile.getPos();
+ testMetaFile.close();
+ StreamCallbackWithID callback2 = pushResolver.receiveBlockDataAsStream(
+ new PushBlockStream(TEST_APP, 0, 1, 0, 0));
+ callback2.onData(callback2.getID(), ByteBuffer.wrap(new byte[5]));
+ // This will complete without any IOExceptions because number of IOExceptions are less than
+ // the threshold but the update to index and meta file will be unsuccessful.
+ callback2.onComplete(callback2.getID());
+ MergeShuffleFile indexFile = partitionInfo.getIndexFile();
+ assertEquals("index position", 16, indexFile.getPos());
+ assertEquals("meta position", metaPosBeforeClose, testMetaFile.getPos());
+ // Restore the meta stream so it can write successfully again.
+ testMetaFile.restore();
+ MergeStatuses statuses = pushResolver.finalizeShuffleMerge(
+ new FinalizeShuffleMerge(TEST_APP, 0));
+ assertEquals("index position", 24, indexFile.getPos());
+ assertTrue("meta position", testMetaFile.getPos() > metaPosBeforeClose);
+ validateMergeStatuses(statuses, new int[] {0}, new long[] {9});
+ MergedBlockMeta blockMeta = pushResolver.getMergedBlockMeta(TEST_APP, 0, 0);
+ validateChunks(TEST_APP, 0, 0, blockMeta, new int[] {4, 5}, new int[][] {{0}, {1}});
+ }
+
+ @Test (expected = RuntimeException.class)
+ public void testIOExceptionsExceededThreshold() throws IOException {
+ RemoteBlockPushResolver.PushBlockStreamCallback callback =
+ (RemoteBlockPushResolver.PushBlockStreamCallback) pushResolver.receiveBlockDataAsStream(
+ new PushBlockStream(TEST_APP, 0, 0, 0, 0));
+ RemoteBlockPushResolver.AppShufflePartitionInfo partitionInfo = callback.getPartitionInfo();
+ callback.onData(callback.getID(), ByteBuffer.wrap(new byte[4]));
+ callback.onComplete(callback.getID());
+ // Close the data stream so it throws continuous IOException
+ partitionInfo.getDataChannel().close();
+ for (int i = 1; i < 5; i++) {
+ RemoteBlockPushResolver.PushBlockStreamCallback callback1 =
+ (RemoteBlockPushResolver.PushBlockStreamCallback) pushResolver.receiveBlockDataAsStream(
+ new PushBlockStream(TEST_APP, 0, i, 0, 0));
+ try {
+ callback1.onData(callback1.getID(), ByteBuffer.wrap(new byte[2]));
+ } catch (IOException ioe) {
+ // this will throw IOException so the client can retry.
+ callback1.onFailure(callback1.getID(), ioe);
+ }
+ }
+ assertEquals(4, partitionInfo.getNumIOExceptions());
+ // After 4 IOException, the server will respond with IOExceptions exceeded threshold
+ try {
+ RemoteBlockPushResolver.PushBlockStreamCallback callback2 =
+ (RemoteBlockPushResolver.PushBlockStreamCallback) pushResolver.receiveBlockDataAsStream(
+ new PushBlockStream(TEST_APP, 0, 5, 0, 0));
+ callback2.onData(callback.getID(), ByteBuffer.wrap(new byte[1]));
+ } catch (Throwable t) {
+ assertEquals("IOExceptions exceeded the threshold when merging shufflePush_0_5_0",
+ t.getMessage());
+ throw t;
+ }
Review comment:
Yes, I meant `deferredBufs`.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #30433: [SPARK-32916][SHUFFLE][test-maven][test-hadoop2.7] Ensure the number of chunks in meta file and index file are equal
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #30433:
URL: https://github.com/apache/spark/pull/30433#issuecomment-741974173
**[Test build #132507 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/132507/testReport)** for PR 30433 at commit [`f48d52f`](https://github.com/apache/spark/commit/f48d52f68e52b63742e7cce5dbb4fbcc89845402).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #30433: [SPARK-32916][SHUFFLE][test-maven][test-hadoop2.7] Ensure the number of chunks in meta file and index file are equal
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #30433:
URL: https://github.com/apache/spark/pull/30433#issuecomment-738336375
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/36759/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] mridulm commented on a change in pull request #30433: [SPARK-32916][SHUFFLE][test-maven][test-hadoop2.7] Ensure the number of chunks in meta file and index file are equal
Posted by GitBox <gi...@apache.org>.
mridulm commented on a change in pull request #30433:
URL: https://github.com/apache/spark/pull/30433#discussion_r528175549
##########
File path: common/network-shuffle/src/main/java/org/apache/spark/network/shuffle/RemoteBlockPushResolver.java
##########
@@ -827,13 +833,16 @@ void resetChunkTracker() {
void updateChunkInfo(long chunkOffset, int mapIndex) throws IOException {
long idxStartPos = -1;
try {
- // update the chunk tracker to meta file before index file
- writeChunkTracker(mapIndex);
idxStartPos = indexFile.getFilePointer();
logger.trace("{} shuffleId {} reduceId {} updated index current {} updated {}",
appShuffleId.appId, appShuffleId.shuffleId, reduceId, this.lastChunkOffset,
chunkOffset);
- indexFile.writeLong(chunkOffset);
+ indexFile.write(Longs.toByteArray(chunkOffset));
+ // Chunk bitmap should be written to the meta file after the index file because if there are
+ // any exceptions during writing the offset to the index file, meta file should not be
+ // updated. If the update to the index file is successful but the update to meta file isn't
+ // then the index file position is reset in the catch clause.
+ writeChunkTracker(mapIndex);
Review comment:
Given the focus here is error handling, how are we handling a `seek` failure in `catch` block ?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] otterc edited a comment on pull request #30433: [SPARK-32916][SHUFFLE][test-maven][test-hadoop2.7] Ensure the number of chunks in meta file and index file are equal
Posted by GitBox <gi...@apache.org>.
otterc edited a comment on pull request #30433:
URL: https://github.com/apache/spark/pull/30433#issuecomment-737477581
While making these changes and adding the unit tests I think it's better to not rely on the file pointer from RandomAccessFile and maintain our own pointer. Thus, not use RandomAccessFile at all.
The reason is that let's say `seek` fails for index file and after that we abort merging any new merge blocks. However, when we finalize that partition we still need the pointer to the last successful update made to the index file of this partition.
Since the seek had failed, we can't rely on the pointer from RAF. I think this also simplifies the exception handling code.
Also, `RAF.getFilePointer` can throw IOException and by now it seems to me that using RAF is not simplifying but making the code more complicated.
Let me know if you think otherwise.
@mridulm @Victsm @Ngone51
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #30433: [SPARK-32916][SHUFFLE][test-maven][test-hadoop2.7] Ensure the number of chunks in meta file and index file are equal
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #30433:
URL: https://github.com/apache/spark/pull/30433#issuecomment-740893039
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/37046/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #30433: [SPARK-32916][SHUFFLE][test-maven][test-hadoop2.7] Ensure the number of chunks in meta file and index file are equal
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #30433:
URL: https://github.com/apache/spark/pull/30433#issuecomment-742199827
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/132507/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] otterc commented on a change in pull request #30433: [SPARK-32916][SHUFFLE][test-maven][test-hadoop2.7] Ensure the number of chunks in meta file and index file are equal
Posted by GitBox <gi...@apache.org>.
otterc commented on a change in pull request #30433:
URL: https://github.com/apache/spark/pull/30433#discussion_r528907429
##########
File path: common/network-shuffle/src/main/java/org/apache/spark/network/shuffle/RemoteBlockPushResolver.java
##########
@@ -827,13 +833,16 @@ void resetChunkTracker() {
void updateChunkInfo(long chunkOffset, int mapIndex) throws IOException {
long idxStartPos = -1;
try {
- // update the chunk tracker to meta file before index file
- writeChunkTracker(mapIndex);
idxStartPos = indexFile.getFilePointer();
logger.trace("{} shuffleId {} reduceId {} updated index current {} updated {}",
appShuffleId.appId, appShuffleId.shuffleId, reduceId, this.lastChunkOffset,
chunkOffset);
- indexFile.writeLong(chunkOffset);
+ indexFile.write(Longs.toByteArray(chunkOffset));
+ // Chunk bitmap should be written to the meta file after the index file because if there are
+ // any exceptions during writing the offset to the index file, meta file should not be
+ // updated. If the update to the index file is successful but the update to meta file isn't
+ // then the index file position is reset in the catch clause.
+ writeChunkTracker(mapIndex);
Review comment:
@Victsm The case that you described how often does it happen? How does this cleanup script differentiates shuffle data from temporary user data and how quickly does it remove this data. This has an assumption that cluster is setup in a certain way.
Would you agree that the more common cases are the non-recoverable disk failures, permission issues, file corruption etc?
If yes, then there will be continuous IOExceptions while updating these index/meta file. So if we just let the chunk grow when the index/meta file couldn't be updated, in most of the cases the executor will fail to fetch these.
In the non-push shuffle mechanism, if writing to index file fails does the executor try multiple times to write to the same file locally? I understand that in the push-mechanism, we are updating the index/meta in a continuous manner but these are also just local files.
> I think that policy should require more than 2 consecutive IOExceptions during chunk metadata write to trigger.
Note that, the same issue could affect writing data file as well.
Just a single IOException while updating the index/meta file should be enough to indicate the client to stop pushing. Yes, it affects writing the data file as well. The reason I am not bringing that up because in this followup fix we are focusing on the exceptions while updating metadata. If others think that we should address the failures during writes to data file as well in this followup fix, we can address it.
> If we want to address this issue holistically, it would require some state tracking.
If so, I'd rather track it on the client side instead of the server side.
We are already tracking the state of shuffle partition in `AppShufflePartitionInfo`. Adding few other flags to it shouldn't be an issue.
Would like to know the thoughts of other folks as well.
@mridulm @tgravescs @attilapiros @Ngone51
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on pull request #30433: [SPARK-32916][SHUFFLE][test-maven][test-hadoop2.7] Ensure the number of chunks in meta file and index file are equal
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on pull request #30433:
URL: https://github.com/apache/spark/pull/30433#issuecomment-750727119
Thanks for letting me know @mridulm.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] otterc commented on a change in pull request #30433: [SPARK-32916][SHUFFLE][test-maven][test-hadoop2.7] Ensure the number of chunks in meta file and index file are equal
Posted by GitBox <gi...@apache.org>.
otterc commented on a change in pull request #30433:
URL: https://github.com/apache/spark/pull/30433#discussion_r535532790
##########
File path: common/network-shuffle/src/main/java/org/apache/spark/network/shuffle/RemoteBlockPushResolver.java
##########
@@ -825,45 +881,79 @@ void resetChunkTracker() {
* @param mapIndex the map index to be added to chunk tracker.
*/
void updateChunkInfo(long chunkOffset, int mapIndex) throws IOException {
- long idxStartPos = -1;
try {
- // update the chunk tracker to meta file before index file
- writeChunkTracker(mapIndex);
- idxStartPos = indexFile.getFilePointer();
logger.trace("{} shuffleId {} reduceId {} updated index current {} updated {}",
- appShuffleId.appId, appShuffleId.shuffleId, reduceId, this.lastChunkOffset,
- chunkOffset);
- indexFile.writeLong(chunkOffset);
- } catch (IOException ioe) {
- if (idxStartPos != -1) {
- // reset the position to avoid corrupting index files during exception.
- logger.warn("{} shuffleId {} reduceId {} reset index to position {}",
- appShuffleId.appId, appShuffleId.shuffleId, reduceId, idxStartPos);
- indexFile.seek(idxStartPos);
+ appShuffleId.appId, appShuffleId.shuffleId, reduceId, this.lastChunkOffset, chunkOffset);
+ if (indexMetaUpdateFailed) {
+ indexFile.getChannel().position(indexFile.getPos());
}
+ indexFile.getDos().writeLong(chunkOffset);
+ // Chunk bitmap should be written to the meta file after the index file because if there are
+ // any exceptions during writing the offset to the index file, meta file should not be
+ // updated. If the update to the index file is successful but the update to meta file isn't
+ // then the index file position is reset in the catch clause.
+ writeChunkTracker(mapIndex);
+ indexFile.updatePos(8);
+ this.lastChunkOffset = chunkOffset;
+ indexMetaUpdateFailed = false;
+ } catch (IOException ioe) {
+ logger.warn("{} shuffleId {} reduceId {} update to index/meta failed", appShuffleId.appId,
+ appShuffleId.shuffleId, reduceId);
+ indexMetaUpdateFailed = true;
+ // Any exception here is propagated to the caller and the caller can decide whether to
+ // abort or not.
throw ioe;
}
- this.lastChunkOffset = chunkOffset;
}
private void writeChunkTracker(int mapIndex) throws IOException {
if (mapIndex == -1) {
return;
}
chunkTracker.add(mapIndex);
- long metaStartPos = metaFile.getFilePointer();
try {
logger.trace("{} shuffleId {} reduceId {} mapIndex {} write chunk to meta file",
appShuffleId.appId, appShuffleId.shuffleId, reduceId, mapIndex);
- chunkTracker.serialize(metaFile);
+ if (indexMetaUpdateFailed) {
+ metaFile.getChannel().position(metaFile.getPos());
+ }
+ int chunkTrackerSize = chunkTracker.serializedSizeInBytes();
+ chunkTracker.serialize(metaFile.getDos());
+ metaFile.updatePos(chunkTrackerSize);
} catch (IOException ioe) {
- logger.warn("{} shuffleId {} reduceId {} mapIndex {} reset position of meta file to {}",
- appShuffleId.appId, appShuffleId.shuffleId, reduceId, mapIndex, metaStartPos);
- metaFile.seek(metaStartPos);
+ logger.warn("{} shuffleId {} reduceId {} mapIndex {} update to meta file failed",
+ appShuffleId.appId, appShuffleId.shuffleId, reduceId, mapIndex);
+ // Any exceptions while writing to meta file is propagated to updateChunkInfo so that
+ // the index file position can be reset as well.
throw ioe;
}
}
+ private void incrementIOExceptions() {
+ numIOExceptions++;
+ }
+
+ private boolean shouldAbort(int ioExceptionsThresholdDuringMerge) {
+ return numIOExceptions > ioExceptionsThresholdDuringMerge;
+ }
+
+ private void finalizePartition() throws IOException {
+ if (dataFilePos != lastChunkOffset) {
+ try {
+ updateChunkInfo(dataFilePos, lastMergedMapIndex);
+ } catch (IOException ioe) {
+ // Any exceptions here while updating the meta files can be ignored. If the files
+ // aren't successfully updated they will be truncated.
+ }
+ }
+ // Get rid of any partial block data at the end of the file. This could either
+ // be due to failure, or a request still being processed when the shuffle
+ // merge gets finalized, or any exceptions while updating index/meta files.
+ dataChannel.truncate(lastChunkOffset);
+ indexFile.getChannel().truncate(indexFile.getPos());
+ metaFile.getChannel().truncate(metaFile.getPos());
Review comment:
I don't think we can ignore any IOExceptions during truncation of index/meta file right now. This is because how we serve the number of chunks from index file and we expect the number of chunks from index file and meta file to be consistent.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] mridulm commented on a change in pull request #30433: [SPARK-32916][SHUFFLE][test-maven][test-hadoop2.7] Ensure the number of chunks in meta file and index file are equal
Posted by GitBox <gi...@apache.org>.
mridulm commented on a change in pull request #30433:
URL: https://github.com/apache/spark/pull/30433#discussion_r535492414
##########
File path: common/network-shuffle/src/main/java/org/apache/spark/network/shuffle/RemoteBlockPushResolver.java
##########
@@ -825,45 +881,79 @@ void resetChunkTracker() {
* @param mapIndex the map index to be added to chunk tracker.
*/
void updateChunkInfo(long chunkOffset, int mapIndex) throws IOException {
- long idxStartPos = -1;
try {
- // update the chunk tracker to meta file before index file
- writeChunkTracker(mapIndex);
- idxStartPos = indexFile.getFilePointer();
logger.trace("{} shuffleId {} reduceId {} updated index current {} updated {}",
- appShuffleId.appId, appShuffleId.shuffleId, reduceId, this.lastChunkOffset,
- chunkOffset);
- indexFile.writeLong(chunkOffset);
- } catch (IOException ioe) {
- if (idxStartPos != -1) {
- // reset the position to avoid corrupting index files during exception.
- logger.warn("{} shuffleId {} reduceId {} reset index to position {}",
- appShuffleId.appId, appShuffleId.shuffleId, reduceId, idxStartPos);
- indexFile.seek(idxStartPos);
+ appShuffleId.appId, appShuffleId.shuffleId, reduceId, this.lastChunkOffset, chunkOffset);
+ if (indexMetaUpdateFailed) {
+ indexFile.getChannel().position(indexFile.getPos());
}
+ indexFile.getDos().writeLong(chunkOffset);
+ // Chunk bitmap should be written to the meta file after the index file because if there are
+ // any exceptions during writing the offset to the index file, meta file should not be
+ // updated. If the update to the index file is successful but the update to meta file isn't
+ // then the index file position is reset in the catch clause.
+ writeChunkTracker(mapIndex);
+ indexFile.updatePos(8);
+ this.lastChunkOffset = chunkOffset;
+ indexMetaUpdateFailed = false;
+ } catch (IOException ioe) {
+ logger.warn("{} shuffleId {} reduceId {} update to index/meta failed", appShuffleId.appId,
+ appShuffleId.shuffleId, reduceId);
+ indexMetaUpdateFailed = true;
+ // Any exception here is propagated to the caller and the caller can decide whether to
+ // abort or not.
throw ioe;
}
- this.lastChunkOffset = chunkOffset;
}
private void writeChunkTracker(int mapIndex) throws IOException {
if (mapIndex == -1) {
return;
}
chunkTracker.add(mapIndex);
- long metaStartPos = metaFile.getFilePointer();
try {
logger.trace("{} shuffleId {} reduceId {} mapIndex {} write chunk to meta file",
appShuffleId.appId, appShuffleId.shuffleId, reduceId, mapIndex);
- chunkTracker.serialize(metaFile);
+ if (indexMetaUpdateFailed) {
+ metaFile.getChannel().position(metaFile.getPos());
+ }
+ int chunkTrackerSize = chunkTracker.serializedSizeInBytes();
+ chunkTracker.serialize(metaFile.getDos());
+ metaFile.updatePos(chunkTrackerSize);
} catch (IOException ioe) {
- logger.warn("{} shuffleId {} reduceId {} mapIndex {} reset position of meta file to {}",
- appShuffleId.appId, appShuffleId.shuffleId, reduceId, mapIndex, metaStartPos);
- metaFile.seek(metaStartPos);
+ logger.warn("{} shuffleId {} reduceId {} mapIndex {} update to meta file failed",
Review comment:
We are logging this multiple times, remove from here and rely on caller to log
##########
File path: common/network-shuffle/src/main/java/org/apache/spark/network/shuffle/RemoteBlockPushResolver.java
##########
@@ -825,45 +881,79 @@ void resetChunkTracker() {
* @param mapIndex the map index to be added to chunk tracker.
*/
void updateChunkInfo(long chunkOffset, int mapIndex) throws IOException {
- long idxStartPos = -1;
try {
- // update the chunk tracker to meta file before index file
- writeChunkTracker(mapIndex);
- idxStartPos = indexFile.getFilePointer();
logger.trace("{} shuffleId {} reduceId {} updated index current {} updated {}",
- appShuffleId.appId, appShuffleId.shuffleId, reduceId, this.lastChunkOffset,
- chunkOffset);
- indexFile.writeLong(chunkOffset);
- } catch (IOException ioe) {
- if (idxStartPos != -1) {
- // reset the position to avoid corrupting index files during exception.
- logger.warn("{} shuffleId {} reduceId {} reset index to position {}",
- appShuffleId.appId, appShuffleId.shuffleId, reduceId, idxStartPos);
- indexFile.seek(idxStartPos);
+ appShuffleId.appId, appShuffleId.shuffleId, reduceId, this.lastChunkOffset, chunkOffset);
+ if (indexMetaUpdateFailed) {
+ indexFile.getChannel().position(indexFile.getPos());
}
+ indexFile.getDos().writeLong(chunkOffset);
+ // Chunk bitmap should be written to the meta file after the index file because if there are
+ // any exceptions during writing the offset to the index file, meta file should not be
+ // updated. If the update to the index file is successful but the update to meta file isn't
+ // then the index file position is reset in the catch clause.
+ writeChunkTracker(mapIndex);
+ indexFile.updatePos(8);
+ this.lastChunkOffset = chunkOffset;
+ indexMetaUpdateFailed = false;
+ } catch (IOException ioe) {
+ logger.warn("{} shuffleId {} reduceId {} update to index/meta failed", appShuffleId.appId,
+ appShuffleId.shuffleId, reduceId);
+ indexMetaUpdateFailed = true;
+ // Any exception here is propagated to the caller and the caller can decide whether to
+ // abort or not.
throw ioe;
}
- this.lastChunkOffset = chunkOffset;
}
private void writeChunkTracker(int mapIndex) throws IOException {
if (mapIndex == -1) {
return;
}
chunkTracker.add(mapIndex);
- long metaStartPos = metaFile.getFilePointer();
try {
logger.trace("{} shuffleId {} reduceId {} mapIndex {} write chunk to meta file",
appShuffleId.appId, appShuffleId.shuffleId, reduceId, mapIndex);
- chunkTracker.serialize(metaFile);
+ if (indexMetaUpdateFailed) {
+ metaFile.getChannel().position(metaFile.getPos());
+ }
+ int chunkTrackerSize = chunkTracker.serializedSizeInBytes();
+ chunkTracker.serialize(metaFile.getDos());
+ metaFile.updatePos(chunkTrackerSize);
Review comment:
Use `fileChannel.position()` to find position after write to find `chunkTrackerSize` and avoid `serializedSizeInBytes`
##########
File path: common/network-shuffle/src/main/java/org/apache/spark/network/shuffle/RemoteBlockPushResolver.java
##########
@@ -825,45 +881,79 @@ void resetChunkTracker() {
* @param mapIndex the map index to be added to chunk tracker.
*/
void updateChunkInfo(long chunkOffset, int mapIndex) throws IOException {
- long idxStartPos = -1;
try {
- // update the chunk tracker to meta file before index file
- writeChunkTracker(mapIndex);
- idxStartPos = indexFile.getFilePointer();
logger.trace("{} shuffleId {} reduceId {} updated index current {} updated {}",
- appShuffleId.appId, appShuffleId.shuffleId, reduceId, this.lastChunkOffset,
- chunkOffset);
- indexFile.writeLong(chunkOffset);
- } catch (IOException ioe) {
- if (idxStartPos != -1) {
- // reset the position to avoid corrupting index files during exception.
- logger.warn("{} shuffleId {} reduceId {} reset index to position {}",
- appShuffleId.appId, appShuffleId.shuffleId, reduceId, idxStartPos);
- indexFile.seek(idxStartPos);
+ appShuffleId.appId, appShuffleId.shuffleId, reduceId, this.lastChunkOffset, chunkOffset);
+ if (indexMetaUpdateFailed) {
+ indexFile.getChannel().position(indexFile.getPos());
}
+ indexFile.getDos().writeLong(chunkOffset);
+ // Chunk bitmap should be written to the meta file after the index file because if there are
+ // any exceptions during writing the offset to the index file, meta file should not be
+ // updated. If the update to the index file is successful but the update to meta file isn't
+ // then the index file position is reset in the catch clause.
+ writeChunkTracker(mapIndex);
+ indexFile.updatePos(8);
+ this.lastChunkOffset = chunkOffset;
+ indexMetaUpdateFailed = false;
+ } catch (IOException ioe) {
+ logger.warn("{} shuffleId {} reduceId {} update to index/meta failed", appShuffleId.appId,
+ appShuffleId.shuffleId, reduceId);
+ indexMetaUpdateFailed = true;
+ // Any exception here is propagated to the caller and the caller can decide whether to
+ // abort or not.
throw ioe;
}
- this.lastChunkOffset = chunkOffset;
}
private void writeChunkTracker(int mapIndex) throws IOException {
if (mapIndex == -1) {
return;
}
chunkTracker.add(mapIndex);
- long metaStartPos = metaFile.getFilePointer();
try {
logger.trace("{} shuffleId {} reduceId {} mapIndex {} write chunk to meta file",
appShuffleId.appId, appShuffleId.shuffleId, reduceId, mapIndex);
- chunkTracker.serialize(metaFile);
+ if (indexMetaUpdateFailed) {
+ metaFile.getChannel().position(metaFile.getPos());
+ }
+ int chunkTrackerSize = chunkTracker.serializedSizeInBytes();
+ chunkTracker.serialize(metaFile.getDos());
+ metaFile.updatePos(chunkTrackerSize);
} catch (IOException ioe) {
- logger.warn("{} shuffleId {} reduceId {} mapIndex {} reset position of meta file to {}",
- appShuffleId.appId, appShuffleId.shuffleId, reduceId, mapIndex, metaStartPos);
- metaFile.seek(metaStartPos);
+ logger.warn("{} shuffleId {} reduceId {} mapIndex {} update to meta file failed",
+ appShuffleId.appId, appShuffleId.shuffleId, reduceId, mapIndex);
+ // Any exceptions while writing to meta file is propagated to updateChunkInfo so that
+ // the index file position can be reset as well.
throw ioe;
}
}
+ private void incrementIOExceptions() {
+ numIOExceptions++;
+ }
+
+ private boolean shouldAbort(int ioExceptionsThresholdDuringMerge) {
+ return numIOExceptions > ioExceptionsThresholdDuringMerge;
+ }
+
+ private void finalizePartition() throws IOException {
+ if (dataFilePos != lastChunkOffset) {
+ try {
+ updateChunkInfo(dataFilePos, lastMergedMapIndex);
+ } catch (IOException ioe) {
+ // Any exceptions here while updating the meta files can be ignored. If the files
+ // aren't successfully updated they will be truncated.
+ }
+ }
+ // Get rid of any partial block data at the end of the file. This could either
+ // be due to failure, or a request still being processed when the shuffle
+ // merge gets finalized, or any exceptions while updating index/meta files.
+ dataChannel.truncate(lastChunkOffset);
+ indexFile.getChannel().truncate(indexFile.getPos());
+ metaFile.getChannel().truncate(metaFile.getPos());
Review comment:
Do we want to ignore exceptions here while doing truncate (for each call) ? This is best case effort without impact on functionality in case there is unread suffix data, right ?
##########
File path: common/network-shuffle/src/main/java/org/apache/spark/network/shuffle/RemoteBlockPushResolver.java
##########
@@ -369,27 +388,19 @@ public MergeStatuses finalizeShuffleMerge(FinalizeShuffleMerge msg) throws IOExc
mergeStatuses =
new MergeStatuses(msg.shuffleId, new RoaringBitmap[0], new int[0], new long[0]);
} else {
- Collection<AppShufflePartitionInfo> partitionsToFinalize = shufflePartitions.values();
- int totalPartitions = partitionsToFinalize.size();
- RoaringBitmap[] bitmaps = new RoaringBitmap[totalPartitions];
- int[] reduceIds = new int[totalPartitions];
- long[] sizes = new long[totalPartitions];
- Iterator<AppShufflePartitionInfo> partitionsIter = partitionsToFinalize.iterator();
- int idx = 0;
+ List<RoaringBitmap> bitmaps = new LinkedList<>();
+ List<Integer> reduceIds = new LinkedList<>();
+ List<Long> sizes = new LinkedList<>();
Review comment:
nit: Use `ArrayList` instead of `LinkedList` (and size it to `shufflePartitions.size()`).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #30433: [SPARK-32916][SHUFFLE][test-maven][test-hadoop2.7] Ensure the number of chunks in meta file and index file are equal
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #30433:
URL: https://github.com/apache/spark/pull/30433#issuecomment-737997320
**[Test build #132098 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/132098/testReport)** for PR 30433 at commit [`f5783a9`](https://github.com/apache/spark/commit/f5783a9e3f83d91308572230a7d050e000c49fa7).
* This patch **fails SparkR unit tests**.
* This patch merges cleanly.
* This patch adds the following public classes _(experimental)_:
* ` static class MergeShuffleFile `
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] otterc commented on a change in pull request #30433: [SPARK-32916][SHUFFLE][test-maven][test-hadoop2.7] Ensure the number of chunks in meta file and index file are equal
Posted by GitBox <gi...@apache.org>.
otterc commented on a change in pull request #30433:
URL: https://github.com/apache/spark/pull/30433#discussion_r536368742
##########
File path: common/network-shuffle/src/main/java/org/apache/spark/network/shuffle/RemoteBlockPushResolver.java
##########
@@ -827,13 +833,16 @@ void resetChunkTracker() {
void updateChunkInfo(long chunkOffset, int mapIndex) throws IOException {
long idxStartPos = -1;
try {
- // update the chunk tracker to meta file before index file
- writeChunkTracker(mapIndex);
idxStartPos = indexFile.getFilePointer();
logger.trace("{} shuffleId {} reduceId {} updated index current {} updated {}",
appShuffleId.appId, appShuffleId.shuffleId, reduceId, this.lastChunkOffset,
chunkOffset);
- indexFile.writeLong(chunkOffset);
+ indexFile.write(Longs.toByteArray(chunkOffset));
+ // Chunk bitmap should be written to the meta file after the index file because if there are
+ // any exceptions during writing the offset to the index file, meta file should not be
+ // updated. If the update to the index file is successful but the update to meta file isn't
+ // then the index file position is reset in the catch clause.
+ writeChunkTracker(mapIndex);
Review comment:
I created this Jira:
https://issues.apache.org/jira/browse/SPARK-33665
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #30433: [SPARK-32916][SHUFFLE][test-maven][test-hadoop2.7] Ensure the number of chunks in meta file and index file are equal
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #30433:
URL: https://github.com/apache/spark/pull/30433#issuecomment-741828632
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/132471/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] Ngone51 commented on a change in pull request #30433: [SPARK-32916][SHUFFLE][test-maven][test-hadoop2.7] Ensure the number of chunks in meta file and index file are equal
Posted by GitBox <gi...@apache.org>.
Ngone51 commented on a change in pull request #30433:
URL: https://github.com/apache/spark/pull/30433#discussion_r532436309
##########
File path: common/network-shuffle/src/main/java/org/apache/spark/network/shuffle/RemoteBlockPushResolver.java
##########
@@ -827,13 +833,16 @@ void resetChunkTracker() {
void updateChunkInfo(long chunkOffset, int mapIndex) throws IOException {
long idxStartPos = -1;
try {
- // update the chunk tracker to meta file before index file
- writeChunkTracker(mapIndex);
idxStartPos = indexFile.getFilePointer();
logger.trace("{} shuffleId {} reduceId {} updated index current {} updated {}",
appShuffleId.appId, appShuffleId.shuffleId, reduceId, this.lastChunkOffset,
chunkOffset);
- indexFile.writeLong(chunkOffset);
+ indexFile.write(Longs.toByteArray(chunkOffset));
+ // Chunk bitmap should be written to the meta file after the index file because if there are
+ // any exceptions during writing the offset to the index file, meta file should not be
+ // updated. If the update to the index file is successful but the update to meta file isn't
+ // then the index file position is reset in the catch clause.
Review comment:
Ah, I see. Thanks!
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] otterc commented on a change in pull request #30433: [SPARK-32916][SHUFFLE][test-maven][test-hadoop2.7] Ensure the number of chunks in meta file and index file are equal
Posted by GitBox <gi...@apache.org>.
otterc commented on a change in pull request #30433:
URL: https://github.com/apache/spark/pull/30433#discussion_r539630414
##########
File path: common/network-shuffle/src/main/java/org/apache/spark/network/shuffle/RemoteBlockPushResolver.java
##########
@@ -586,6 +592,7 @@ public void onData(String streamId, ByteBuffer buf) throws IOException {
deferredBufs = null;
return;
}
+ abortIfNecessary();
Review comment:
hmmm. I think the intention is to not have the server attempt writing if the threshold is reached for the partition. Probably this check here makes this behavior more accurate.
However, I don't have a strong opinion on this since the assumption is that if these number of IOExceptions have already reached the threshold, any further writes will result in an IOException as well. With that assumption, the write in `onData` after this threshold is met, will very likely throw `IOException` as well and since the threshold is already met, the server will instead throw `IOExceptions exceeded threshold`.
I can remove it from `onData`. @Ngone51 What do you think?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] otterc commented on a change in pull request #30433: [SPARK-32916][SHUFFLE][test-maven][test-hadoop2.7] Ensure the number of chunks in meta file and index file are equal
Posted by GitBox <gi...@apache.org>.
otterc commented on a change in pull request #30433:
URL: https://github.com/apache/spark/pull/30433#discussion_r539050327
##########
File path: common/network-shuffle/src/test/java/org/apache/spark/network/shuffle/RemoteBlockPushResolverSuite.java
##########
@@ -411,6 +415,269 @@ void deleteExecutorDirs(Path[] dirs) {
}
}
+ @Test
+ public void testRecoverIndexFileAfterIOExceptions() throws IOException {
+ useTestFiles(true, false);
+ RemoteBlockPushResolver.PushBlockStreamCallback callback1 =
+ (RemoteBlockPushResolver.PushBlockStreamCallback) pushResolver.receiveBlockDataAsStream(
+ new PushBlockStream(TEST_APP, 0, 0, 0, 0));
+ callback1.onData(callback1.getID(), ByteBuffer.wrap(new byte[4]));
+ callback1.onComplete(callback1.getID());
+ RemoteBlockPushResolver.AppShufflePartitionInfo partitionInfo = callback1.getPartitionInfo();
+ // Close the index stream so it throws IOException
+ TestMergeShuffleFile testIndexFile = (TestMergeShuffleFile) partitionInfo.getIndexFile();
+ testIndexFile.close();
+ StreamCallbackWithID callback2 = pushResolver.receiveBlockDataAsStream(
+ new PushBlockStream(TEST_APP, 0, 1, 0, 0));
+ callback2.onData(callback2.getID(), ByteBuffer.wrap(new byte[5]));
+ // This will complete without any IOExceptions because number of IOExceptions are less than
+ // the threshold but the update to index file will be unsuccessful.
+ callback2.onComplete(callback2.getID());
+ assertEquals("index position", 16, testIndexFile.getPos());
+ // Restore the index stream so it can write successfully again.
+ testIndexFile.restore();
+ StreamCallbackWithID callback3 = pushResolver.receiveBlockDataAsStream(
+ new PushBlockStream(TEST_APP, 0, 2, 0, 0));
+ callback3.onData(callback3.getID(), ByteBuffer.wrap(new byte[2]));
+ callback3.onComplete(callback3.getID());
+ assertEquals("index position", 24, testIndexFile.getPos());
+ MergeStatuses statuses = pushResolver.finalizeShuffleMerge(
+ new FinalizeShuffleMerge(TEST_APP, 0));
+ validateMergeStatuses(statuses, new int[] {0}, new long[] {11});
+ MergedBlockMeta blockMeta = pushResolver.getMergedBlockMeta(TEST_APP, 0, 0);
+ validateChunks(TEST_APP, 0, 0, blockMeta, new int[] {4, 7}, new int[][] {{0}, {1, 2}});
+ }
+
+ @Test
+ public void testRecoverIndexFileAfterIOExceptionsInFinalize() throws IOException {
+ useTestFiles(true, false);
+ registerExecutor(TEST_APP, prepareLocalDirs(localDirs));
+ RemoteBlockPushResolver.PushBlockStreamCallback callback1 =
+ (RemoteBlockPushResolver.PushBlockStreamCallback) pushResolver.receiveBlockDataAsStream(
+ new PushBlockStream(TEST_APP, 0, 0, 0, 0));
+ callback1.onData(callback1.getID(), ByteBuffer.wrap(new byte[4]));
+ callback1.onComplete(callback1.getID());
+ RemoteBlockPushResolver.AppShufflePartitionInfo partitionInfo = callback1.getPartitionInfo();
+ // Close the index stream so it throws IOException
+ TestMergeShuffleFile testIndexFile = (TestMergeShuffleFile) partitionInfo.getIndexFile();
+ testIndexFile.close();
+ StreamCallbackWithID callback2 = pushResolver.receiveBlockDataAsStream(
+ new PushBlockStream(TEST_APP, 0, 1, 0, 0));
+ callback2.onData(callback2.getID(), ByteBuffer.wrap(new byte[5]));
+ // This will complete without any IOExceptions because number of IOExceptions are less than
+ // the threshold but the update to index file will be unsuccessful.
+ callback2.onComplete(callback2.getID());
+ assertEquals("index position", 16, testIndexFile.getPos());
+ // The last update to index was unsuccessful however any further updates will be successful.
+ // Restore the index stream so it can write successfully again.
+ testIndexFile.restore();
+ MergeStatuses statuses = pushResolver.finalizeShuffleMerge(
+ new FinalizeShuffleMerge(TEST_APP, 0));
+ assertEquals("index position", 24, testIndexFile.getPos());
+ validateMergeStatuses(statuses, new int[] {0}, new long[] {9});
+ MergedBlockMeta blockMeta = pushResolver.getMergedBlockMeta(TEST_APP, 0, 0);
+ validateChunks(TEST_APP, 0, 0, blockMeta, new int[] {4, 5}, new int[][] {{0}, {1}});
+ }
+
+ @Test
+ public void testRecoverMetaFileAfterIOExceptions() throws IOException {
+ useTestFiles(false, true);
+ RemoteBlockPushResolver.PushBlockStreamCallback callback1 =
+ (RemoteBlockPushResolver.PushBlockStreamCallback) pushResolver.receiveBlockDataAsStream(
+ new PushBlockStream(TEST_APP, 0, 0, 0, 0));
+ callback1.onData(callback1.getID(), ByteBuffer.wrap(new byte[4]));
+ callback1.onComplete(callback1.getID());
+ RemoteBlockPushResolver.AppShufflePartitionInfo partitionInfo = callback1.getPartitionInfo();
+ // Close the meta stream so it throws IOException
+ TestMergeShuffleFile testMetaFile = (TestMergeShuffleFile) partitionInfo.getMetaFile();
+ long metaPosBeforeClose = testMetaFile.getPos();
+ testMetaFile.close();
+ StreamCallbackWithID callback2 = pushResolver.receiveBlockDataAsStream(
+ new PushBlockStream(TEST_APP, 0, 1, 0, 0));
+ callback2.onData(callback2.getID(), ByteBuffer.wrap(new byte[5]));
+ // This will complete without any IOExceptions because number of IOExceptions are less than
+ // the threshold but the update to index and meta file will be unsuccessful.
+ callback2.onComplete(callback2.getID());
+ assertEquals("index position", 16, partitionInfo.getIndexFile().getPos());
+ assertEquals("meta position", metaPosBeforeClose, testMetaFile.getPos());
+ // Restore the meta stream so it can write successfully again.
+ testMetaFile.restore();
+ StreamCallbackWithID callback3 = pushResolver.receiveBlockDataAsStream(
+ new PushBlockStream(TEST_APP, 0, 2, 0, 0));
+ callback3.onData(callback3.getID(), ByteBuffer.wrap(new byte[2]));
+ callback3.onComplete(callback3.getID());
+ assertEquals("index position", 24, partitionInfo.getIndexFile().getPos());
+ assertTrue("meta position", testMetaFile.getPos() > metaPosBeforeClose);
+ MergeStatuses statuses = pushResolver.finalizeShuffleMerge(
+ new FinalizeShuffleMerge(TEST_APP, 0));
+ validateMergeStatuses(statuses, new int[] {0}, new long[] {11});
+ MergedBlockMeta blockMeta = pushResolver.getMergedBlockMeta(TEST_APP, 0, 0);
+ validateChunks(TEST_APP, 0, 0, blockMeta, new int[] {4, 7}, new int[][] {{0}, {1, 2}});
+ }
+
+ @Test
+ public void testRecoverMetaFileAfterIOExceptionsInFinalize() throws IOException {
+ useTestFiles(false, true);
+ RemoteBlockPushResolver.PushBlockStreamCallback callback1 =
+ (RemoteBlockPushResolver.PushBlockStreamCallback) pushResolver.receiveBlockDataAsStream(
+ new PushBlockStream(TEST_APP, 0, 0, 0, 0));
+ callback1.onData(callback1.getID(), ByteBuffer.wrap(new byte[4]));
+ callback1.onComplete(callback1.getID());
+ RemoteBlockPushResolver.AppShufflePartitionInfo partitionInfo = callback1.getPartitionInfo();
+ // Close the meta stream so it throws IOException
+ TestMergeShuffleFile testMetaFile = (TestMergeShuffleFile) partitionInfo.getMetaFile();
+ long metaPosBeforeClose = testMetaFile.getPos();
+ testMetaFile.close();
+ StreamCallbackWithID callback2 = pushResolver.receiveBlockDataAsStream(
+ new PushBlockStream(TEST_APP, 0, 1, 0, 0));
+ callback2.onData(callback2.getID(), ByteBuffer.wrap(new byte[5]));
+ // This will complete without any IOExceptions because number of IOExceptions are less than
+ // the threshold but the update to index and meta file will be unsuccessful.
+ callback2.onComplete(callback2.getID());
+ MergeShuffleFile indexFile = partitionInfo.getIndexFile();
+ assertEquals("index position", 16, indexFile.getPos());
+ assertEquals("meta position", metaPosBeforeClose, testMetaFile.getPos());
+ // Restore the meta stream so it can write successfully again.
+ testMetaFile.restore();
+ MergeStatuses statuses = pushResolver.finalizeShuffleMerge(
+ new FinalizeShuffleMerge(TEST_APP, 0));
+ assertEquals("index position", 24, indexFile.getPos());
+ assertTrue("meta position", testMetaFile.getPos() > metaPosBeforeClose);
+ validateMergeStatuses(statuses, new int[] {0}, new long[] {9});
+ MergedBlockMeta blockMeta = pushResolver.getMergedBlockMeta(TEST_APP, 0, 0);
+ validateChunks(TEST_APP, 0, 0, blockMeta, new int[] {4, 5}, new int[][] {{0}, {1}});
+ }
+
+ @Test (expected = RuntimeException.class)
+ public void testIOExceptionsExceededThreshold() throws IOException {
+ RemoteBlockPushResolver.PushBlockStreamCallback callback =
+ (RemoteBlockPushResolver.PushBlockStreamCallback) pushResolver.receiveBlockDataAsStream(
+ new PushBlockStream(TEST_APP, 0, 0, 0, 0));
+ RemoteBlockPushResolver.AppShufflePartitionInfo partitionInfo = callback.getPartitionInfo();
+ callback.onData(callback.getID(), ByteBuffer.wrap(new byte[4]));
+ callback.onComplete(callback.getID());
+ // Close the data stream so it throws continuous IOException
+ partitionInfo.getDataChannel().close();
+ for (int i = 1; i < 5; i++) {
+ RemoteBlockPushResolver.PushBlockStreamCallback callback1 =
+ (RemoteBlockPushResolver.PushBlockStreamCallback) pushResolver.receiveBlockDataAsStream(
+ new PushBlockStream(TEST_APP, 0, i, 0, 0));
+ try {
+ callback1.onData(callback1.getID(), ByteBuffer.wrap(new byte[2]));
+ } catch (IOException ioe) {
+ // this will throw IOException so the client can retry.
+ callback1.onFailure(callback1.getID(), ioe);
+ }
+ }
+ assertEquals(4, partitionInfo.getNumIOExceptions());
+ // After 4 IOException, the server will respond with IOExceptions exceeded threshold
+ try {
+ RemoteBlockPushResolver.PushBlockStreamCallback callback2 =
+ (RemoteBlockPushResolver.PushBlockStreamCallback) pushResolver.receiveBlockDataAsStream(
+ new PushBlockStream(TEST_APP, 0, 5, 0, 0));
+ callback2.onData(callback.getID(), ByteBuffer.wrap(new byte[1]));
+ } catch (Throwable t) {
+ assertEquals("IOExceptions exceeded the threshold when merging shufflePush_0_5_0",
+ t.getMessage());
+ throw t;
+ }
Review comment:
I have updated the PR to fail any pending blocks immediately (before attempting to write) if the shuffle partition has hit the IOException threshold. PTAL
> And stop writing when the partition already hits IO exceptions threshold.
This should already be happening. Not sure what you mean.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] otterc commented on a change in pull request #30433: [SPARK-32916][SHUFFLE][test-maven][test-hadoop2.7] Ensure the number of chunks in meta file and index file are equal
Posted by GitBox <gi...@apache.org>.
otterc commented on a change in pull request #30433:
URL: https://github.com/apache/spark/pull/30433#discussion_r528907429
##########
File path: common/network-shuffle/src/main/java/org/apache/spark/network/shuffle/RemoteBlockPushResolver.java
##########
@@ -827,13 +833,16 @@ void resetChunkTracker() {
void updateChunkInfo(long chunkOffset, int mapIndex) throws IOException {
long idxStartPos = -1;
try {
- // update the chunk tracker to meta file before index file
- writeChunkTracker(mapIndex);
idxStartPos = indexFile.getFilePointer();
logger.trace("{} shuffleId {} reduceId {} updated index current {} updated {}",
appShuffleId.appId, appShuffleId.shuffleId, reduceId, this.lastChunkOffset,
chunkOffset);
- indexFile.writeLong(chunkOffset);
+ indexFile.write(Longs.toByteArray(chunkOffset));
+ // Chunk bitmap should be written to the meta file after the index file because if there are
+ // any exceptions during writing the offset to the index file, meta file should not be
+ // updated. If the update to the index file is successful but the update to meta file isn't
+ // then the index file position is reset in the catch clause.
+ writeChunkTracker(mapIndex);
Review comment:
@Victsm The case that you described how often does it happen? How does this cleanup script differentiates shuffle data from temporary user data and how quickly does it remove this data. This has an assumption that cluster is setup in a certain way.
Would you agree that the more common cases are the non-recoverable disk failures, permission issues, file corruption etc?
If yes, then there will be continuous IOExceptions while updating these index/meta file. So if we just let the chunk grow when the index/meta file couldn't be updated, in most of the cases the executor will fail to fetch these.
In the non-push shuffle mechanism, if writing to index file fails does the executor try multiple times to write to the same file locally? I understand that in the push-mechanism, we are updating the index/meta in a continuous manner but even here it writing to an a local file.
> I think that policy should require more than 2 consecutive IOExceptions during chunk metadata write to trigger.
Note that, the same issue could affect writing data file as well.
Just a single IOException while updating the index/meta file should be enough to indicate the client to stop pushing. Yes, it affects writing the data file as well. The reason I am not bringing that up because in this followup fix we are focusing on the exceptions while updating metadata. If others think that we should address the failures during writes to data file as well in this followup fix, we can address it.
> If we want to address this issue holistically, it would require some state tracking.
If so, I'd rather track it on the client side instead of the server side.
We are already tracking the state of shuffle partition in `AppShufflePartitionInfo`. Adding few other flags to it shouldn't be an issue.
Would like to know the thoughts of other folks as well.
@mridulm @tgravescs @attilapiros @Ngone51
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #30433: [SPARK-32916][SHUFFLE][test-maven][test-hadoop2.7] Ensure the number of chunks in meta file and index file are equal
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #30433:
URL: https://github.com/apache/spark/pull/30433#issuecomment-732503804
**[Test build #131594 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/131594/testReport)** for PR 30433 at commit [`30c8f83`](https://github.com/apache/spark/commit/30c8f83b7ba3e39724c90865fd1318b41876f927).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] Victsm commented on a change in pull request #30433: [SPARK-32916][SHUFFLE][test-maven][test-hadoop2.7] Ensure the number of chunks in meta file and index file are equal
Posted by GitBox <gi...@apache.org>.
Victsm commented on a change in pull request #30433:
URL: https://github.com/apache/spark/pull/30433#discussion_r539082197
##########
File path: common/network-shuffle/src/main/java/org/apache/spark/network/shuffle/RemoteBlockPushResolver.java
##########
@@ -674,7 +683,10 @@ public void onComplete(String streamId) throws IOException {
}
if (partitionInfo.getCurrentMapIndex() < 0) {
try {
- writeAnyDeferredBufs();
+ if (deferredBufs != null && !deferredBufs.isEmpty()) {
+ abortIfNecessary();
Review comment:
Why does this need to be inside the if block, instead of keeping `writeDeferredBufs` as is and just calling `abortIfNecessary` before `writeDeferredBufs`?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #30433: [SPARK-32916][SHUFFLE][test-maven][test-hadoop2.7] Ensure the number of chunks in meta file and index file are equal
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #30433:
URL: https://github.com/apache/spark/pull/30433#issuecomment-737692908
**[Test build #132098 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/132098/testReport)** for PR 30433 at commit [`f5783a9`](https://github.com/apache/spark/commit/f5783a9e3f83d91308572230a7d050e000c49fa7).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #30433: [SPARK-32916][SHUFFLE][test-maven][test-hadoop2.7] Ensure the number of chunks in meta file and index file are equal
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #30433:
URL: https://github.com/apache/spark/pull/30433#issuecomment-742011336
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/37109/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #30433: [SPARK-32916][SHUFFLE][test-maven][test-hadoop2.7] Ensure the number of chunks in meta file and index file are equal
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #30433:
URL: https://github.com/apache/spark/pull/30433#issuecomment-741569139
**[Test build #132471 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/132471/testReport)** for PR 30433 at commit [`0bce631`](https://github.com/apache/spark/commit/0bce63198e2e5304dd31aa39602f16d9d7e90af4).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] otterc commented on a change in pull request #30433: [SPARK-32916][SHUFFLE][test-maven][test-hadoop2.7] Ensure the number of chunks in meta file and index file are equal
Posted by GitBox <gi...@apache.org>.
otterc commented on a change in pull request #30433:
URL: https://github.com/apache/spark/pull/30433#discussion_r538952136
##########
File path: common/network-shuffle/src/test/java/org/apache/spark/network/shuffle/RemoteBlockPushResolverSuite.java
##########
@@ -411,6 +415,269 @@ void deleteExecutorDirs(Path[] dirs) {
}
}
+ @Test
+ public void testRecoverIndexFileAfterIOExceptions() throws IOException {
+ useTestFiles(true, false);
+ RemoteBlockPushResolver.PushBlockStreamCallback callback1 =
+ (RemoteBlockPushResolver.PushBlockStreamCallback) pushResolver.receiveBlockDataAsStream(
+ new PushBlockStream(TEST_APP, 0, 0, 0, 0));
+ callback1.onData(callback1.getID(), ByteBuffer.wrap(new byte[4]));
+ callback1.onComplete(callback1.getID());
+ RemoteBlockPushResolver.AppShufflePartitionInfo partitionInfo = callback1.getPartitionInfo();
+ // Close the index stream so it throws IOException
+ TestMergeShuffleFile testIndexFile = (TestMergeShuffleFile) partitionInfo.getIndexFile();
+ testIndexFile.close();
+ StreamCallbackWithID callback2 = pushResolver.receiveBlockDataAsStream(
+ new PushBlockStream(TEST_APP, 0, 1, 0, 0));
+ callback2.onData(callback2.getID(), ByteBuffer.wrap(new byte[5]));
+ // This will complete without any IOExceptions because number of IOExceptions are less than
+ // the threshold but the update to index file will be unsuccessful.
+ callback2.onComplete(callback2.getID());
+ assertEquals("index position", 16, testIndexFile.getPos());
+ // Restore the index stream so it can write successfully again.
+ testIndexFile.restore();
+ StreamCallbackWithID callback3 = pushResolver.receiveBlockDataAsStream(
+ new PushBlockStream(TEST_APP, 0, 2, 0, 0));
+ callback3.onData(callback3.getID(), ByteBuffer.wrap(new byte[2]));
+ callback3.onComplete(callback3.getID());
+ assertEquals("index position", 24, testIndexFile.getPos());
+ MergeStatuses statuses = pushResolver.finalizeShuffleMerge(
+ new FinalizeShuffleMerge(TEST_APP, 0));
+ validateMergeStatuses(statuses, new int[] {0}, new long[] {11});
+ MergedBlockMeta blockMeta = pushResolver.getMergedBlockMeta(TEST_APP, 0, 0);
+ validateChunks(TEST_APP, 0, 0, blockMeta, new int[] {4, 7}, new int[][] {{0}, {1, 2}});
+ }
+
+ @Test
+ public void testRecoverIndexFileAfterIOExceptionsInFinalize() throws IOException {
+ useTestFiles(true, false);
+ registerExecutor(TEST_APP, prepareLocalDirs(localDirs));
+ RemoteBlockPushResolver.PushBlockStreamCallback callback1 =
+ (RemoteBlockPushResolver.PushBlockStreamCallback) pushResolver.receiveBlockDataAsStream(
+ new PushBlockStream(TEST_APP, 0, 0, 0, 0));
+ callback1.onData(callback1.getID(), ByteBuffer.wrap(new byte[4]));
+ callback1.onComplete(callback1.getID());
+ RemoteBlockPushResolver.AppShufflePartitionInfo partitionInfo = callback1.getPartitionInfo();
+ // Close the index stream so it throws IOException
+ TestMergeShuffleFile testIndexFile = (TestMergeShuffleFile) partitionInfo.getIndexFile();
+ testIndexFile.close();
+ StreamCallbackWithID callback2 = pushResolver.receiveBlockDataAsStream(
+ new PushBlockStream(TEST_APP, 0, 1, 0, 0));
+ callback2.onData(callback2.getID(), ByteBuffer.wrap(new byte[5]));
+ // This will complete without any IOExceptions because number of IOExceptions are less than
+ // the threshold but the update to index file will be unsuccessful.
+ callback2.onComplete(callback2.getID());
+ assertEquals("index position", 16, testIndexFile.getPos());
+ // The last update to index was unsuccessful however any further updates will be successful.
+ // Restore the index stream so it can write successfully again.
+ testIndexFile.restore();
+ MergeStatuses statuses = pushResolver.finalizeShuffleMerge(
+ new FinalizeShuffleMerge(TEST_APP, 0));
+ assertEquals("index position", 24, testIndexFile.getPos());
+ validateMergeStatuses(statuses, new int[] {0}, new long[] {9});
+ MergedBlockMeta blockMeta = pushResolver.getMergedBlockMeta(TEST_APP, 0, 0);
+ validateChunks(TEST_APP, 0, 0, blockMeta, new int[] {4, 5}, new int[][] {{0}, {1}});
+ }
+
+ @Test
+ public void testRecoverMetaFileAfterIOExceptions() throws IOException {
+ useTestFiles(false, true);
+ RemoteBlockPushResolver.PushBlockStreamCallback callback1 =
+ (RemoteBlockPushResolver.PushBlockStreamCallback) pushResolver.receiveBlockDataAsStream(
+ new PushBlockStream(TEST_APP, 0, 0, 0, 0));
+ callback1.onData(callback1.getID(), ByteBuffer.wrap(new byte[4]));
+ callback1.onComplete(callback1.getID());
+ RemoteBlockPushResolver.AppShufflePartitionInfo partitionInfo = callback1.getPartitionInfo();
+ // Close the meta stream so it throws IOException
+ TestMergeShuffleFile testMetaFile = (TestMergeShuffleFile) partitionInfo.getMetaFile();
+ long metaPosBeforeClose = testMetaFile.getPos();
+ testMetaFile.close();
+ StreamCallbackWithID callback2 = pushResolver.receiveBlockDataAsStream(
+ new PushBlockStream(TEST_APP, 0, 1, 0, 0));
+ callback2.onData(callback2.getID(), ByteBuffer.wrap(new byte[5]));
+ // This will complete without any IOExceptions because number of IOExceptions are less than
+ // the threshold but the update to index and meta file will be unsuccessful.
+ callback2.onComplete(callback2.getID());
+ assertEquals("index position", 16, partitionInfo.getIndexFile().getPos());
+ assertEquals("meta position", metaPosBeforeClose, testMetaFile.getPos());
+ // Restore the meta stream so it can write successfully again.
+ testMetaFile.restore();
+ StreamCallbackWithID callback3 = pushResolver.receiveBlockDataAsStream(
+ new PushBlockStream(TEST_APP, 0, 2, 0, 0));
+ callback3.onData(callback3.getID(), ByteBuffer.wrap(new byte[2]));
+ callback3.onComplete(callback3.getID());
+ assertEquals("index position", 24, partitionInfo.getIndexFile().getPos());
+ assertTrue("meta position", testMetaFile.getPos() > metaPosBeforeClose);
+ MergeStatuses statuses = pushResolver.finalizeShuffleMerge(
+ new FinalizeShuffleMerge(TEST_APP, 0));
+ validateMergeStatuses(statuses, new int[] {0}, new long[] {11});
+ MergedBlockMeta blockMeta = pushResolver.getMergedBlockMeta(TEST_APP, 0, 0);
+ validateChunks(TEST_APP, 0, 0, blockMeta, new int[] {4, 7}, new int[][] {{0}, {1, 2}});
+ }
+
+ @Test
+ public void testRecoverMetaFileAfterIOExceptionsInFinalize() throws IOException {
+ useTestFiles(false, true);
+ RemoteBlockPushResolver.PushBlockStreamCallback callback1 =
+ (RemoteBlockPushResolver.PushBlockStreamCallback) pushResolver.receiveBlockDataAsStream(
+ new PushBlockStream(TEST_APP, 0, 0, 0, 0));
+ callback1.onData(callback1.getID(), ByteBuffer.wrap(new byte[4]));
+ callback1.onComplete(callback1.getID());
+ RemoteBlockPushResolver.AppShufflePartitionInfo partitionInfo = callback1.getPartitionInfo();
+ // Close the meta stream so it throws IOException
+ TestMergeShuffleFile testMetaFile = (TestMergeShuffleFile) partitionInfo.getMetaFile();
+ long metaPosBeforeClose = testMetaFile.getPos();
+ testMetaFile.close();
+ StreamCallbackWithID callback2 = pushResolver.receiveBlockDataAsStream(
+ new PushBlockStream(TEST_APP, 0, 1, 0, 0));
+ callback2.onData(callback2.getID(), ByteBuffer.wrap(new byte[5]));
+ // This will complete without any IOExceptions because number of IOExceptions are less than
+ // the threshold but the update to index and meta file will be unsuccessful.
+ callback2.onComplete(callback2.getID());
+ MergeShuffleFile indexFile = partitionInfo.getIndexFile();
+ assertEquals("index position", 16, indexFile.getPos());
+ assertEquals("meta position", metaPosBeforeClose, testMetaFile.getPos());
+ // Restore the meta stream so it can write successfully again.
+ testMetaFile.restore();
+ MergeStatuses statuses = pushResolver.finalizeShuffleMerge(
+ new FinalizeShuffleMerge(TEST_APP, 0));
+ assertEquals("index position", 24, indexFile.getPos());
+ assertTrue("meta position", testMetaFile.getPos() > metaPosBeforeClose);
+ validateMergeStatuses(statuses, new int[] {0}, new long[] {9});
+ MergedBlockMeta blockMeta = pushResolver.getMergedBlockMeta(TEST_APP, 0, 0);
+ validateChunks(TEST_APP, 0, 0, blockMeta, new int[] {4, 5}, new int[][] {{0}, {1}});
+ }
+
+ @Test (expected = RuntimeException.class)
+ public void testIOExceptionsExceededThreshold() throws IOException {
+ RemoteBlockPushResolver.PushBlockStreamCallback callback =
+ (RemoteBlockPushResolver.PushBlockStreamCallback) pushResolver.receiveBlockDataAsStream(
+ new PushBlockStream(TEST_APP, 0, 0, 0, 0));
+ RemoteBlockPushResolver.AppShufflePartitionInfo partitionInfo = callback.getPartitionInfo();
+ callback.onData(callback.getID(), ByteBuffer.wrap(new byte[4]));
+ callback.onComplete(callback.getID());
+ // Close the data stream so it throws continuous IOException
+ partitionInfo.getDataChannel().close();
+ for (int i = 1; i < 5; i++) {
+ RemoteBlockPushResolver.PushBlockStreamCallback callback1 =
+ (RemoteBlockPushResolver.PushBlockStreamCallback) pushResolver.receiveBlockDataAsStream(
+ new PushBlockStream(TEST_APP, 0, i, 0, 0));
+ try {
+ callback1.onData(callback1.getID(), ByteBuffer.wrap(new byte[2]));
+ } catch (IOException ioe) {
+ // this will throw IOException so the client can retry.
+ callback1.onFailure(callback1.getID(), ioe);
+ }
+ }
+ assertEquals(4, partitionInfo.getNumIOExceptions());
+ // After 4 IOException, the server will respond with IOExceptions exceeded threshold
+ try {
+ RemoteBlockPushResolver.PushBlockStreamCallback callback2 =
+ (RemoteBlockPushResolver.PushBlockStreamCallback) pushResolver.receiveBlockDataAsStream(
+ new PushBlockStream(TEST_APP, 0, 5, 0, 0));
+ callback2.onData(callback.getID(), ByteBuffer.wrap(new byte[1]));
+ } catch (Throwable t) {
+ assertEquals("IOExceptions exceeded the threshold when merging shufflePush_0_5_0",
+ t.getMessage());
+ throw t;
+ }
Review comment:
> I actually mean those blocks which are waiting for the partition lock at the same time.
I see. That's a good point. The server would attempt to merge them. If they don't see any IOExceptions, which is unlikely, they will be merged. Do you think we should check before writing them?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #30433: [SPARK-32916][SHUFFLE][test-maven][test-hadoop2.7] Ensure the number of chunks in meta file and index file are equal
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #30433:
URL: https://github.com/apache/spark/pull/30433#issuecomment-741092420
Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/37050/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #30433: [SPARK-32916][SHUFFLE][test-maven][test-hadoop2.7] Ensure the number of chunks in meta file and index file are equal
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #30433:
URL: https://github.com/apache/spark/pull/30433#issuecomment-749689156
Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/37832/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #30433: [SPARK-32916][SHUFFLE][test-maven][test-hadoop2.7] Ensure the number of chunks in meta file and index file are equal
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #30433:
URL: https://github.com/apache/spark/pull/30433#issuecomment-732475511
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] asfgit closed pull request #30433: [SPARK-32916][SHUFFLE][test-maven][test-hadoop2.7] Ensure the number of chunks in meta file and index file are equal
Posted by GitBox <gi...@apache.org>.
asfgit closed pull request #30433:
URL: https://github.com/apache/spark/pull/30433
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] otterc commented on a change in pull request #30433: [SPARK-32916][SHUFFLE][test-maven][test-hadoop2.7] Ensure the number of chunks in meta file and index file are equal
Posted by GitBox <gi...@apache.org>.
otterc commented on a change in pull request #30433:
URL: https://github.com/apache/spark/pull/30433#discussion_r528519186
##########
File path: common/network-shuffle/src/main/java/org/apache/spark/network/shuffle/RemoteBlockPushResolver.java
##########
@@ -827,13 +833,16 @@ void resetChunkTracker() {
void updateChunkInfo(long chunkOffset, int mapIndex) throws IOException {
long idxStartPos = -1;
try {
- // update the chunk tracker to meta file before index file
- writeChunkTracker(mapIndex);
idxStartPos = indexFile.getFilePointer();
logger.trace("{} shuffleId {} reduceId {} updated index current {} updated {}",
appShuffleId.appId, appShuffleId.shuffleId, reduceId, this.lastChunkOffset,
chunkOffset);
- indexFile.writeLong(chunkOffset);
+ indexFile.write(Longs.toByteArray(chunkOffset));
+ // Chunk bitmap should be written to the meta file after the index file because if there are
+ // any exceptions during writing the offset to the index file, meta file should not be
+ // updated. If the update to the index file is successful but the update to meta file isn't
+ // then the index file position is reset in the catch clause.
+ writeChunkTracker(mapIndex);
Review comment:
> I think we should catch the chunk level IOExceptions inside onComplete, setting certain flags to make the onFailure handling logic know about whether we encountered a block level failure or a chunk level failure.
For a chunk level failure, we shouldn't overwrite the previous block, we should effectively only delay the closure of the chunk.
The closure of chunk write now depends on the minimum size of the chunk. If we keep growing the chunk then eventually it will be very large and the executors will get killed fetching them because they will exceed the physical memory limit. This was the main reason to serve data in manageable-sized chunks.
> Stopping merging blocks only after 2 consecutive IOExceptions seems to be giving up too early.
With local file systems the IOExceptions that I can think of would be due to disk failures, file corruption, permission and these will not go away when retrying to update these files. Are there any other scenarios that I am missing?
> If there's a need to stop early due to too many disk failures, I think that logic should be on the client side inside ErrorHandler.
Sure, however the server here can respond with a similar runtime exception that it can't update metadata just like the `TooLate` runtime exception and the error handle for the push side can stop pushing the data.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] otterc commented on a change in pull request #30433: [SPARK-32916][SHUFFLE][test-maven][test-hadoop2.7] Ensure the number of chunks in meta file and index file are equal
Posted by GitBox <gi...@apache.org>.
otterc commented on a change in pull request #30433:
URL: https://github.com/apache/spark/pull/30433#discussion_r538685008
##########
File path: common/network-shuffle/src/main/java/org/apache/spark/network/shuffle/RemoteBlockPushResolver.java
##########
@@ -825,43 +884,68 @@ void resetChunkTracker() {
* @param mapIndex the map index to be added to chunk tracker.
*/
void updateChunkInfo(long chunkOffset, int mapIndex) throws IOException {
- long idxStartPos = -1;
try {
- // update the chunk tracker to meta file before index file
- writeChunkTracker(mapIndex);
- idxStartPos = indexFile.getFilePointer();
logger.trace("{} shuffleId {} reduceId {} updated index current {} updated {}",
- appShuffleId.appId, appShuffleId.shuffleId, reduceId, this.lastChunkOffset,
- chunkOffset);
- indexFile.writeLong(chunkOffset);
- } catch (IOException ioe) {
- if (idxStartPos != -1) {
- // reset the position to avoid corrupting index files during exception.
- logger.warn("{} shuffleId {} reduceId {} reset index to position {}",
- appShuffleId.appId, appShuffleId.shuffleId, reduceId, idxStartPos);
- indexFile.seek(idxStartPos);
+ appShuffleId.appId, appShuffleId.shuffleId, reduceId, this.lastChunkOffset, chunkOffset);
+ if (indexMetaUpdateFailed) {
+ indexFile.getChannel().position(indexFile.getPos());
}
+ indexFile.getDos().writeLong(chunkOffset);
+ // Chunk bitmap should be written to the meta file after the index file because if there are
+ // any exceptions during writing the offset to the index file, meta file should not be
+ // updated. If the update to the index file is successful but the update to meta file isn't
+ // then the index file position is reset in the catch clause.
Review comment:
Done.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] mridulm commented on pull request #30433: [SPARK-32916][SHUFFLE][test-maven][test-hadoop2.7] Ensure the number of chunks in meta file and index file are equal
Posted by GitBox <gi...@apache.org>.
mridulm commented on pull request #30433:
URL: https://github.com/apache/spark/pull/30433#issuecomment-732447250
ok to test
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] otterc commented on pull request #30433: [SPARK-32916][SHUFFLE][test-maven][test-hadoop2.7] Ensure the number of chunks in meta file and index file are equal
Posted by GitBox <gi...@apache.org>.
otterc commented on pull request #30433:
URL: https://github.com/apache/spark/pull/30433#issuecomment-730609019
@tgravescs @Ngone51 @attilapiros @mridulm @Victsm
Please help review these bug fixes
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] Victsm commented on a change in pull request #30433: [SPARK-32916][SHUFFLE][test-maven][test-hadoop2.7] Ensure the number of chunks in meta file and index file are equal
Posted by GitBox <gi...@apache.org>.
Victsm commented on a change in pull request #30433:
URL: https://github.com/apache/spark/pull/30433#discussion_r538908610
##########
File path: common/network-shuffle/src/main/java/org/apache/spark/network/shuffle/RemoteBlockPushResolver.java
##########
@@ -370,26 +391,19 @@ public MergeStatuses finalizeShuffleMerge(FinalizeShuffleMerge msg) throws IOExc
new MergeStatuses(msg.shuffleId, new RoaringBitmap[0], new int[0], new long[0]);
} else {
Collection<AppShufflePartitionInfo> partitionsToFinalize = shufflePartitions.values();
- int totalPartitions = partitionsToFinalize.size();
- RoaringBitmap[] bitmaps = new RoaringBitmap[totalPartitions];
- int[] reduceIds = new int[totalPartitions];
- long[] sizes = new long[totalPartitions];
+ List<RoaringBitmap> bitmaps = new ArrayList<>(partitionsToFinalize.size());
+ List<Integer> reduceIds = new ArrayList<>(partitionsToFinalize.size());
+ List<Long> sizes = new ArrayList<>(partitionsToFinalize.size());
Review comment:
We could alternatively still maintain the arrays to avoid array copying towards the end.
We can handle the null bitmaps on the driver side after it receives the MergeStatuses.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #30433: [SPARK-32916][SHUFFLE][test-maven][test-hadoop2.7] Ensure the number of chunks in meta file and index file are equal
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #30433:
URL: https://github.com/apache/spark/pull/30433#issuecomment-737698932
Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/36690/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] otterc commented on a change in pull request #30433: [SPARK-32916][SHUFFLE][test-maven][test-hadoop2.7] Ensure the number of chunks in meta file and index file are equal
Posted by GitBox <gi...@apache.org>.
otterc commented on a change in pull request #30433:
URL: https://github.com/apache/spark/pull/30433#discussion_r528504439
##########
File path: common/network-shuffle/src/main/java/org/apache/spark/network/shuffle/RemoteBlockPushResolver.java
##########
@@ -827,13 +833,16 @@ void resetChunkTracker() {
void updateChunkInfo(long chunkOffset, int mapIndex) throws IOException {
long idxStartPos = -1;
try {
- // update the chunk tracker to meta file before index file
- writeChunkTracker(mapIndex);
idxStartPos = indexFile.getFilePointer();
logger.trace("{} shuffleId {} reduceId {} updated index current {} updated {}",
appShuffleId.appId, appShuffleId.shuffleId, reduceId, this.lastChunkOffset,
chunkOffset);
- indexFile.writeLong(chunkOffset);
+ indexFile.write(Longs.toByteArray(chunkOffset));
+ // Chunk bitmap should be written to the meta file after the index file because if there are
+ // any exceptions during writing the offset to the index file, meta file should not be
+ // updated. If the update to the index file is successful but the update to meta file isn't
+ // then the index file position is reset in the catch clause.
+ writeChunkTracker(mapIndex);
Review comment:
> I don't think we should assume that.
We could have taken an approach where we give up on writing merged shuffle data file as well.
Here the shuffle service is writing to a local file. It's not a network filesystem/hdfs where because of network there could be transient exceptions. If during updating the local index/meta file the shuffle service sees an IOException and then there is an IOException again while seeking to a position, chances that the updates to the same file will fail are very high. If index/meta files are not successfully updated then those merged blocks cannot be served to the executors when they read/fetch it.
If most of the time `IOExceptions` while writing to local filesystem can only be because of issues like file permission, disk full, etc, I don't see the advantage of still trying to keep merging blocks to the shuffle partition.
It might be better for the server to reply to the executor to stop pushing in this case. Otherwise, they keep pushing more blocks and in most of the cases they will fail as well.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #30433: [SPARK-32916][SHUFFLE][test-maven][test-hadoop2.7] Ensure the number of chunks in meta file and index file are equal
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #30433:
URL: https://github.com/apache/spark/pull/30433#issuecomment-740830147
**[Test build #132444 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/132444/testReport)** for PR 30433 at commit [`22c7a32`](https://github.com/apache/spark/commit/22c7a328c48bf64de758fa7fa97f8b34333bc641).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] otterc commented on pull request #30433: [SPARK-32916][SHUFFLE][test-maven][test-hadoop2.7] Ensure the number of chunks in meta file and index file are equal
Posted by GitBox <gi...@apache.org>.
otterc commented on pull request #30433:
URL: https://github.com/apache/spark/pull/30433#issuecomment-737477581
While making these changes and adding the unit tests I think it's better to not rely on the file pointer from RandomAccessFile and maintain our own pointer. Thus, not use RandomAccessFile at all.
The reason is that let's say `seek` fails for index file and after that we abort merging any new merge blocks. However, when we finalize that partition we still need the pointer to the last successful update made to the index file of this partition.
Since the seek had failed, we can't rely on the pointer from RAF. I think this also simplifies the exception handling code.
Let me know if you think otherwise.
@mridulm @Victsm @Ngone51
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #30433: [SPARK-32916][SHUFFLE][test-maven][test-hadoop2.7] Ensure the number of chunks in meta file and index file are equal
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #30433:
URL: https://github.com/apache/spark/pull/30433#issuecomment-737721294
Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/36698/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #30433: [SPARK-32916][SHUFFLE][test-maven][test-hadoop2.7] Ensure the number of chunks in meta file and index file are equal
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #30433:
URL: https://github.com/apache/spark/pull/30433#issuecomment-730608398
Can one of the admins verify this patch?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #30433: [SPARK-32916][SHUFFLE][test-maven][test-hadoop2.7] Ensure the number of chunks in meta file and index file are equal
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #30433:
URL: https://github.com/apache/spark/pull/30433#issuecomment-741974173
**[Test build #132507 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/132507/testReport)** for PR 30433 at commit [`f48d52f`](https://github.com/apache/spark/commit/f48d52f68e52b63742e7cce5dbb4fbcc89845402).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #30433: [SPARK-32916][SHUFFLE][test-maven][test-hadoop2.7] Ensure the number of chunks in meta file and index file are equal
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #30433:
URL: https://github.com/apache/spark/pull/30433#issuecomment-741521075
**[Test build #132448 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/132448/testReport)** for PR 30433 at commit [`5422f10`](https://github.com/apache/spark/commit/5422f100a197d96f646864c75fc1a0beb2d41af6).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] otterc commented on a change in pull request #30433: [SPARK-32916][SHUFFLE][test-maven][test-hadoop2.7] Ensure the number of chunks in meta file and index file are equal
Posted by GitBox <gi...@apache.org>.
otterc commented on a change in pull request #30433:
URL: https://github.com/apache/spark/pull/30433#discussion_r529823126
##########
File path: common/network-shuffle/src/main/java/org/apache/spark/network/shuffle/RemoteBlockPushResolver.java
##########
@@ -827,13 +833,16 @@ void resetChunkTracker() {
void updateChunkInfo(long chunkOffset, int mapIndex) throws IOException {
long idxStartPos = -1;
try {
- // update the chunk tracker to meta file before index file
- writeChunkTracker(mapIndex);
idxStartPos = indexFile.getFilePointer();
logger.trace("{} shuffleId {} reduceId {} updated index current {} updated {}",
appShuffleId.appId, appShuffleId.shuffleId, reduceId, this.lastChunkOffset,
chunkOffset);
- indexFile.writeLong(chunkOffset);
+ indexFile.write(Longs.toByteArray(chunkOffset));
+ // Chunk bitmap should be written to the meta file after the index file because if there are
+ // any exceptions during writing the offset to the index file, meta file should not be
+ // updated. If the update to the index file is successful but the update to meta file isn't
+ // then the index file position is reset in the catch clause.
+ writeChunkTracker(mapIndex);
Review comment:
Had a discussion with @Victsm @mridulm yesterday. This is the approach currently we are thinking of:
1. Verify if `seek` is not recoverable. If it is not then let the clients know to stop pushing and stop merging this partition.
2. Have a threshold on number of `IOExceptions` from writes and when this threshold is reached for a single partition, inform the client to stop pushing and stop merging this partition.
3. When the update to metadata fails, not propagate this exception back to client so that they push the block again. The size of the current chunk may grow but with (2) in place it will still be of a manageable size.
These changes will not be that complex.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] Victsm commented on a change in pull request #30433: [SPARK-32916][SHUFFLE][test-maven][test-hadoop2.7] Ensure the number of chunks in meta file and index file are equal
Posted by GitBox <gi...@apache.org>.
Victsm commented on a change in pull request #30433:
URL: https://github.com/apache/spark/pull/30433#discussion_r539083309
##########
File path: common/network-shuffle/src/main/java/org/apache/spark/network/shuffle/RemoteBlockPushResolver.java
##########
@@ -586,6 +592,7 @@ public void onData(String streamId, ByteBuffer buf) throws IOException {
deferredBufs = null;
return;
}
+ abortIfNecessary();
Review comment:
I don't think we should invoke this here.
Invoking inside `onComplete` should be sufficient.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #30433: [SPARK-32916][SHUFFLE][test-maven][test-hadoop2.7] Ensure the number of chunks in meta file and index file are equal
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #30433:
URL: https://github.com/apache/spark/pull/30433#issuecomment-732465585
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/36176/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] Victsm commented on a change in pull request #30433: [SPARK-32916][SHUFFLE][test-maven][test-hadoop2.7] Ensure the number of chunks in meta file and index file are equal
Posted by GitBox <gi...@apache.org>.
Victsm commented on a change in pull request #30433:
URL: https://github.com/apache/spark/pull/30433#discussion_r528475145
##########
File path: common/network-shuffle/src/main/java/org/apache/spark/network/shuffle/RemoteBlockPushResolver.java
##########
@@ -827,13 +833,16 @@ void resetChunkTracker() {
void updateChunkInfo(long chunkOffset, int mapIndex) throws IOException {
long idxStartPos = -1;
try {
- // update the chunk tracker to meta file before index file
- writeChunkTracker(mapIndex);
idxStartPos = indexFile.getFilePointer();
logger.trace("{} shuffleId {} reduceId {} updated index current {} updated {}",
appShuffleId.appId, appShuffleId.shuffleId, reduceId, this.lastChunkOffset,
chunkOffset);
- indexFile.writeLong(chunkOffset);
+ indexFile.write(Longs.toByteArray(chunkOffset));
+ // Chunk bitmap should be written to the meta file after the index file because if there are
+ // any exceptions during writing the offset to the index file, meta file should not be
+ // updated. If the update to the index file is successful but the update to meta file isn't
+ // then the index file position is reset in the catch clause.
+ writeChunkTracker(mapIndex);
Review comment:
I don't think we should assume that.
We could have taken an approach where we give up on writing merged shuffle data file as well.
The issue I see here is that we are mixing block level exception with chunk level exception.
Right now, both will throw exceptions triggering the `onFailure` handling.
However, the current `onFailure` handling is meant for block level only.
I think we should catch the chunk level IOExceptions inside `onComplete`, setting certain flags to make the `onFailure` handling logic know about whether we encountered a block level failure or a chunk level failure.
For a chunk level failure, we shouldn't overwrite the previous block, we should effectively only delay the closure of the chunk.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] Victsm commented on a change in pull request #30433: [SPARK-32916][SHUFFLE][test-maven][test-hadoop2.7] Ensure the number of chunks in meta file and index file are equal
Posted by GitBox <gi...@apache.org>.
Victsm commented on a change in pull request #30433:
URL: https://github.com/apache/spark/pull/30433#discussion_r528923185
##########
File path: common/network-shuffle/src/main/java/org/apache/spark/network/shuffle/RemoteBlockPushResolver.java
##########
@@ -827,13 +833,16 @@ void resetChunkTracker() {
void updateChunkInfo(long chunkOffset, int mapIndex) throws IOException {
long idxStartPos = -1;
try {
- // update the chunk tracker to meta file before index file
- writeChunkTracker(mapIndex);
idxStartPos = indexFile.getFilePointer();
logger.trace("{} shuffleId {} reduceId {} updated index current {} updated {}",
appShuffleId.appId, appShuffleId.shuffleId, reduceId, this.lastChunkOffset,
chunkOffset);
- indexFile.writeLong(chunkOffset);
+ indexFile.write(Longs.toByteArray(chunkOffset));
+ // Chunk bitmap should be written to the meta file after the index file because if there are
+ // any exceptions during writing the offset to the index file, meta file should not be
+ // updated. If the update to the index file is successful but the update to meta file isn't
+ // then the index file position is reset in the catch clause.
+ writeChunkTracker(mapIndex);
Review comment:
Also, the script I described is just one potential way to mitigate the issue faster.
That's temporary data which will be deleted by YARN at the end of the job anyway, so the cluster already has ways to self-recover from that type of issues.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] Victsm commented on a change in pull request #30433: [SPARK-32916][SHUFFLE][test-maven][test-hadoop2.7] Ensure the number of chunks in meta file and index file are equal
Posted by GitBox <gi...@apache.org>.
Victsm commented on a change in pull request #30433:
URL: https://github.com/apache/spark/pull/30433#discussion_r528925984
##########
File path: common/network-shuffle/src/main/java/org/apache/spark/network/shuffle/RemoteBlockPushResolver.java
##########
@@ -827,13 +833,16 @@ void resetChunkTracker() {
void updateChunkInfo(long chunkOffset, int mapIndex) throws IOException {
long idxStartPos = -1;
try {
- // update the chunk tracker to meta file before index file
- writeChunkTracker(mapIndex);
idxStartPos = indexFile.getFilePointer();
logger.trace("{} shuffleId {} reduceId {} updated index current {} updated {}",
appShuffleId.appId, appShuffleId.shuffleId, reduceId, this.lastChunkOffset,
chunkOffset);
- indexFile.writeLong(chunkOffset);
+ indexFile.write(Longs.toByteArray(chunkOffset));
+ // Chunk bitmap should be written to the meta file after the index file because if there are
+ // any exceptions during writing the offset to the index file, meta file should not be
+ // updated. If the update to the index file is successful but the update to meta file isn't
+ // then the index file position is reset in the catch clause.
+ writeChunkTracker(mapIndex);
Review comment:
Giving up on the merge write just after one IOException is too expensive.
We could have merged much more blocks if such an issue is only temporary.
I think this is going against our original design to make the shuffle partition merge write tolerant of duplication, collision, and failure.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #30433: [SPARK-32916][SHUFFLE][test-maven][test-hadoop2.7] Ensure the number of chunks in meta file and index file are equal
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #30433:
URL: https://github.com/apache/spark/pull/30433#issuecomment-738000093
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] Victsm commented on a change in pull request #30433: [SPARK-32916][SHUFFLE][test-maven][test-hadoop2.7] Ensure the number of chunks in meta file and index file are equal
Posted by GitBox <gi...@apache.org>.
Victsm commented on a change in pull request #30433:
URL: https://github.com/apache/spark/pull/30433#discussion_r530765093
##########
File path: common/network-shuffle/src/main/java/org/apache/spark/network/shuffle/RemoteBlockPushResolver.java
##########
@@ -827,13 +833,16 @@ void resetChunkTracker() {
void updateChunkInfo(long chunkOffset, int mapIndex) throws IOException {
long idxStartPos = -1;
try {
- // update the chunk tracker to meta file before index file
- writeChunkTracker(mapIndex);
idxStartPos = indexFile.getFilePointer();
logger.trace("{} shuffleId {} reduceId {} updated index current {} updated {}",
appShuffleId.appId, appShuffleId.shuffleId, reduceId, this.lastChunkOffset,
chunkOffset);
- indexFile.writeLong(chunkOffset);
+ indexFile.write(Longs.toByteArray(chunkOffset));
+ // Chunk bitmap should be written to the meta file after the index file because if there are
+ // any exceptions during writing the offset to the index file, meta file should not be
+ // updated. If the update to the index file is successful but the update to meta file isn't
+ // then the index file position is reset in the catch clause.
Review comment:
If exception happens, line 853 would throw the exception skipping line 855.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] otterc commented on a change in pull request #30433: [SPARK-32916][SHUFFLE][test-maven][test-hadoop2.7] Ensure the number of chunks in meta file and index file are equal
Posted by GitBox <gi...@apache.org>.
otterc commented on a change in pull request #30433:
URL: https://github.com/apache/spark/pull/30433#discussion_r539504808
##########
File path: common/network-shuffle/src/main/java/org/apache/spark/network/shuffle/RemoteBlockPushResolver.java
##########
@@ -674,7 +683,10 @@ public void onComplete(String streamId) throws IOException {
}
if (partitionInfo.getCurrentMapIndex() < 0) {
try {
- writeAnyDeferredBufs();
+ if (deferredBufs != null && !deferredBufs.isEmpty()) {
+ abortIfNecessary();
Review comment:
I was thinking that if a stream is completing but doesn't need to write any deferred bufs or update the metadata then we should let it complete without failures.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] mridulm commented on pull request #30433: [SPARK-32916][SHUFFLE][test-maven][test-hadoop2.7] Ensure the number of chunks in meta file and index file are equal
Posted by GitBox <gi...@apache.org>.
mridulm commented on pull request #30433:
URL: https://github.com/apache/spark/pull/30433#issuecomment-750425099
The failure is unrelated.
Merging to master and branch-3.1
+CC @HyukjinKwon who is RM for 3.1
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] mridulm commented on pull request #30433: [SPARK-32916][SHUFFLE][test-maven][test-hadoop2.7] Ensure the number of chunks in meta file and index file are equal
Posted by GitBox <gi...@apache.org>.
mridulm commented on pull request #30433:
URL: https://github.com/apache/spark/pull/30433#issuecomment-749638114
Kicked off test once more, will merge once it goes through.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] otterc commented on pull request #30433: [SPARK-32916][SHUFFLE][test-maven][test-hadoop2.7] Ensure the number of chunks in meta file and index file are equal
Posted by GitBox <gi...@apache.org>.
otterc commented on pull request #30433:
URL: https://github.com/apache/spark/pull/30433#issuecomment-738592653
> So we get rid of the `seek` now? And `ioExceptionsThresholdDuringMerge` is used for other potential recoverable IOExceptions?
There is a need to maintain the positions in the index/meta file, since we can't rely on `RAF.getFilePointer`. We have to maintain our own pointer, so it just seems better to avoid RAF all together. This gets rid of `RAF.seek`.
> And ioExceptionsThresholdDuringMerge is used for other potential recoverable IOExceptions?
This is a limit on the number of IOExceptions for a particular shuffle partition. Once this limit is breached, the server will respond with a failure similar to "TOO Late".
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] otterc commented on a change in pull request #30433: [SPARK-32916][SHUFFLE][test-maven][test-hadoop2.7] Ensure the number of chunks in meta file and index file are equal
Posted by GitBox <gi...@apache.org>.
otterc commented on a change in pull request #30433:
URL: https://github.com/apache/spark/pull/30433#discussion_r539871040
##########
File path: common/network-shuffle/src/main/java/org/apache/spark/network/shuffle/RemoteBlockPushResolver.java
##########
@@ -586,6 +592,7 @@ public void onData(String streamId, ByteBuffer buf) throws IOException {
deferredBufs = null;
return;
}
+ abortIfNecessary();
Review comment:
Even If the write succeeds, I don't see an issue. If there are no further failures with this block it will be merged successfully. If there are more failures later, then they will be handled in the same way as now. I could add UTs to verify this as well.
> we still receive data in onData but just not write when the partition already reach IO threshold and only call abortIfNecessary inside `onComplete`.
My thought about this is that if a block is completing and doesn't have any deferred bufs then we should let it complete without failures if possible.
Again, I don't have a strong opinion about this. IMO reaching the threshold is indicative of a persistent issue and any further writes will faill.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] Victsm commented on pull request #30433: [SPARK-32916][SHUFFLE][test-maven][test-hadoop2.7] Ensure the number of chunks in meta file and index file are equal
Posted by GitBox <gi...@apache.org>.
Victsm commented on pull request #30433:
URL: https://github.com/apache/spark/pull/30433#issuecomment-749180623
Current change looks good to me as well.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] mridulm commented on pull request #30433: [SPARK-32916][SHUFFLE][test-maven][test-hadoop2.7] Ensure the number of chunks in meta file and index file are equal
Posted by GitBox <gi...@apache.org>.
mridulm commented on pull request #30433:
URL: https://github.com/apache/spark/pull/30433#issuecomment-749637911
ok to test
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] otterc commented on a change in pull request #30433: [SPARK-32916][SHUFFLE][test-maven][test-hadoop2.7] Ensure the number of chunks in meta file and index file are equal
Posted by GitBox <gi...@apache.org>.
otterc commented on a change in pull request #30433:
URL: https://github.com/apache/spark/pull/30433#discussion_r539516440
##########
File path: common/network-shuffle/src/main/java/org/apache/spark/network/shuffle/RemoteBlockPushResolver.java
##########
@@ -370,26 +391,19 @@ public MergeStatuses finalizeShuffleMerge(FinalizeShuffleMerge msg) throws IOExc
new MergeStatuses(msg.shuffleId, new RoaringBitmap[0], new int[0], new long[0]);
} else {
Collection<AppShufflePartitionInfo> partitionsToFinalize = shufflePartitions.values();
- int totalPartitions = partitionsToFinalize.size();
- RoaringBitmap[] bitmaps = new RoaringBitmap[totalPartitions];
- int[] reduceIds = new int[totalPartitions];
- long[] sizes = new long[totalPartitions];
+ List<RoaringBitmap> bitmaps = new ArrayList<>(partitionsToFinalize.size());
+ List<Integer> reduceIds = new ArrayList<>(partitionsToFinalize.size());
+ List<Long> sizes = new ArrayList<>(partitionsToFinalize.size());
Review comment:
I think that was my mistake. I changed it overlooking the fact that there could be fewer partitions because of the exceptions when truncating data file or updating index/meta.
I don't think it is better to have `null`s in the array that is returned though.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #30433: [SPARK-32916][SHUFFLE][test-maven][test-hadoop2.7] Ensure the number of chunks in meta file and index file are equal
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #30433:
URL: https://github.com/apache/spark/pull/30433#issuecomment-737687656
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/36690/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] otterc commented on pull request #30433: [SPARK-32916][SHUFFLE][test-maven][test-hadoop2.7] Ensure the number of chunks in meta file and index file are equal
Posted by GitBox <gi...@apache.org>.
otterc commented on pull request #30433:
URL: https://github.com/apache/spark/pull/30433#issuecomment-750471241
Thanks @mridulm !
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] Ngone51 commented on a change in pull request #30433: [SPARK-32916][SHUFFLE][test-maven][test-hadoop2.7] Ensure the number of chunks in meta file and index file are equal
Posted by GitBox <gi...@apache.org>.
Ngone51 commented on a change in pull request #30433:
URL: https://github.com/apache/spark/pull/30433#discussion_r535849402
##########
File path: common/network-shuffle/src/main/java/org/apache/spark/network/shuffle/RemoteBlockPushResolver.java
##########
@@ -827,13 +833,16 @@ void resetChunkTracker() {
void updateChunkInfo(long chunkOffset, int mapIndex) throws IOException {
long idxStartPos = -1;
try {
- // update the chunk tracker to meta file before index file
- writeChunkTracker(mapIndex);
idxStartPos = indexFile.getFilePointer();
logger.trace("{} shuffleId {} reduceId {} updated index current {} updated {}",
appShuffleId.appId, appShuffleId.shuffleId, reduceId, this.lastChunkOffset,
chunkOffset);
- indexFile.writeLong(chunkOffset);
+ indexFile.write(Longs.toByteArray(chunkOffset));
+ // Chunk bitmap should be written to the meta file after the index file because if there are
+ // any exceptions during writing the offset to the index file, meta file should not be
+ // updated. If the update to the index file is successful but the update to meta file isn't
+ // then the index file position is reset in the catch clause.
+ writeChunkTracker(mapIndex);
Review comment:
Could you file a JIRA ticket for the corresponding work for the client?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] mridulm commented on pull request #30433: [SPARK-32916][SHUFFLE][test-maven][test-hadoop2.7] Ensure the number of chunks in meta file and index file are equal
Posted by GitBox <gi...@apache.org>.
mridulm commented on pull request #30433:
URL: https://github.com/apache/spark/pull/30433#issuecomment-731555135
> Any particular reason for this change ? writeLong should be able to do it without the intermediate byte[] creation ?
Scratch that, missed it in the description
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] Victsm commented on a change in pull request #30433: [SPARK-32916][SHUFFLE][test-maven][test-hadoop2.7] Ensure the number of chunks in meta file and index file are equal
Posted by GitBox <gi...@apache.org>.
Victsm commented on a change in pull request #30433:
URL: https://github.com/apache/spark/pull/30433#discussion_r534192619
##########
File path: common/network-shuffle/src/main/java/org/apache/spark/network/shuffle/RemoteBlockPushResolver.java
##########
@@ -877,7 +886,7 @@ void closeAllFiles() {
}
Review comment:
That truncation is to truncate to the last successfully appended block.
However, if the index/meta files are not in sync, we would have these blocks in the end of the file that won't be consumed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #30433: [SPARK-32916][SHUFFLE][test-maven][test-hadoop2.7] Ensure the number of chunks in meta file and index file are equal
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #30433:
URL: https://github.com/apache/spark/pull/30433#issuecomment-737709304
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/36698/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #30433: [SPARK-32916][SHUFFLE][test-maven][test-hadoop2.7] Ensure the number of chunks in meta file and index file are equal
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #30433:
URL: https://github.com/apache/spark/pull/30433#issuecomment-740915871
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/132444/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #30433: [SPARK-32916][SHUFFLE][test-maven][test-hadoop2.7] Ensure the number of chunks in meta file and index file are equal
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #30433:
URL: https://github.com/apache/spark/pull/30433#issuecomment-749641378
**[Test build #133233 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/133233/testReport)** for PR 30433 at commit [`f48d52f`](https://github.com/apache/spark/commit/f48d52f68e52b63742e7cce5dbb4fbcc89845402).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #30433: [SPARK-32916][SHUFFLE][test-maven][test-hadoop2.7] Ensure the number of chunks in meta file and index file are equal
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #30433:
URL: https://github.com/apache/spark/pull/30433#issuecomment-749673594
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/37832/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] otterc commented on pull request #30433: [SPARK-32916][SHUFFLE][test-maven][test-hadoop2.7] Ensure the number of chunks in meta file and index file are equal
Posted by GitBox <gi...@apache.org>.
otterc commented on pull request #30433:
URL: https://github.com/apache/spark/pull/30433#issuecomment-737665325
@mridulm @Victsm @Ngone51 @tgravescs @attilapiros Please take a look when you get a chance.
There are a few changes here which revert few recommendations in https://github.com/apache/spark/pull/30062 to address bugs.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] otterc commented on a change in pull request #30433: [SPARK-32916][SHUFFLE][test-maven][test-hadoop2.7] Ensure the number of chunks in meta file and index file are equal
Posted by GitBox <gi...@apache.org>.
otterc commented on a change in pull request #30433:
URL: https://github.com/apache/spark/pull/30433#discussion_r528907429
##########
File path: common/network-shuffle/src/main/java/org/apache/spark/network/shuffle/RemoteBlockPushResolver.java
##########
@@ -827,13 +833,16 @@ void resetChunkTracker() {
void updateChunkInfo(long chunkOffset, int mapIndex) throws IOException {
long idxStartPos = -1;
try {
- // update the chunk tracker to meta file before index file
- writeChunkTracker(mapIndex);
idxStartPos = indexFile.getFilePointer();
logger.trace("{} shuffleId {} reduceId {} updated index current {} updated {}",
appShuffleId.appId, appShuffleId.shuffleId, reduceId, this.lastChunkOffset,
chunkOffset);
- indexFile.writeLong(chunkOffset);
+ indexFile.write(Longs.toByteArray(chunkOffset));
+ // Chunk bitmap should be written to the meta file after the index file because if there are
+ // any exceptions during writing the offset to the index file, meta file should not be
+ // updated. If the update to the index file is successful but the update to meta file isn't
+ // then the index file position is reset in the catch clause.
+ writeChunkTracker(mapIndex);
Review comment:
@Victsm The case that you described how often does it happen? How does this cleanup script differentiates shuffle data from temporary user data and how quickly does it remove this data. This has an assumption that cluster is setup in a certain way.
Would you agree that the more common cases are the non-recoverable disk failures, permission issues, file corruption etc?
If yes, then there will be continuous IOExceptions while updating these index/meta file. So if we just let the chunk grow when the index/meta file couldn't be updated, in most of the cases the executor will fail to fetch these.
In the non-push shuffle mechanism, if writing to index file fails does the executor try multiple times to write to the same file locally? I understand that in the push-mechanism, we are updating the index/meta in a continuous manner but these are also just local files.
> I think that policy should require more than 2 consecutive IOExceptions during chunk metadata write to trigger.
Note that, the same issue could affect writing data file as well.
Just a single IOException while updating the index/meta file should be enough to indicate the client to stop pushing. Yes, it affects writing the data file as well. The reason I am not bringing that up because in this change we are focusing on the exceptions while updating metadata. If others think that we should address the failures during writes to data file as well in this followup fix, we can address it.
> If we want to address this issue holistically, it would require some state tracking.
If so, I'd rather track it on the client side instead of the server side.
We are already tracking the state of shuffle partition in `AppShufflePartitionInfo`. Adding few other flags to it shouldn't be an issue.
Would like to know the thoughts of other folks as well.
@mridulm @tgravescs @attilapiros @Ngone51
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] otterc commented on a change in pull request #30433: [SPARK-32916][SHUFFLE][test-maven][test-hadoop2.7] Ensure the number of chunks in meta file and index file are equal
Posted by GitBox <gi...@apache.org>.
otterc commented on a change in pull request #30433:
URL: https://github.com/apache/spark/pull/30433#discussion_r528519186
##########
File path: common/network-shuffle/src/main/java/org/apache/spark/network/shuffle/RemoteBlockPushResolver.java
##########
@@ -827,13 +833,16 @@ void resetChunkTracker() {
void updateChunkInfo(long chunkOffset, int mapIndex) throws IOException {
long idxStartPos = -1;
try {
- // update the chunk tracker to meta file before index file
- writeChunkTracker(mapIndex);
idxStartPos = indexFile.getFilePointer();
logger.trace("{} shuffleId {} reduceId {} updated index current {} updated {}",
appShuffleId.appId, appShuffleId.shuffleId, reduceId, this.lastChunkOffset,
chunkOffset);
- indexFile.writeLong(chunkOffset);
+ indexFile.write(Longs.toByteArray(chunkOffset));
+ // Chunk bitmap should be written to the meta file after the index file because if there are
+ // any exceptions during writing the offset to the index file, meta file should not be
+ // updated. If the update to the index file is successful but the update to meta file isn't
+ // then the index file position is reset in the catch clause.
+ writeChunkTracker(mapIndex);
Review comment:
> I think we should catch the chunk level IOExceptions inside onComplete, setting certain flags to make the onFailure handling logic know about whether we encountered a block level failure or a chunk level failure.
For a chunk level failure, we shouldn't overwrite the previous block, we should effectively only delay the closure of the chunk.
The closure of chunk write now depends on the minimum size of the chunk. If we keep growing the chunk then eventually it will be very large and the executors will get killed fetching them because they will exceed the physical memory limit. This was the main reason to serve data in manageable-sized chunks.
> Stopping merging blocks only after 2 consecutive IOExceptions seems to be giving up too early.
With local file systems the IOExceptions that I can think of would be due to disk failures, file corruption, permission and these will not go away when retrying to write to these. Are there any other scenarios that I am missing?
> If there's a need to stop early due to too many disk failures, I think that logic should be on the client side inside ErrorHandler.
Sure, however the server here can respond with a similar runtime exception that it can't update merge file like the `TooLate` runtime exception and the error handle for the push side can stop pushing the data.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #30433: [SPARK-32916][SHUFFLE][test-maven][test-hadoop2.7] Ensure the number of chunks in meta file and index file are equal
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #30433:
URL: https://github.com/apache/spark/pull/30433#issuecomment-749834474
**[Test build #133233 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/133233/testReport)** for PR 30433 at commit [`f48d52f`](https://github.com/apache/spark/commit/f48d52f68e52b63742e7cce5dbb4fbcc89845402).
* This patch **fails Spark unit tests**.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] otterc commented on a change in pull request #30433: [SPARK-32916][SHUFFLE][test-maven][test-hadoop2.7] Ensure the number of chunks in meta file and index file are equal
Posted by GitBox <gi...@apache.org>.
otterc commented on a change in pull request #30433:
URL: https://github.com/apache/spark/pull/30433#discussion_r536347122
##########
File path: common/network-shuffle/src/main/java/org/apache/spark/network/shuffle/RemoteBlockPushResolver.java
##########
@@ -825,45 +881,79 @@ void resetChunkTracker() {
* @param mapIndex the map index to be added to chunk tracker.
*/
void updateChunkInfo(long chunkOffset, int mapIndex) throws IOException {
- long idxStartPos = -1;
try {
- // update the chunk tracker to meta file before index file
- writeChunkTracker(mapIndex);
- idxStartPos = indexFile.getFilePointer();
logger.trace("{} shuffleId {} reduceId {} updated index current {} updated {}",
- appShuffleId.appId, appShuffleId.shuffleId, reduceId, this.lastChunkOffset,
- chunkOffset);
- indexFile.writeLong(chunkOffset);
- } catch (IOException ioe) {
- if (idxStartPos != -1) {
- // reset the position to avoid corrupting index files during exception.
- logger.warn("{} shuffleId {} reduceId {} reset index to position {}",
- appShuffleId.appId, appShuffleId.shuffleId, reduceId, idxStartPos);
- indexFile.seek(idxStartPos);
+ appShuffleId.appId, appShuffleId.shuffleId, reduceId, this.lastChunkOffset, chunkOffset);
+ if (indexMetaUpdateFailed) {
+ indexFile.getChannel().position(indexFile.getPos());
}
+ indexFile.getDos().writeLong(chunkOffset);
+ // Chunk bitmap should be written to the meta file after the index file because if there are
+ // any exceptions during writing the offset to the index file, meta file should not be
+ // updated. If the update to the index file is successful but the update to meta file isn't
+ // then the index file position is reset in the catch clause.
+ writeChunkTracker(mapIndex);
+ indexFile.updatePos(8);
+ this.lastChunkOffset = chunkOffset;
+ indexMetaUpdateFailed = false;
+ } catch (IOException ioe) {
+ logger.warn("{} shuffleId {} reduceId {} update to index/meta failed", appShuffleId.appId,
+ appShuffleId.shuffleId, reduceId);
+ indexMetaUpdateFailed = true;
+ // Any exception here is propagated to the caller and the caller can decide whether to
+ // abort or not.
throw ioe;
}
- this.lastChunkOffset = chunkOffset;
}
private void writeChunkTracker(int mapIndex) throws IOException {
if (mapIndex == -1) {
return;
}
chunkTracker.add(mapIndex);
- long metaStartPos = metaFile.getFilePointer();
try {
logger.trace("{} shuffleId {} reduceId {} mapIndex {} write chunk to meta file",
appShuffleId.appId, appShuffleId.shuffleId, reduceId, mapIndex);
- chunkTracker.serialize(metaFile);
+ if (indexMetaUpdateFailed) {
+ metaFile.getChannel().position(metaFile.getPos());
+ }
+ int chunkTrackerSize = chunkTracker.serializedSizeInBytes();
+ chunkTracker.serialize(metaFile.getDos());
+ metaFile.updatePos(chunkTrackerSize);
} catch (IOException ioe) {
- logger.warn("{} shuffleId {} reduceId {} mapIndex {} reset position of meta file to {}",
- appShuffleId.appId, appShuffleId.shuffleId, reduceId, mapIndex, metaStartPos);
- metaFile.seek(metaStartPos);
+ logger.warn("{} shuffleId {} reduceId {} mapIndex {} update to meta file failed",
Review comment:
done
##########
File path: common/network-shuffle/src/main/java/org/apache/spark/network/shuffle/RemoteBlockPushResolver.java
##########
@@ -825,45 +881,79 @@ void resetChunkTracker() {
* @param mapIndex the map index to be added to chunk tracker.
*/
void updateChunkInfo(long chunkOffset, int mapIndex) throws IOException {
- long idxStartPos = -1;
try {
- // update the chunk tracker to meta file before index file
- writeChunkTracker(mapIndex);
- idxStartPos = indexFile.getFilePointer();
logger.trace("{} shuffleId {} reduceId {} updated index current {} updated {}",
- appShuffleId.appId, appShuffleId.shuffleId, reduceId, this.lastChunkOffset,
- chunkOffset);
- indexFile.writeLong(chunkOffset);
- } catch (IOException ioe) {
- if (idxStartPos != -1) {
- // reset the position to avoid corrupting index files during exception.
- logger.warn("{} shuffleId {} reduceId {} reset index to position {}",
- appShuffleId.appId, appShuffleId.shuffleId, reduceId, idxStartPos);
- indexFile.seek(idxStartPos);
+ appShuffleId.appId, appShuffleId.shuffleId, reduceId, this.lastChunkOffset, chunkOffset);
+ if (indexMetaUpdateFailed) {
+ indexFile.getChannel().position(indexFile.getPos());
}
+ indexFile.getDos().writeLong(chunkOffset);
+ // Chunk bitmap should be written to the meta file after the index file because if there are
+ // any exceptions during writing the offset to the index file, meta file should not be
+ // updated. If the update to the index file is successful but the update to meta file isn't
+ // then the index file position is reset in the catch clause.
+ writeChunkTracker(mapIndex);
+ indexFile.updatePos(8);
+ this.lastChunkOffset = chunkOffset;
+ indexMetaUpdateFailed = false;
+ } catch (IOException ioe) {
+ logger.warn("{} shuffleId {} reduceId {} update to index/meta failed", appShuffleId.appId,
+ appShuffleId.shuffleId, reduceId);
+ indexMetaUpdateFailed = true;
+ // Any exception here is propagated to the caller and the caller can decide whether to
+ // abort or not.
throw ioe;
}
- this.lastChunkOffset = chunkOffset;
}
private void writeChunkTracker(int mapIndex) throws IOException {
if (mapIndex == -1) {
return;
}
chunkTracker.add(mapIndex);
- long metaStartPos = metaFile.getFilePointer();
try {
logger.trace("{} shuffleId {} reduceId {} mapIndex {} write chunk to meta file",
appShuffleId.appId, appShuffleId.shuffleId, reduceId, mapIndex);
- chunkTracker.serialize(metaFile);
+ if (indexMetaUpdateFailed) {
+ metaFile.getChannel().position(metaFile.getPos());
+ }
+ int chunkTrackerSize = chunkTracker.serializedSizeInBytes();
+ chunkTracker.serialize(metaFile.getDos());
+ metaFile.updatePos(chunkTrackerSize);
Review comment:
done
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #30433: [SPARK-32916][SHUFFLE][test-maven][test-hadoop2.7] Ensure the number of chunks in meta file and index file are equal
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #30433:
URL: https://github.com/apache/spark/pull/30433#issuecomment-732504488
**[Test build #131594 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/131594/testReport)** for PR 30433 at commit [`30c8f83`](https://github.com/apache/spark/commit/30c8f83b7ba3e39724c90865fd1318b41876f927).
* This patch **fails to build**.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #30433: [SPARK-32916][SHUFFLE][test-maven][test-hadoop2.7] Ensure the number of chunks in meta file and index file are equal
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #30433:
URL: https://github.com/apache/spark/pull/30433#issuecomment-741826993
**[Test build #132471 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/132471/testReport)** for PR 30433 at commit [`0bce631`](https://github.com/apache/spark/commit/0bce63198e2e5304dd31aa39602f16d9d7e90af4).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #30433: [SPARK-32916][SHUFFLE][test-maven][test-hadoop2.7] Ensure the number of chunks in meta file and index file are equal
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #30433:
URL: https://github.com/apache/spark/pull/30433#issuecomment-741051011
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/37050/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #30433: [SPARK-32916][SHUFFLE][test-maven][test-hadoop2.7] Ensure the number of chunks in meta file and index file are equal
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #30433:
URL: https://github.com/apache/spark/pull/30433#issuecomment-730608398
Can one of the admins verify this patch?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #30433: [SPARK-32916][SHUFFLE][test-maven][test-hadoop2.7] Ensure the number of chunks in meta file and index file are equal
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #30433:
URL: https://github.com/apache/spark/pull/30433#issuecomment-741605563
Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/37073/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] mridulm commented on a change in pull request #30433: [SPARK-32916][SHUFFLE][test-maven][test-hadoop2.7] Ensure the number of chunks in meta file and index file are equal
Posted by GitBox <gi...@apache.org>.
mridulm commented on a change in pull request #30433:
URL: https://github.com/apache/spark/pull/30433#discussion_r529015078
##########
File path: common/network-shuffle/src/main/java/org/apache/spark/network/shuffle/RemoteBlockPushResolver.java
##########
@@ -827,13 +833,16 @@ void resetChunkTracker() {
void updateChunkInfo(long chunkOffset, int mapIndex) throws IOException {
long idxStartPos = -1;
try {
- // update the chunk tracker to meta file before index file
- writeChunkTracker(mapIndex);
idxStartPos = indexFile.getFilePointer();
logger.trace("{} shuffleId {} reduceId {} updated index current {} updated {}",
appShuffleId.appId, appShuffleId.shuffleId, reduceId, this.lastChunkOffset,
chunkOffset);
- indexFile.writeLong(chunkOffset);
+ indexFile.write(Longs.toByteArray(chunkOffset));
Review comment:
There are multiple cases of this in spark; but nothing which is reusable. That is ok, the cost is not very high, and this is infrequent enough in comparison to be an issue :)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] otterc commented on a change in pull request #30433: [SPARK-32916][SHUFFLE][test-maven][test-hadoop2.7] Ensure the number of chunks in meta file and index file are equal
Posted by GitBox <gi...@apache.org>.
otterc commented on a change in pull request #30433:
URL: https://github.com/apache/spark/pull/30433#discussion_r533783839
##########
File path: common/network-shuffle/src/main/java/org/apache/spark/network/shuffle/RemoteBlockPushResolver.java
##########
@@ -877,7 +886,7 @@ void closeAllFiles() {
}
Review comment:
What do we do when the truncate of data file here fails?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] mridulm commented on a change in pull request #30433: [SPARK-32916][SHUFFLE][test-maven][test-hadoop2.7] Ensure the number of chunks in meta file and index file are equal
Posted by GitBox <gi...@apache.org>.
mridulm commented on a change in pull request #30433:
URL: https://github.com/apache/spark/pull/30433#discussion_r532973975
##########
File path: common/network-shuffle/src/main/java/org/apache/spark/network/shuffle/RemoteBlockPushResolver.java
##########
@@ -877,7 +886,7 @@ void closeAllFiles() {
}
Review comment:
Truncating would be good, it will keep the file consistent with the index files as well.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] mridulm commented on a change in pull request #30433: [SPARK-32916][SHUFFLE][test-maven][test-hadoop2.7] Ensure the number of chunks in meta file and index file are equal
Posted by GitBox <gi...@apache.org>.
mridulm commented on a change in pull request #30433:
URL: https://github.com/apache/spark/pull/30433#discussion_r528176231
##########
File path: common/network-shuffle/src/main/java/org/apache/spark/network/shuffle/RemoteBlockPushResolver.java
##########
@@ -827,13 +833,16 @@ void resetChunkTracker() {
void updateChunkInfo(long chunkOffset, int mapIndex) throws IOException {
long idxStartPos = -1;
try {
- // update the chunk tracker to meta file before index file
- writeChunkTracker(mapIndex);
idxStartPos = indexFile.getFilePointer();
logger.trace("{} shuffleId {} reduceId {} updated index current {} updated {}",
appShuffleId.appId, appShuffleId.shuffleId, reduceId, this.lastChunkOffset,
chunkOffset);
- indexFile.writeLong(chunkOffset);
+ indexFile.write(Longs.toByteArray(chunkOffset));
Review comment:
I wish `Longs` took a parameter - we can skip allocation/clear of the array (cheap as it might be).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] Victsm commented on a change in pull request #30433: [SPARK-32916][SHUFFLE][test-maven][test-hadoop2.7] Ensure the number of chunks in meta file and index file are equal
Posted by GitBox <gi...@apache.org>.
Victsm commented on a change in pull request #30433:
URL: https://github.com/apache/spark/pull/30433#discussion_r528921687
##########
File path: common/network-shuffle/src/main/java/org/apache/spark/network/shuffle/RemoteBlockPushResolver.java
##########
@@ -827,13 +833,16 @@ void resetChunkTracker() {
void updateChunkInfo(long chunkOffset, int mapIndex) throws IOException {
long idxStartPos = -1;
try {
- // update the chunk tracker to meta file before index file
- writeChunkTracker(mapIndex);
idxStartPos = indexFile.getFilePointer();
logger.trace("{} shuffleId {} reduceId {} updated index current {} updated {}",
appShuffleId.appId, appShuffleId.shuffleId, reduceId, this.lastChunkOffset,
chunkOffset);
- indexFile.writeLong(chunkOffset);
+ indexFile.write(Longs.toByteArray(chunkOffset));
+ // Chunk bitmap should be written to the meta file after the index file because if there are
+ // any exceptions during writing the offset to the index file, meta file should not be
+ // updated. If the update to the index file is successful but the update to meta file isn't
+ // then the index file position is reset in the catch clause.
+ writeChunkTracker(mapIndex);
Review comment:
According to our operation experiences, the temporary issue I described is more common than the non-recoverable issues you described.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] Victsm commented on a change in pull request #30433: [SPARK-32916][SHUFFLE][test-maven][test-hadoop2.7] Ensure the number of chunks in meta file and index file are equal
Posted by GitBox <gi...@apache.org>.
Victsm commented on a change in pull request #30433:
URL: https://github.com/apache/spark/pull/30433#discussion_r540534695
##########
File path: common/network-shuffle/src/main/java/org/apache/spark/network/shuffle/RemoteBlockPushResolver.java
##########
@@ -586,6 +592,7 @@ public void onData(String streamId, ByteBuffer buf) throws IOException {
deferredBufs = null;
return;
}
+ abortIfNecessary();
Review comment:
Had an offline discussion with @mridulm and @otterc , the concern about closing channels with throwing exception mid-stream seems negligible since it only happens after reaching the max IOException count.
Calling `abortIfNecessary` inside `onData` should also have negligible performance implications.
With those, I think it should be fine to keep this part of the code as is.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] Victsm commented on a change in pull request #30433: [SPARK-32916][SHUFFLE][test-maven][test-hadoop2.7] Ensure the number of chunks in meta file and index file are equal
Posted by GitBox <gi...@apache.org>.
Victsm commented on a change in pull request #30433:
URL: https://github.com/apache/spark/pull/30433#discussion_r539085223
##########
File path: common/network-shuffle/src/main/java/org/apache/spark/network/shuffle/RemoteBlockPushResolver.java
##########
@@ -370,26 +391,19 @@ public MergeStatuses finalizeShuffleMerge(FinalizeShuffleMerge msg) throws IOExc
new MergeStatuses(msg.shuffleId, new RoaringBitmap[0], new int[0], new long[0]);
} else {
Collection<AppShufflePartitionInfo> partitionsToFinalize = shufflePartitions.values();
- int totalPartitions = partitionsToFinalize.size();
- RoaringBitmap[] bitmaps = new RoaringBitmap[totalPartitions];
- int[] reduceIds = new int[totalPartitions];
- long[] sizes = new long[totalPartitions];
+ List<RoaringBitmap> bitmaps = new ArrayList<>(partitionsToFinalize.size());
+ List<Integer> reduceIds = new ArrayList<>(partitionsToFinalize.size());
+ List<Long> sizes = new ArrayList<>(partitionsToFinalize.size());
Review comment:
I just thought that since we made this change in #30062, we might want to keep it if we can.
I don't have strong opinion on this.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #30433: [SPARK-32916][SHUFFLE][test-maven][test-hadoop2.7] Ensure the number of chunks in meta file and index file are equal
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #30433:
URL: https://github.com/apache/spark/pull/30433#issuecomment-737998185
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] Ngone51 commented on a change in pull request #30433: [SPARK-32916][SHUFFLE][test-maven][test-hadoop2.7] Ensure the number of chunks in meta file and index file are equal
Posted by GitBox <gi...@apache.org>.
Ngone51 commented on a change in pull request #30433:
URL: https://github.com/apache/spark/pull/30433#discussion_r538991760
##########
File path: common/network-shuffle/src/test/java/org/apache/spark/network/shuffle/RemoteBlockPushResolverSuite.java
##########
@@ -411,6 +415,269 @@ void deleteExecutorDirs(Path[] dirs) {
}
}
+ @Test
+ public void testRecoverIndexFileAfterIOExceptions() throws IOException {
+ useTestFiles(true, false);
+ RemoteBlockPushResolver.PushBlockStreamCallback callback1 =
+ (RemoteBlockPushResolver.PushBlockStreamCallback) pushResolver.receiveBlockDataAsStream(
+ new PushBlockStream(TEST_APP, 0, 0, 0, 0));
+ callback1.onData(callback1.getID(), ByteBuffer.wrap(new byte[4]));
+ callback1.onComplete(callback1.getID());
+ RemoteBlockPushResolver.AppShufflePartitionInfo partitionInfo = callback1.getPartitionInfo();
+ // Close the index stream so it throws IOException
+ TestMergeShuffleFile testIndexFile = (TestMergeShuffleFile) partitionInfo.getIndexFile();
+ testIndexFile.close();
+ StreamCallbackWithID callback2 = pushResolver.receiveBlockDataAsStream(
+ new PushBlockStream(TEST_APP, 0, 1, 0, 0));
+ callback2.onData(callback2.getID(), ByteBuffer.wrap(new byte[5]));
+ // This will complete without any IOExceptions because number of IOExceptions are less than
+ // the threshold but the update to index file will be unsuccessful.
+ callback2.onComplete(callback2.getID());
+ assertEquals("index position", 16, testIndexFile.getPos());
+ // Restore the index stream so it can write successfully again.
+ testIndexFile.restore();
+ StreamCallbackWithID callback3 = pushResolver.receiveBlockDataAsStream(
+ new PushBlockStream(TEST_APP, 0, 2, 0, 0));
+ callback3.onData(callback3.getID(), ByteBuffer.wrap(new byte[2]));
+ callback3.onComplete(callback3.getID());
+ assertEquals("index position", 24, testIndexFile.getPos());
+ MergeStatuses statuses = pushResolver.finalizeShuffleMerge(
+ new FinalizeShuffleMerge(TEST_APP, 0));
+ validateMergeStatuses(statuses, new int[] {0}, new long[] {11});
+ MergedBlockMeta blockMeta = pushResolver.getMergedBlockMeta(TEST_APP, 0, 0);
+ validateChunks(TEST_APP, 0, 0, blockMeta, new int[] {4, 7}, new int[][] {{0}, {1, 2}});
+ }
+
+ @Test
+ public void testRecoverIndexFileAfterIOExceptionsInFinalize() throws IOException {
+ useTestFiles(true, false);
+ registerExecutor(TEST_APP, prepareLocalDirs(localDirs));
+ RemoteBlockPushResolver.PushBlockStreamCallback callback1 =
+ (RemoteBlockPushResolver.PushBlockStreamCallback) pushResolver.receiveBlockDataAsStream(
+ new PushBlockStream(TEST_APP, 0, 0, 0, 0));
+ callback1.onData(callback1.getID(), ByteBuffer.wrap(new byte[4]));
+ callback1.onComplete(callback1.getID());
+ RemoteBlockPushResolver.AppShufflePartitionInfo partitionInfo = callback1.getPartitionInfo();
+ // Close the index stream so it throws IOException
+ TestMergeShuffleFile testIndexFile = (TestMergeShuffleFile) partitionInfo.getIndexFile();
+ testIndexFile.close();
+ StreamCallbackWithID callback2 = pushResolver.receiveBlockDataAsStream(
+ new PushBlockStream(TEST_APP, 0, 1, 0, 0));
+ callback2.onData(callback2.getID(), ByteBuffer.wrap(new byte[5]));
+ // This will complete without any IOExceptions because number of IOExceptions are less than
+ // the threshold but the update to index file will be unsuccessful.
+ callback2.onComplete(callback2.getID());
+ assertEquals("index position", 16, testIndexFile.getPos());
+ // The last update to index was unsuccessful however any further updates will be successful.
+ // Restore the index stream so it can write successfully again.
+ testIndexFile.restore();
+ MergeStatuses statuses = pushResolver.finalizeShuffleMerge(
+ new FinalizeShuffleMerge(TEST_APP, 0));
+ assertEquals("index position", 24, testIndexFile.getPos());
+ validateMergeStatuses(statuses, new int[] {0}, new long[] {9});
+ MergedBlockMeta blockMeta = pushResolver.getMergedBlockMeta(TEST_APP, 0, 0);
+ validateChunks(TEST_APP, 0, 0, blockMeta, new int[] {4, 5}, new int[][] {{0}, {1}});
+ }
+
+ @Test
+ public void testRecoverMetaFileAfterIOExceptions() throws IOException {
+ useTestFiles(false, true);
+ RemoteBlockPushResolver.PushBlockStreamCallback callback1 =
+ (RemoteBlockPushResolver.PushBlockStreamCallback) pushResolver.receiveBlockDataAsStream(
+ new PushBlockStream(TEST_APP, 0, 0, 0, 0));
+ callback1.onData(callback1.getID(), ByteBuffer.wrap(new byte[4]));
+ callback1.onComplete(callback1.getID());
+ RemoteBlockPushResolver.AppShufflePartitionInfo partitionInfo = callback1.getPartitionInfo();
+ // Close the meta stream so it throws IOException
+ TestMergeShuffleFile testMetaFile = (TestMergeShuffleFile) partitionInfo.getMetaFile();
+ long metaPosBeforeClose = testMetaFile.getPos();
+ testMetaFile.close();
+ StreamCallbackWithID callback2 = pushResolver.receiveBlockDataAsStream(
+ new PushBlockStream(TEST_APP, 0, 1, 0, 0));
+ callback2.onData(callback2.getID(), ByteBuffer.wrap(new byte[5]));
+ // This will complete without any IOExceptions because number of IOExceptions are less than
+ // the threshold but the update to index and meta file will be unsuccessful.
+ callback2.onComplete(callback2.getID());
+ assertEquals("index position", 16, partitionInfo.getIndexFile().getPos());
+ assertEquals("meta position", metaPosBeforeClose, testMetaFile.getPos());
+ // Restore the meta stream so it can write successfully again.
+ testMetaFile.restore();
+ StreamCallbackWithID callback3 = pushResolver.receiveBlockDataAsStream(
+ new PushBlockStream(TEST_APP, 0, 2, 0, 0));
+ callback3.onData(callback3.getID(), ByteBuffer.wrap(new byte[2]));
+ callback3.onComplete(callback3.getID());
+ assertEquals("index position", 24, partitionInfo.getIndexFile().getPos());
+ assertTrue("meta position", testMetaFile.getPos() > metaPosBeforeClose);
+ MergeStatuses statuses = pushResolver.finalizeShuffleMerge(
+ new FinalizeShuffleMerge(TEST_APP, 0));
+ validateMergeStatuses(statuses, new int[] {0}, new long[] {11});
+ MergedBlockMeta blockMeta = pushResolver.getMergedBlockMeta(TEST_APP, 0, 0);
+ validateChunks(TEST_APP, 0, 0, blockMeta, new int[] {4, 7}, new int[][] {{0}, {1, 2}});
+ }
+
+ @Test
+ public void testRecoverMetaFileAfterIOExceptionsInFinalize() throws IOException {
+ useTestFiles(false, true);
+ RemoteBlockPushResolver.PushBlockStreamCallback callback1 =
+ (RemoteBlockPushResolver.PushBlockStreamCallback) pushResolver.receiveBlockDataAsStream(
+ new PushBlockStream(TEST_APP, 0, 0, 0, 0));
+ callback1.onData(callback1.getID(), ByteBuffer.wrap(new byte[4]));
+ callback1.onComplete(callback1.getID());
+ RemoteBlockPushResolver.AppShufflePartitionInfo partitionInfo = callback1.getPartitionInfo();
+ // Close the meta stream so it throws IOException
+ TestMergeShuffleFile testMetaFile = (TestMergeShuffleFile) partitionInfo.getMetaFile();
+ long metaPosBeforeClose = testMetaFile.getPos();
+ testMetaFile.close();
+ StreamCallbackWithID callback2 = pushResolver.receiveBlockDataAsStream(
+ new PushBlockStream(TEST_APP, 0, 1, 0, 0));
+ callback2.onData(callback2.getID(), ByteBuffer.wrap(new byte[5]));
+ // This will complete without any IOExceptions because number of IOExceptions are less than
+ // the threshold but the update to index and meta file will be unsuccessful.
+ callback2.onComplete(callback2.getID());
+ MergeShuffleFile indexFile = partitionInfo.getIndexFile();
+ assertEquals("index position", 16, indexFile.getPos());
+ assertEquals("meta position", metaPosBeforeClose, testMetaFile.getPos());
+ // Restore the meta stream so it can write successfully again.
+ testMetaFile.restore();
+ MergeStatuses statuses = pushResolver.finalizeShuffleMerge(
+ new FinalizeShuffleMerge(TEST_APP, 0));
+ assertEquals("index position", 24, indexFile.getPos());
+ assertTrue("meta position", testMetaFile.getPos() > metaPosBeforeClose);
+ validateMergeStatuses(statuses, new int[] {0}, new long[] {9});
+ MergedBlockMeta blockMeta = pushResolver.getMergedBlockMeta(TEST_APP, 0, 0);
+ validateChunks(TEST_APP, 0, 0, blockMeta, new int[] {4, 5}, new int[][] {{0}, {1}});
+ }
+
+ @Test (expected = RuntimeException.class)
+ public void testIOExceptionsExceededThreshold() throws IOException {
+ RemoteBlockPushResolver.PushBlockStreamCallback callback =
+ (RemoteBlockPushResolver.PushBlockStreamCallback) pushResolver.receiveBlockDataAsStream(
+ new PushBlockStream(TEST_APP, 0, 0, 0, 0));
+ RemoteBlockPushResolver.AppShufflePartitionInfo partitionInfo = callback.getPartitionInfo();
+ callback.onData(callback.getID(), ByteBuffer.wrap(new byte[4]));
+ callback.onComplete(callback.getID());
+ // Close the data stream so it throws continuous IOException
+ partitionInfo.getDataChannel().close();
+ for (int i = 1; i < 5; i++) {
+ RemoteBlockPushResolver.PushBlockStreamCallback callback1 =
+ (RemoteBlockPushResolver.PushBlockStreamCallback) pushResolver.receiveBlockDataAsStream(
+ new PushBlockStream(TEST_APP, 0, i, 0, 0));
+ try {
+ callback1.onData(callback1.getID(), ByteBuffer.wrap(new byte[2]));
+ } catch (IOException ioe) {
+ // this will throw IOException so the client can retry.
+ callback1.onFailure(callback1.getID(), ioe);
+ }
+ }
+ assertEquals(4, partitionInfo.getNumIOExceptions());
+ // After 4 IOException, the server will respond with IOExceptions exceeded threshold
+ try {
+ RemoteBlockPushResolver.PushBlockStreamCallback callback2 =
+ (RemoteBlockPushResolver.PushBlockStreamCallback) pushResolver.receiveBlockDataAsStream(
+ new PushBlockStream(TEST_APP, 0, 5, 0, 0));
+ callback2.onData(callback.getID(), ByteBuffer.wrap(new byte[1]));
+ } catch (Throwable t) {
+ assertEquals("IOExceptions exceeded the threshold when merging shufflePush_0_5_0",
+ t.getMessage());
+ throw t;
+ }
Review comment:
> Do you think we should check before writing them?
I think so.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #30433: [SPARK-32916][SHUFFLE][test-maven][test-hadoop2.7] Ensure the number of chunks in meta file and index file are equal
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #30433:
URL: https://github.com/apache/spark/pull/30433#issuecomment-740936463
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/37046/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #30433: [SPARK-32916][SHUFFLE][test-maven][test-hadoop2.7] Ensure the number of chunks in meta file and index file are equal
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #30433:
URL: https://github.com/apache/spark/pull/30433#issuecomment-740998644
**[Test build #132448 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/132448/testReport)** for PR 30433 at commit [`5422f10`](https://github.com/apache/spark/commit/5422f100a197d96f646864c75fc1a0beb2d41af6).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #30433: [SPARK-32916][SHUFFLE][test-maven][test-hadoop2.7] Ensure the number of chunks in meta file and index file are equal
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #30433:
URL: https://github.com/apache/spark/pull/30433#issuecomment-737698947
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #30433: [SPARK-32916][SHUFFLE][test-maven][test-hadoop2.7] Ensure the number of chunks in meta file and index file are equal
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #30433:
URL: https://github.com/apache/spark/pull/30433#issuecomment-738547222
**[Test build #132158 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/132158/testReport)** for PR 30433 at commit [`72b5976`](https://github.com/apache/spark/commit/72b5976579aa959689aaa78f1b8a841d6c9a09e1).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #30433: [SPARK-32916][SHUFFLE][test-maven][test-hadoop2.7] Ensure the number of chunks in meta file and index file are equal
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #30433:
URL: https://github.com/apache/spark/pull/30433#issuecomment-741622882
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/37073/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] Ngone51 commented on a change in pull request #30433: [SPARK-32916][SHUFFLE][test-maven][test-hadoop2.7] Ensure the number of chunks in meta file and index file are equal
Posted by GitBox <gi...@apache.org>.
Ngone51 commented on a change in pull request #30433:
URL: https://github.com/apache/spark/pull/30433#discussion_r530108377
##########
File path: common/network-shuffle/src/main/java/org/apache/spark/network/shuffle/RemoteBlockPushResolver.java
##########
@@ -827,13 +833,16 @@ void resetChunkTracker() {
void updateChunkInfo(long chunkOffset, int mapIndex) throws IOException {
long idxStartPos = -1;
try {
- // update the chunk tracker to meta file before index file
- writeChunkTracker(mapIndex);
idxStartPos = indexFile.getFilePointer();
logger.trace("{} shuffleId {} reduceId {} updated index current {} updated {}",
appShuffleId.appId, appShuffleId.shuffleId, reduceId, this.lastChunkOffset,
chunkOffset);
- indexFile.writeLong(chunkOffset);
+ indexFile.write(Longs.toByteArray(chunkOffset));
+ // Chunk bitmap should be written to the meta file after the index file because if there are
+ // any exceptions during writing the offset to the index file, meta file should not be
+ // updated. If the update to the index file is successful but the update to meta file isn't
+ // then the index file position is reset in the catch clause.
+ writeChunkTracker(mapIndex);
Review comment:
I mean (1). I know it's hard for the current framework. It requires more refactor if we want to do so.
(2) sounds good to me. But to clarify, it only benefits the following push but doesn't help ease the problem(the IO exception) in the current push.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #30433: [SPARK-32916][SHUFFLE][test-maven][test-hadoop2.7] Ensure the number of chunks in meta file and index file are equal
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #30433:
URL: https://github.com/apache/spark/pull/30433#issuecomment-742199827
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/132507/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] Ngone51 commented on pull request #30433: [SPARK-32916][SHUFFLE][test-maven][test-hadoop2.7] Ensure the number of chunks in meta file and index file are equal
Posted by GitBox <gi...@apache.org>.
Ngone51 commented on pull request #30433:
URL: https://github.com/apache/spark/pull/30433#issuecomment-738578007
So we get rid of the `seek` now? And `ioExceptionsThresholdDuringMerge` is used for other potential recoverable IOExceptions?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] otterc commented on a change in pull request #30433: [SPARK-32916][SHUFFLE][test-maven][test-hadoop2.7] Ensure the number of chunks in meta file and index file are equal
Posted by GitBox <gi...@apache.org>.
otterc commented on a change in pull request #30433:
URL: https://github.com/apache/spark/pull/30433#discussion_r528229428
##########
File path: common/network-shuffle/src/main/java/org/apache/spark/network/shuffle/RemoteBlockPushResolver.java
##########
@@ -827,13 +833,16 @@ void resetChunkTracker() {
void updateChunkInfo(long chunkOffset, int mapIndex) throws IOException {
long idxStartPos = -1;
try {
- // update the chunk tracker to meta file before index file
- writeChunkTracker(mapIndex);
idxStartPos = indexFile.getFilePointer();
logger.trace("{} shuffleId {} reduceId {} updated index current {} updated {}",
appShuffleId.appId, appShuffleId.shuffleId, reduceId, this.lastChunkOffset,
chunkOffset);
- indexFile.writeLong(chunkOffset);
+ indexFile.write(Longs.toByteArray(chunkOffset));
+ // Chunk bitmap should be written to the meta file after the index file because if there are
+ // any exceptions during writing the offset to the index file, meta file should not be
+ // updated. If the update to the index file is successful but the update to meta file isn't
+ // then the index file position is reset in the catch clause.
+ writeChunkTracker(mapIndex);
Review comment:
This is a good point. I have overlooked IOExceptions from the seeks. Any IOException from the seek would not guarantee that the position is updated. So any more updates to the file will not overwrite the corrupt data which means that the files are corrupted.
One way of handling this I think is maintain the expected length of both index and meta files in AppShufflePartitionInfo instead of relying on the FilePointer. Updates to these files would be similar to writing to the merged data file as below:
```
if (partitionInfo.isEncounteredFailure()) {
long updatedPos = partitionInfo.getPosition() + length;
logger.debug(
"{} shuffleId {} reduceId {} encountered failure current pos {} updated pos {}",
partitionInfo.appShuffleId.appId, partitionInfo.appShuffleId.shuffleId,
partitionInfo.reduceId, partitionInfo.getPosition(), updatedPos);
length += partitionInfo.dataChannel.write(buf, updatedPos);
} else {
length += partitionInfo.dataChannel.write(buf);
}
```
Please let me know if there are any other suggestions?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] Victsm commented on a change in pull request #30433: [SPARK-32916][SHUFFLE][test-maven][test-hadoop2.7] Ensure the number of chunks in meta file and index file are equal
Posted by GitBox <gi...@apache.org>.
Victsm commented on a change in pull request #30433:
URL: https://github.com/apache/spark/pull/30433#discussion_r528472651
##########
File path: common/network-shuffle/src/main/java/org/apache/spark/network/shuffle/RemoteBlockPushResolver.java
##########
@@ -17,6 +17,7 @@
package org.apache.spark.network.shuffle;
+import com.google.common.primitives.Longs;
Review comment:
Nit: import order
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #30433: [SPARK-32916][SHUFFLE][test-maven][test-hadoop2.7] Ensure the number of chunks in meta file and index file are equal
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #30433:
URL: https://github.com/apache/spark/pull/30433#issuecomment-730611561
Can one of the admins verify this patch?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org