You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by GitBox <gi...@apache.org> on 2021/07/22 06:21:28 UTC
[GitHub] [spark] otterc opened a new pull request #33477: [SPARK-36255][SHUFFLE][CORE] Stop pushing and retrying on FileNotFound exceptions
otterc opened a new pull request #33477:
URL: https://github.com/apache/spark/pull/33477
### What changes were proposed in this pull request?
Once a job in a Spark application completes, the shuffle files are cleaned up by the executors. This is because the driver instructs the executors to clean up the shuffle. In this case, the push of the shuffle data by the executors can throw `FileNotFoundException`s because the shuffle files are deleted. When this exception is thrown from the `shuffle-block-push-thread`, it causes the executor to exit. Both the `shuffle-block-push` threads and the netty event-loops will encounter `FileNotFoundException`s in this case. The fix here stops these threads from pushing more blocks when they encounter `FileNotFoundException`. When the exception is from the `shuffle-block-push-thread`, it will get handled and logged as warning instead of failing the executor.
### Why are the changes needed?
This fixes the bug which causes executor to exits when they are instructed to clean up shuffle data by the driver
Below is the stacktrace of this exception:
```
21/06/17 16:03:57 ERROR util.SparkUncaughtExceptionHandler: Uncaught exception in thread Thread[block-push-thread-1,5,main]
java.lang.Error: java.io.IOException: Error in opening FileSegmentManagedBuffer
{file=********/application_1619720975011_11057757/blockmgr-560cb4cf-9918-4ea7-a007-a16c5e3a35fe/0a/shuffle_1_690_0.data, offset=10640, length=190}
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1155)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.io.IOException: Error in opening FileSegmentManagedBuffer\{file=*******/application_1619720975011_11057757/blockmgr-560cb4cf-9918-4ea7-a007-a16c5e3a35fe/0a/shuffle_1_690_0.data, offset=10640, length=190}
at org.apache.spark.network.buffer.FileSegmentManagedBuffer.nioByteBuffer(FileSegmentManagedBuffer.java:89)
at org.apache.spark.shuffle.ShuffleWriter.sliceReqBufferIntoBlockBuffers(ShuffleWriter.scala:294)
at org.apache.spark.shuffle.ShuffleWriter.org$apache$spark$shuffle$ShuffleWriter$$sendRequest(ShuffleWriter.scala:270)
at org.apache.spark.shuffle.ShuffleWriter.org$apache$spark$shuffle$ShuffleWriter$$pushUpToMax(ShuffleWriter.scala:191)
at org.apache.spark.shuffle.ShuffleWriter$$anon$2$$anon$4.run(ShuffleWriter.scala:244)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
... 2 more
Caused by: java.io.FileNotFoundException: ******/application_1619720975011_11057757/blockmgr-560cb4cf-9918-4ea7-a007-a16c5e3a35fe/0a/shuffle_1_690_0.data (No such file or directory)
at java.io.RandomAccessFile.open0(Native Method)
at java.io.RandomAccessFile.open(RandomAccessFile.java:316)
at java.io.RandomAccessFile.<init>(RandomAccessFile.java:243)
at org.apache.spark.network.buffer.FileSegmentManagedBuffer.nioByteBuffer(FileSegmentManagedBuffer.java:62)
```
### Does this PR introduce _any_ user-facing change?
No
### How was this patch tested?
Added a unit to verify no more data is pushed when `FileNotFoundException` is encountered. Have also verified in our environment.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] mridulm commented on pull request #33477: [SPARK-36255][SHUFFLE][CORE] Stop pushing and retrying on FileNotFound exceptions
Posted by GitBox <gi...@apache.org>.
mridulm commented on pull request #33477:
URL: https://github.com/apache/spark/pull/33477#issuecomment-886154604
Late LGTM. Thanks for fixing this @otterc !
Thanks for the reviews @Ngone51, @srowen, @gengliangwang
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] Ngone51 commented on pull request #33477: [SPARK-36255][SHUFFLE][CORE] Stop pushing and retrying on FileNotFound exceptions
Posted by GitBox <gi...@apache.org>.
Ngone51 commented on pull request #33477:
URL: https://github.com/apache/spark/pull/33477#issuecomment-886051730
Thanks, merged to master/3.2.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] Ngone51 closed pull request #33477: [SPARK-36255][SHUFFLE][CORE] Stop pushing and retrying on FileNotFound exceptions
Posted by GitBox <gi...@apache.org>.
Ngone51 closed pull request #33477:
URL: https://github.com/apache/spark/pull/33477
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] otterc commented on pull request #33477: [SPARK-36255][SHUFFLE][CORE] Stop pushing and retrying on FileNotFound exceptions
Posted by GitBox <gi...@apache.org>.
otterc commented on pull request #33477:
URL: https://github.com/apache/spark/pull/33477#issuecomment-886099620
Thanks @Ngone51 @gengliangwang @srowen for the reviews
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33477: [SPARK-36255][SHUFFLE][CORE] Stop pushing and retrying on FileNotFound exceptions
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33477:
URL: https://github.com/apache/spark/pull/33477#issuecomment-885709819
**[Test build #141563 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/141563/testReport)** for PR 33477 at commit [`6bb9709`](https://github.com/apache/spark/commit/6bb97096c3973619eb18b95951738085af712ccc).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33477: [SPARK-36255][SHUFFLE][CORE] Stop pushing and retrying on FileNotFound exceptions
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33477:
URL: https://github.com/apache/spark/pull/33477#issuecomment-885775985
Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/46081/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #33477: [SPARK-36255][SHUFFLE][CORE] Stop pushing and retrying on FileNotFound exceptions
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #33477:
URL: https://github.com/apache/spark/pull/33477#issuecomment-885815950
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/141563/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] Ngone51 commented on pull request #33477: [SPARK-36255][SHUFFLE][CORE] Stop pushing and retrying on FileNotFound exceptions
Posted by GitBox <gi...@apache.org>.
Ngone51 commented on pull request #33477:
URL: https://github.com/apache/spark/pull/33477#issuecomment-884712661
> Once a job in a Spark application completes, the shuffle files are cleaned up by the executors.
Actually, it's cleaned up by the `ContextCleaner` when the shuffle dependency can be GCed. Job completion doesn't ensure that. Could you update the description to make it more accurate?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33477: [SPARK-36255][SHUFFLE][CORE] Stop pushing and retrying on FileNotFound exceptions
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33477:
URL: https://github.com/apache/spark/pull/33477#issuecomment-885814680
**[Test build #141563 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/141563/testReport)** for PR 33477 at commit [`6bb9709`](https://github.com/apache/spark/commit/6bb97096c3973619eb18b95951738085af712ccc).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] Ngone51 commented on a change in pull request #33477: [SPARK-36255][SHUFFLE][CORE] Stop pushing and retrying on FileNotFound exceptions
Posted by GitBox <gi...@apache.org>.
Ngone51 commented on a change in pull request #33477:
URL: https://github.com/apache/spark/pull/33477#discussion_r674569528
##########
File path: core/src/main/scala/org/apache/spark/shuffle/ShuffleBlockPusher.scala
##########
@@ -71,6 +71,12 @@ private[spark] class ShuffleBlockPusher(conf: SparkConf) extends Logging {
// blocks to just that host and continue push blocks to other hosts. So, here push of
// all blocks will only stop when it is "Too Late". Also see updateStateAndCheckIfPushMore.
override def shouldRetryError(t: Throwable): Boolean = {
+ // If it is a FileNotFound exception originating from the client while pushing the shuffle
Review comment:
nit: `FileNotFound exception` -> `FileNotFoundException`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] otterc commented on a change in pull request #33477: [SPARK-36255][SHUFFLE][CORE] Stop pushing and retrying on FileNotFound exceptions
Posted by GitBox <gi...@apache.org>.
otterc commented on a change in pull request #33477:
URL: https://github.com/apache/spark/pull/33477#discussion_r674976476
##########
File path: core/src/main/scala/org/apache/spark/shuffle/ShuffleBlockPusher.scala
##########
@@ -100,7 +106,15 @@ private[spark] class ShuffleBlockPusher(conf: SparkConf) extends Logging {
pushRequests ++= Utils.randomize(requests)
submitTask(() => {
- pushUpToMax()
+ try {
+ pushUpToMax()
+ } catch {
+ case e: FileNotFoundException =>
Review comment:
Sure, I will catch it there as well because there is a chance that `pushUpToMax()` there can throw FNF. We mostly saw the `result` from push with FNF as failure so `updateStateAndCheckIfPushMore(...)` would return false. But it might just happen, that the files got deleted after that.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #33477: [SPARK-36255][SHUFFLE][CORE] Stop pushing and retrying on FileNotFound exceptions
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #33477:
URL: https://github.com/apache/spark/pull/33477#issuecomment-885742194
Can one of the admins verify this patch?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #33477: [SPARK-36255][SHUFFLE][CORE] Stop pushing and retrying on FileNotFound exceptions
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #33477:
URL: https://github.com/apache/spark/pull/33477#issuecomment-884697196
Can one of the admins verify this patch?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #33477: [SPARK-36255][SHUFFLE][CORE] Stop pushing and retrying on FileNotFound exceptions
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #33477:
URL: https://github.com/apache/spark/pull/33477#issuecomment-885815950
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/141563/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] otterc commented on a change in pull request #33477: [SPARK-36255][SHUFFLE][CORE] Stop pushing and retrying on FileNotFound exceptions
Posted by GitBox <gi...@apache.org>.
otterc commented on a change in pull request #33477:
URL: https://github.com/apache/spark/pull/33477#discussion_r674976476
##########
File path: core/src/main/scala/org/apache/spark/shuffle/ShuffleBlockPusher.scala
##########
@@ -100,7 +106,15 @@ private[spark] class ShuffleBlockPusher(conf: SparkConf) extends Logging {
pushRequests ++= Utils.randomize(requests)
submitTask(() => {
- pushUpToMax()
+ try {
+ pushUpToMax()
+ } catch {
+ case e: FileNotFoundException =>
Review comment:
Good point! Sure, I will catch it there as well because there is a chance that `pushUpToMax()` there can throw FNF. We mostly saw the `result` from push with FNF as failure so `updateStateAndCheckIfPushMore(...)` would return false. But it might just happen, that the files got deleted after that.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] Ngone51 commented on pull request #33477: [SPARK-36255][SHUFFLE][CORE] Stop pushing and retrying on FileNotFound exceptions
Posted by GitBox <gi...@apache.org>.
Ngone51 commented on pull request #33477:
URL: https://github.com/apache/spark/pull/33477#issuecomment-885709025
ok to test
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] Ngone51 commented on a change in pull request #33477: [SPARK-36255][SHUFFLE][CORE] Stop pushing and retrying on FileNotFound exceptions
Posted by GitBox <gi...@apache.org>.
Ngone51 commented on a change in pull request #33477:
URL: https://github.com/apache/spark/pull/33477#discussion_r674957998
##########
File path: core/src/main/scala/org/apache/spark/shuffle/ShuffleBlockPusher.scala
##########
@@ -100,7 +106,15 @@ private[spark] class ShuffleBlockPusher(conf: SparkConf) extends Logging {
pushRequests ++= Utils.randomize(requests)
submitTask(() => {
- pushUpToMax()
+ try {
+ pushUpToMax()
+ } catch {
+ case e: FileNotFoundException =>
Review comment:
```
def handleResult(result: PushResult): Unit = {
submitTask(() => {
if (updateStateAndCheckIfPushMore(
sizeMap(result.blockId), address, remainingBlocks, result)) {
pushUpToMax()
}
})
```
We should also catch the exception there?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #33477: [SPARK-36255][SHUFFLE][CORE] Stop pushing and retrying on FileNotFound exceptions
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33477:
URL: https://github.com/apache/spark/pull/33477#issuecomment-885753052
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/46081/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #33477: [SPARK-36255][SHUFFLE][CORE] Stop pushing and retrying on FileNotFound exceptions
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #33477:
URL: https://github.com/apache/spark/pull/33477#issuecomment-884697196
Can one of the admins verify this patch?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #33477: [SPARK-36255][SHUFFLE][CORE] Stop pushing and retrying on FileNotFound exceptions
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #33477:
URL: https://github.com/apache/spark/pull/33477#issuecomment-885777285
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/46081/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] otterc commented on pull request #33477: [SPARK-36255][SHUFFLE][CORE] Stop pushing and retrying on FileNotFound exceptions
Posted by GitBox <gi...@apache.org>.
otterc commented on pull request #33477:
URL: https://github.com/apache/spark/pull/33477#issuecomment-884683878
@Ngone51 @mridulm Please help review this bug fix.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #33477: [SPARK-36255][SHUFFLE][CORE] Stop pushing and retrying on FileNotFound exceptions
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #33477:
URL: https://github.com/apache/spark/pull/33477#issuecomment-885742194
Can one of the admins verify this patch?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] Ngone51 commented on a change in pull request #33477: [SPARK-36255][SHUFFLE][CORE] Stop pushing and retrying on FileNotFound exceptions
Posted by GitBox <gi...@apache.org>.
Ngone51 commented on a change in pull request #33477:
URL: https://github.com/apache/spark/pull/33477#discussion_r674567533
##########
File path: core/src/main/scala/org/apache/spark/shuffle/ShuffleBlockPusher.scala
##########
@@ -100,7 +106,15 @@ private[spark] class ShuffleBlockPusher(conf: SparkConf) extends Logging {
pushRequests ++= Utils.randomize(requests)
submitTask(() => {
- pushUpToMax()
+ try {
+ pushUpToMax()
+ } catch {
+ case e: FileNotFoundException =>
+ logWarning("The shuffle files got deleted when this task was reading from them " +
Review comment:
`this task` -> `this shuffle-block-push-thread`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] gengliangwang commented on a change in pull request #33477: [SPARK-36255][SHUFFLE][CORE] Stop pushing and retrying on FileNotFound exceptions
Posted by GitBox <gi...@apache.org>.
gengliangwang commented on a change in pull request #33477:
URL: https://github.com/apache/spark/pull/33477#discussion_r674587837
##########
File path: core/src/test/scala/org/apache/spark/shuffle/ShuffleBlockPusherSuite.scala
##########
@@ -324,8 +324,32 @@ class ShuffleBlockPusherSuite extends SparkFunSuite with BeforeAndAfterEach {
assert(pusher.unreachableBlockMgrs.size == 2)
}
+ test("FileNotFound exceptions stops the push") {
Review comment:
Nit: let add jira ID SPARK-36255 here.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #33477: [SPARK-36255][SHUFFLE][CORE] Stop pushing and retrying on FileNotFound exceptions
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #33477:
URL: https://github.com/apache/spark/pull/33477#issuecomment-885709819
**[Test build #141563 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/141563/testReport)** for PR 33477 at commit [`6bb9709`](https://github.com/apache/spark/commit/6bb97096c3973619eb18b95951738085af712ccc).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org