You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by GitBox <gi...@apache.org> on 2021/01/22 20:32:36 UTC
[GitHub] [spark] holdenk opened a new pull request #31298: [SPARK-34193][CORE] TorrentBroadcast block manager decommissioning race fix
holdenk opened a new pull request #31298:
URL: https://github.com/apache/spark/pull/31298
### What changes were proposed in this pull request?
Allow broadcast blocks to be put during decommissioning since migrations don't apply to them and they may be stored as part of job exec.
### Why are the changes needed?
Potential race condition.
### Does this PR introduce _any_ user-facing change?
Removal of race condition.
### How was this patch tested?
New unit test.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on pull request #31298: [SPARK-34193][CORE] TorrentBroadcast block manager decommissioning race fix
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on pull request #31298:
URL: https://github.com/apache/spark/pull/31298#issuecomment-766285048
cc @Ngone51 who also works on decommissioning and torrent broadcast stuff.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] holdenk commented on a change in pull request #31298: [SPARK-34193][CORE] TorrentBroadcast block manager decommissioning race fix
Posted by GitBox <gi...@apache.org>.
holdenk commented on a change in pull request #31298:
URL: https://github.com/apache/spark/pull/31298#discussion_r564621830
##########
File path: core/src/main/scala/org/apache/spark/storage/BlockManager.scala
##########
@@ -258,6 +258,17 @@ private[spark] class BlockManager(
@inline final private def isDecommissioning() = {
decommissioner.isDefined
}
+
+ @inline final private def checkShouldStore(blockId: BlockId) = {
+ if (isDecommissioning()) {
Review comment:
Oh yeah I did the nested if for branch prediction but yeah I'll collapse it down.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #31298: [SPARK-34193][CORE] TorrentBroadcast block manager decommissioning race fix
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #31298:
URL: https://github.com/apache/spark/pull/31298#issuecomment-766279632
**[Test build #134406 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/134406/testReport)** for PR 31298 at commit [`305b3fc`](https://github.com/apache/spark/commit/305b3fc5f70ce8b2d2ab1f9dfe13f0402783b03b).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #31298: [SPARK-34193][CORE] TorrentBroadcast block manager decommissioning race fix
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #31298:
URL: https://github.com/apache/spark/pull/31298#issuecomment-765687452
**[Test build #134384 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/134384/testReport)** for PR 31298 at commit [`305b3fc`](https://github.com/apache/spark/commit/305b3fc5f70ce8b2d2ab1f9dfe13f0402783b03b).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #31298: [SPARK-34193][CORE] TorrentBroadcast block manager decommissioning race fix
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #31298:
URL: https://github.com/apache/spark/pull/31298#issuecomment-766283831
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/134406/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #31298: [SPARK-34193][CORE] TorrentBroadcast block manager decommissioning race fix
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #31298:
URL: https://github.com/apache/spark/pull/31298#issuecomment-765796559
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/38973/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #31298: [SPARK-34193][CORE] TorrentBroadcast block manager decommissioning race fix
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #31298:
URL: https://github.com/apache/spark/pull/31298#issuecomment-768562463
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/134560/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #31298: [SPARK-34193][CORE] TorrentBroadcast block manager decommissioning race fix
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #31298:
URL: https://github.com/apache/spark/pull/31298#issuecomment-766273202
Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/38992/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #31298: [SPARK-34193][CORE] TorrentBroadcast block manager decommissioning race fix
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #31298:
URL: https://github.com/apache/spark/pull/31298#issuecomment-765718275
Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/38970/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #31298: [SPARK-34193][CORE] TorrentBroadcast block manager decommissioning race fix
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #31298:
URL: https://github.com/apache/spark/pull/31298#issuecomment-768562463
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/134560/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #31298: [SPARK-34193][CORE] TorrentBroadcast block manager decommissioning race fix
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #31298:
URL: https://github.com/apache/spark/pull/31298#issuecomment-765737030
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #31298: [SPARK-34193][CORE] TorrentBroadcast block manager decommissioning race fix
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #31298:
URL: https://github.com/apache/spark/pull/31298#issuecomment-765808832
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/134388/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] Ngone51 edited a comment on pull request #31298: [SPARK-34193][CORE] TorrentBroadcast block manager decommissioning race fix
Posted by GitBox <gi...@apache.org>.
Ngone51 edited a comment on pull request #31298:
URL: https://github.com/apache/spark/pull/31298#issuecomment-766596881
``` scala
[info] org.apache.spark.SparkException: Job aborted due to stage failure:
Task 1 in stage 3.0 failed 4 times, most recent failure: Lost task 1.3 in stage 3.0 (TID 8, 192.168.1.57, executor 1):
java.io.IOException: org.apache.spark.storage.BlockSavedOnDecommissionedBlockManagerException:
Block broadcast_2_piece0 cannot be saved on decommissioned executor
[info] org.apache.spark.SparkException: Job aborted due to stage failure:
Task 1 in stage 3.0 failed 4 times, most recent failure: Lost task 1.3 in stage 3.0 (TID 8, 192.168.1.57, executor 1): java.io.IOException: org.apache.spark.storage.BlockSavedOnDecommissionedBlockManagerException: Block broadcast_2_piece0 cannot be saved on decommissioned executor[info] at
...
```
Usually, the task fails with the max failure times would due to the same reason. But in the case of `BlockSavedOnDecommissionedBlockManagerException`, I'd not expect the 4 times failures are all caused by `BlockSavedOnDecommissionedBlockManagerException` since we don't schedule tasks on the decommissioning executor. (Unless all the task failures always happen before the decommission notice arrives at the driver. But this seems very impossible.)
So to double check that we still ensure that we don't schedule tasks on the decommissioning executor (or there's really a race condition or it's just a coincidence), I'd like to know the previous task failures for this task (TID = 8). e.g., what are failures reasons for previous attempts? And which executors are scheduled?
BTW, the change itself lgtm.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] mridulm commented on pull request #31298: [SPARK-34193][CORE] TorrentBroadcast block manager decommissioning race fix
Posted by GitBox <gi...@apache.org>.
mridulm commented on pull request #31298:
URL: https://github.com/apache/spark/pull/31298#issuecomment-771240887
Late LGTM, thanks for the fix @holdenk !
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] holdenk commented on pull request #31298: [SPARK-34193][CORE] TorrentBroadcast block manager decommissioning race fix
Posted by GitBox <gi...@apache.org>.
holdenk commented on pull request #31298:
URL: https://github.com/apache/spark/pull/31298#issuecomment-768549645
@HyukjinKwon do you want me to wait for @jiangxb1987's review? Personally I'm comfortable with @attilapiros's sign-off.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #31298: [SPARK-34193][CORE] TorrentBroadcast block manager decommissioning race fix
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #31298:
URL: https://github.com/apache/spark/pull/31298#issuecomment-765687452
**[Test build #134384 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/134384/testReport)** for PR 31298 at commit [`305b3fc`](https://github.com/apache/spark/commit/305b3fc5f70ce8b2d2ab1f9dfe13f0402783b03b).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on pull request #31298: [SPARK-34193][CORE] TorrentBroadcast block manager decommissioning race fix
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on pull request #31298:
URL: https://github.com/apache/spark/pull/31298#issuecomment-768581888
I think it's fine to merge and leave it for a post review given that the RC will be cut out soon. I reviewed too and signed off.
Merged to master and branch-3.1.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] holdenk commented on pull request #31298: [SPARK-34193][CORE] TorrentBroadcast block manager decommissioning race fix
Posted by GitBox <gi...@apache.org>.
holdenk commented on pull request #31298:
URL: https://github.com/apache/spark/pull/31298#issuecomment-765732226
Hive failures are probably unrelated, jenkins retest this please.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #31298: [SPARK-34193][CORE] TorrentBroadcast block manager decommissioning race fix
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #31298:
URL: https://github.com/apache/spark/pull/31298#issuecomment-766256745
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/38992/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] Ngone51 edited a comment on pull request #31298: [SPARK-34193][CORE] TorrentBroadcast block manager decommissioning race fix
Posted by GitBox <gi...@apache.org>.
Ngone51 edited a comment on pull request #31298:
URL: https://github.com/apache/spark/pull/31298#issuecomment-766596881
``` scala
[info] org.apache.spark.SparkException: Job aborted due to stage failure:
Task 1 in stage 3.0 failed 4 times, most recent failure: Lost task 1.3 in stage 3.0 (TID 8, 192.168.1.57, executor 1):
java.io.IOException: org.apache.spark.storage.BlockSavedOnDecommissionedBlockManagerException:
Block broadcast_2_piece0 cannot be saved on decommissioned executor
[info] org.apache.spark.SparkException: Job aborted due to stage failure:
Task 1 in stage 3.0 failed 4 times, most recent failure: Lost task 1.3 in stage 3.0 (TID 8, 192.168.1.57, executor 1): java.io.IOException: org.apache.spark.storage.BlockSavedOnDecommissionedBlockManagerException: Block broadcast_2_piece0 cannot be saved on decommissioned executor[info] at
...
```
Usually, the task fails with the max failure times would due to the same reason. But in the case of `BlockSavedOnDecommissionedBlockManagerException`, I'd not expect the 4 times failures are all caused by `BlockSavedOnDecommissionedBlockManagerException` since we don't schedule tasks on the decommissioning executor. (Unless all the task failures always happen before the decommission notice arrives at the driver. But this seems very impossible.)
So to double check that we still ensure that we don't schedule tasks on the decommissioning executor (or there's really a race condition or it's just a coincidence), I'd like to know the previous task failures for this task (TID = 8). e.g., what are failures reasons for previous attempts? And which executors are scheduled?
BTW, the change itself lgtm.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #31298: [SPARK-34193][CORE] TorrentBroadcast block manager decommissioning race fix
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #31298:
URL: https://github.com/apache/spark/pull/31298#issuecomment-766276489
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/38992/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #31298: [SPARK-34193][CORE] TorrentBroadcast block manager decommissioning race fix
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #31298:
URL: https://github.com/apache/spark/pull/31298#issuecomment-765737030
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] attilapiros commented on a change in pull request #31298: [SPARK-34193][CORE] TorrentBroadcast block manager decommissioning race fix
Posted by GitBox <gi...@apache.org>.
attilapiros commented on a change in pull request #31298:
URL: https://github.com/apache/spark/pull/31298#discussion_r564250920
##########
File path: core/src/main/scala/org/apache/spark/storage/BlockManager.scala
##########
@@ -258,6 +258,17 @@ private[spark] class BlockManager(
@inline final private def isDecommissioning() = {
decommissioner.isDefined
}
+
+ @inline final private def checkShouldStore(blockId: BlockId) = {
+ if (isDecommissioning()) {
Review comment:
Nit: I would avoid the nested if.
```suggestion
if (isDecommissioning() && !blockId.isBroadcast) {
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] mridulm commented on pull request #31298: [SPARK-34193][CORE] TorrentBroadcast block manager decommissioning race fix
Posted by GitBox <gi...@apache.org>.
mridulm commented on pull request #31298:
URL: https://github.com/apache/spark/pull/31298#issuecomment-771240887
Late LGTM, thanks for the fix @holdenk !
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #31298: [SPARK-34193][CORE] TorrentBroadcast block manager decommissioning race fix
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #31298:
URL: https://github.com/apache/spark/pull/31298#issuecomment-765771164
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/38973/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] holdenk commented on pull request #31298: [SPARK-34193][CORE] TorrentBroadcast block manager decommissioning race fix
Posted by GitBox <gi...@apache.org>.
holdenk commented on pull request #31298:
URL: https://github.com/apache/spark/pull/31298#issuecomment-766242622
Jenkins retest this please
(kafka failure unrelated).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #31298: [SPARK-34193][CORE] TorrentBroadcast block manager decommissioning race fix
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #31298:
URL: https://github.com/apache/spark/pull/31298#issuecomment-768484943
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/39146/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] holdenk commented on pull request #31298: [SPARK-34193][CORE] TorrentBroadcast block manager decommissioning race fix
Posted by GitBox <gi...@apache.org>.
holdenk commented on pull request #31298:
URL: https://github.com/apache/spark/pull/31298#issuecomment-767082051
> So to double check that we still ensure that we don't schedule tasks on the decommissioning executor (or there's really a race condition or it's just a coincidence), I'd like to know the previous task failures for this task (TID = 8). e.g., what are failures reasons for previous attempts? And which executors are scheduled?
Yes so to be clear I got this while executing in a tight loop of test code wherein I intentionally scheduled the job. I'm not sure this race condition can occur in the "real world" to the point the task actually fails, but we might abort some started computation which is undeseriable (but not a major issue).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #31298: [SPARK-34193][CORE] TorrentBroadcast block manager decommissioning race fix
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #31298:
URL: https://github.com/apache/spark/pull/31298#issuecomment-768523389
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/39146/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #31298: [SPARK-34193][CORE] TorrentBroadcast block manager decommissioning race fix
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #31298:
URL: https://github.com/apache/spark/pull/31298#issuecomment-765808536
**[Test build #134388 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/134388/testReport)** for PR 31298 at commit [`305b3fc`](https://github.com/apache/spark/commit/305b3fc5f70ce8b2d2ab1f9dfe13f0402783b03b).
* This patch **fails Spark unit tests**.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] Ngone51 commented on pull request #31298: [SPARK-34193][CORE] TorrentBroadcast block manager decommissioning race fix
Posted by GitBox <gi...@apache.org>.
Ngone51 commented on pull request #31298:
URL: https://github.com/apache/spark/pull/31298#issuecomment-766596881
``` scala
[info] org.apache.spark.SparkException: Job aborted due to stage failure:
Task 1 in stage 3.0 failed 4 times, most recent failure: Lost task 1.3 in stage 3.0 (TID 8, 192.168.1.57, executor 1):
java.io.IOException: org.apache.spark.storage.BlockSavedOnDecommissionedBlockManagerException:
Block broadcast_2_piece0 cannot be saved on decommissioned executor
[info] org.apache.spark.SparkException: Job aborted due to stage failure:
Task 1 in stage 3.0 failed 4 times, most recent failure: Lost task 1.3 in stage 3.0 (TID 8, 192.168.1.57, executor 1): java.io.IOException: org.apache.spark.storage.BlockSavedOnDecommissionedBlockManagerException: Block broadcast_2_piece0 cannot be saved on decommissioned executor[info] at
...
```
Usually, the task fails with the max failure times would due to the same reason. But in the case of `BlockSavedOnDecommissionedBlockManagerException`, I'd not expect the 4 times failures are all caused by `BlockSavedOnDecommissionedBlockManagerException` since we don't schedule tasks on the decommissioning executor. (Unless all the task failures always happen before the decommission notice arrives at the driver. But this seems very impossible.)
So to ensure that we still ensure that we don't schedule tasks on the decommissioning executor (or there's really a race condition or it's just a coincidence), I'd like to know the previous task failures for this task (TID = 8). e.g., what are failures reasons for previous attempts? And which executors are scheduled?
BTW, the change itself lgtm.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon closed pull request #31298: [SPARK-34193][CORE] TorrentBroadcast block manager decommissioning race fix
Posted by GitBox <gi...@apache.org>.
HyukjinKwon closed pull request #31298:
URL: https://github.com/apache/spark/pull/31298
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #31298: [SPARK-34193][CORE] TorrentBroadcast block manager decommissioning race fix
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #31298:
URL: https://github.com/apache/spark/pull/31298#issuecomment-766247355
**[Test build #134406 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/134406/testReport)** for PR 31298 at commit [`305b3fc`](https://github.com/apache/spark/commit/305b3fc5f70ce8b2d2ab1f9dfe13f0402783b03b).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #31298: [SPARK-34193][CORE] TorrentBroadcast block manager decommissioning race fix
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #31298:
URL: https://github.com/apache/spark/pull/31298#issuecomment-766283831
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/134406/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #31298: [SPARK-34193][CORE] TorrentBroadcast block manager decommissioning race fix
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #31298:
URL: https://github.com/apache/spark/pull/31298#issuecomment-768460247
**[Test build #134560 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/134560/testReport)** for PR 31298 at commit [`de11694`](https://github.com/apache/spark/commit/de116945c98eec0194b53c5d2ccfebdff84a74bb).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #31298: [SPARK-34193][CORE] TorrentBroadcast block manager decommissioning race fix
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #31298:
URL: https://github.com/apache/spark/pull/31298#issuecomment-765796545
Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/38973/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] holdenk commented on pull request #31298: [SPARK-34193][CORE] TorrentBroadcast block manager decommissioning race fix
Posted by GitBox <gi...@apache.org>.
holdenk commented on pull request #31298:
URL: https://github.com/apache/spark/pull/31298#issuecomment-767185174
cc @attilapiros
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #31298: [SPARK-34193][CORE] TorrentBroadcast block manager decommissioning race fix
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #31298:
URL: https://github.com/apache/spark/pull/31298#issuecomment-768499457
Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/39146/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] Ngone51 commented on pull request #31298: [SPARK-34193][CORE] TorrentBroadcast block manager decommissioning race fix
Posted by GitBox <gi...@apache.org>.
Ngone51 commented on pull request #31298:
URL: https://github.com/apache/spark/pull/31298#issuecomment-767362213
Thanks for the explanation @holdenk . +1 to address @attilapiros 's suggestion, otherwise LGTM.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #31298: [SPARK-34193][CORE] TorrentBroadcast block manager decommissioning race fix
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #31298:
URL: https://github.com/apache/spark/pull/31298#issuecomment-765796559
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/38973/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] holdenk commented on pull request #31298: [SPARK-34193][CORE] TorrentBroadcast block manager decommissioning race fix
Posted by GitBox <gi...@apache.org>.
holdenk commented on pull request #31298:
URL: https://github.com/apache/spark/pull/31298#issuecomment-767082051
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on pull request #31298: [SPARK-34193][CORE] TorrentBroadcast block manager decommissioning race fix
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on pull request #31298:
URL: https://github.com/apache/spark/pull/31298#issuecomment-768017199
cc @jiangxb1987 would you mind taking a look and merging this one when you find some time please?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #31298: [SPARK-34193][CORE] TorrentBroadcast block manager decommissioning race fix
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #31298:
URL: https://github.com/apache/spark/pull/31298#issuecomment-765740513
**[Test build #134388 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/134388/testReport)** for PR 31298 at commit [`305b3fc`](https://github.com/apache/spark/commit/305b3fc5f70ce8b2d2ab1f9dfe13f0402783b03b).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #31298: [SPARK-34193][CORE] TorrentBroadcast block manager decommissioning race fix
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #31298:
URL: https://github.com/apache/spark/pull/31298#issuecomment-766276489
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/38992/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #31298: [SPARK-34193][CORE] TorrentBroadcast block manager decommissioning race fix
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #31298:
URL: https://github.com/apache/spark/pull/31298#issuecomment-768460247
**[Test build #134560 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/134560/testReport)** for PR 31298 at commit [`de11694`](https://github.com/apache/spark/commit/de116945c98eec0194b53c5d2ccfebdff84a74bb).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #31298: [SPARK-34193][CORE] TorrentBroadcast block manager decommissioning race fix
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #31298:
URL: https://github.com/apache/spark/pull/31298#issuecomment-768561202
**[Test build #134560 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/134560/testReport)** for PR 31298 at commit [`de11694`](https://github.com/apache/spark/commit/de116945c98eec0194b53c5d2ccfebdff84a74bb).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #31298: [SPARK-34193][CORE] TorrentBroadcast block manager decommissioning race fix
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #31298:
URL: https://github.com/apache/spark/pull/31298#issuecomment-766247355
**[Test build #134406 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/134406/testReport)** for PR 31298 at commit [`305b3fc`](https://github.com/apache/spark/commit/305b3fc5f70ce8b2d2ab1f9dfe13f0402783b03b).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #31298: [SPARK-34193][CORE] TorrentBroadcast block manager decommissioning race fix
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #31298:
URL: https://github.com/apache/spark/pull/31298#issuecomment-768523389
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/39146/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #31298: [SPARK-34193][CORE] TorrentBroadcast block manager decommissioning race fix
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #31298:
URL: https://github.com/apache/spark/pull/31298#issuecomment-765740513
**[Test build #134388 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/134388/testReport)** for PR 31298 at commit [`305b3fc`](https://github.com/apache/spark/commit/305b3fc5f70ce8b2d2ab1f9dfe13f0402783b03b).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #31298: [SPARK-34193][CORE] TorrentBroadcast block manager decommissioning race fix
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #31298:
URL: https://github.com/apache/spark/pull/31298#issuecomment-765808832
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/134388/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #31298: [SPARK-34193][CORE] TorrentBroadcast block manager decommissioning race fix
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #31298:
URL: https://github.com/apache/spark/pull/31298#issuecomment-765707212
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/38970/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #31298: [SPARK-34193][CORE] TorrentBroadcast block manager decommissioning race fix
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #31298:
URL: https://github.com/apache/spark/pull/31298#issuecomment-765729523
**[Test build #134384 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/134384/testReport)** for PR 31298 at commit [`305b3fc`](https://github.com/apache/spark/commit/305b3fc5f70ce8b2d2ab1f9dfe13f0402783b03b).
* This patch **fails Spark unit tests**.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] Ngone51 commented on pull request #31298: [SPARK-34193][CORE] TorrentBroadcast block manager decommissioning race fix
Posted by GitBox <gi...@apache.org>.
Ngone51 commented on pull request #31298:
URL: https://github.com/apache/spark/pull/31298#issuecomment-766596881
``` scala
[info] org.apache.spark.SparkException: Job aborted due to stage failure:
Task 1 in stage 3.0 failed 4 times, most recent failure: Lost task 1.3 in stage 3.0 (TID 8, 192.168.1.57, executor 1):
java.io.IOException: org.apache.spark.storage.BlockSavedOnDecommissionedBlockManagerException:
Block broadcast_2_piece0 cannot be saved on decommissioned executor
[info] org.apache.spark.SparkException: Job aborted due to stage failure:
Task 1 in stage 3.0 failed 4 times, most recent failure: Lost task 1.3 in stage 3.0 (TID 8, 192.168.1.57, executor 1): java.io.IOException: org.apache.spark.storage.BlockSavedOnDecommissionedBlockManagerException: Block broadcast_2_piece0 cannot be saved on decommissioned executor[info] at
...
```
Usually, the task fails with the max failure times would due to the same reason. But in the case of `BlockSavedOnDecommissionedBlockManagerException`, I'd not expect the 4 times failures are all caused by `BlockSavedOnDecommissionedBlockManagerException` since we don't schedule tasks on the decommissioning executor. (Unless all the task failures always happen before the decommission notice arrives at the driver. But this seems very impossible.)
So to ensure that we still ensure that we don't schedule tasks on the decommissioning executor (or there's really a race condition or it's just a coincidence), I'd like to know the previous task failures for this task (TID = 8). e.g., what are failures reasons for previous attempts? And which executors are scheduled?
BTW, the change itself lgtm.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org