You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Aaruna Godthi (Jira)" <ji...@apache.org> on 2021/01/14 04:21:00 UTC
[jira] [Updated] (SPARK-34109) Killing executors excluded on
failure, results in additional executors being marked as excluded due to
fetch failures
[ https://issues.apache.org/jira/browse/SPARK-34109?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Aaruna Godthi updated SPARK-34109:
----------------------------------
Description:
Configuration:
{code:java}
spark.excludeOnFailure.enabled: true # aka deprecated spark.blacklist.enabled
spark.excludeOnFailure.application.fetchFailure.enabled: true # aka deprecated spark.blacklist.application.fetchFailure.enabled
spark.excludeOnFailure.killExcludedExecutors: true # aka deprecated spark.blacklist.killBlacklistedExecutors
{code}
In this case, we have noticed when a few executors are excluded due to task failures (maybe due to host issues), then those executors are killed after being excluded.
However, when other executors try to fetch shuffle blocks from these killed executors, then these other executors also end up getting excluded due to `spark.excludeOnFailure.application.fetchFailure.enabled`.
Instead, the fetch failures in case of fetch from these excluded executors should not be considered when excluding executors based on `spark.excludeOnFailure.application.fetchFailure.enabled`
was:
Configuration:
```
spark.excludeOnFailure.enabled: true # aka deprecated spark.blacklist.enabled
spark.excludeOnFailure.application.fetchFailure.enabled: true # aka deprecated spark.blacklist.application.fetchFailure.enabled
spark.excludeOnFailure.killExcludedExecutors: true # aka deprecated spark.blacklist.killBlacklistedExecutors
```
In this case, we have noticed when a few executors are excluded due to task failures (maybe due to host issues), then those executors are killed after being excluded.
However, when other executors try to fetch shuffle blocks from these killed executors, then these other executors also end up getting excluded due to `spark.excludeOnFailure.application.fetchFailure.enabled`.
Instead, the fetch failures in case of fetch from these excluded executors should not be considered when excluding executors based on `spark.excludeOnFailure.application.fetchFailure.enabled`
> Killing executors excluded on failure, results in additional executors being marked as excluded due to fetch failures
> ---------------------------------------------------------------------------------------------------------------------
>
> Key: SPARK-34109
> URL: https://issues.apache.org/jira/browse/SPARK-34109
> Project: Spark
> Issue Type: Bug
> Components: Kubernetes, Shuffle, Spark Core
> Affects Versions: 3.0.0, 3.0.1
> Reporter: Aaruna Godthi
> Priority: Major
>
> Configuration:
>
> {code:java}
> spark.excludeOnFailure.enabled: true # aka deprecated spark.blacklist.enabled
> spark.excludeOnFailure.application.fetchFailure.enabled: true # aka deprecated spark.blacklist.application.fetchFailure.enabled
> spark.excludeOnFailure.killExcludedExecutors: true # aka deprecated spark.blacklist.killBlacklistedExecutors
> {code}
>
>
>
> In this case, we have noticed when a few executors are excluded due to task failures (maybe due to host issues), then those executors are killed after being excluded.
> However, when other executors try to fetch shuffle blocks from these killed executors, then these other executors also end up getting excluded due to `spark.excludeOnFailure.application.fetchFailure.enabled`.
> Instead, the fetch failures in case of fetch from these excluded executors should not be considered when excluding executors based on `spark.excludeOnFailure.application.fetchFailure.enabled`
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org