You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "ASF GitHub Bot (Jira)" <ji...@apache.org> on 2024/01/11 12:17:00 UTC
[jira] [Updated] (SPARK-46681) Refactor `ExecutorFailureTracker#maxNumExecutorFailures` to avoid unnecessary computations when `MAX_EXECUTOR_FAILURES` is configured
[ https://issues.apache.org/jira/browse/SPARK-46681?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
ASF GitHub Bot updated SPARK-46681:
-----------------------------------
Labels: pull-request-available (was: )
> Refactor `ExecutorFailureTracker#maxNumExecutorFailures` to avoid unnecessary computations when `MAX_EXECUTOR_FAILURES` is configured
> -------------------------------------------------------------------------------------------------------------------------------------
>
> Key: SPARK-46681
> URL: https://issues.apache.org/jira/browse/SPARK-46681
> Project: Spark
> Issue Type: Improvement
> Components: Spark Core
> Affects Versions: 4.0.0
> Reporter: Yang Jie
> Priority: Minor
> Labels: pull-request-available
>
> {code:java}
> def maxNumExecutorFailures(sparkConf: SparkConf): Int = {
> val effectiveNumExecutors =
> if (Utils.isStreamingDynamicAllocationEnabled(sparkConf)) {
> sparkConf.get(STREAMING_DYN_ALLOCATION_MAX_EXECUTORS)
> } else if (Utils.isDynamicAllocationEnabled(sparkConf)) {
> sparkConf.get(DYN_ALLOCATION_MAX_EXECUTORS)
> } else {
> sparkConf.get(EXECUTOR_INSTANCES).getOrElse(0)
> }
> // By default, effectiveNumExecutors is Int.MaxValue if dynamic allocation is enabled. We need
> // avoid the integer overflow here.
> val defaultMaxNumExecutorFailures = math.max(3,
> if (effectiveNumExecutors > Int.MaxValue / 2) Int.MaxValue else 2 * effectiveNumExecutors)
> sparkConf.get(MAX_EXECUTOR_FAILURES).getOrElse(defaultMaxNumExecutorFailures)
> } {code}
> The result of defaultMaxNumExecutorFailures is calculated first, even if {{MAX_EXECUTOR_FAILURES}} is configured now
>
>
>
>
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org