You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by GitBox <gi...@apache.org> on 2022/01/06 05:19:07 UTC
[GitHub] [spark] viirya commented on a change in pull request #35109: [SPARK-37819][K8S] Add OUTLIER executor roll policy and use it by default
viirya commented on a change in pull request #35109:
URL: https://github.com/apache/spark/pull/35109#discussion_r779305398
##########
File path: resource-managers/kubernetes/core/src/main/scala/org/apache/spark/deploy/k8s/Config.scala
##########
@@ -147,26 +147,30 @@ private[spark] object Config extends Logging {
.createWithDefault(0)
object ExecutorRollPolicy extends Enumeration {
- val ID, ADD_TIME, TOTAL_GC_TIME, TOTAL_DURATION, AVERAGE_DURATION, FAILED_TASKS = Value
+ val ID, ADD_TIME, TOTAL_GC_TIME, TOTAL_DURATION, AVERAGE_DURATION, FAILED_TASKS, OUTLIER = Value
}
val EXECUTOR_ROLL_POLICY =
ConfigBuilder("spark.kubernetes.executor.rollPolicy")
- .doc("Executor roll policy: Valid values are ID, ADD_TIME, TOTAL_GC_TIME (default), " +
- "TOTAL_DURATION, and FAILED_TASKS. " +
+ .doc("Executor roll policy: Valid values are ID, ADD_TIME, TOTAL_GC_TIME, " +
+ "TOTAL_DURATION, FAILED_TASKS, and OUTLIER (default). " +
"When executor roll happens, Spark uses this policy to choose " +
"an executor and decommission it. The built-in policies are based on executor summary." +
"ID policy chooses an executor with the smallest executor ID. " +
"ADD_TIME policy chooses an executor with the smallest add-time. " +
"TOTAL_GC_TIME policy chooses an executor with the biggest total task GC time. " +
"TOTAL_DURATION policy chooses an executor with the biggest total task time. " +
"AVERAGE_DURATION policy chooses an executor with the biggest average task time. " +
- "FAILED_TASKS policy chooses an executor with the most number of failed tasks.")
+ "FAILED_TASKS policy chooses an executor with the most number of failed tasks. " +
+ "OUTLIER policy chooses an executor with outstanding statistics which is bigger than" +
+ "at least two standard deviation from the mean in average task time, " +
+ "total task time, total task GC time, and the number of failed tasks if exists. " +
+ "If there is no outlier, it works like TOTAL_DURATION policy.")
Review comment:
previously the default is TOTAL_GC_TIME, why don't we fallback to it?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org