You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@spark.apache.org by do...@apache.org on 2026/10/02 19:59:44 UTC

(spark) branch branch-4.x updated: [SPARK-59924][K8S][DOC] Fix stale "Valid values" list for `spark.kubernetes.executor.rollPolicy`

This is an automated email from the ASF dual-hosted git repository.

dongjoon-hyun pushed a commit to branch branch-4.x
in repository https://gitbox.apache.org/repos/asf/spark.git


The following commit(s) were added to refs/heads/branch-4.x by this push:
     new 39f1c3aaade7 [SPARK-59924][K8S][DOC] Fix stale "Valid values" list for `spark.kubernetes.executor.rollPolicy`
39f1c3aaade7 is described below

commit 39f1c3aaade77ac9cfb7fcce4fba0ccb2aa1fff4
Author: Kousuke Saruta <sa...@apache.org>
AuthorDate: Fri Oct 2 12:59:24 2026 -0700

    [SPARK-59924][K8S][DOC] Fix stale "Valid values" list for `spark.kubernetes.executor.rollPolicy`
    
    ### What changes were proposed in this pull request?
    This PR fixes the "Valid values" list for `spark.kubernetes.executor.rollPolicy`, which was out of sync with the `ExecutorRollPolicy` enum in both `Config.scala` and `running-on-kubernetes.md`.
    
    - `Config.scala`: added the missing `TOTAL_SHUFFLE_WRITE` and `DISK_USED`.
    - `running-on-kubernetes.md`: added the missing `AVERAGE_DURATION`, `PEAK_JVM_ONHEAP_MEMORY`, `PEAK_JVM_OFFHEAP_MEMORY`, `TOTAL_SHUFFLE_WRITE`, `DISK_USED`, `ACTIVE_TASKS` and `OUTLIER_NO_FALLBACK`, including their per-policy descriptions.
    
    Both lists now follow the enum definition order.
    
    ### Why are the changes needed?
    The documented list of valid values listed only a subset of the supported policies, which is misleading.
    
    ### Does this PR introduce _any_ user-facing change?
    No. Docs only. No behavior change.
    
    ### How was this patch tested?
    Verified by inspection that the lists in both files match the `ExecutorRollPolicy` enum, and that the `OUTLIER` dimensions, their checking order, and the `DISK_USED` wording match `ExecutorRollPlugin` and `AppStatusListener`.
    
    ### Was this patch authored or co-authored using generative AI tooling?
    Generated-by: Kiro CLI / Claude
    
    Closes #59187 from sarutak/rollpolicy-valid-values-docfix.
    
    Authored-by: Kousuke Saruta <sa...@apache.org>
    Signed-off-by: Dongjoon Hyun <do...@apache.org>
    (cherry picked from commit 25a9aa7861642a834af8e045de6416626b12bb15)
    Signed-off-by: Dongjoon Hyun <do...@apache.org>
---
 docs/running-on-kubernetes.md                            | 16 ++++++++++++++--
 .../main/scala/org/apache/spark/deploy/k8s/Config.scala  | 15 ++++++++++-----
 2 files changed, 24 insertions(+), 7 deletions(-)

diff --git a/docs/running-on-kubernetes.md b/docs/running-on-kubernetes.md
index 322b299b38de..07bf8294a097 100644
--- a/docs/running-on-kubernetes.md
+++ b/docs/running-on-kubernetes.md
@@ -2100,7 +2100,9 @@ See the [configuration page](configuration.html) for information on Spark config
   <td><code>OUTLIER</code></td>
   <td>
     Executor roll policy: Valid values are ID, ADD_TIME, TOTAL_GC_TIME,
-    TOTAL_DURATION, FAILED_TASKS, and OUTLIER (default).
+    TOTAL_DURATION, AVERAGE_DURATION, FAILED_TASKS, PEAK_JVM_ONHEAP_MEMORY,
+    PEAK_JVM_OFFHEAP_MEMORY, TOTAL_SHUFFLE_WRITE, DISK_USED, ACTIVE_TASKS,
+    OUTLIER (default), and OUTLIER_NO_FALLBACK.
     When executor roll happens, Spark uses this policy to choose
     an executor and decommission it. The built-in policies are based on executor summary
     and newly started executors are protected by spark.kubernetes.executor.minTasksPerExecutorBeforeRolling.
@@ -2110,14 +2112,24 @@ See the [configuration page](configuration.html) for information on Spark config
     TOTAL_DURATION policy chooses an executor with the biggest total task time.
     AVERAGE_DURATION policy chooses an executor with the biggest average task time.
     FAILED_TASKS policy chooses an executor with the most number of failed tasks.
+    PEAK_JVM_ONHEAP_MEMORY policy chooses an executor with the biggest peak JVM on-heap memory.
+    PEAK_JVM_OFFHEAP_MEMORY policy chooses an executor with the biggest peak JVM off-heap memory.
+    TOTAL_SHUFFLE_WRITE policy chooses an executor with the biggest total shuffle write.
+    DISK_USED policy chooses an executor with the biggest disk size used by its
+    stored blocks (e.g., disk-persisted RDD blocks).
     ACTIVE_TASKS policy chooses an executor with the smallest number of active tasks.
     If there is a tie, it chooses an executor with the smallest add-time.
     It is recommended to use it with spark.kubernetes.executor.minTasksPerExecutorBeforeRolling
     because newly started executors usually have no active tasks.
     OUTLIER policy chooses an executor with outstanding statistics which is bigger than
     at least two standard deviation from the mean in average task time,
-    total task time, total task GC time, and the number of failed tasks if exists.
+    total task time, total task GC time, the number of failed tasks,
+    peak JVM on-heap memory, peak JVM off-heap memory, total shuffle write,
+    and disk used if exists.
+    The dimensions are checked in this order and the first outlier found is chosen.
     If there is no outlier, it works like TOTAL_DURATION policy.
+    OUTLIER_NO_FALLBACK policy picks an outlier using the OUTLIER policy above.
+    If there is no outlier then no executor will be rolled.
   </td>
   <td>3.3.0</td>
 </tr>
diff --git a/resource-managers/kubernetes/core/src/main/scala/org/apache/spark/deploy/k8s/Config.scala b/resource-managers/kubernetes/core/src/main/scala/org/apache/spark/deploy/k8s/Config.scala
index 705bc475d310..5162640de39a 100644
--- a/resource-managers/kubernetes/core/src/main/scala/org/apache/spark/deploy/k8s/Config.scala
+++ b/resource-managers/kubernetes/core/src/main/scala/org/apache/spark/deploy/k8s/Config.scala
@@ -247,9 +247,10 @@ private[spark] object Config extends Logging {
     ConfigBuilder("spark.kubernetes.executor.rollPolicy")
       .doc("Executor roll policy: Valid values are ID, ADD_TIME, TOTAL_GC_TIME, " +
         "TOTAL_DURATION, AVERAGE_DURATION, FAILED_TASKS, PEAK_JVM_ONHEAP_MEMORY, " +
-        "PEAK_JVM_OFFHEAP_MEMORY, ACTIVE_TASKS, OUTLIER (default), and OUTLIER_NO_FALLBACK. " +
+        "PEAK_JVM_OFFHEAP_MEMORY, TOTAL_SHUFFLE_WRITE, DISK_USED, ACTIVE_TASKS, " +
+        "OUTLIER (default), and OUTLIER_NO_FALLBACK. " +
         "When executor roll happens, Spark uses this policy to choose " +
-        "an executor and decommission it. The built-in policies are based on executor summary." +
+        "an executor and decommission it. The built-in policies are based on executor summary. " +
         "ID policy chooses an executor with the smallest executor ID. " +
         "ADD_TIME policy chooses an executor with the smallest add-time. " +
         "TOTAL_GC_TIME policy chooses an executor with the biggest total task GC time. " +
@@ -260,15 +261,19 @@ private[spark] object Config extends Logging {
         "memory. PEAK_JVM_OFFHEAP_MEMORY policy chooses an executor with the biggest peak JVM " +
         "off-heap memory. " +
         "TOTAL_SHUFFLE_WRITE policy chooses an executor with the biggest total shuffle write. " +
-        "DISK_USED policy chooses an executor with the biggest used disk size. " +
+        "DISK_USED policy chooses an executor with the biggest disk size used by its " +
+        "stored blocks (e.g., disk-persisted RDD blocks). " +
         "ACTIVE_TASKS policy chooses an executor with the smallest number of active tasks. " +
         "If there is a tie, it chooses an executor with the smallest add-time. " +
         "It is recommended to use it with " +
         "spark.kubernetes.executor.minTasksPerExecutorBeforeRolling " +
         "because newly started executors usually have no active tasks. " +
-        "OUTLIER policy chooses an executor with outstanding statistics which is bigger than" +
+        "OUTLIER policy chooses an executor with outstanding statistics which is bigger than " +
         "at least two standard deviation from the mean in average task time, " +
-        "total task time, total task GC time, and the number of failed tasks if exists. " +
+        "total task time, total task GC time, the number of failed tasks, " +
+        "peak JVM on-heap memory, peak JVM off-heap memory, total shuffle write, " +
+        "and disk used if exists. " +
+        "The dimensions are checked in this order and the first outlier found is chosen. " +
         "If there is no outlier it works like TOTAL_DURATION policy. " +
         "OUTLIER_NO_FALLBACK policy picks an outlier using the OUTLIER policy above. " +
         "If there is no outlier then no executor will be rolled.")


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org