You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@spark.apache.org by do...@apache.org on 2026/04/29 05:09:17 UTC

(spark) branch master updated: [SPARK-56653][K8S][DOCS] Document `spark.kubernetes.(executor.useDriverPodIP|allocation.maxPendingPodsPerRp|driver.annotateExitException)`

This is an automated email from the ASF dual-hosted git repository.

dongjoon-hyun pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git


The following commit(s) were added to refs/heads/master by this push:
     new 68456a6aa686 [SPARK-56653][K8S][DOCS] Document `spark.kubernetes.(executor.useDriverPodIP|allocation.maxPendingPodsPerRp|driver.annotateExitException)`
68456a6aa686 is described below

commit 68456a6aa686dc32eda0a6a2cd1a2c9e841762eb
Author: Dongjoon Hyun <do...@apache.org>
AuthorDate: Tue Apr 28 22:09:08 2026 -0700

    [SPARK-56653][K8S][DOCS] Document `spark.kubernetes.(executor.useDriverPodIP|allocation.maxPendingPodsPerRp|driver.annotateExitException)`
    
    ### What changes were proposed in this pull request?
    
    This PR documents three missing Spark configurations introduced in Spark 4.1.0.
    
    | Property | Default | Since | Introduced in |
    | --- | --- | --- | --- |
    | `spark.kubernetes.executor.useDriverPodIP` | `false` | 4.1.0 | SPARK-53944 |
    | `spark.kubernetes.driver.annotateExitException` | `false` | 4.1.0 | SPARK-53335 |
    | `spark.kubernetes.allocation.maxPendingPodsPerRp` | `Int.MaxValue` | 4.1.0 | SPARK-53324 |
    
    ### Why are the changes needed?
    
    Three user-facing Kubernetes configurations added in Spark 4.1.0 but weren't documented yet.
    
    - https://github.com/apache/spark/pull/52650
    - https://github.com/apache/spark/pull/52068
    - https://github.com/apache/spark/pull/51913
    
    ### Does this PR introduce _any_ user-facing change?
    
    No. Documentation-only update.
    
    ### How was this patch tested?
    
    Manual review.
    
    ### Was this patch authored or co-authored using generative AI tooling?
    
    Generated-by: Claude Code (Opus 4.7)
    
    Closes #55591 from dongjoon-hyun/SPARK-56653.
    
    Authored-by: Dongjoon Hyun <do...@apache.org>
    Signed-off-by: Dongjoon Hyun <do...@apache.org>
---
 docs/running-on-kubernetes.md | 27 +++++++++++++++++++++++++++
 1 file changed, 27 insertions(+)

diff --git a/docs/running-on-kubernetes.md b/docs/running-on-kubernetes.md
index 5aa906f9f37d..b575a127199c 100644
--- a/docs/running-on-kubernetes.md
+++ b/docs/running-on-kubernetes.md
@@ -1557,6 +1557,14 @@ See the [configuration page](configuration.html) for information on Spark config
   </td>
   <td>3.2.0</td>
 </tr>
+<tr>
+  <td><code>spark.kubernetes.driver.annotateExitException</code></td>
+  <td><code>false</code></td>
+  <td>
+    If set to true, Spark will store the exit exception failed applications in the Kubernetes API server using the <code>spark.exit-exception</code> annotation.
+  </td>
+  <td>4.1.0</td>
+</tr>
 <tr>
   <td><code>spark.kubernetes.driver.service.ipFamilyPolicy</code></td>
   <td><code>SingleStack</code></td>
@@ -1575,6 +1583,14 @@ See the [configuration page](configuration.html) for information on Spark config
   </td>
   <td>3.4.0</td>
 </tr>
+<tr>
+  <td><code>spark.kubernetes.executor.useDriverPodIP</code></td>
+  <td><code>false</code></td>
+  <td>
+    If true, executor pods use Driver pod IP directly instead of Driver Service.
+  </td>
+  <td>4.1.0</td>
+</tr>
 <tr>
   <td><code>spark.kubernetes.driver.ownPersistentVolumeClaim</code></td>
   <td><code>true</code></td>
@@ -1672,6 +1688,17 @@ See the [configuration page](configuration.html) for information on Spark config
   </td>
   <td>3.2.0</td>
 </tr>
+<tr>
+  <td><code>spark.kubernetes.allocation.maxPendingPodsPerRp</code></td>
+  <td><code>Int.MaxValue</code></td>
+  <td>
+    Maximum number of pending PODs allowed per resource profile ID during executor
+    allocation. This provides finer-grained control over pending pods by limiting them
+    per resource profile rather than globally. When set, this limit is enforced
+    independently for each resource profile ID.
+  </td>
+  <td>4.1.0</td>
+</tr>
 <tr>
   <td><code>spark.kubernetes.allocation.pods.allocator</code></td>
   <td><code>direct</code></td>


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org