You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@spark.apache.org by do...@apache.org on 2026/04/29 05:09:17 UTC
(spark) branch master updated: [SPARK-56653][K8S][DOCS] Document `spark.kubernetes.(executor.useDriverPodIP|allocation.maxPendingPodsPerRp|driver.annotateExitException)`
This is an automated email from the ASF dual-hosted git repository.
dongjoon-hyun pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/master by this push:
new 68456a6aa686 [SPARK-56653][K8S][DOCS] Document `spark.kubernetes.(executor.useDriverPodIP|allocation.maxPendingPodsPerRp|driver.annotateExitException)`
68456a6aa686 is described below
commit 68456a6aa686dc32eda0a6a2cd1a2c9e841762eb
Author: Dongjoon Hyun <do...@apache.org>
AuthorDate: Tue Apr 28 22:09:08 2026 -0700
[SPARK-56653][K8S][DOCS] Document `spark.kubernetes.(executor.useDriverPodIP|allocation.maxPendingPodsPerRp|driver.annotateExitException)`
### What changes were proposed in this pull request?
This PR documents three missing Spark configurations introduced in Spark 4.1.0.
| Property | Default | Since | Introduced in |
| --- | --- | --- | --- |
| `spark.kubernetes.executor.useDriverPodIP` | `false` | 4.1.0 | SPARK-53944 |
| `spark.kubernetes.driver.annotateExitException` | `false` | 4.1.0 | SPARK-53335 |
| `spark.kubernetes.allocation.maxPendingPodsPerRp` | `Int.MaxValue` | 4.1.0 | SPARK-53324 |
### Why are the changes needed?
Three user-facing Kubernetes configurations added in Spark 4.1.0 but weren't documented yet.
- https://github.com/apache/spark/pull/52650
- https://github.com/apache/spark/pull/52068
- https://github.com/apache/spark/pull/51913
### Does this PR introduce _any_ user-facing change?
No. Documentation-only update.
### How was this patch tested?
Manual review.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Code (Opus 4.7)
Closes #55591 from dongjoon-hyun/SPARK-56653.
Authored-by: Dongjoon Hyun <do...@apache.org>
Signed-off-by: Dongjoon Hyun <do...@apache.org>
---
docs/running-on-kubernetes.md | 27 +++++++++++++++++++++++++++
1 file changed, 27 insertions(+)
diff --git a/docs/running-on-kubernetes.md b/docs/running-on-kubernetes.md
index 5aa906f9f37d..b575a127199c 100644
--- a/docs/running-on-kubernetes.md
+++ b/docs/running-on-kubernetes.md
@@ -1557,6 +1557,14 @@ See the [configuration page](configuration.html) for information on Spark config
</td>
<td>3.2.0</td>
</tr>
+<tr>
+ <td><code>spark.kubernetes.driver.annotateExitException</code></td>
+ <td><code>false</code></td>
+ <td>
+ If set to true, Spark will store the exit exception failed applications in the Kubernetes API server using the <code>spark.exit-exception</code> annotation.
+ </td>
+ <td>4.1.0</td>
+</tr>
<tr>
<td><code>spark.kubernetes.driver.service.ipFamilyPolicy</code></td>
<td><code>SingleStack</code></td>
@@ -1575,6 +1583,14 @@ See the [configuration page](configuration.html) for information on Spark config
</td>
<td>3.4.0</td>
</tr>
+<tr>
+ <td><code>spark.kubernetes.executor.useDriverPodIP</code></td>
+ <td><code>false</code></td>
+ <td>
+ If true, executor pods use Driver pod IP directly instead of Driver Service.
+ </td>
+ <td>4.1.0</td>
+</tr>
<tr>
<td><code>spark.kubernetes.driver.ownPersistentVolumeClaim</code></td>
<td><code>true</code></td>
@@ -1672,6 +1688,17 @@ See the [configuration page](configuration.html) for information on Spark config
</td>
<td>3.2.0</td>
</tr>
+<tr>
+ <td><code>spark.kubernetes.allocation.maxPendingPodsPerRp</code></td>
+ <td><code>Int.MaxValue</code></td>
+ <td>
+ Maximum number of pending PODs allowed per resource profile ID during executor
+ allocation. This provides finer-grained control over pending pods by limiting them
+ per resource profile rather than globally. When set, this limit is enforced
+ independently for each resource profile ID.
+ </td>
+ <td>4.1.0</td>
+</tr>
<tr>
<td><code>spark.kubernetes.allocation.pods.allocator</code></td>
<td><code>direct</code></td>
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org