You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@spark.apache.org by ur...@apache.org on 2026/10/05 13:03:28 UTC
(spark) branch branch-4.x updated: [SPARK-58881][CORE][FOLLOWUP] Document checkpoint part-file filtering
This is an automated email from the ASF dual-hosted git repository.
uros-b pushed a commit to branch branch-4.x
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/branch-4.x by this push:
new 1f605b06e37f [SPARK-58881][CORE][FOLLOWUP] Document checkpoint part-file filtering
1f605b06e37f is described below
commit 1f605b06e37f6d7728cae0b1d9c834a7e6a4b5cf
Author: Charlie Chen <jc...@uw.edu>
AuthorDate: Mon Oct 5 13:03:02 2026 +0000
[SPARK-58881][CORE][FOLLOWUP] Document checkpoint part-file filtering
### What changes were proposed in this pull request?
Add one entry to the Spark Core 4.3-to-4.4 migration guide describing the RDD checkpoint-file filtering change from #58535.
The note explains that names such as `part-00000.bak`, `part-backup`, and `part-` are now ignored instead of causing `NumberFormatException`. Validation of numeric partition names and the contiguous partition sequence is unchanged.
### Why are the changes needed?
This documents the behavior change already merged for [SPARK-58881](https://issues.apache.org/jira/browse/SPARK-58881), following the [post-merge review request](https://github.com/apache/spark/pull/58535#issuecomment-5870625252). The fix was merged into master and branch-4.x; the latter targets Spark 4.4.
### Does this PR introduce _any_ user-facing change?
No. Documentation only; no additional runtime behavior change.
### How was this patch tested?
- Checked the wording against the merged `ReliableCheckpointRDD.getPartitions` / `isCheckpointFile` implementation and the SPARK-58881 regression in `CheckpointStorageSuite`.
- `git diff --check` passed; the only changed file is `docs/core-migration-guide.md`.
- Parsed the page's YAML front matter and checked for unresolved conflict markers.
- Did not rerun runtime tests because no code changes.
- The full Jekyll documentation build was not run locally: the machine has Ruby 2.6 and does not have the required Bundler 2.4.22 / Ruby 3 documentation environment. Remote documentation CI is pending.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: OpenAI Codex (GPT-5)
This contribution is submitted under the project's Apache License 2.0.
Closes #59191 from charliec05/codex/spark-58881-checkpoint-migration.
Authored-by: Charlie Chen <jc...@uw.edu>
Signed-off-by: Uros Bojanic <22...@users.noreply.github.com>
(cherry picked from commit ffb268f6c57fab88b82e8f13af8ab7bcd42ec614)
Signed-off-by: Uros Bojanic <22...@users.noreply.github.com>
---
docs/core-migration-guide.md | 2 ++
1 file changed, 2 insertions(+)
diff --git a/docs/core-migration-guide.md b/docs/core-migration-guide.md
index 6650f4f02c4a..8fac8b3d4902 100644
--- a/docs/core-migration-guide.md
+++ b/docs/core-migration-guide.md
@@ -24,6 +24,8 @@ license: |
## Upgrading from Core 4.3 to 4.4
+- Since Spark 4.4, reading an RDD checkpoint ignores names beginning with `part-` unless the suffix consists of one or more ASCII digits. Previously, names such as `part-00000.bak`, `part-backup`, or `part-` caused checkpoint recovery to fail with `NumberFormatException`. Validation of numeric partition names and the contiguous partition sequence is unchanged.
+
- Since Spark 4.4, an executor pod template that names a service account in `serviceAccountName` keeps it: Spark applies `spark.kubernetes.authenticate.executor.serviceAccountName`, or `spark.kubernetes.authenticate.driver.serviceAccountName` as a fallback, only when the template names no account in either `serviceAccount` or `serviceAccountName`. Earlier versions decided by reading the deprecated `serviceAccount` field alone, so with either configuration set, a template that named the a [...]
- Since Spark 4.4, an empty service account name in an executor pod template counts as naming no account, which is how Kubernetes itself defaults the two fields. A template whose deprecated `serviceAccount` field is present but empty, with `serviceAccountName` empty or absent, used to count as naming an account: the configured account was dropped and the executor pods ran as the namespace's default account. They now run as `spark.kubernetes.authenticate.executor.serviceAccountName` or, f [...]
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org