You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Uroš Bojanić (Jira)" <ji...@apache.org> on 2026/09/28 12:44:00 UTC
[jira] [Resolved] (SPARK-58881) Ignore non-numeric part-* files when reading a checkpoint directory
[ https://issues.apache.org/jira/browse/SPARK-58881?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Uroš Bojanić resolved SPARK-58881.
----------------------------------
Fix Version/s: 4.4.0
Resolution: Fixed
Issue resolved by pull request 58535
[https://github.com/apache/spark/pull/58535]
> Ignore non-numeric part-* files when reading a checkpoint directory
> -------------------------------------------------------------------
>
> Key: SPARK-58881
> URL: https://issues.apache.org/jira/browse/SPARK-58881
> Project: Spark
> Issue Type: Improvement
> Components: Spark Core
> Affects Versions: 4.4.0
> Reporter: Yang Jie
> Assignee: Jichuan Chen
> Priority: Major
> Labels: pull-request-available
> Fix For: 4.4.0
>
>
> {code:none}
> $ ls /data/cp/rdd-3
> part-00000 part-00000.bak part-00001
> $ sc.checkpointFile[Int]("/data/cp/rdd-3").partitions
> java.lang.NumberFormatException: For input string: "00000.bak"
> {code}
> {{ReliableCheckpointRDD.getPartitions}} filters the directory listing with {{startsWith("part-")}} and then sorts with {{sortBy(_.getName.stripPrefix("part-").toInt)}}. A file whose name starts with {{part-}} but does not parse as an integer, such as a leftover {{part-00000.bak}}, passes the filter and throws a raw {{NumberFormatException}} out of the sort, before the contiguity check that would raise {{INVALID_CHECKPOINT_DIRECTORY}}.
> The exception carries no indication of which directory or which file is at fault, and one unrelated leftover file makes an otherwise complete checkpoint unreadable.
> Spark never writes such a name itself: partition files come from {{"part-%05d".format(i)}} and in-progress writes use a leading dot ({{.part-00000-attempt-7}}), which the filter already excludes. So these names only appear when a user or an external tool leaves something behind.
> Proposal: tighten the filter to accept only {{part-}} followed by digits, so a non-numeric name is ignored rather than fatal. Note the width is not fixed at five: {{%05d}} is a minimum width, so a job with more than 100000 partitions produces {{part-100000}}. A pattern anchored to exactly five digits would break those.
> Names that are numeric but not zero-padded to five digits, such as {{part-1}}, already reach {{INVALID_CHECKPOINT_DIRECTORY}} today and should keep doing so.
> Found while reviewing SPARK-58770 (https://github.com/apache/spark/pull/58004), which named this condition. Left out of that PR because changing which files fail a read is a behaviour decision rather than a naming one.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org