You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Uroš Bojanić (Jira)" <ji...@apache.org> on 2026/09/28 12:44:00 UTC

[jira] [Resolved] (SPARK-58881) Ignore non-numeric part-* files when reading a checkpoint directory

     [ https://issues.apache.org/jira/browse/SPARK-58881?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Uroš Bojanić resolved SPARK-58881.
----------------------------------
    Fix Version/s: 4.4.0
       Resolution: Fixed

Issue resolved by pull request 58535
[https://github.com/apache/spark/pull/58535]

> Ignore non-numeric part-* files when reading a checkpoint directory
> -------------------------------------------------------------------
>
>                 Key: SPARK-58881
>                 URL: https://issues.apache.org/jira/browse/SPARK-58881
>             Project: Spark
>          Issue Type: Improvement
>          Components: Spark Core
>    Affects Versions: 4.4.0
>            Reporter: Yang Jie
>            Assignee: Jichuan Chen
>            Priority: Major
>              Labels: pull-request-available
>             Fix For: 4.4.0
>
>
> {code:none}
> $ ls /data/cp/rdd-3
> part-00000  part-00000.bak  part-00001
> $ sc.checkpointFile[Int]("/data/cp/rdd-3").partitions
> java.lang.NumberFormatException: For input string: "00000.bak"
> {code}
> {{ReliableCheckpointRDD.getPartitions}} filters the directory listing with {{startsWith("part-")}} and then sorts with {{sortBy(_.getName.stripPrefix("part-").toInt)}}. A file whose name starts with {{part-}} but does not parse as an integer, such as a leftover {{part-00000.bak}}, passes the filter and throws a raw {{NumberFormatException}} out of the sort, before the contiguity check that would raise {{INVALID_CHECKPOINT_DIRECTORY}}.
> The exception carries no indication of which directory or which file is at fault, and one unrelated leftover file makes an otherwise complete checkpoint unreadable.
> Spark never writes such a name itself: partition files come from {{"part-%05d".format(i)}} and in-progress writes use a leading dot ({{.part-00000-attempt-7}}), which the filter already excludes. So these names only appear when a user or an external tool leaves something behind.
> Proposal: tighten the filter to accept only {{part-}} followed by digits, so a non-numeric name is ignored rather than fatal. Note the width is not fixed at five: {{%05d}} is a minimum width, so a job with more than 100000 partitions produces {{part-100000}}. A pattern anchored to exactly five digits would break those.
> Names that are numeric but not zero-padded to five digits, such as {{part-1}}, already reach {{INVALID_CHECKPOINT_DIRECTORY}} today and should keep doing so.
> Found while reviewing SPARK-58770 (https://github.com/apache/spark/pull/58004), which named this condition. Left out of that PR because changing which files fail a read is a behaviour decision rather than a naming one.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org