You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@spark.apache.org by do...@apache.org on 2026/10/22 22:28:15 UTC
(spark) branch branch-4.2 updated: [SPARK-60467][CORE] Fix `FileSystemPersistenceEngine` to clean up failed writes and skip unreadable files
This is an automated email from the ASF dual-hosted git repository.
dongjoon-hyun pushed a commit to branch branch-4.2
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/branch-4.2 by this push:
new 26ef9b8216b3 [SPARK-60467][CORE] Fix `FileSystemPersistenceEngine` to clean up failed writes and skip unreadable files
26ef9b8216b3 is described below
commit 26ef9b8216b38c9d1276679d36094f2f2a50d03e
Author: Dongjoon Hyun <do...@apache.org>
AuthorDate: Thu Oct 22 15:23:50 2026 -0700
[SPARK-60467][CORE] Fix `FileSystemPersistenceEngine` to clean up failed writes and skip unreadable files
### What changes were proposed in this pull request?
This PR fixes two robustness issues in `FileSystemPersistenceEngine`, which backs the `FILESYSTEM` recovery mode of the standalone Master.
1. `serializeIntoFile` keeps the raw `FileOutputStream` as `rawOut` and closes the serialization stream, the codec stream and `rawOut`, in this order, each guaranteed by a nested `Utils.tryWithSafeFinally`. The close order from SPARK-46205, where the serializer and the codec flush before the file closes, is unchanged. The codec stream is now created inside the `tryWithSafeFinally` body (as SPARK-60419 does on the read side), so the raw stream is closed even when the codec stream const [...]
2. `serializeIntoFile` deletes the file and rethrows when writing fails after it created the file. It never deletes an existing file: `File already exists` and `Could not create file` are thrown before the file is created.
3. `read` skips a file that cannot be deserialized instead of failing as a whole. It logs a warning with the path and the exception, and renames the file to `corrupted_<name>`, which no longer matches the `app_`, `driver_` or `worker_` prefix and is kept for inspection.
Items 2 and 3 together are the smallest design that unblocks recovery in all cases:
- Deleting on write failures alone does not cover a crash or a power loss in the middle of a write, or files that released versions already left behind.
- Skipping on read alone keeps a failed write's file under its real name until the next start, which then warns about it and renames it.
- Writing to a temporary file, then `fsync` and `ATOMIC_MOVE`, would add an `fsync` to every app, driver and worker registration on the Master's RPC thread, which is slow on an NFS-mounted recovery directory. It also needs cleanup of leftover temporary files, `ATOMIC_MOVE` support from the file system, and a directory `fsync` that Java cannot do portably, and it still does not cover files that are already broken. Durability is out of the scope of this PR.
Unlike `ZooKeeperPersistenceEngine`, which deletes unreadable entries, `read` renames them. This keeps the data when the cause is not a broken file, e.g. an incompatible class, a changed `spark.deploy.recoveryCompressionCodec`, or a transient I/O error.
SPARK-60419 (https://github.com/apache/spark/pull/59670) fixed the double open in `deserializeFromFile`. This PR keeps that method as is, and fixes the write side and the per-file handling in `read`.
### Why are the changes needed?
**Stream leaks on write failures.** When the last write fails during close, e.g. with `ENOSPC` or `EDQUOT`, `out.close()` throws. `ObjectOutputStream.close()` calls `flush()` first, which pushes the buffered data through the codec, so it never reaches the codec stream's `close()`. The `finally` block then skips `fileOut.close()`:
- The raw `FileOutputStream` stays open until GC, with or without a codec.
- With zstd, `ZstdOutputStreamNoFinalizer` is never closed, so its native `CStream` is never freed. The class has no finalizer as a fallback.
Reaching `fileOut.close()` is not enough. When their final write fails, the `close()` methods of `LZ4BlockOutputStream`, `ZstdOutputStreamNoFinalizer` and the non-parallel `LZFOutputStream` throw before closing the stream they wrap. So does `SnappyOutputStream` in snappy-java 1.1.10.8 and older, which `branch-4.x`, `branch-4.1` and `branch-4.0` use; snappy-java 1.1.10.9 fixed it in https://github.com/xerial/snappy-java/pull/744. Nothing else referenced the raw `FileOutputStream`. The [...]
**Partially written files block recovery.** `serializeIntoFile` creates the file before writing it, never deletes it on failure, and does not `fsync`. `read` fails as a whole when any file cannot be deserialized. So an empty or truncated `app_`, `driver_` or `worker_` file, left by a failed write, a crash in the middle of a write, or a power loss, makes `readPersistedData` throw in the `ElectedLeader` handler on the next start. The RPC inbox only logs `Ignoring error`, the Master stay [...]
### Does this PR introduce _any_ user-facing change?
Yes, in the `FILESYSTEM` recovery mode, compared with the released versions:
- A failed persist no longer leaves an empty or truncated file in the recovery directory, and no longer leaks its file descriptor.
- An unreadable file in the recovery directory no longer keeps the Master in `STANDBY`. The Master logs a warning, renames the file to `corrupted_<name>`, and recovers the other apps, drivers and workers. This also applies when the recovery directory was not cleared after changing `spark.deploy.recoveryMode` or `spark.deploy.recoveryCompressionCodec`, which the documentation asks users to do.
### How was this patch tested?
Pass the CIs with the newly added test cases, which fail without the main code change:
- `FileSystemPersistenceEngine closes and deletes the file when persist fails` passes a test-only `CompressionCodec` through the engine's `codec` constructor parameter. Its output stream fails on `flush()` and `close()` without closing the stream it wraps, as the lz4, zstd and lzf streams do on a full disk, or it fails to be created. The test checks that the codec stream's `close()` is called, that the raw `FileOutputStream` is closed (`getFD.valid()` is `false`), and that no file is [...]
- `FileSystemPersistenceEngine skips and renames unreadable files` writes an empty file and a truncated file next to a valid one. `read` returns only the valid object, the two files are renamed to `corrupted_<name>`, and their names can be persisted again. Without the fix, `read` throws `java.io.EOFException`.
- The existing `SPARK-46191` test case also checks that a persist failing with `File already exists` keeps the existing file.
```
$ build/sbt 'core/testOnly *PersistenceEngineSuite *RecoverySuite'
...
[info] Tests: succeeded 28, failed 0, canceled 0, ignored 0, pending 0
```
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Opus 5.5
Closes #59709 from dongjoon-hyun/SPARK-60467.
Authored-by: Dongjoon Hyun <do...@apache.org>
Signed-off-by: Dongjoon Hyun <do...@apache.org>
(cherry picked from commit eacf31bf9a429d28e34ebe20ad7c7444c3d026bd)
Signed-off-by: Dongjoon Hyun <do...@apache.org>
---
.../master/FileSystemPersistenceEngine.scala | 59 ++++++++++++++++----
.../deploy/master/PersistenceEngineSuite.scala | 64 ++++++++++++++++++++++
2 files changed, 112 insertions(+), 11 deletions(-)
diff --git a/core/src/main/scala/org/apache/spark/deploy/master/FileSystemPersistenceEngine.scala b/core/src/main/scala/org/apache/spark/deploy/master/FileSystemPersistenceEngine.scala
index 71b6a625f6d8..e39eefa1db1a 100644
--- a/core/src/main/scala/org/apache/spark/deploy/master/FileSystemPersistenceEngine.scala
+++ b/core/src/main/scala/org/apache/spark/deploy/master/FileSystemPersistenceEngine.scala
@@ -21,6 +21,7 @@ import java.io._
import java.nio.file.{FileAlreadyExistsException, Files, Paths}
import scala.reflect.ClassTag
+import scala.util.control.NonFatal
import org.apache.spark.internal.Logging
import org.apache.spark.internal.LogKeys._
@@ -63,24 +64,60 @@ private[master] class FileSystemPersistenceEngine(
override def read[T: ClassTag](prefix: String): Seq[T] = {
val files = new File(dir).listFiles().filter(_.getName.startsWith(prefix))
- files.map(deserializeFromFile[T]).toImmutableArraySeq
+ files.flatMap { file =>
+ try {
+ Some(deserializeFromFile[T](file))
+ } catch {
+ case NonFatal(e) =>
+ // Skip an unreadable file, e.g. one left truncated by a crash, so that it cannot block
+ // recovery. Rename it, instead of deleting it, so that it is kept for inspection but no
+ // longer matches the app_, driver_ or worker_ prefix.
+ val corrupted = new File(dir, "corrupted_" + file.getName)
+ logWarning(log"Exception while reading persisted file ${MDC(PATH, file.getPath())}, " +
+ log"renaming it to ${MDC(NEW_PATH, corrupted.getPath())}", e)
+ if (!file.renameTo(corrupted)) {
+ logWarning(log"Error renaming ${MDC(PATH, file.getPath())}")
+ }
+ None
+ }
+ }.toImmutableArraySeq
}
private def serializeIntoFile(file: File, value: AnyRef): Unit = {
if (file.exists()) { throw new IllegalStateException("File already exists: " + file) }
val created = file.createNewFile()
if (!created) { throw new IllegalStateException("Could not create file: " + file) }
- var fileOut: OutputStream = new FileOutputStream(file)
- codec.foreach { c => fileOut = c.compressedOutputStream(fileOut) }
- var out: SerializationStream = null
- Utils.tryWithSafeFinally {
- out = serializer.newInstance().serializeStream(fileOut)
- out.writeObject(value)
- } {
- if (out != null) {
- out.close()
+ try {
+ val rawOut = new FileOutputStream(file)
+ var fileOut: OutputStream = rawOut
+ var out: SerializationStream = null
+ Utils.tryWithSafeFinally {
+ codec.foreach { c => fileOut = c.compressedOutputStream(fileOut) }
+ out = serializer.newInstance().serializeStream(fileOut)
+ out.writeObject(value)
+ } {
+ // Close the serializer and the codec before the file so that they flush first. Close
+ // each one even if the previous close fails, e.g. on a full disk, because some codec
+ // streams do not close the stream they wrap when their final write fails.
+ Utils.tryWithSafeFinally {
+ Utils.tryWithSafeFinally {
+ if (out != null) {
+ out.close()
+ }
+ } {
+ fileOut.close()
+ }
+ } {
+ rawOut.close()
+ }
}
- fileOut.close()
+ } catch {
+ case e: Throwable =>
+ // Remove the file created above so that a partially written file cannot block recovery.
+ if (!file.delete()) {
+ logWarning(log"Error deleting ${MDC(PATH, file.getPath())}")
+ }
+ throw e
}
}
diff --git a/core/src/test/scala/org/apache/spark/deploy/master/PersistenceEngineSuite.scala b/core/src/test/scala/org/apache/spark/deploy/master/PersistenceEngineSuite.scala
index 6839afdeeff8..f594040d2e57 100644
--- a/core/src/test/scala/org/apache/spark/deploy/master/PersistenceEngineSuite.scala
+++ b/core/src/test/scala/org/apache/spark/deploy/master/PersistenceEngineSuite.scala
@@ -18,6 +18,7 @@
package org.apache.spark.deploy.master
+import java.io.{File, FileOutputStream, FilterOutputStream, InputStream, IOException, OutputStream}
import java.net.ServerSocket
import java.nio.file.{Files, Paths}
import java.util.concurrent.ThreadLocalRandom
@@ -61,6 +62,8 @@ class PersistenceEngineSuite extends SparkFunSuite {
engine.persist("test_1", "test_1_value")
}.getMessage
assert(m.contains("File already exists"))
+ // The failed persist does not delete the existing file.
+ assert(engine.read[String]("test_1") === Seq("test_1_value"))
}
}
@@ -115,6 +118,42 @@ class PersistenceEngineSuite extends SparkFunSuite {
}
}
+ test("SPARK-60467: FileSystemPersistenceEngine closes and deletes the file when persist fails") {
+ Seq(false, true).foreach { failOnCreate =>
+ withTempDir { dir =>
+ val codec = new FailingCompressionCodec(failOnCreate)
+ val engine = new FileSystemPersistenceEngine(
+ dir.getAbsolutePath, new JavaSerializer(new SparkConf()), Some(codec))
+ intercept[IOException] {
+ engine.persist("test_1", "test_1_value")
+ }
+ // The codec stream is closed if it was created, and the file stream is closed although
+ // the codec stream does not close it.
+ assert(codec.closeCalled === !failOnCreate)
+ assert(!codec.fileOut.getFD.valid())
+ // No partially written file is left behind.
+ assert(dir.list().isEmpty)
+ }
+ }
+ }
+
+ test("SPARK-60467: FileSystemPersistenceEngine skips and renames unreadable files") {
+ withTempDir { dir =>
+ val engine = new FileSystemPersistenceEngine(
+ dir.getAbsolutePath, new JavaSerializer(new SparkConf()))
+ engine.persist("test_1", "test_1_value")
+ // An empty file and a truncated file, as a crash in the middle of a persist can leave.
+ val bytes = Files.readAllBytes(new File(dir, "test_1").toPath)
+ Files.write(new File(dir, "test_2").toPath, Array.emptyByteArray)
+ Files.write(new File(dir, "test_3").toPath, bytes.take(bytes.length / 2))
+ assert(engine.read[String]("test_") === Seq("test_1_value"))
+ assert(dir.list().toSet === Set("test_1", "corrupted_test_2", "corrupted_test_3"))
+ // The renamed files neither block reads nor persisting the same names again.
+ engine.persist("test_2", "test_2_value")
+ assert(engine.read[String]("test_").toSet === Set("test_1_value", "test_2_value"))
+ }
+ }
+
test("ZooKeeperPersistenceEngine") {
val conf = new SparkConf()
// TestingServer logs the port conflict exception rather than throwing an exception.
@@ -199,4 +238,29 @@ class PersistenceEngineSuite extends SparkFunSuite {
(null, trialPort)
}, conf)._2
}
+
+ /**
+ * A codec that fails to create its output stream, or whose output stream fails to flush and
+ * close, as on a full disk, without closing the file stream it wraps.
+ */
+ private class FailingCompressionCodec(failOnCreate: Boolean) extends CompressionCodec {
+ var fileOut: FileOutputStream = _
+ var closeCalled = false
+
+ override def compressedOutputStream(s: OutputStream): OutputStream = {
+ fileOut = s.asInstanceOf[FileOutputStream]
+ if (failOnCreate) {
+ throw new IOException("Cannot create the compressed stream")
+ }
+ new FilterOutputStream(s) {
+ override def flush(): Unit = throw new IOException("No space left on device")
+ override def close(): Unit = {
+ closeCalled = true
+ throw new IOException("No space left on device")
+ }
+ }
+ }
+
+ override def compressedInputStream(s: InputStream): InputStream = s
+ }
}
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org