You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@spark.apache.org by do...@apache.org on 2026/10/22 03:19:34 UTC
(spark) branch branch-4.3 updated: [SPARK-60430][CORE] Fix `FileSystemRecoveryModeFactory` to handle `none` as a codec
This is an automated email from the ASF dual-hosted git repository.
dongjoon-hyun pushed a commit to branch branch-4.3
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/branch-4.3 by this push:
new a7c96a8bb64d [SPARK-60430][CORE] Fix `FileSystemRecoveryModeFactory` to handle `none` as a codec
a7c96a8bb64d is described below
commit a7c96a8bb64dba2b0972cf917d6640b6f1995314
Author: Dongjoon Hyun <do...@apache.org>
AuthorDate: Wed Oct 21 20:18:59 2026 -0700
[SPARK-60430][CORE] Fix `FileSystemRecoveryModeFactory` to handle `none` as a codec
### What changes were proposed in this pull request?
This PR aims to fix `FileSystemRecoveryModeFactory` to handle `none` as a codec of `spark.deploy.recoveryCompressionCodec` instead of throwing an exception. Like the other short codec names, `none` is matched case-insensitively.
### Why are the changes needed?
Since Apache Spark 4.0.0 ([SPARK-46216](https://github.com/apache/spark/pull/44129)), the documentation of `spark.deploy.recoveryCompressionCodec` lists `none` as a valid value.
https://github.com/apache/spark/blob/cd06735c008285bddb9bd976b3e931f37eb4990f/docs/spark-standalone.md?plain=1#L939-L942
> A compression codec for persistence engines. none (default), lz4, lzf, snappy, and zstd. Currently, only FILESYSTEM mode supports this configuration.
However, `FileSystemRecoveryModeFactory` passes any configured value to `CompressionCodec.createCodec`, which doesn't know `none`. As a result, `Master.onStart` fails with the following error when `none` is given explicitly, and the Master never becomes `ALIVE`.
```
ERROR Inbox: Ignoring error
org.apache.spark.SparkIllegalArgumentException: [CODEC_NOT_AVAILABLE.WITH_CONF_SUGGESTION] The codec none is not available. Consider to set the config "spark.io.compression.codec" to "snappy". SQLSTATE: 56038
at org.apache.spark.errors.SparkCoreErrors$.codecNotAvailableError(SparkCoreErrors.scala:476)
at org.apache.spark.io.CompressionCodec$.$anonfun$createCodec$2(CompressionCodec.scala:96)
at scala.Option.getOrElse(Option.scala:201)
at org.apache.spark.io.CompressionCodec$.createCodec(CompressionCodec.scala:96)
at org.apache.spark.deploy.master.FileSystemRecoveryModeFactory.$anonfun$createPersistenceEngine$2(RecoveryModeFactory.scala:61)
at scala.Option.map(Option.scala:242)
at org.apache.spark.deploy.master.FileSystemRecoveryModeFactory.createPersistenceEngine(RecoveryModeFactory.scala:61)
at org.apache.spark.deploy.master.Master.onStart(Master.scala:189)
...
```
This is consistent with `spark.eventLog.compression.codec`, which handles `none` in the same way since [SPARK-45261](https://github.com/apache/spark/pull/43038).
### Does this PR introduce _any_ user-facing change?
Yes. `spark.deploy.recoveryCompressionCodec=none` works as documented now, which means no compression like the default (unset). Previously, the Master failed to initialize the `FILESYSTEM` recovery mode with `CODEC_NOT_AVAILABLE` and never became `ALIVE`. This bug exists since [v4.0.0](https://github.com/apache/spark/releases/tag/v4.0.0) (2025-05-19).
### How was this patch tested?
Pass the CIs with the newly added test cases. The new test cases fail without the main code change.
```
$ build/sbt 'core/testOnly org.apache.spark.deploy.master.RecoverySuite org.apache.spark.deploy.master.PersistenceEngineSuite'
...
[info] - SPARK-46216: Recovery without compression (62 milliseconds)
[info] - SPARK-46216: Recovery with compression (51 milliseconds)
[info] - SPARK-60430: Recovery with explicit none codec (none) (55 milliseconds)
[info] - SPARK-60430: Recovery with explicit none codec (NONE) (79 milliseconds)
...
[info] Tests: succeeded 26, failed 0, canceled 0, ignored 0, pending 0
[info] All tests passed.
```
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Opus 5.5
Closes #59680 from dongjoon-hyun/SPARK-60430.
Authored-by: Dongjoon Hyun <do...@apache.org>
Signed-off-by: Dongjoon Hyun <do...@apache.org>
(cherry picked from commit 4e2fc4a79865ea86f24a5f1487820a719ba70a5c)
Signed-off-by: Dongjoon Hyun <do...@apache.org>
---
.../spark/deploy/master/RecoveryModeFactory.scala | 4 +++-
.../apache/spark/deploy/master/RecoverySuite.scala | 20 ++++++++++++++++++++
2 files changed, 23 insertions(+), 1 deletion(-)
diff --git a/core/src/main/scala/org/apache/spark/deploy/master/RecoveryModeFactory.scala b/core/src/main/scala/org/apache/spark/deploy/master/RecoveryModeFactory.scala
index a751c7bfc67c..99042ad841aa 100644
--- a/core/src/main/scala/org/apache/spark/deploy/master/RecoveryModeFactory.scala
+++ b/core/src/main/scala/org/apache/spark/deploy/master/RecoveryModeFactory.scala
@@ -58,7 +58,9 @@ private[master] class FileSystemRecoveryModeFactory(conf: SparkConf, serializer:
def createPersistenceEngine(): PersistenceEngine = {
logInfo(log"Persisting recovery state to directory: ${MDC(LogKeys.PATH, recoveryDir)}")
- val codec = conf.get(RECOVERY_COMPRESSION_CODEC).map(c => CompressionCodec.createCodec(conf, c))
+ val codec = conf.get(RECOVERY_COMPRESSION_CODEC)
+ .filterNot(_.equalsIgnoreCase("none"))
+ .map(c => CompressionCodec.createCodec(conf, c))
new FileSystemPersistenceEngine(recoveryDir, serializer, codec)
}
diff --git a/core/src/test/scala/org/apache/spark/deploy/master/RecoverySuite.scala b/core/src/test/scala/org/apache/spark/deploy/master/RecoverySuite.scala
index f8beee7b2a34..cbe20c58f7c9 100644
--- a/core/src/test/scala/org/apache/spark/deploy/master/RecoverySuite.scala
+++ b/core/src/test/scala/org/apache/spark/deploy/master/RecoverySuite.scala
@@ -513,6 +513,26 @@ class RecoverySuite extends MasterSuiteBase {
}
}
+ gridTest("SPARK-60430: Recovery with explicit none codec")(Seq("none", "NONE")) { codecName =>
+ val conf = new SparkConf(loadDefaults = false)
+ conf.set(RECOVERY_MODE, "FILESYSTEM")
+ conf.set(RECOVERY_DIRECTORY, System.getProperty("java.io.tmpdir"))
+ conf.set(RECOVERY_COMPRESSION_CODEC, codecName)
+
+ var master: Master = null
+ try {
+ master = makeAliveMaster(conf)
+ val e = master.persistenceEngine.asInstanceOf[FileSystemPersistenceEngine]
+ assert(e.codec.isEmpty)
+ } finally {
+ if (master != null) {
+ master.rpcEnv.shutdown()
+ master.rpcEnv.awaitTermination()
+ master = null
+ }
+ }
+ }
+
test("SPARK-46258: Recovery with RocksDB") {
val conf = new SparkConf(loadDefaults = false)
conf.set(RECOVERY_MODE, "ROCKSDB")
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org