You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@spark.apache.org by do...@apache.org on 2023/12/03 08:26:26 UTC

(spark) branch master updated: [SPARK-46216][CORE] Improve `FileSystemPersistenceEngine` to support compressions

This is an automated email from the ASF dual-hosted git repository.

dongjoon pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git


The following commit(s) were added to refs/heads/master by this push:
     new 3da2e5c6324 [SPARK-46216][CORE] Improve `FileSystemPersistenceEngine` to support compressions
3da2e5c6324 is described below

commit 3da2e5c632468ec7cf7001255c1a44197b46ce30
Author: Dongjoon Hyun <dh...@apple.com>
AuthorDate: Sun Dec 3 00:26:16 2023 -0800

    [SPARK-46216][CORE] Improve `FileSystemPersistenceEngine` to support compressions
    
    ### What changes were proposed in this pull request?
    
    This PR aims to improve `FileSystemPersistenceEngine` to support compressions via a new configuration, `spark.deploy.recoveryCompressionCodec`.
    
    ### Why are the changes needed?
    
    To allow the users to choose a proper compression codec for their workloads. For `JavaSerializer` case, `LZ4` compression is **2x** faster than the baseline (no compression).
    ```
    OpenJDK 64-Bit Server VM 17.0.9+9-LTS on Linux 5.15.0-1051-azure
    AMD EPYC 7763 64-Core Processor
    2000 Workers:                                             Best Time(ms)   Avg Time(ms)   Stdev(ms)    Rate(M/s)   Per Row(ns)   Relative
    ----------------------------------------------------------------------------------------------------------------------------------------
    ZooKeeperPersistenceEngine with JavaSerializer                     2276           2360         115          0.0     1137909.6       1.0X
    ZooKeeperPersistenceEngine with KryoSerializer                     1883           1906          34          0.0      941364.2       1.2X
    FileSystemPersistenceEngine with JavaSerializer                     431            436           7          0.0      215436.9       5.3X
    FileSystemPersistenceEngine with JavaSerializer (lz4)               209            216           9          0.0      104404.1      10.9X
    FileSystemPersistenceEngine with JavaSerializer (lzf)               199            202           2          0.0       99489.5      11.4X
    FileSystemPersistenceEngine with JavaSerializer (snappy)            192            199           9          0.0       95872.9      11.9X
    FileSystemPersistenceEngine with JavaSerializer (zstd)              258            264           6          0.0      129249.4       8.8X
    FileSystemPersistenceEngine with KryoSerializer                     139            151          13          0.0       69374.5      16.4X
    FileSystemPersistenceEngine with KryoSerializer (lz4)               159            165           8          0.0       79588.9      14.3X
    FileSystemPersistenceEngine with KryoSerializer (lzf)               180            195          18          0.0       89844.0      12.7X
    FileSystemPersistenceEngine with KryoSerializer (snappy)            164            183          18          0.0       82016.0      13.9X
    FileSystemPersistenceEngine with KryoSerializer (zstd)              206            218          11          0.0      102838.9      11.1X
    BlackHolePersistenceEngine                                            0              0           0         35.1          28.5   39908.5X
    ```
    
    ### Does this PR introduce _any_ user-facing change?
    
    No, this is a new feature.
    
    ### How was this patch tested?
    
    Pass the CIs with the newly added test case.
    
    ### Was this patch authored or co-authored using generative AI tooling?
    
    No.
    
    Closes #44129 from dongjoon-hyun/SPARK-46216.
    
    Authored-by: Dongjoon Hyun <dh...@apple.com>
    Signed-off-by: Dongjoon Hyun <dh...@apple.com>
---
 .../PersistenceEngineBenchmark-jdk21-results.txt   | 22 ++++++++----
 .../PersistenceEngineBenchmark-results.txt         | 22 ++++++++----
 .../master/FileSystemPersistenceEngine.scala       | 10 ++++--
 .../spark/deploy/master/RecoveryModeFactory.scala  |  6 ++--
 .../org/apache/spark/internal/config/Deploy.scala  |  7 ++++
 .../apache/spark/deploy/master/MasterSuite.scala   | 40 ++++++++++++++++++++++
 .../deploy/master/PersistenceEngineBenchmark.scala | 19 ++++++++--
 .../deploy/master/PersistenceEngineSuite.scala     | 13 +++++++
 8 files changed, 118 insertions(+), 21 deletions(-)

diff --git a/core/benchmarks/PersistenceEngineBenchmark-jdk21-results.txt b/core/benchmarks/PersistenceEngineBenchmark-jdk21-results.txt
index 65dbfd0990d..38e74ed6b53 100644
--- a/core/benchmarks/PersistenceEngineBenchmark-jdk21-results.txt
+++ b/core/benchmarks/PersistenceEngineBenchmark-jdk21-results.txt
@@ -4,12 +4,20 @@ PersistenceEngineBenchmark
 
 OpenJDK 64-Bit Server VM 21.0.1+12-LTS on Linux 5.15.0-1051-azure
 AMD EPYC 7763 64-Core Processor
-1000 Workers:                                    Best Time(ms)   Avg Time(ms)   Stdev(ms)    Rate(M/s)   Per Row(ns)   Relative
--------------------------------------------------------------------------------------------------------------------------------
-ZooKeeperPersistenceEngine with JavaSerializer            1100           1255         150          0.0     1099532.9       1.0X
-ZooKeeperPersistenceEngine with KryoSerializer             946            967          20          0.0      946367.3       1.2X
-FileSystemPersistenceEngine with JavaSerializer            218            223           4          0.0      217851.5       5.0X
-FileSystemPersistenceEngine with KryoSerializer             79             87          12          0.0       78611.1      14.0X
-BlackHolePersistenceEngine                                   0              0           0         42.0          23.8   46191.1X
+2000 Workers:                                             Best Time(ms)   Avg Time(ms)   Stdev(ms)    Rate(M/s)   Per Row(ns)   Relative
+----------------------------------------------------------------------------------------------------------------------------------------
+ZooKeeperPersistenceEngine with JavaSerializer                     2254           2329         119          0.0     1126867.1       1.0X
+ZooKeeperPersistenceEngine with KryoSerializer                     1911           1912           1          0.0      955667.1       1.2X
+FileSystemPersistenceEngine with JavaSerializer                     438            448          15          0.0      218868.1       5.1X
+FileSystemPersistenceEngine with JavaSerializer (lz4)               187            195           8          0.0       93337.8      12.1X
+FileSystemPersistenceEngine with JavaSerializer (lzf)               193            216          20          0.0       96678.8      11.7X
+FileSystemPersistenceEngine with JavaSerializer (snappy)            175            183          10          0.0       87652.3      12.9X
+FileSystemPersistenceEngine with JavaSerializer (zstd)              243            255          14          0.0      121695.2       9.3X
+FileSystemPersistenceEngine with KryoSerializer                     150            160          15          0.0       75089.7      15.0X
+FileSystemPersistenceEngine with KryoSerializer (lz4)               170            177          10          0.0       84996.7      13.3X
+FileSystemPersistenceEngine with KryoSerializer (lzf)               192            203          12          0.0       96019.1      11.7X
+FileSystemPersistenceEngine with KryoSerializer (snappy)            184            202          16          0.0       92241.3      12.2X
+FileSystemPersistenceEngine with KryoSerializer (zstd)              232            238           5          0.0      116075.2       9.7X
+BlackHolePersistenceEngine                                            0              0           0         27.3          36.6   30761.0X
 
 
diff --git a/core/benchmarks/PersistenceEngineBenchmark-results.txt b/core/benchmarks/PersistenceEngineBenchmark-results.txt
index eb63d07b4df..fc743da46a8 100644
--- a/core/benchmarks/PersistenceEngineBenchmark-results.txt
+++ b/core/benchmarks/PersistenceEngineBenchmark-results.txt
@@ -4,12 +4,20 @@ PersistenceEngineBenchmark
 
 OpenJDK 64-Bit Server VM 17.0.9+9-LTS on Linux 5.15.0-1051-azure
 AMD EPYC 7763 64-Core Processor
-1000 Workers:                                    Best Time(ms)   Avg Time(ms)   Stdev(ms)    Rate(M/s)   Per Row(ns)   Relative
--------------------------------------------------------------------------------------------------------------------------------
-ZooKeeperPersistenceEngine with JavaSerializer            1202           1298         138          0.0     1201614.2       1.0X
-ZooKeeperPersistenceEngine with KryoSerializer             951           1004          48          0.0      950559.0       1.3X
-FileSystemPersistenceEngine with JavaSerializer            212            217           6          0.0      211623.2       5.7X
-FileSystemPersistenceEngine with KryoSerializer             79             81           2          0.0       79132.5      15.2X
-BlackHolePersistenceEngine                                   0              0           0         30.9          32.4   37109.8X
+2000 Workers:                                             Best Time(ms)   Avg Time(ms)   Stdev(ms)    Rate(M/s)   Per Row(ns)   Relative
+----------------------------------------------------------------------------------------------------------------------------------------
+ZooKeeperPersistenceEngine with JavaSerializer                     2276           2360         115          0.0     1137909.6       1.0X
+ZooKeeperPersistenceEngine with KryoSerializer                     1883           1906          34          0.0      941364.2       1.2X
+FileSystemPersistenceEngine with JavaSerializer                     431            436           7          0.0      215436.9       5.3X
+FileSystemPersistenceEngine with JavaSerializer (lz4)               209            216           9          0.0      104404.1      10.9X
+FileSystemPersistenceEngine with JavaSerializer (lzf)               199            202           2          0.0       99489.5      11.4X
+FileSystemPersistenceEngine with JavaSerializer (snappy)            192            199           9          0.0       95872.9      11.9X
+FileSystemPersistenceEngine with JavaSerializer (zstd)              258            264           6          0.0      129249.4       8.8X
+FileSystemPersistenceEngine with KryoSerializer                     139            151          13          0.0       69374.5      16.4X
+FileSystemPersistenceEngine with KryoSerializer (lz4)               159            165           8          0.0       79588.9      14.3X
+FileSystemPersistenceEngine with KryoSerializer (lzf)               180            195          18          0.0       89844.0      12.7X
+FileSystemPersistenceEngine with KryoSerializer (snappy)            164            183          18          0.0       82016.0      13.9X
+FileSystemPersistenceEngine with KryoSerializer (zstd)              206            218          11          0.0      102838.9      11.1X
+BlackHolePersistenceEngine                                            0              0           0         35.1          28.5   39908.5X
 
 
diff --git a/core/src/main/scala/org/apache/spark/deploy/master/FileSystemPersistenceEngine.scala b/core/src/main/scala/org/apache/spark/deploy/master/FileSystemPersistenceEngine.scala
index 094136ea274..785367a0dee 100644
--- a/core/src/main/scala/org/apache/spark/deploy/master/FileSystemPersistenceEngine.scala
+++ b/core/src/main/scala/org/apache/spark/deploy/master/FileSystemPersistenceEngine.scala
@@ -23,6 +23,7 @@ import java.nio.file.{Files, Paths}
 import scala.reflect.ClassTag
 
 import org.apache.spark.internal.Logging
+import org.apache.spark.io.CompressionCodec
 import org.apache.spark.serializer.{DeserializationStream, SerializationStream, Serializer}
 import org.apache.spark.util.ArrayImplicits._
 import org.apache.spark.util.Utils
@@ -37,7 +38,8 @@ import org.apache.spark.util.Utils
  */
 private[master] class FileSystemPersistenceEngine(
     val dir: String,
-    val serializer: Serializer)
+    val serializer: Serializer,
+    val codec: Option[CompressionCodec] = None)
   extends PersistenceEngine with Logging {
 
   Files.createDirectories(Paths.get(dir))
@@ -62,7 +64,8 @@ private[master] class FileSystemPersistenceEngine(
     if (file.exists()) { throw new IllegalStateException("File already exists: " + file) }
     val created = file.createNewFile()
     if (!created) { throw new IllegalStateException("Could not create file: " + file) }
-    val fileOut = new FileOutputStream(file)
+    var fileOut: OutputStream = new FileOutputStream(file)
+    codec.foreach { c => fileOut = c.compressedOutputStream(fileOut) }
     var out: SerializationStream = null
     Utils.tryWithSafeFinally {
       out = serializer.newInstance().serializeStream(fileOut)
@@ -76,7 +79,8 @@ private[master] class FileSystemPersistenceEngine(
   }
 
   private def deserializeFromFile[T](file: File)(implicit m: ClassTag[T]): T = {
-    val fileIn = new FileInputStream(file)
+    var fileIn: InputStream = new FileInputStream(file)
+    codec.foreach { c => fileIn = c.compressedInputStream(new FileInputStream(file)) }
     var in: DeserializationStream = null
     try {
       in = serializer.newInstance().deserializeStream(fileIn)
diff --git a/core/src/main/scala/org/apache/spark/deploy/master/RecoveryModeFactory.scala b/core/src/main/scala/org/apache/spark/deploy/master/RecoveryModeFactory.scala
index 470798793ce..c24c4e5fe6b 100644
--- a/core/src/main/scala/org/apache/spark/deploy/master/RecoveryModeFactory.scala
+++ b/core/src/main/scala/org/apache/spark/deploy/master/RecoveryModeFactory.scala
@@ -20,7 +20,8 @@ package org.apache.spark.deploy.master
 import org.apache.spark.SparkConf
 import org.apache.spark.annotation.DeveloperApi
 import org.apache.spark.internal.Logging
-import org.apache.spark.internal.config.Deploy.RECOVERY_DIRECTORY
+import org.apache.spark.internal.config.Deploy.{RECOVERY_COMPRESSION_CODEC, RECOVERY_DIRECTORY}
+import org.apache.spark.io.CompressionCodec
 import org.apache.spark.serializer.Serializer
 
 /**
@@ -57,7 +58,8 @@ private[master] class FileSystemRecoveryModeFactory(conf: SparkConf, serializer:
 
   def createPersistenceEngine(): PersistenceEngine = {
     logInfo("Persisting recovery state to directory: " + recoveryDir)
-    new FileSystemPersistenceEngine(recoveryDir, serializer)
+    val codec = conf.get(RECOVERY_COMPRESSION_CODEC).map(c => CompressionCodec.createCodec(conf, c))
+    new FileSystemPersistenceEngine(recoveryDir, serializer, codec)
   }
 
   def createLeaderElectionAgent(master: LeaderElectable): LeaderElectionAgent = {
diff --git a/core/src/main/scala/org/apache/spark/internal/config/Deploy.scala b/core/src/main/scala/org/apache/spark/internal/config/Deploy.scala
index ae9e8b79dfe..b52ea356789 100644
--- a/core/src/main/scala/org/apache/spark/internal/config/Deploy.scala
+++ b/core/src/main/scala/org/apache/spark/internal/config/Deploy.scala
@@ -39,6 +39,13 @@ private[spark] object Deploy {
     .checkValues(RecoverySerializer.values.map(_.toString))
     .createWithDefault(RecoverySerializer.JAVA.toString)
 
+  val RECOVERY_COMPRESSION_CODEC = ConfigBuilder("spark.deploy.recoveryCompressionCodec")
+    .doc("A compression codec for persistence engines. none (default), lz4, lzf, snappy, and " +
+      "zstd. Currently, only FILESYSTEM mode supports this configuration.")
+    .version("4.0.0")
+    .stringConf
+    .createOptional
+
   val RECOVERY_MODE_FACTORY = ConfigBuilder("spark.deploy.recoveryMode.factory")
     .version("1.2.0")
     .stringConf
diff --git a/core/src/test/scala/org/apache/spark/deploy/master/MasterSuite.scala b/core/src/test/scala/org/apache/spark/deploy/master/MasterSuite.scala
index 272bcad3b19..81756d807b2 100644
--- a/core/src/test/scala/org/apache/spark/deploy/master/MasterSuite.scala
+++ b/core/src/test/scala/org/apache/spark/deploy/master/MasterSuite.scala
@@ -45,6 +45,7 @@ import org.apache.spark.internal.config._
 import org.apache.spark.internal.config.Deploy._
 import org.apache.spark.internal.config.UI._
 import org.apache.spark.internal.config.Worker._
+import org.apache.spark.io.LZ4CompressionCodec
 import org.apache.spark.resource.{ResourceInformation, ResourceProfile, ResourceRequirement}
 import org.apache.spark.resource.ResourceProfile.DEFAULT_RESOURCE_PROFILE_ID
 import org.apache.spark.resource.ResourceUtils.{FPGA, GPU}
@@ -346,6 +347,45 @@ class MasterSuite extends SparkFunSuite
     }
   }
 
+  test("SPARK-46216: Recovery without compression") {
+    val conf = new SparkConf(loadDefaults = false)
+    conf.set(RECOVERY_MODE, "FILESYSTEM")
+    conf.set(RECOVERY_DIRECTORY, System.getProperty("java.io.tmpdir"))
+
+    var master: Master = null
+    try {
+      master = makeAliveMaster(conf)
+      val e = master.invokePrivate(_persistenceEngine()).asInstanceOf[FileSystemPersistenceEngine]
+      assert(e.codec.isEmpty)
+    } finally {
+      if (master != null) {
+        master.rpcEnv.shutdown()
+        master.rpcEnv.awaitTermination()
+        master = null
+      }
+    }
+  }
+
+  test("SPARK-46216: Recovery with compression") {
+    val conf = new SparkConf(loadDefaults = false)
+    conf.set(RECOVERY_MODE, "FILESYSTEM")
+    conf.set(RECOVERY_DIRECTORY, System.getProperty("java.io.tmpdir"))
+    conf.set(RECOVERY_COMPRESSION_CODEC, "lz4")
+
+    var master: Master = null
+    try {
+      master = makeAliveMaster(conf)
+      val e = master.invokePrivate(_persistenceEngine()).asInstanceOf[FileSystemPersistenceEngine]
+      assert(e.codec.get.isInstanceOf[LZ4CompressionCodec])
+    } finally {
+      if (master != null) {
+        master.rpcEnv.shutdown()
+        master.rpcEnv.awaitTermination()
+        master = null
+      }
+    }
+  }
+
   test("master/worker web ui available") {
     implicit val formats = org.json4s.DefaultFormats
     val conf = new SparkConf()
diff --git a/core/src/test/scala/org/apache/spark/deploy/master/PersistenceEngineBenchmark.scala b/core/src/test/scala/org/apache/spark/deploy/master/PersistenceEngineBenchmark.scala
index 2f8e9a8eff2..a1df75b846d 100644
--- a/core/src/test/scala/org/apache/spark/deploy/master/PersistenceEngineBenchmark.scala
+++ b/core/src/test/scala/org/apache/spark/deploy/master/PersistenceEngineBenchmark.scala
@@ -25,6 +25,7 @@ import org.apache.curator.test.TestingServer
 import org.apache.spark.SparkConf
 import org.apache.spark.benchmark.{Benchmark, BenchmarkBase}
 import org.apache.spark.internal.config.Deploy.ZOOKEEPER_URL
+import org.apache.spark.io.CompressionCodec
 import org.apache.spark.resource.ResourceUtils.{FPGA, GPU}
 import org.apache.spark.serializer.{JavaSerializer, KryoSerializer}
 import org.apache.spark.util.Utils
@@ -52,7 +53,7 @@ object PersistenceEngineBenchmark extends BenchmarkBase {
   override def runBenchmarkSuite(mainArgs: Array[String]): Unit = {
 
     val numIters = 3
-    val numWorkers = 1000
+    val numWorkers = 2000
     val workers = (1 to numWorkers).map(createWorkerInfo).toArray
 
     conf.set(ZOOKEEPER_URL, zkTestServer.getConnectString)
@@ -73,7 +74,8 @@ object PersistenceEngineBenchmark extends BenchmarkBase {
 
       serializers.foreach { serializer =>
         val serializerName = serializer.getClass.getSimpleName
-        benchmark.addCase(s"FileSystemPersistenceEngine with $serializerName", numIters) { _ =>
+        val name = s"FileSystemPersistenceEngine with $serializerName"
+        benchmark.addCase(name, numIters) { _ =>
           val dir = Utils.createTempDir().getAbsolutePath
           val engine = new FileSystemPersistenceEngine(dir, serializer)
           workers.foreach(engine.addWorker)
@@ -81,6 +83,19 @@ object PersistenceEngineBenchmark extends BenchmarkBase {
           workers.foreach(engine.removeWorker)
           engine.close()
         }
+        CompressionCodec.ALL_COMPRESSION_CODECS.foreach { c =>
+          val codec = CompressionCodec.createCodec(conf, c)
+          val shortCodecName = CompressionCodec.getShortName(c)
+          val name = s"FileSystemPersistenceEngine with $serializerName ($shortCodecName)"
+          benchmark.addCase(name, numIters) { _ =>
+            val dir = Utils.createTempDir().getAbsolutePath
+            val engine = new FileSystemPersistenceEngine(dir, serializer, Some(codec))
+            workers.foreach(engine.addWorker)
+            engine.read[WorkerInfo]("worker_")
+            workers.foreach(engine.removeWorker)
+            engine.close()
+          }
+        }
       }
 
       benchmark.addCase("BlackHolePersistenceEngine", numIters) { _ =>
diff --git a/core/src/test/scala/org/apache/spark/deploy/master/PersistenceEngineSuite.scala b/core/src/test/scala/org/apache/spark/deploy/master/PersistenceEngineSuite.scala
index 8546f4e01f3..c02b84fc1a6 100644
--- a/core/src/test/scala/org/apache/spark/deploy/master/PersistenceEngineSuite.scala
+++ b/core/src/test/scala/org/apache/spark/deploy/master/PersistenceEngineSuite.scala
@@ -25,6 +25,7 @@ import org.apache.curator.test.TestingServer
 
 import org.apache.spark.{SecurityManager, SparkConf, SparkFunSuite}
 import org.apache.spark.internal.config.Deploy.ZOOKEEPER_URL
+import org.apache.spark.io.CompressionCodec
 import org.apache.spark.rpc.{RpcEndpoint, RpcEnv}
 import org.apache.spark.serializer.{JavaSerializer, KryoSerializer, Serializer}
 import org.apache.spark.util.Utils
@@ -74,6 +75,18 @@ class PersistenceEngineSuite extends SparkFunSuite {
     }
   }
 
+  test("SPARK-46216: FileSystemPersistenceEngine with compression") {
+    val conf = new SparkConf()
+    CompressionCodec.ALL_COMPRESSION_CODECS.foreach { c =>
+      val codec = CompressionCodec.createCodec(conf, c)
+      withTempDir { dir =>
+        testPersistenceEngine(conf, serializer =>
+          new FileSystemPersistenceEngine(dir.getAbsolutePath, serializer, Some(codec))
+        )
+      }
+    }
+  }
+
   test("ZooKeeperPersistenceEngine") {
     val conf = new SparkConf()
     // TestingServer logs the port conflict exception rather than throwing an exception.


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org