You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by GitBox <gi...@apache.org> on 2021/05/18 14:55:23 UTC
[GitHub] [spark] xuanyuanking opened a new pull request #32582: [WIP][SPARK-35436] RocksDBFileManager - save checkpoint to DFS
xuanyuanking opened a new pull request #32582:
URL: https://github.com/apache/spark/pull/32582
### What changes were proposed in this pull request?
The implementation for the save operation of RocksDBFileManager.
### Why are the changes needed?
Save all the files in the given local checkpoint directory as a committed version in DFS.
### Does this PR introduce _any_ user-facing change?
No.
### How was this patch tested?
New UT added.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32582: [SPARK-35436][SS] RocksDBFileManager - save checkpoint to DFS
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32582:
URL: https://github.com/apache/spark/pull/32582#issuecomment-850087350
**[Test build #139041 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/139041/testReport)** for PR 32582 at commit [`703f59f`](https://github.com/apache/spark/commit/703f59fd4055d68e5bf957e1dc4f17159256a65a).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HeartSaVioR commented on pull request #32582: [SPARK-35436][SS] RocksDBFileManager - save checkpoint to DFS
Posted by GitBox <gi...@apache.org>.
HeartSaVioR commented on pull request #32582:
URL: https://github.com/apache/spark/pull/32582#issuecomment-857377223
Thanks! Merging to master.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32582: [SPARK-35436][SS] RocksDBFileManager - save checkpoint to DFS
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32582:
URL: https://github.com/apache/spark/pull/32582#issuecomment-853022843
**[Test build #139221 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/139221/testReport)** for PR 32582 at commit [`3800c51`](https://github.com/apache/spark/commit/3800c5173f056b6b63abf4865a6bdf9da16136d5).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] viirya commented on a change in pull request #32582: [SPARK-35436][SS] RocksDBFileManager - save checkpoint to DFS
Posted by GitBox <gi...@apache.org>.
viirya commented on a change in pull request #32582:
URL: https://github.com/apache/spark/pull/32582#discussion_r645326904
##########
File path: sql/core/src/test/scala/org/apache/spark/sql/execution/streaming/state/RocksDBSuite.scala
##########
@@ -20,12 +20,88 @@ package org.apache.spark.sql.execution.streaming.state
import java.io._
import java.nio.charset.Charset
+import scala.language.implicitConversions
+
import org.apache.commons.io.FileUtils
+import org.apache.hadoop.conf.Configuration
import org.apache.spark._
+import org.apache.spark.sql.catalyst.util.quietly
+import org.apache.spark.sql.execution.streaming.CreateAtomicTestManager
+import org.apache.spark.sql.internal.SQLConf
+import org.apache.spark.util.Utils
class RocksDBSuite extends SparkFunSuite {
+ test("RocksDBFileManager: upload only new immutable files") {
+ withTempDir { dir =>
+ val dfsRootDir = dir.getAbsolutePath
+ val fileManager = new RocksDBFileManager(
+ dfsRootDir, Utils.createTempDir(), new Configuration)
+ val sstDir = s"$dfsRootDir/SSTs"
+ def numRemoteSSTFiles: Int = listFiles(sstDir).length
+ val logDir = s"$dfsRootDir/logs"
+ def numRemoteLogFiles: Int = listFiles(logDir).length
+
+ // Save a version of checkpoint files
+ val cpFiles1 = Seq(
+ "sst-file1.sst" -> 10,
+ "sst-file2.sst" -> 20,
+ "other-file1" -> 100,
+ "other-file2" -> 200,
+ "archive/00001.log" -> 1000,
+ "archive/00002.log" -> 2000
+ )
+ saveCheckpointFiles(fileManager, cpFiles1, version = 1, numKeys = 101)
+ assert(numRemoteSSTFiles == 2) // 2 sst files copied
+ assert(numRemoteLogFiles == 2) // 2 log files copied
+
+ // Save SAME version again with different checkpoint files and verify
+ val cpFiles1_ = Seq(
+ "sst-file1.sst" -> 10, // same SST file as before, should not get copied
+ "sst-file2.sst" -> 25, // new SST file with same name as before, but different length
+ "sst-file3.sst" -> 30, // new SST file
+ "other-file1" -> 100, // same non-SST file as before, should not get copied
+ "other-file2" -> 210, // new non-SST file with same name as before, but different length
+ "other-file3" -> 300, // new non-SST file
+ "archive/00001.log" -> 1000, // same log file as before, should not get copied
+ "archive/00002.log" -> 2500, // new log file with same name as before, but different length
+ "archive/00003.log" -> 3000 // new log file
+ )
+ saveCheckpointFiles(fileManager, cpFiles1_, version = 1, numKeys = 1001)
+ assert(numRemoteSSTFiles === 4, "shouldn't copy same files again") // 2 old + 2 new SST files
+ assert(numRemoteLogFiles === 4, "shouldn't copy same files again") // 2 old + 2 new log files
+
+ // Save another version and verify
+ val cpFiles2 = Seq(
+ "sst-file4.sst" -> 40,
+ "other-file4" -> 400,
+ "archive/00004.log" -> 4000
+ )
+ saveCheckpointFiles(fileManager, cpFiles2, version = 2, numKeys = 1501)
+ assert(numRemoteSSTFiles === 5) // 1 new file over earlier 4 files
+ assert(numRemoteLogFiles === 5) // 1 new file over earlier 4 files
+ }
+ }
+
+ test("RocksDBFileManager: error writing [version].zip cancels the output stream") {
+ quietly {
+ val hadoopConf = new Configuration()
+ hadoopConf.set(
+ SQLConf.STREAMING_CHECKPOINT_FILE_MANAGER_CLASS.parent.key,
+ classOf[CreateAtomicTestManager].getName)
+ val dfsRootDir = Utils.createTempDir().getAbsolutePath
+ val fileManager = new RocksDBFileManager(dfsRootDir, Utils.createTempDir(), hadoopConf)
+ val cpFiles = Seq("sst-file1.sst" -> 10, "sst-file2.sst" -> 20, "other-file1" -> 100)
+ CreateAtomicTestManager.shouldFailInCreateAtomic = true
+ CreateAtomicTestManager.cancelCalledInCreateAtomic = false
+ intercept[IOException] {
+ saveCheckpointFiles(fileManager, cpFiles, version = 1, numKeys = 101)
+ }
Review comment:
I'm not sure why this will fail?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #32582: [SPARK-35436][SS] RocksDBFileManager - save checkpoint to DFS
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #32582:
URL: https://github.com/apache/spark/pull/32582#issuecomment-853863870
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/139285/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #32582: [SPARK-35436][SS] RocksDBFileManager - save checkpoint to DFS
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #32582:
URL: https://github.com/apache/spark/pull/32582#issuecomment-853850149
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #32582: [SPARK-35436][SS] RocksDBFileManager - save checkpoint to DFS
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #32582:
URL: https://github.com/apache/spark/pull/32582#issuecomment-857361732
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/139516/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32582: [SPARK-35436][SS] RocksDBFileManager - save checkpoint to DFS
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32582:
URL: https://github.com/apache/spark/pull/32582#issuecomment-853827082
Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/43809/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #32582: [SPARK-35436][SS] RocksDBFileManager - save checkpoint to DFS
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #32582:
URL: https://github.com/apache/spark/pull/32582#issuecomment-850124238
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/43559/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #32582: [SPARK-35436][SS] RocksDBFileManager - save checkpoint to DFS
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #32582:
URL: https://github.com/apache/spark/pull/32582#issuecomment-857385133
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/44054/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32582: [SPARK-35436][SS] RocksDBFileManager - save checkpoint to DFS
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32582:
URL: https://github.com/apache/spark/pull/32582#issuecomment-850164095
**[Test build #139041 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/139041/testReport)** for PR 32582 at commit [`703f59f`](https://github.com/apache/spark/commit/703f59fd4055d68e5bf957e1dc4f17159256a65a).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds the following public classes _(experimental)_:
* `class RocksDBFileManager(`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32582: [SPARK-35436][SS] RocksDBFileManager - save checkpoint to DFS
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32582:
URL: https://github.com/apache/spark/pull/32582#issuecomment-857301345
**[Test build #139516 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/139516/testReport)** for PR 32582 at commit [`f86cea4`](https://github.com/apache/spark/commit/f86cea4fb9c75d3dc29ec83d6f05184431d8584e).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] xuanyuanking commented on pull request #32582: [SPARK-35436][SS] RocksDBFileManager - save checkpoint to DFS
Posted by GitBox <gi...@apache.org>.
xuanyuanking commented on pull request #32582:
URL: https://github.com/apache/spark/pull/32582#issuecomment-853535338
```
It would be nice if we can do the same we did for reviewing the first PR; if remaining review comments are planned to be addressed via further PR, please raise the PR so that we can refer the PR and finally approve this one.
```
Agree, let me submit the 3rd PR today.
The failure tests are not related, they can pass locally.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32582: [SPARK-35436][SS] RocksDBFileManager - save checkpoint to DFS
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32582:
URL: https://github.com/apache/spark/pull/32582#issuecomment-850121547
Kubernetes integration test unable to build dist.
exiting with code: 1
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/43559/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] xuanyuanking commented on a change in pull request #32582: [SPARK-35436][SS] RocksDBFileManager - save checkpoint to DFS
Posted by GitBox <gi...@apache.org>.
xuanyuanking commented on a change in pull request #32582:
URL: https://github.com/apache/spark/pull/32582#discussion_r645673106
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/state/RocksDBFileManager.scala
##########
@@ -17,18 +17,270 @@
package org.apache.spark.sql.execution.streaming.state
-import java.io.File
+import java.io.{File, FileInputStream, InputStream}
import java.nio.charset.StandardCharsets.UTF_8
import java.nio.file.Files
+import java.util.UUID
+import java.util.concurrent.ConcurrentHashMap
+import java.util.zip.{ZipEntry, ZipOutputStream}
+import scala.collection.JavaConverters._
import scala.collection.Seq
import com.fasterxml.jackson.annotation.JsonInclude.Include
import com.fasterxml.jackson.databind.{DeserializationFeature, ObjectMapper}
import com.fasterxml.jackson.module.scala.{DefaultScalaModule, ScalaObjectMapper}
+import org.apache.commons.io.{FilenameUtils, IOUtils}
+import org.apache.hadoop.conf.Configuration
+import org.apache.hadoop.fs.Path
import org.json4s.NoTypeHints
import org.json4s.jackson.Serialization
+import org.apache.spark.internal.Logging
+import org.apache.spark.sql.execution.streaming.CheckpointFileManager
+import org.apache.spark.util.Utils
+
+/**
+ * Class responsible for syncing RocksDB checkpoint files from local disk to DFS.
+ * Every version checkpoint is saved in specific directory structure that allows successive
+ * versions to reuse to SST data files and archived log files. This allows each commit to be
+ * incremental, only new SST files and archived log files generated by RocksDB will be uploaded.
+ * The directory structures on local disk and in DFS are as follows.
+ *
+ * Local checkpoint dir structure
+ * ------------------------------
+ * RocksDB generates a bunch of files in the local checkpoint directory. The most important among
+ * them are the SST files; they are the actual log structured data files. Rest of the files contain
+ * the metadata necessary for RocksDB to read the SST files and start from the checkpoint.
+ * Note that the SST files are hard links to files in the RocksDB's working directory, and therefore
+ * successive checkpoints can share some of the SST files. So these SST files have to be copied to
+ * DFS in shared directory such that different committed versions can save them.
+ *
+ * We consider both SST files and archived log files as immutable files which can be shared between
+ * different checkpoints.
+ *
+ * localCheckpointDir
+ * |
+ * +-- OPTIONS-000005
+ * +-- MANIFEST-000008
+ * +-- CURRENT
+ * +-- 00007.sst
+ * +-- 00011.sst
+ * +-- archive
+ * | +-- 00008.log
+ * | +-- 00013.log
+ * ...
+ *
+ *
+ * DFS directory structure after saving to DFS as version 10
+ * -----------------------------------------------------------
+ * The SST and archived log files are given unique file names and copied to the shared subdirectory.
+ * Every version maintains a mapping of local immutable file name to the unique file name in DFS.
+ * This mapping is saved in a JSON file (named `metadata`), which is zipped along with other
+ * checkpoint files into a single file `[version].zip`.
+ *
+ * dfsRootDir
+ * |
+ * +-- SSTs
+ * | +-- 00007-[uuid1].sst
+ * | +-- 00011-[uuid2].sst
+ * +-- logs
+ * | +-- 00008-[uuid3].log
+ * | +-- 00013-[uuid4].log
+ * +-- 10.zip
+ * | +-- metadata <--- contains mapping between 00007.sst and [uuid1].sst,
+ * and the mapping between 00008.log and [uuid3].log
+ * | +-- OPTIONS-000005
+ * | +-- MANIFEST-000008
+ * | +-- CURRENT
+ * | ...
+ * |
+ * +-- 9.zip
+ * +-- 8.zip
+ * ...
+ *
+ * Note the following.
+ * - Each [version].zip is a complete description of all the data and metadata needed to recover
+ * a RocksDB instance at the corresponding version. This is unlike the [version].delta files of
+ * HDFSBackedStateStore where previous delta files needs to be read to be recovered.
+ * - This is safe wrt speculatively executed tasks running concurrently in different executors
+ * as each task would upload a different copy of the generated immutable files and
+ * atomically update the [version].zip.
+ * - Immutable files are identified uniquely based on their file name and file size.
+ * - Immutable files can be reused only across adjacent checkpoints/versions.
+ * - This class is thread-safe. Specifically, it is safe to concurrently delete old files from a
+ * different thread than the task thread saving files.
+ *
+ * @param dfsRootDir Directory where the [version].zip files will be stored
+ * @param localTempDir Local directory for temporary work
+ * @param hadoopConf Hadoop configuration for talking to DFS
+ * @param loggingId Id that will be prepended in logs for isolating concurrent RocksDBs
+ */
+class RocksDBFileManager(
+ dfsRootDir: String,
+ localTempDir: File,
+ hadoopConf: Configuration,
+ loggingId: String = "")
+ extends Logging {
+
+ import RocksDBImmutableFile._
+
+ private val versionToRocksDBFiles = new ConcurrentHashMap[Long, Seq[RocksDBImmutableFile]]
+ private lazy val fm = CheckpointFileManager.create(new Path(dfsRootDir), hadoopConf)
+ private val fs = new Path(dfsRootDir).getFileSystem(hadoopConf)
+
+ /** Save all the files in given local checkpoint directory as a committed version in DFS */
+ def saveCheckpointToDfs(checkpointDir: File, version: Long, numKeys: Long): Unit = {
+ logFilesInDir(checkpointDir, s"Saving checkpoint files for version $version")
+ val allCheckpointFiles = listRocksDBFiles(checkpointDir)
+ val (localImmutableFiles, localOtherFiles) = allCheckpointFiles.partition(isImmutableFile)
+ val rocksDBFiles = saveImmutableFilesToDfs(version, checkpointDir, localImmutableFiles)
+ val metadata = RocksDBCheckpointMetadata(rocksDBFiles, numKeys)
+ rememberImmutableFiles(version, rocksDBFiles)
+ val metadataFile = localMetadataFile(checkpointDir)
+ metadata.writeToFile(metadataFile)
+ logInfo(s"Written metadata for version $version:\n${metadata.prettyJson}")
+
+ if (version <= 0 && numKeys == 0) {
Review comment:
Yeah, let's change the version to 1 for avoiding the extra cost of file operations. Will change it in the next commit. Done in bdd9e8e
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] viirya commented on a change in pull request #32582: [SPARK-35436][SS] RocksDBFileManager - save checkpoint to DFS
Posted by GitBox <gi...@apache.org>.
viirya commented on a change in pull request #32582:
URL: https://github.com/apache/spark/pull/32582#discussion_r645325536
##########
File path: sql/core/src/test/scala/org/apache/spark/sql/execution/streaming/state/RocksDBSuite.scala
##########
@@ -20,12 +20,88 @@ package org.apache.spark.sql.execution.streaming.state
import java.io._
import java.nio.charset.Charset
+import scala.language.implicitConversions
+
import org.apache.commons.io.FileUtils
+import org.apache.hadoop.conf.Configuration
import org.apache.spark._
+import org.apache.spark.sql.catalyst.util.quietly
+import org.apache.spark.sql.execution.streaming.CreateAtomicTestManager
+import org.apache.spark.sql.internal.SQLConf
+import org.apache.spark.util.Utils
class RocksDBSuite extends SparkFunSuite {
+ test("RocksDBFileManager: upload only new immutable files") {
+ withTempDir { dir =>
+ val dfsRootDir = dir.getAbsolutePath
+ val fileManager = new RocksDBFileManager(
+ dfsRootDir, Utils.createTempDir(), new Configuration)
+ val sstDir = s"$dfsRootDir/SSTs"
+ def numRemoteSSTFiles: Int = listFiles(sstDir).length
+ val logDir = s"$dfsRootDir/logs"
+ def numRemoteLogFiles: Int = listFiles(logDir).length
+
+ // Save a version of checkpoint files
+ val cpFiles1 = Seq(
+ "sst-file1.sst" -> 10,
+ "sst-file2.sst" -> 20,
+ "other-file1" -> 100,
+ "other-file2" -> 200,
+ "archive/00001.log" -> 1000,
+ "archive/00002.log" -> 2000
+ )
+ saveCheckpointFiles(fileManager, cpFiles1, version = 1, numKeys = 101)
+ assert(numRemoteSSTFiles == 2) // 2 sst files copied
+ assert(numRemoteLogFiles == 2) // 2 log files copied
+
+ // Save SAME version again with different checkpoint files and verify
+ val cpFiles1_ = Seq(
+ "sst-file1.sst" -> 10, // same SST file as before, should not get copied
+ "sst-file2.sst" -> 25, // new SST file with same name as before, but different length
+ "sst-file3.sst" -> 30, // new SST file
+ "other-file1" -> 100, // same non-SST file as before, should not get copied
+ "other-file2" -> 210, // new non-SST file with same name as before, but different length
+ "other-file3" -> 300, // new non-SST file
+ "archive/00001.log" -> 1000, // same log file as before, should not get copied
+ "archive/00002.log" -> 2500, // new log file with same name as before, but different length
+ "archive/00003.log" -> 3000 // new log file
+ )
+ saveCheckpointFiles(fileManager, cpFiles1_, version = 1, numKeys = 1001)
+ assert(numRemoteSSTFiles === 4, "shouldn't copy same files again") // 2 old + 2 new SST files
+ assert(numRemoteLogFiles === 4, "shouldn't copy same files again") // 2 old + 2 new log files
+
+ // Save another version and verify
+ val cpFiles2 = Seq(
+ "sst-file4.sst" -> 40,
+ "other-file4" -> 400,
+ "archive/00004.log" -> 4000
+ )
+ saveCheckpointFiles(fileManager, cpFiles2, version = 2, numKeys = 1501)
+ assert(numRemoteSSTFiles === 5) // 1 new file over earlier 4 files
+ assert(numRemoteLogFiles === 5) // 1 new file over earlier 4 files
+ }
+ }
+
+ test("RocksDBFileManager: error writing [version].zip cancels the output stream") {
+ quietly {
+ val hadoopConf = new Configuration()
+ hadoopConf.set(
+ SQLConf.STREAMING_CHECKPOINT_FILE_MANAGER_CLASS.parent.key,
+ classOf[CreateAtomicTestManager].getName)
+ val dfsRootDir = Utils.createTempDir().getAbsolutePath
Review comment:
Use `withTempDir` too?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] xuanyuanking commented on a change in pull request #32582: [SPARK-35436][SS] RocksDBFileManager - save checkpoint to DFS
Posted by GitBox <gi...@apache.org>.
xuanyuanking commented on a change in pull request #32582:
URL: https://github.com/apache/spark/pull/32582#discussion_r643850985
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/state/RocksDBFileManager.scala
##########
@@ -17,18 +17,270 @@
package org.apache.spark.sql.execution.streaming.state
-import java.io.File
+import java.io.{File, FileInputStream, InputStream}
import java.nio.charset.StandardCharsets.UTF_8
import java.nio.file.Files
+import java.util.UUID
+import java.util.concurrent.ConcurrentHashMap
+import java.util.zip.{ZipEntry, ZipOutputStream}
+import scala.collection.JavaConverters._
import scala.collection.Seq
import com.fasterxml.jackson.annotation.JsonInclude.Include
import com.fasterxml.jackson.databind.{DeserializationFeature, ObjectMapper}
import com.fasterxml.jackson.module.scala.{DefaultScalaModule, ScalaObjectMapper}
+import org.apache.commons.io.{FilenameUtils, IOUtils}
+import org.apache.hadoop.conf.Configuration
+import org.apache.hadoop.fs.Path
import org.json4s.NoTypeHints
import org.json4s.jackson.Serialization
+import org.apache.spark.internal.Logging
+import org.apache.spark.sql.execution.streaming.CheckpointFileManager
+import org.apache.spark.util.Utils
+
+/**
+ * Class responsible for syncing RocksDB checkpoint files from local disk to DFS.
+ * Every version checkpoint is saved in specific directory structure that allows successive
+ * versions to reuse to SST data files and archived log files. This allows each commit to be
+ * incremental, only new SST files and archived log files generated by RocksDB will be uploaded.
+ * The directory structures on local disk and in DFS are as follows.
+ *
+ * Local checkpoint dir structure
+ * ------------------------------
+ * RocksDB generates a bunch of files in the local checkpoint directory. The most important among
+ * them are the SST files; they are the actual log structured data files. Rest of the files contain
+ * the metadata necessary for RocksDB to read the SST files and start from the checkpoint.
+ * Note that the SST files are hard links to files in the RocksDB's working directory, and therefore
+ * successive checkpoints can share some of the SST files. So these SST files have to be copied to
+ * DFS in shared directory such that different committed versions can save them.
+ *
+ * We consider both SST files and archived log files as immutable files which can be shared between
+ * different checkpoints.
+ *
+ * localCheckpointDir
+ * |
+ * +-- OPTIONS-000005
+ * +-- MANIFEST-000008
+ * +-- CURRENT
+ * +-- 00007.sst
+ * +-- 00011.sst
+ * +-- archive
+ * | +-- 00008.log
+ * | +-- 00013.log
+ * ...
+ *
+ *
+ * DFS directory structure after saving to DFS as version 10
+ * -----------------------------------------------------------
+ * The SST and archived log files are given unique file names and copied to the shared subdirectory.
+ * Every version maintains a mapping of local immutable file name to the unique file name in DFS.
+ * This mapping is saved in a JSON file (named `metadata`), which is zipped along with other
+ * checkpoint files into a single file `[version].zip`.
+ *
+ * dfsRootDir
+ * |
+ * +-- SSTs
+ * | +-- 00007-[uuid1].sst
+ * | +-- 00011-[uuid2].sst
+ * +-- logs
+ * | +-- 00008-[uuid3].log
+ * | +-- 00013-[uuid4].log
+ * +-- 10.zip
+ * | +-- metadata <--- contains mapping between 00007.sst and [uuid1].sst,
+ * and the mapping between 00008.log and [uuid3].log
+ * | +-- OPTIONS-000005
+ * | +-- MANIFEST-000008
+ * | +-- CURRENT
+ * | ...
+ * |
+ * +-- 9.zip
+ * +-- 8.zip
+ * ...
+ *
+ * Note the following.
+ * - Each [version].zip is a complete description of all the data and metadata needed to recover
+ * a RocksDB instance at the corresponding version. This is unlike the [version].delta files of
+ * HDFSBackedStateStore where previous delta files needs to be read to be recovered.
+ * - This is safe wrt speculatively executed tasks running concurrently in different executors
+ * as each task would upload a different copy of the generated immutable files and
+ * atomically update the [version].zip.
+ * - Immutable files are identified uniquely based on their file name and file size.
+ * - Immutable files can be reused only across adjacent checkpoints/versions.
+ * - This class is thread-safe. Specifically, it is safe to concurrently delete old files from a
+ * different thread than the task thread saving files.
+ *
+ * @param dfsRootDir Directory where the [version].zip files will be stored
+ * @param localTempDir Local directory for temporary work
+ * @param hadoopConf Hadoop configuration for talking to DFS
+ * @param loggingId Id that will be prepended in logs for isolating concurrent RocksDBs
+ */
+class RocksDBFileManager(
+ dfsRootDir: String,
+ localTempDir: File,
+ hadoopConf: Configuration,
+ loggingId: String = "")
+ extends Logging {
+
+ import RocksDBImmutableFile._
+
+ private val versionToRocksDBFiles = new ConcurrentHashMap[Long, Seq[RocksDBImmutableFile]]
Review comment:
That's right. `versionToRocksDBFiles` was touched in the following 3 places:
- saveCheckpointToDfs (this PR)
- deleteOldVersions
- loadCheckpointFromDfs
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/state/RocksDBFileManager.scala
##########
@@ -17,18 +17,270 @@
package org.apache.spark.sql.execution.streaming.state
-import java.io.File
+import java.io.{File, FileInputStream, InputStream}
import java.nio.charset.StandardCharsets.UTF_8
import java.nio.file.Files
+import java.util.UUID
+import java.util.concurrent.ConcurrentHashMap
+import java.util.zip.{ZipEntry, ZipOutputStream}
+import scala.collection.JavaConverters._
import scala.collection.Seq
import com.fasterxml.jackson.annotation.JsonInclude.Include
import com.fasterxml.jackson.databind.{DeserializationFeature, ObjectMapper}
import com.fasterxml.jackson.module.scala.{DefaultScalaModule, ScalaObjectMapper}
+import org.apache.commons.io.{FilenameUtils, IOUtils}
+import org.apache.hadoop.conf.Configuration
+import org.apache.hadoop.fs.Path
import org.json4s.NoTypeHints
import org.json4s.jackson.Serialization
+import org.apache.spark.internal.Logging
+import org.apache.spark.sql.execution.streaming.CheckpointFileManager
+import org.apache.spark.util.Utils
+
+/**
+ * Class responsible for syncing RocksDB checkpoint files from local disk to DFS.
+ * Every version checkpoint is saved in specific directory structure that allows successive
+ * versions to reuse to SST data files and archived log files. This allows each commit to be
+ * incremental, only new SST files and archived log files generated by RocksDB will be uploaded.
+ * The directory structures on local disk and in DFS are as follows.
+ *
+ * Local checkpoint dir structure
+ * ------------------------------
+ * RocksDB generates a bunch of files in the local checkpoint directory. The most important among
+ * them are the SST files; they are the actual log structured data files. Rest of the files contain
+ * the metadata necessary for RocksDB to read the SST files and start from the checkpoint.
+ * Note that the SST files are hard links to files in the RocksDB's working directory, and therefore
+ * successive checkpoints can share some of the SST files. So these SST files have to be copied to
+ * DFS in shared directory such that different committed versions can save them.
+ *
+ * We consider both SST files and archived log files as immutable files which can be shared between
+ * different checkpoints.
+ *
+ * localCheckpointDir
+ * |
+ * +-- OPTIONS-000005
+ * +-- MANIFEST-000008
+ * +-- CURRENT
+ * +-- 00007.sst
+ * +-- 00011.sst
+ * +-- archive
+ * | +-- 00008.log
+ * | +-- 00013.log
+ * ...
+ *
+ *
+ * DFS directory structure after saving to DFS as version 10
+ * -----------------------------------------------------------
+ * The SST and archived log files are given unique file names and copied to the shared subdirectory.
+ * Every version maintains a mapping of local immutable file name to the unique file name in DFS.
+ * This mapping is saved in a JSON file (named `metadata`), which is zipped along with other
+ * checkpoint files into a single file `[version].zip`.
+ *
+ * dfsRootDir
+ * |
+ * +-- SSTs
+ * | +-- 00007-[uuid1].sst
+ * | +-- 00011-[uuid2].sst
+ * +-- logs
+ * | +-- 00008-[uuid3].log
+ * | +-- 00013-[uuid4].log
+ * +-- 10.zip
+ * | +-- metadata <--- contains mapping between 00007.sst and [uuid1].sst,
+ * and the mapping between 00008.log and [uuid3].log
+ * | +-- OPTIONS-000005
+ * | +-- MANIFEST-000008
+ * | +-- CURRENT
+ * | ...
+ * |
+ * +-- 9.zip
+ * +-- 8.zip
+ * ...
+ *
+ * Note the following.
+ * - Each [version].zip is a complete description of all the data and metadata needed to recover
+ * a RocksDB instance at the corresponding version. This is unlike the [version].delta files of
+ * HDFSBackedStateStore where previous delta files needs to be read to be recovered.
+ * - This is safe wrt speculatively executed tasks running concurrently in different executors
+ * as each task would upload a different copy of the generated immutable files and
+ * atomically update the [version].zip.
+ * - Immutable files are identified uniquely based on their file name and file size.
+ * - Immutable files can be reused only across adjacent checkpoints/versions.
+ * - This class is thread-safe. Specifically, it is safe to concurrently delete old files from a
+ * different thread than the task thread saving files.
+ *
+ * @param dfsRootDir Directory where the [version].zip files will be stored
+ * @param localTempDir Local directory for temporary work
+ * @param hadoopConf Hadoop configuration for talking to DFS
+ * @param loggingId Id that will be prepended in logs for isolating concurrent RocksDBs
+ */
+class RocksDBFileManager(
+ dfsRootDir: String,
+ localTempDir: File,
+ hadoopConf: Configuration,
+ loggingId: String = "")
+ extends Logging {
+
+ import RocksDBImmutableFile._
+
+ private val versionToRocksDBFiles = new ConcurrentHashMap[Long, Seq[RocksDBImmutableFile]]
+ private lazy val fm = CheckpointFileManager.create(new Path(dfsRootDir), hadoopConf)
+ private val fs = new Path(dfsRootDir).getFileSystem(hadoopConf)
+
+ /** Save all the files in given local checkpoint directory as a committed version in DFS */
+ def saveCheckpointToDfs(checkpointDir: File, version: Long, numKeys: Long): Unit = {
+ logFilesInDir(checkpointDir, s"Saving checkpoint files for version $version")
+ val allCheckpointFiles = listRocksDBFiles(checkpointDir)
+ val (localImmutableFiles, localOtherFiles) = allCheckpointFiles.partition(isImmutableFile)
+ val rocksDBFiles = saveImmutableFilesToDfs(version, checkpointDir, localImmutableFiles)
+ val metadata = RocksDBCheckpointMetadata(rocksDBFiles, numKeys)
+ rememberImmutableFiles(version, rocksDBFiles)
+ val metadataFile = localMetadataFile(checkpointDir)
+ metadata.writeToFile(metadataFile)
+ logInfo(s"Written metadata for version $version:\n${metadata.prettyJson}")
+
+ if (version <= 0 && numKeys == 0) {
Review comment:
We also use batch ID + 1 as the version for the RocksDB state store. Since the batch ID starts from -1 so for the corner case of the first empty batch (batch ID 0), we add the safeguard here to make sure the working dir is created.
##########
File path: sql/core/src/test/scala/org/apache/spark/sql/execution/streaming/state/RocksDBSuite.scala
##########
@@ -54,4 +128,31 @@ class RocksDBSuite extends SparkFunSuite {
"""{"sstFiles":[{"localFileName":"00001.sst","dfsSstFileName":"00001-uuid.sst","sizeBytes":12345678901234}],"logFiles":[{"localFileName":"00001.log","dfsLogFileName":"00001-uuid.log","sizeBytes":12345678901234}],"numKeys":12345678901234}""")
// scalastyle:on line.size.limit
}
+
+
+ def generateFiles(dir: String, fileToLengths: Seq[(String, Int)]): Unit = {
+ fileToLengths.foreach { case (fileName, length) =>
+ val file = new File(dir, fileName)
+ FileUtils.write(file, "a" * length)
+ }
+ }
+
+ def saveCheckpointFiles(
+ fileManager: RocksDBFileManager,
+ fileToLengths: Seq[(String, Int)],
+ version: Int,
+ numKeys: Int): Unit = {
+ val checkpointDir = Utils.createTempDir().getAbsolutePath // local dir to create checkpoints
+ generateFiles(checkpointDir, fileToLengths)
+ fileManager.saveCheckpointToDfs(checkpointDir, version, numKeys)
+ }
+
+ implicit def toFile(path: String): File = new File(path)
Review comment:
That's right in this PR. In the further PRs, we will have more caller for the `listFiles(file: String)`. Maybe let's keep it for now.
##########
File path: sql/core/src/test/scala/org/apache/spark/sql/execution/streaming/state/RocksDBSuite.scala
##########
@@ -20,12 +20,86 @@ package org.apache.spark.sql.execution.streaming.state
import java.io._
import java.nio.charset.Charset
+import scala.language.implicitConversions
+
import org.apache.commons.io.FileUtils
+import org.apache.hadoop.conf.Configuration
import org.apache.spark._
+import org.apache.spark.sql.catalyst.util.quietly
+import org.apache.spark.sql.execution.streaming.CreateAtomicTestManager
+import org.apache.spark.sql.internal.SQLConf
+import org.apache.spark.util.Utils
class RocksDBSuite extends SparkFunSuite {
+ test("RocksDBFileManager: upload only new immutable files") {
+ val dfsRootDir = Utils.createTempDir().getAbsolutePath
+ val fileManager = new RocksDBFileManager(
+ dfsRootDir, Utils.createTempDir(), new Configuration)
+ val sstDir = s"$dfsRootDir/SSTs"
+ def numRemoteSSTFiles: Int = listFiles(sstDir).length
+ val logDir = s"$dfsRootDir/logs"
+ def numRemoteLogFiles: Int = listFiles(logDir).length
Review comment:
We'll enhance this test in the next PR, `RocksDBFileManager - load checkpoint from DFS`, by load back the metadata and check the other files. Let me keep this conversation open and be referenced in the next PR.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32582: [SPARK-35436][SS] RocksDBFileManager - save checkpoint to DFS
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32582:
URL: https://github.com/apache/spark/pull/32582#issuecomment-852961517
Kubernetes integration test unable to build dist.
exiting with code: 1
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/43744/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #32582: [SPARK-35436][SS] RocksDBFileManager - save checkpoint to DFS
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #32582:
URL: https://github.com/apache/spark/pull/32582#issuecomment-857342024
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/44041/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #32582: [SPARK-35436][SS] RocksDBFileManager - save checkpoint to DFS
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #32582:
URL: https://github.com/apache/spark/pull/32582#issuecomment-854910753
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/43862/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] viirya commented on a change in pull request #32582: [SPARK-35436][SS] RocksDBFileManager - save checkpoint to DFS
Posted by GitBox <gi...@apache.org>.
viirya commented on a change in pull request #32582:
URL: https://github.com/apache/spark/pull/32582#discussion_r645325161
##########
File path: sql/core/src/test/scala/org/apache/spark/sql/execution/streaming/state/RocksDBSuite.scala
##########
@@ -20,12 +20,88 @@ package org.apache.spark.sql.execution.streaming.state
import java.io._
import java.nio.charset.Charset
+import scala.language.implicitConversions
+
import org.apache.commons.io.FileUtils
+import org.apache.hadoop.conf.Configuration
import org.apache.spark._
+import org.apache.spark.sql.catalyst.util.quietly
+import org.apache.spark.sql.execution.streaming.CreateAtomicTestManager
+import org.apache.spark.sql.internal.SQLConf
+import org.apache.spark.util.Utils
class RocksDBSuite extends SparkFunSuite {
+ test("RocksDBFileManager: upload only new immutable files") {
+ withTempDir { dir =>
+ val dfsRootDir = dir.getAbsolutePath
+ val fileManager = new RocksDBFileManager(
+ dfsRootDir, Utils.createTempDir(), new Configuration)
+ val sstDir = s"$dfsRootDir/SSTs"
+ def numRemoteSSTFiles: Int = listFiles(sstDir).length
+ val logDir = s"$dfsRootDir/logs"
+ def numRemoteLogFiles: Int = listFiles(logDir).length
+
+ // Save a version of checkpoint files
+ val cpFiles1 = Seq(
+ "sst-file1.sst" -> 10,
+ "sst-file2.sst" -> 20,
+ "other-file1" -> 100,
+ "other-file2" -> 200,
+ "archive/00001.log" -> 1000,
+ "archive/00002.log" -> 2000
+ )
+ saveCheckpointFiles(fileManager, cpFiles1, version = 1, numKeys = 101)
+ assert(numRemoteSSTFiles == 2) // 2 sst files copied
+ assert(numRemoteLogFiles == 2) // 2 log files copied
+
+ // Save SAME version again with different checkpoint files and verify
+ val cpFiles1_ = Seq(
+ "sst-file1.sst" -> 10, // same SST file as before, should not get copied
+ "sst-file2.sst" -> 25, // new SST file with same name as before, but different length
+ "sst-file3.sst" -> 30, // new SST file
+ "other-file1" -> 100, // same non-SST file as before, should not get copied
+ "other-file2" -> 210, // new non-SST file with same name as before, but different length
+ "other-file3" -> 300, // new non-SST file
+ "archive/00001.log" -> 1000, // same log file as before, should not get copied
+ "archive/00002.log" -> 2500, // new log file with same name as before, but different length
+ "archive/00003.log" -> 3000 // new log file
+ )
+ saveCheckpointFiles(fileManager, cpFiles1_, version = 1, numKeys = 1001)
Review comment:
`version = 1`? In practice, will we save checkpoint files for same version? I.e. for same micro-batch?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] xuanyuanking commented on a change in pull request #32582: [SPARK-35436][SS] RocksDBFileManager - save checkpoint to DFS
Posted by GitBox <gi...@apache.org>.
xuanyuanking commented on a change in pull request #32582:
URL: https://github.com/apache/spark/pull/32582#discussion_r644675514
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/state/RocksDBFileManager.scala
##########
@@ -17,18 +17,265 @@
package org.apache.spark.sql.execution.streaming.state
-import java.io.File
+import java.io.{File, FileInputStream, InputStream}
import java.nio.charset.StandardCharsets.UTF_8
import java.nio.file.Files
+import java.util.UUID
+import java.util.concurrent.ConcurrentHashMap
+import java.util.zip.{ZipEntry, ZipOutputStream}
+import scala.collection.JavaConverters._
import scala.collection.Seq
import com.fasterxml.jackson.annotation.JsonInclude.Include
import com.fasterxml.jackson.databind.{DeserializationFeature, ObjectMapper}
import com.fasterxml.jackson.module.scala.{DefaultScalaModule, ScalaObjectMapper}
+import org.apache.commons.io.{FilenameUtils, IOUtils}
+import org.apache.hadoop.conf.Configuration
+import org.apache.hadoop.fs.Path
import org.json4s.NoTypeHints
import org.json4s.jackson.Serialization
+import org.apache.spark.internal.Logging
+import org.apache.spark.sql.execution.streaming.CheckpointFileManager
+import org.apache.spark.util.Utils
+
+/**
+ * Class responsible for syncing RocksDB checkpoint files from local disk to DFS.
+ * For each version, checkpoint is saved in specific directory structure that allows successive
+ * versions to reuse to SST data files and archived log files. This allows each commit to be
+ * incremental, only new SST files and archived log files generated by RocksDB will be uploaded.
+ * The directory structures on local disk and in DFS are as follows.
+ *
+ * Local checkpoint dir structure
+ * ------------------------------
+ * RocksDB generates a bunch of files in the local checkpoint directory. The most important among
+ * them are the SST files; they are the actual log structured data files. Rest of the files contain
+ * the metadata necessary for RocksDB to read the SST files and start from the checkpoint.
+ * Note that the SST files are hard links to files in the RocksDB's working directory, and therefore
+ * successive checkpoints can share some of the SST files. So these SST files have to be copied to
+ * DFS in shared directory such that different committed versions can save them.
+ *
+ * We consider both SST files and archived log files as immutable files which can be shared between
+ * different checkpoints.
+ *
+ * localCheckpointDir
+ * |
+ * +-- OPTIONS-000005
+ * +-- MANIFEST-000008
+ * +-- CURRENT
+ * +-- 00007.sst
+ * +-- 00011.sst
+ * +-- archive
+ * | +-- 00008.log
+ * | +-- 00013.log
+ * ...
+ *
+ *
+ * DFS directory structure after saving to DFS as version 10
+ * -----------------------------------------------------------
+ * The SST and archived log files are given unique file names and copied to the shared subdirectory.
+ * Every version maintains a mapping of local immutable file name to the unique file name in DFS.
+ * This mapping is saved in a JSON file (named `metadata`), which is zipped along with other
+ * checkpoint files into a single file `[version].zip`.
+ *
+ * dfsRootDir
+ * |
+ * +-- SSTs
+ * | +-- 00007-[uuid1].sst
+ * | +-- 00011-[uuid2].sst
+ * +-- logs
+ * | +-- 00008-[uuid3].log
+ * | +-- 00013-[uuid4].log
+ * +-- 10.zip
+ * | +-- metadata <--- contains mapping between 00007.sst and [uuid1].sst,
+ * and the mapping between 00008.log and [uuid3].log
+ * | +-- OPTIONS-000005
+ * | +-- MANIFEST-000008
+ * | +-- CURRENT
+ * | ...
+ * |
+ * +-- 9.zip
+ * +-- 8.zip
+ * ...
+ *
+ * Note the following.
+ * - Each [version].zip is a complete description of all the data and metadata needed to recover
+ * a RocksDB instance at the corresponding version. The SST files and log files are not included
+ * in the zip files, they can be shared cross different versions. This is unlike the
+ * [version].delta files of HDFSBackedStateStore where previous delta files needs to be read
+ * to be recovered.
+ * - This is safe wrt speculatively executed tasks running concurrently in different executors
+ * as each task would upload a different copy of the generated immutable files and
+ * atomically update the [version].zip.
+ * - Immutable files are identified uniquely based on their file name and file size.
+ * - Immutable files can be reused only across adjacent checkpoints/versions.
+ * - This class is thread-safe. Specifically, it is safe to concurrently delete old files from a
+ * different thread than the task thread saving files.
Review comment:
Yes. It refers to delete the files contained in the old versions. Here's the description of the deleteOldVersions method of RocksDBFileManager, which will be called in RocksDBStateStoreProvider.doMaintenance. As we did before, I'll also refer this comment when the PR for delete path submitted.
```
* Delete old versions by deleting the associated version and SST files.
* At a high-level, this method finds which versions to delete, and which SST files that were
* last used in those versions. Its safe to delete these SST files because a SST file can
* be reused only in successive versions. Therefore, if a SST file F was last used in version
* V, then it wont be used in version V+1 or later, and if version V can be deleted, then
* F can safely be deleted as well.
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] viirya commented on a change in pull request #32582: [SPARK-35436][SS] RocksDBFileManager - save checkpoint to DFS
Posted by GitBox <gi...@apache.org>.
viirya commented on a change in pull request #32582:
URL: https://github.com/apache/spark/pull/32582#discussion_r645685787
##########
File path: sql/core/src/test/scala/org/apache/spark/sql/execution/streaming/state/RocksDBSuite.scala
##########
@@ -54,4 +130,30 @@ class RocksDBSuite extends SparkFunSuite {
"""{"sstFiles":[{"localFileName":"00001.sst","dfsSstFileName":"00001-uuid.sst","sizeBytes":12345678901234}],"logFiles":[{"localFileName":"00001.log","dfsLogFileName":"00001-uuid.log","sizeBytes":12345678901234}],"numKeys":12345678901234}""")
// scalastyle:on line.size.limit
}
+
+ def generateFiles(dir: String, fileToLengths: Seq[(String, Int)]): Unit = {
+ fileToLengths.foreach { case (fileName, length) =>
+ val file = new File(dir, fileName)
+ FileUtils.write(file, "a" * length)
+ }
+ }
+
+ def saveCheckpointFiles(
+ fileManager: RocksDBFileManager,
+ fileToLengths: Seq[(String, Int)],
+ version: Int,
+ numKeys: Int): Unit = {
+ val checkpointDir = Utils.createTempDir().getAbsolutePath // local dir to create checkpoints
Review comment:
Is it `localTempDir`?
```scala
class RocksDBFileManager(
dfsRootDir: String,
localTempDir: File,
hadoopConf: Configuration,
loggingId: String = "")
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] xuanyuanking commented on pull request #32582: [SPARK-35436][SS] RocksDBFileManager - save checkpoint to DFS
Posted by GitBox <gi...@apache.org>.
xuanyuanking commented on pull request #32582:
URL: https://github.com/apache/spark/pull/32582#issuecomment-857294184
Thanks @viirya. Let me retrigger the test.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #32582: [SPARK-35436][SS] RocksDBFileManager - save checkpoint to DFS
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #32582:
URL: https://github.com/apache/spark/pull/32582#issuecomment-857385133
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/44054/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HeartSaVioR commented on a change in pull request #32582: [SPARK-35436][SS] RocksDBFileManager - save checkpoint to DFS
Posted by GitBox <gi...@apache.org>.
HeartSaVioR commented on a change in pull request #32582:
URL: https://github.com/apache/spark/pull/32582#discussion_r645312635
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/state/RocksDBFileManager.scala
##########
@@ -17,18 +17,270 @@
package org.apache.spark.sql.execution.streaming.state
-import java.io.File
+import java.io.{File, FileInputStream, InputStream}
import java.nio.charset.StandardCharsets.UTF_8
import java.nio.file.Files
+import java.util.UUID
+import java.util.concurrent.ConcurrentHashMap
+import java.util.zip.{ZipEntry, ZipOutputStream}
+import scala.collection.JavaConverters._
import scala.collection.Seq
import com.fasterxml.jackson.annotation.JsonInclude.Include
import com.fasterxml.jackson.databind.{DeserializationFeature, ObjectMapper}
import com.fasterxml.jackson.module.scala.{DefaultScalaModule, ScalaObjectMapper}
+import org.apache.commons.io.{FilenameUtils, IOUtils}
+import org.apache.hadoop.conf.Configuration
+import org.apache.hadoop.fs.Path
import org.json4s.NoTypeHints
import org.json4s.jackson.Serialization
+import org.apache.spark.internal.Logging
+import org.apache.spark.sql.execution.streaming.CheckpointFileManager
+import org.apache.spark.util.Utils
+
+/**
+ * Class responsible for syncing RocksDB checkpoint files from local disk to DFS.
+ * Every version checkpoint is saved in specific directory structure that allows successive
+ * versions to reuse to SST data files and archived log files. This allows each commit to be
+ * incremental, only new SST files and archived log files generated by RocksDB will be uploaded.
+ * The directory structures on local disk and in DFS are as follows.
+ *
+ * Local checkpoint dir structure
+ * ------------------------------
+ * RocksDB generates a bunch of files in the local checkpoint directory. The most important among
+ * them are the SST files; they are the actual log structured data files. Rest of the files contain
+ * the metadata necessary for RocksDB to read the SST files and start from the checkpoint.
+ * Note that the SST files are hard links to files in the RocksDB's working directory, and therefore
+ * successive checkpoints can share some of the SST files. So these SST files have to be copied to
+ * DFS in shared directory such that different committed versions can save them.
+ *
+ * We consider both SST files and archived log files as immutable files which can be shared between
+ * different checkpoints.
+ *
+ * localCheckpointDir
+ * |
+ * +-- OPTIONS-000005
+ * +-- MANIFEST-000008
+ * +-- CURRENT
+ * +-- 00007.sst
+ * +-- 00011.sst
+ * +-- archive
+ * | +-- 00008.log
+ * | +-- 00013.log
+ * ...
+ *
+ *
+ * DFS directory structure after saving to DFS as version 10
+ * -----------------------------------------------------------
+ * The SST and archived log files are given unique file names and copied to the shared subdirectory.
+ * Every version maintains a mapping of local immutable file name to the unique file name in DFS.
+ * This mapping is saved in a JSON file (named `metadata`), which is zipped along with other
+ * checkpoint files into a single file `[version].zip`.
+ *
+ * dfsRootDir
+ * |
+ * +-- SSTs
+ * | +-- 00007-[uuid1].sst
+ * | +-- 00011-[uuid2].sst
+ * +-- logs
+ * | +-- 00008-[uuid3].log
+ * | +-- 00013-[uuid4].log
+ * +-- 10.zip
+ * | +-- metadata <--- contains mapping between 00007.sst and [uuid1].sst,
+ * and the mapping between 00008.log and [uuid3].log
+ * | +-- OPTIONS-000005
+ * | +-- MANIFEST-000008
+ * | +-- CURRENT
+ * | ...
+ * |
+ * +-- 9.zip
+ * +-- 8.zip
+ * ...
+ *
+ * Note the following.
+ * - Each [version].zip is a complete description of all the data and metadata needed to recover
+ * a RocksDB instance at the corresponding version. This is unlike the [version].delta files of
+ * HDFSBackedStateStore where previous delta files needs to be read to be recovered.
+ * - This is safe wrt speculatively executed tasks running concurrently in different executors
+ * as each task would upload a different copy of the generated immutable files and
+ * atomically update the [version].zip.
+ * - Immutable files are identified uniquely based on their file name and file size.
+ * - Immutable files can be reused only across adjacent checkpoints/versions.
+ * - This class is thread-safe. Specifically, it is safe to concurrently delete old files from a
+ * different thread than the task thread saving files.
+ *
+ * @param dfsRootDir Directory where the [version].zip files will be stored
+ * @param localTempDir Local directory for temporary work
+ * @param hadoopConf Hadoop configuration for talking to DFS
+ * @param loggingId Id that will be prepended in logs for isolating concurrent RocksDBs
+ */
+class RocksDBFileManager(
+ dfsRootDir: String,
+ localTempDir: File,
+ hadoopConf: Configuration,
+ loggingId: String = "")
+ extends Logging {
+
+ import RocksDBImmutableFile._
+
+ private val versionToRocksDBFiles = new ConcurrentHashMap[Long, Seq[RocksDBImmutableFile]]
+ private lazy val fm = CheckpointFileManager.create(new Path(dfsRootDir), hadoopConf)
+ private val fs = new Path(dfsRootDir).getFileSystem(hadoopConf)
+
+ /** Save all the files in given local checkpoint directory as a committed version in DFS */
+ def saveCheckpointToDfs(checkpointDir: File, version: Long, numKeys: Long): Unit = {
+ logFilesInDir(checkpointDir, s"Saving checkpoint files for version $version")
+ val allCheckpointFiles = listRocksDBFiles(checkpointDir)
+ val (localImmutableFiles, localOtherFiles) = allCheckpointFiles.partition(isImmutableFile)
+ val rocksDBFiles = saveImmutableFilesToDfs(version, checkpointDir, localImmutableFiles)
+ val metadata = RocksDBCheckpointMetadata(rocksDBFiles, numKeys)
+ rememberImmutableFiles(version, rocksDBFiles)
+ val metadataFile = localMetadataFile(checkpointDir)
+ metadata.writeToFile(metadataFile)
+ logInfo(s"Written metadata for version $version:\n${metadata.prettyJson}")
+
+ if (version <= 0 && numKeys == 0) {
Review comment:
What about simply making sure the directory has been created in prior? We can add below in initialization and don't bother with handling specific case.
```
// create a root directory on initialization if necessary
if (!fm.exists(new Path(dfsRootDir))) {
fm.mkdirs(new Path(dfsRootDir))
}
```
If we'd like to avoid paying the cost on calling exist, let's keep the code and change the version to 1 as I met the case the version starts from 1 during test.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] xuanyuanking commented on a change in pull request #32582: [SPARK-35436][SS] RocksDBFileManager - save checkpoint to DFS
Posted by GitBox <gi...@apache.org>.
xuanyuanking commented on a change in pull request #32582:
URL: https://github.com/apache/spark/pull/32582#discussion_r644679717
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/state/RocksDBFileManager.scala
##########
@@ -17,18 +17,265 @@
package org.apache.spark.sql.execution.streaming.state
-import java.io.File
+import java.io.{File, FileInputStream, InputStream}
import java.nio.charset.StandardCharsets.UTF_8
import java.nio.file.Files
+import java.util.UUID
+import java.util.concurrent.ConcurrentHashMap
+import java.util.zip.{ZipEntry, ZipOutputStream}
+import scala.collection.JavaConverters._
import scala.collection.Seq
import com.fasterxml.jackson.annotation.JsonInclude.Include
import com.fasterxml.jackson.databind.{DeserializationFeature, ObjectMapper}
import com.fasterxml.jackson.module.scala.{DefaultScalaModule, ScalaObjectMapper}
+import org.apache.commons.io.{FilenameUtils, IOUtils}
+import org.apache.hadoop.conf.Configuration
+import org.apache.hadoop.fs.Path
import org.json4s.NoTypeHints
import org.json4s.jackson.Serialization
+import org.apache.spark.internal.Logging
+import org.apache.spark.sql.execution.streaming.CheckpointFileManager
+import org.apache.spark.util.Utils
+
+/**
+ * Class responsible for syncing RocksDB checkpoint files from local disk to DFS.
+ * For each version, checkpoint is saved in specific directory structure that allows successive
+ * versions to reuse to SST data files and archived log files. This allows each commit to be
+ * incremental, only new SST files and archived log files generated by RocksDB will be uploaded.
+ * The directory structures on local disk and in DFS are as follows.
+ *
+ * Local checkpoint dir structure
+ * ------------------------------
+ * RocksDB generates a bunch of files in the local checkpoint directory. The most important among
+ * them are the SST files; they are the actual log structured data files. Rest of the files contain
+ * the metadata necessary for RocksDB to read the SST files and start from the checkpoint.
+ * Note that the SST files are hard links to files in the RocksDB's working directory, and therefore
+ * successive checkpoints can share some of the SST files. So these SST files have to be copied to
+ * DFS in shared directory such that different committed versions can save them.
+ *
+ * We consider both SST files and archived log files as immutable files which can be shared between
+ * different checkpoints.
+ *
+ * localCheckpointDir
+ * |
+ * +-- OPTIONS-000005
+ * +-- MANIFEST-000008
+ * +-- CURRENT
+ * +-- 00007.sst
+ * +-- 00011.sst
+ * +-- archive
+ * | +-- 00008.log
+ * | +-- 00013.log
+ * ...
+ *
+ *
+ * DFS directory structure after saving to DFS as version 10
+ * -----------------------------------------------------------
+ * The SST and archived log files are given unique file names and copied to the shared subdirectory.
+ * Every version maintains a mapping of local immutable file name to the unique file name in DFS.
+ * This mapping is saved in a JSON file (named `metadata`), which is zipped along with other
+ * checkpoint files into a single file `[version].zip`.
+ *
+ * dfsRootDir
+ * |
+ * +-- SSTs
+ * | +-- 00007-[uuid1].sst
+ * | +-- 00011-[uuid2].sst
+ * +-- logs
+ * | +-- 00008-[uuid3].log
+ * | +-- 00013-[uuid4].log
+ * +-- 10.zip
+ * | +-- metadata <--- contains mapping between 00007.sst and [uuid1].sst,
+ * and the mapping between 00008.log and [uuid3].log
+ * | +-- OPTIONS-000005
+ * | +-- MANIFEST-000008
+ * | +-- CURRENT
+ * | ...
+ * |
+ * +-- 9.zip
+ * +-- 8.zip
+ * ...
+ *
+ * Note the following.
+ * - Each [version].zip is a complete description of all the data and metadata needed to recover
+ * a RocksDB instance at the corresponding version. The SST files and log files are not included
+ * in the zip files, they can be shared cross different versions. This is unlike the
+ * [version].delta files of HDFSBackedStateStore where previous delta files needs to be read
+ * to be recovered.
+ * - This is safe wrt speculatively executed tasks running concurrently in different executors
+ * as each task would upload a different copy of the generated immutable files and
+ * atomically update the [version].zip.
+ * - Immutable files are identified uniquely based on their file name and file size.
+ * - Immutable files can be reused only across adjacent checkpoints/versions.
+ * - This class is thread-safe. Specifically, it is safe to concurrently delete old files from a
+ * different thread than the task thread saving files.
+ *
+ * @param dfsRootDir Directory where the [version].zip files will be stored
+ * @param localTempDir Local directory for temporary work
+ * @param hadoopConf Hadoop configuration for talking to DFS
+ * @param loggingId Id that will be prepended in logs for isolating concurrent RocksDBs
+ */
+class RocksDBFileManager(
+ dfsRootDir: String,
+ localTempDir: File,
+ hadoopConf: Configuration,
+ loggingId: String = "")
+ extends Logging {
+
+ import RocksDBImmutableFile._
+
+ private val versionToRocksDBFiles = new ConcurrentHashMap[Long, Seq[RocksDBImmutableFile]]
+ private lazy val fm = CheckpointFileManager.create(new Path(dfsRootDir), hadoopConf)
+ private val fs = new Path(dfsRootDir).getFileSystem(hadoopConf)
+
+ /** Save all the files in given local checkpoint directory as a committed version in DFS */
+ def saveCheckpointToDfs(checkpointDir: File, version: Long, numKeys: Long): Unit = {
+ logFilesInDir(checkpointDir, s"Saving checkpoint files for version $version")
+ val (localImmutableFiles, localOtherFiles) = listRocksDBFiles(checkpointDir)
+ val rocksDBFiles = saveImmutableFilesToDfs(version, localImmutableFiles)
+ val metadata = RocksDBCheckpointMetadata(rocksDBFiles, numKeys)
+ val metadataFile = localMetadataFile(checkpointDir)
+ metadata.writeToFile(metadataFile)
+ logInfo(s"Written metadata for version $version:\n${metadata.prettyJson}")
+
+ if (version <= 0 && numKeys == 0) {
+ // If we're writing the initial version and there's no data, we have to explicitly initialize
+ // the root directory. Normally saveImmutableFilesToDfs will do this initialization, but
+ // when there's no data that method won't write any files, and zipToDfsFile uses the
+ // CheckpointFileManager.createAtomic API which doesn't auto-initialize parent directories.
+ fm.mkdirs(new Path(dfsRootDir))
+ }
+ zipToDfsFile(localOtherFiles :+ metadataFile, dfsBatchZipFile(version))
+ logInfo(s"Saved checkpoint file for version $version")
+ }
+
+ /** Save immutable files to DFS directory */
+ private def saveImmutableFilesToDfs(
+ version: Long,
+ localFiles: Seq[File]): Seq[RocksDBImmutableFile] = {
+ // Get the immutable files used in previous versions, as some of those uploaded files can be
+ // reused for this version
+ logInfo(s"Saving RocksDB files to DFS for $version")
+ val prevFilesToSizes = versionToRocksDBFiles.values.asScala.flatten.map { f =>
+ f.localFileName -> f
+ }.toMap
+
+ var bytesCopied = 0L
+ var filesCopied = 0L
+ var filesReused = 0L
+
+ val immutableFiles = localFiles.map { localFile =>
+ prevFilesToSizes
+ .get(localFile.getName)
+ .filter(_.isSameFile(localFile))
+ .map { reusable =>
+ filesReused += 1
+ reusable
+ }.getOrElse {
+ val localFileName = localFile.getName
+ val dfsFileName = newDFSFileName(localFileName)
+ val dfsFile = dfsFilePath(dfsFileName)
+ // Note: The implementation of copyFromLocalFile() closes the output stream when there is
+ // any exception while copying. So this may generate partial files on DFS. But that is
+ // okay because until the main [version].zip file is written, those partial files are
+ // not going to be used at all. Eventually these files should get cleared.
+ fs.copyFromLocalFile(
+ new Path(localFile.getAbsoluteFile.toURI), dfsFile)
+ val localFileSize = localFile.length()
+ logInfo(s"Copied $localFile to $dfsFile - $localFileSize bytes")
+ filesCopied += 1
+ bytesCopied += localFileSize
+
+ RocksDBImmutableFile(localFile.getName, dfsFileName, localFileSize)
+ }
+ }
+ logInfo(s"Copied $filesCopied files ($bytesCopied bytes) from local to" +
+ s" DFS for version $version. $filesReused files reused without copying.")
+ versionToRocksDBFiles.put(version, immutableFiles)
+
+ immutableFiles
+ }
+
+ /**
+ * Compress files to a single zip file in DFS. Only the file names are embedded in the zip.
+ * Any error while writing will ensure that the file is not written.
+ */
+ private def zipToDfsFile(files: Seq[File], dfsZipFile: Path): Unit = {
+ lazy val filesStr = s"$dfsZipFile\n\t${files.mkString("\n\t")}"
+ var in: InputStream = null
+ val out = fm.createAtomic(dfsZipFile, overwriteIfPossible = true)
+ var totalBytes = 0L
+ val zout = new ZipOutputStream(out)
+ try {
+ files.foreach { file =>
+ zout.putNextEntry(new ZipEntry(file.getName))
+ in = new FileInputStream(file)
+ val bytes = IOUtils.copy(in, zout)
+ in.close()
+ zout.closeEntry()
+ totalBytes += bytes
+ }
+ zout.close() // so that any error in closing also cancels the output stream
+ logInfo(s"Zipped $totalBytes bytes (before compression) to $filesStr")
Review comment:
the `fileStr` contains the `dfsZipFile` name. The log format here is `${dfsZipFile} \n ${listing all the file names}`
##########
File path: sql/core/src/test/scala/org/apache/spark/sql/execution/streaming/state/RocksDBSuite.scala
##########
@@ -20,12 +20,86 @@ package org.apache.spark.sql.execution.streaming.state
import java.io._
import java.nio.charset.Charset
+import scala.language.implicitConversions
+
import org.apache.commons.io.FileUtils
+import org.apache.hadoop.conf.Configuration
import org.apache.spark._
+import org.apache.spark.sql.catalyst.util.quietly
+import org.apache.spark.sql.execution.streaming.CreateAtomicTestManager
+import org.apache.spark.sql.internal.SQLConf
+import org.apache.spark.util.Utils
class RocksDBSuite extends SparkFunSuite {
+ test("RocksDBFileManager: upload only new immutable files") {
+ val dfsRootDir = Utils.createTempDir().getAbsolutePath
Review comment:
Sure, done in f5a6104
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32582: [SPARK-35436][SS] RocksDBFileManager - save checkpoint to DFS
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32582:
URL: https://github.com/apache/spark/pull/32582#issuecomment-853783833
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #32582: [SPARK-35436][SS] RocksDBFileManager - save checkpoint to DFS
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #32582:
URL: https://github.com/apache/spark/pull/32582#issuecomment-854910753
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/43862/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] xuanyuanking commented on a change in pull request #32582: [SPARK-35436][SS] RocksDBFileManager - save checkpoint to DFS
Posted by GitBox <gi...@apache.org>.
xuanyuanking commented on a change in pull request #32582:
URL: https://github.com/apache/spark/pull/32582#discussion_r645666674
##########
File path: sql/core/src/test/scala/org/apache/spark/sql/execution/streaming/state/RocksDBSuite.scala
##########
@@ -20,12 +20,88 @@ package org.apache.spark.sql.execution.streaming.state
import java.io._
import java.nio.charset.Charset
+import scala.language.implicitConversions
+
import org.apache.commons.io.FileUtils
+import org.apache.hadoop.conf.Configuration
import org.apache.spark._
+import org.apache.spark.sql.catalyst.util.quietly
+import org.apache.spark.sql.execution.streaming.CreateAtomicTestManager
+import org.apache.spark.sql.internal.SQLConf
+import org.apache.spark.util.Utils
class RocksDBSuite extends SparkFunSuite {
+ test("RocksDBFileManager: upload only new immutable files") {
+ withTempDir { dir =>
+ val dfsRootDir = dir.getAbsolutePath
+ val fileManager = new RocksDBFileManager(
+ dfsRootDir, Utils.createTempDir(), new Configuration)
+ val sstDir = s"$dfsRootDir/SSTs"
+ def numRemoteSSTFiles: Int = listFiles(sstDir).length
+ val logDir = s"$dfsRootDir/logs"
+ def numRemoteLogFiles: Int = listFiles(logDir).length
+
+ // Save a version of checkpoint files
+ val cpFiles1 = Seq(
+ "sst-file1.sst" -> 10,
+ "sst-file2.sst" -> 20,
+ "other-file1" -> 100,
+ "other-file2" -> 200,
+ "archive/00001.log" -> 1000,
+ "archive/00002.log" -> 2000
+ )
+ saveCheckpointFiles(fileManager, cpFiles1, version = 1, numKeys = 101)
+ assert(numRemoteSSTFiles == 2) // 2 sst files copied
+ assert(numRemoteLogFiles == 2) // 2 log files copied
+
+ // Save SAME version again with different checkpoint files and verify
+ val cpFiles1_ = Seq(
+ "sst-file1.sst" -> 10, // same SST file as before, should not get copied
+ "sst-file2.sst" -> 25, // new SST file with same name as before, but different length
+ "sst-file3.sst" -> 30, // new SST file
+ "other-file1" -> 100, // same non-SST file as before, should not get copied
+ "other-file2" -> 210, // new non-SST file with same name as before, but different length
+ "other-file3" -> 300, // new non-SST file
+ "archive/00001.log" -> 1000, // same log file as before, should not get copied
+ "archive/00002.log" -> 2500, // new log file with same name as before, but different length
+ "archive/00003.log" -> 3000 // new log file
+ )
+ saveCheckpointFiles(fileManager, cpFiles1_, version = 1, numKeys = 1001)
Review comment:
For task failover, it's possible that the tasks failed during uploading. The new task uses the same version and may schedule to the same executor but with different checkpoint dir.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32582: [WIP][SPARK-35436] RocksDBFileManager - save checkpoint to DFS
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32582:
URL: https://github.com/apache/spark/pull/32582#issuecomment-843280476
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/43206/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #32582: [SPARK-35436][SS] RocksDBFileManager - save checkpoint to DFS
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #32582:
URL: https://github.com/apache/spark/pull/32582#issuecomment-853051027
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/139221/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #32582: [WIP][SPARK-35436] RocksDBFileManager - save checkpoint to DFS
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #32582:
URL: https://github.com/apache/spark/pull/32582#issuecomment-843243715
**[Test build #138685 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/138685/testReport)** for PR 32582 at commit [`5e6bf13`](https://github.com/apache/spark/commit/5e6bf130ae1ae104a48020cd8e08c07b3968c4c0).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] xuanyuanking commented on a change in pull request #32582: [SPARK-35436][SS] RocksDBFileManager - save checkpoint to DFS
Posted by GitBox <gi...@apache.org>.
xuanyuanking commented on a change in pull request #32582:
URL: https://github.com/apache/spark/pull/32582#discussion_r645661110
##########
File path: sql/core/src/test/scala/org/apache/spark/sql/execution/streaming/state/RocksDBSuite.scala
##########
@@ -54,4 +130,30 @@ class RocksDBSuite extends SparkFunSuite {
"""{"sstFiles":[{"localFileName":"00001.sst","dfsSstFileName":"00001-uuid.sst","sizeBytes":12345678901234}],"logFiles":[{"localFileName":"00001.log","dfsLogFileName":"00001-uuid.log","sizeBytes":12345678901234}],"numKeys":12345678901234}""")
// scalastyle:on line.size.limit
}
+
+ def generateFiles(dir: String, fileToLengths: Seq[(String, Int)]): Unit = {
+ fileToLengths.foreach { case (fileName, length) =>
+ val file = new File(dir, fileName)
+ FileUtils.write(file, "a" * length)
+ }
+ }
+
+ def saveCheckpointFiles(
+ fileManager: RocksDBFileManager,
+ fileToLengths: Seq[(String, Int)],
+ version: Int,
+ numKeys: Int): Unit = {
+ val checkpointDir = Utils.createTempDir().getAbsolutePath // local dir to create checkpoints
Review comment:
The `checkpointDir` here is a temp dir for a specific version of checkpointing. This checkpoint here corresponds to a RocksDB checkpoint. The method, `saveCheckpointToDfs` will be called in the `commit` logic for the RocksDBStateStoreProvider. The commit follows these steps:
```
* - Write all the updates to the native RocksDB
* - Flush all changes to disk
* - Create a RocksDB checkpoint in a new local dir [Done in this method]
* - Sync the checkpoint dir files to DFS [Done in this method]
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #32582: [SPARK-35436][SS] RocksDBFileManager - save checkpoint to DFS
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #32582:
URL: https://github.com/apache/spark/pull/32582#issuecomment-853850149
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/43809/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] xuanyuanking commented on a change in pull request #32582: [SPARK-35436][SS] RocksDBFileManager - save checkpoint to DFS
Posted by GitBox <gi...@apache.org>.
xuanyuanking commented on a change in pull request #32582:
URL: https://github.com/apache/spark/pull/32582#discussion_r644679178
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/state/RocksDBFileManager.scala
##########
@@ -17,18 +17,265 @@
package org.apache.spark.sql.execution.streaming.state
-import java.io.File
+import java.io.{File, FileInputStream, InputStream}
import java.nio.charset.StandardCharsets.UTF_8
import java.nio.file.Files
+import java.util.UUID
+import java.util.concurrent.ConcurrentHashMap
+import java.util.zip.{ZipEntry, ZipOutputStream}
+import scala.collection.JavaConverters._
import scala.collection.Seq
import com.fasterxml.jackson.annotation.JsonInclude.Include
import com.fasterxml.jackson.databind.{DeserializationFeature, ObjectMapper}
import com.fasterxml.jackson.module.scala.{DefaultScalaModule, ScalaObjectMapper}
+import org.apache.commons.io.{FilenameUtils, IOUtils}
+import org.apache.hadoop.conf.Configuration
+import org.apache.hadoop.fs.Path
import org.json4s.NoTypeHints
import org.json4s.jackson.Serialization
+import org.apache.spark.internal.Logging
+import org.apache.spark.sql.execution.streaming.CheckpointFileManager
+import org.apache.spark.util.Utils
+
+/**
+ * Class responsible for syncing RocksDB checkpoint files from local disk to DFS.
+ * For each version, checkpoint is saved in specific directory structure that allows successive
+ * versions to reuse to SST data files and archived log files. This allows each commit to be
+ * incremental, only new SST files and archived log files generated by RocksDB will be uploaded.
+ * The directory structures on local disk and in DFS are as follows.
+ *
+ * Local checkpoint dir structure
+ * ------------------------------
+ * RocksDB generates a bunch of files in the local checkpoint directory. The most important among
+ * them are the SST files; they are the actual log structured data files. Rest of the files contain
+ * the metadata necessary for RocksDB to read the SST files and start from the checkpoint.
+ * Note that the SST files are hard links to files in the RocksDB's working directory, and therefore
+ * successive checkpoints can share some of the SST files. So these SST files have to be copied to
+ * DFS in shared directory such that different committed versions can save them.
+ *
+ * We consider both SST files and archived log files as immutable files which can be shared between
+ * different checkpoints.
+ *
+ * localCheckpointDir
+ * |
+ * +-- OPTIONS-000005
+ * +-- MANIFEST-000008
+ * +-- CURRENT
+ * +-- 00007.sst
+ * +-- 00011.sst
+ * +-- archive
+ * | +-- 00008.log
+ * | +-- 00013.log
+ * ...
+ *
+ *
+ * DFS directory structure after saving to DFS as version 10
+ * -----------------------------------------------------------
+ * The SST and archived log files are given unique file names and copied to the shared subdirectory.
+ * Every version maintains a mapping of local immutable file name to the unique file name in DFS.
+ * This mapping is saved in a JSON file (named `metadata`), which is zipped along with other
+ * checkpoint files into a single file `[version].zip`.
+ *
+ * dfsRootDir
+ * |
+ * +-- SSTs
+ * | +-- 00007-[uuid1].sst
+ * | +-- 00011-[uuid2].sst
+ * +-- logs
+ * | +-- 00008-[uuid3].log
+ * | +-- 00013-[uuid4].log
+ * +-- 10.zip
+ * | +-- metadata <--- contains mapping between 00007.sst and [uuid1].sst,
+ * and the mapping between 00008.log and [uuid3].log
+ * | +-- OPTIONS-000005
+ * | +-- MANIFEST-000008
+ * | +-- CURRENT
+ * | ...
+ * |
+ * +-- 9.zip
+ * +-- 8.zip
+ * ...
+ *
+ * Note the following.
+ * - Each [version].zip is a complete description of all the data and metadata needed to recover
+ * a RocksDB instance at the corresponding version. The SST files and log files are not included
+ * in the zip files, they can be shared cross different versions. This is unlike the
+ * [version].delta files of HDFSBackedStateStore where previous delta files needs to be read
+ * to be recovered.
+ * - This is safe wrt speculatively executed tasks running concurrently in different executors
+ * as each task would upload a different copy of the generated immutable files and
+ * atomically update the [version].zip.
+ * - Immutable files are identified uniquely based on their file name and file size.
+ * - Immutable files can be reused only across adjacent checkpoints/versions.
+ * - This class is thread-safe. Specifically, it is safe to concurrently delete old files from a
+ * different thread than the task thread saving files.
+ *
+ * @param dfsRootDir Directory where the [version].zip files will be stored
+ * @param localTempDir Local directory for temporary work
+ * @param hadoopConf Hadoop configuration for talking to DFS
+ * @param loggingId Id that will be prepended in logs for isolating concurrent RocksDBs
+ */
+class RocksDBFileManager(
+ dfsRootDir: String,
+ localTempDir: File,
+ hadoopConf: Configuration,
+ loggingId: String = "")
+ extends Logging {
+
+ import RocksDBImmutableFile._
+
+ private val versionToRocksDBFiles = new ConcurrentHashMap[Long, Seq[RocksDBImmutableFile]]
+ private lazy val fm = CheckpointFileManager.create(new Path(dfsRootDir), hadoopConf)
+ private val fs = new Path(dfsRootDir).getFileSystem(hadoopConf)
+
+ /** Save all the files in given local checkpoint directory as a committed version in DFS */
+ def saveCheckpointToDfs(checkpointDir: File, version: Long, numKeys: Long): Unit = {
+ logFilesInDir(checkpointDir, s"Saving checkpoint files for version $version")
+ val (localImmutableFiles, localOtherFiles) = listRocksDBFiles(checkpointDir)
+ val rocksDBFiles = saveImmutableFilesToDfs(version, localImmutableFiles)
+ val metadata = RocksDBCheckpointMetadata(rocksDBFiles, numKeys)
+ val metadataFile = localMetadataFile(checkpointDir)
+ metadata.writeToFile(metadataFile)
+ logInfo(s"Written metadata for version $version:\n${metadata.prettyJson}")
+
+ if (version <= 0 && numKeys == 0) {
+ // If we're writing the initial version and there's no data, we have to explicitly initialize
+ // the root directory. Normally saveImmutableFilesToDfs will do this initialization, but
+ // when there's no data that method won't write any files, and zipToDfsFile uses the
+ // CheckpointFileManager.createAtomic API which doesn't auto-initialize parent directories.
+ fm.mkdirs(new Path(dfsRootDir))
Review comment:
Make sense, done in f5a6104
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] xuanyuanking commented on pull request #32582: [SPARK-35436][SS] RocksDBFileManager - save checkpoint to DFS
Posted by GitBox <gi...@apache.org>.
xuanyuanking commented on pull request #32582:
URL: https://github.com/apache/spark/pull/32582#issuecomment-853535338
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #32582: [SPARK-35436][SS] RocksDBFileManager - save checkpoint to DFS
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #32582:
URL: https://github.com/apache/spark/pull/32582#issuecomment-852965676
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/43744/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #32582: [SPARK-35436][SS] RocksDBFileManager - save checkpoint to DFS
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #32582:
URL: https://github.com/apache/spark/pull/32582#issuecomment-853850149
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/43809/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #32582: [SPARK-35436][SS] RocksDBFileManager - save checkpoint to DFS
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #32582:
URL: https://github.com/apache/spark/pull/32582#issuecomment-850124238
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/43559/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #32582: [WIP][SPARK-35436] RocksDBFileManager - save checkpoint to DFS
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #32582:
URL: https://github.com/apache/spark/pull/32582#issuecomment-843398502
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] xuanyuanking commented on a change in pull request #32582: [SPARK-35436][SS] RocksDBFileManager - save checkpoint to DFS
Posted by GitBox <gi...@apache.org>.
xuanyuanking commented on a change in pull request #32582:
URL: https://github.com/apache/spark/pull/32582#discussion_r643852204
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/state/RocksDBFileManager.scala
##########
@@ -17,18 +17,270 @@
package org.apache.spark.sql.execution.streaming.state
-import java.io.File
+import java.io.{File, FileInputStream, InputStream}
import java.nio.charset.StandardCharsets.UTF_8
import java.nio.file.Files
+import java.util.UUID
+import java.util.concurrent.ConcurrentHashMap
+import java.util.zip.{ZipEntry, ZipOutputStream}
+import scala.collection.JavaConverters._
import scala.collection.Seq
import com.fasterxml.jackson.annotation.JsonInclude.Include
import com.fasterxml.jackson.databind.{DeserializationFeature, ObjectMapper}
import com.fasterxml.jackson.module.scala.{DefaultScalaModule, ScalaObjectMapper}
+import org.apache.commons.io.{FilenameUtils, IOUtils}
+import org.apache.hadoop.conf.Configuration
+import org.apache.hadoop.fs.Path
import org.json4s.NoTypeHints
import org.json4s.jackson.Serialization
+import org.apache.spark.internal.Logging
+import org.apache.spark.sql.execution.streaming.CheckpointFileManager
+import org.apache.spark.util.Utils
+
+/**
+ * Class responsible for syncing RocksDB checkpoint files from local disk to DFS.
+ * Every version checkpoint is saved in specific directory structure that allows successive
+ * versions to reuse to SST data files and archived log files. This allows each commit to be
+ * incremental, only new SST files and archived log files generated by RocksDB will be uploaded.
+ * The directory structures on local disk and in DFS are as follows.
+ *
+ * Local checkpoint dir structure
+ * ------------------------------
+ * RocksDB generates a bunch of files in the local checkpoint directory. The most important among
+ * them are the SST files; they are the actual log structured data files. Rest of the files contain
+ * the metadata necessary for RocksDB to read the SST files and start from the checkpoint.
+ * Note that the SST files are hard links to files in the RocksDB's working directory, and therefore
+ * successive checkpoints can share some of the SST files. So these SST files have to be copied to
+ * DFS in shared directory such that different committed versions can save them.
+ *
+ * We consider both SST files and archived log files as immutable files which can be shared between
+ * different checkpoints.
+ *
+ * localCheckpointDir
+ * |
+ * +-- OPTIONS-000005
+ * +-- MANIFEST-000008
+ * +-- CURRENT
+ * +-- 00007.sst
+ * +-- 00011.sst
+ * +-- archive
+ * | +-- 00008.log
+ * | +-- 00013.log
+ * ...
+ *
+ *
+ * DFS directory structure after saving to DFS as version 10
+ * -----------------------------------------------------------
+ * The SST and archived log files are given unique file names and copied to the shared subdirectory.
+ * Every version maintains a mapping of local immutable file name to the unique file name in DFS.
+ * This mapping is saved in a JSON file (named `metadata`), which is zipped along with other
+ * checkpoint files into a single file `[version].zip`.
+ *
+ * dfsRootDir
+ * |
+ * +-- SSTs
+ * | +-- 00007-[uuid1].sst
+ * | +-- 00011-[uuid2].sst
+ * +-- logs
+ * | +-- 00008-[uuid3].log
+ * | +-- 00013-[uuid4].log
+ * +-- 10.zip
+ * | +-- metadata <--- contains mapping between 00007.sst and [uuid1].sst,
+ * and the mapping between 00008.log and [uuid3].log
+ * | +-- OPTIONS-000005
+ * | +-- MANIFEST-000008
+ * | +-- CURRENT
+ * | ...
+ * |
+ * +-- 9.zip
+ * +-- 8.zip
+ * ...
+ *
+ * Note the following.
+ * - Each [version].zip is a complete description of all the data and metadata needed to recover
+ * a RocksDB instance at the corresponding version. This is unlike the [version].delta files of
+ * HDFSBackedStateStore where previous delta files needs to be read to be recovered.
+ * - This is safe wrt speculatively executed tasks running concurrently in different executors
+ * as each task would upload a different copy of the generated immutable files and
+ * atomically update the [version].zip.
+ * - Immutable files are identified uniquely based on their file name and file size.
+ * - Immutable files can be reused only across adjacent checkpoints/versions.
+ * - This class is thread-safe. Specifically, it is safe to concurrently delete old files from a
+ * different thread than the task thread saving files.
+ *
+ * @param dfsRootDir Directory where the [version].zip files will be stored
+ * @param localTempDir Local directory for temporary work
+ * @param hadoopConf Hadoop configuration for talking to DFS
+ * @param loggingId Id that will be prepended in logs for isolating concurrent RocksDBs
+ */
+class RocksDBFileManager(
+ dfsRootDir: String,
+ localTempDir: File,
+ hadoopConf: Configuration,
+ loggingId: String = "")
+ extends Logging {
+
+ import RocksDBImmutableFile._
+
+ private val versionToRocksDBFiles = new ConcurrentHashMap[Long, Seq[RocksDBImmutableFile]]
+ private lazy val fm = CheckpointFileManager.create(new Path(dfsRootDir), hadoopConf)
+ private val fs = new Path(dfsRootDir).getFileSystem(hadoopConf)
+
+ /** Save all the files in given local checkpoint directory as a committed version in DFS */
+ def saveCheckpointToDfs(checkpointDir: File, version: Long, numKeys: Long): Unit = {
+ logFilesInDir(checkpointDir, s"Saving checkpoint files for version $version")
+ val allCheckpointFiles = listRocksDBFiles(checkpointDir)
+ val (localImmutableFiles, localOtherFiles) = allCheckpointFiles.partition(isImmutableFile)
+ val rocksDBFiles = saveImmutableFilesToDfs(version, checkpointDir, localImmutableFiles)
+ val metadata = RocksDBCheckpointMetadata(rocksDBFiles, numKeys)
+ rememberImmutableFiles(version, rocksDBFiles)
+ val metadataFile = localMetadataFile(checkpointDir)
+ metadata.writeToFile(metadataFile)
+ logInfo(s"Written metadata for version $version:\n${metadata.prettyJson}")
+
+ if (version <= 0 && numKeys == 0) {
+ // If we're writing the initial version and there's no data, we have to explicitly initialize
+ // the root directory. Normally saveImmutableFilesToDfs will do this initialization, but
+ // when there's no data that method won't write any files, and zipToDfsFile uses the
+ // CheckpointFileManager.createAtomic API which doesn't auto-initialize parent directories.
+ fm.mkdirs(new Path(dfsRootDir))
+ }
+ zipToDfsFile(localOtherFiles :+ metadataFile, dfsBatchZipFile(version))
+ logInfo(s"Saved checkpoint file for version $version")
+ }
+
+ /** Save immutable files to DFS directory */
+ private def saveImmutableFilesToDfs(
+ version: Long,
+ localDir: File,
Review comment:
Done in 3800c51
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] xuanyuanking commented on a change in pull request #32582: [SPARK-35436][SS] RocksDBFileManager - save checkpoint to DFS
Posted by GitBox <gi...@apache.org>.
xuanyuanking commented on a change in pull request #32582:
URL: https://github.com/apache/spark/pull/32582#discussion_r643852096
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/state/RocksDBFileManager.scala
##########
@@ -17,18 +17,270 @@
package org.apache.spark.sql.execution.streaming.state
-import java.io.File
+import java.io.{File, FileInputStream, InputStream}
import java.nio.charset.StandardCharsets.UTF_8
import java.nio.file.Files
+import java.util.UUID
+import java.util.concurrent.ConcurrentHashMap
+import java.util.zip.{ZipEntry, ZipOutputStream}
+import scala.collection.JavaConverters._
import scala.collection.Seq
import com.fasterxml.jackson.annotation.JsonInclude.Include
import com.fasterxml.jackson.databind.{DeserializationFeature, ObjectMapper}
import com.fasterxml.jackson.module.scala.{DefaultScalaModule, ScalaObjectMapper}
+import org.apache.commons.io.{FilenameUtils, IOUtils}
+import org.apache.hadoop.conf.Configuration
+import org.apache.hadoop.fs.Path
import org.json4s.NoTypeHints
import org.json4s.jackson.Serialization
+import org.apache.spark.internal.Logging
+import org.apache.spark.sql.execution.streaming.CheckpointFileManager
+import org.apache.spark.util.Utils
+
+/**
+ * Class responsible for syncing RocksDB checkpoint files from local disk to DFS.
+ * Every version checkpoint is saved in specific directory structure that allows successive
+ * versions to reuse to SST data files and archived log files. This allows each commit to be
+ * incremental, only new SST files and archived log files generated by RocksDB will be uploaded.
+ * The directory structures on local disk and in DFS are as follows.
+ *
+ * Local checkpoint dir structure
+ * ------------------------------
+ * RocksDB generates a bunch of files in the local checkpoint directory. The most important among
+ * them are the SST files; they are the actual log structured data files. Rest of the files contain
+ * the metadata necessary for RocksDB to read the SST files and start from the checkpoint.
+ * Note that the SST files are hard links to files in the RocksDB's working directory, and therefore
+ * successive checkpoints can share some of the SST files. So these SST files have to be copied to
+ * DFS in shared directory such that different committed versions can save them.
+ *
+ * We consider both SST files and archived log files as immutable files which can be shared between
+ * different checkpoints.
+ *
+ * localCheckpointDir
+ * |
+ * +-- OPTIONS-000005
+ * +-- MANIFEST-000008
+ * +-- CURRENT
+ * +-- 00007.sst
+ * +-- 00011.sst
+ * +-- archive
+ * | +-- 00008.log
+ * | +-- 00013.log
+ * ...
+ *
+ *
+ * DFS directory structure after saving to DFS as version 10
+ * -----------------------------------------------------------
+ * The SST and archived log files are given unique file names and copied to the shared subdirectory.
+ * Every version maintains a mapping of local immutable file name to the unique file name in DFS.
+ * This mapping is saved in a JSON file (named `metadata`), which is zipped along with other
+ * checkpoint files into a single file `[version].zip`.
+ *
+ * dfsRootDir
+ * |
+ * +-- SSTs
+ * | +-- 00007-[uuid1].sst
+ * | +-- 00011-[uuid2].sst
+ * +-- logs
+ * | +-- 00008-[uuid3].log
+ * | +-- 00013-[uuid4].log
+ * +-- 10.zip
+ * | +-- metadata <--- contains mapping between 00007.sst and [uuid1].sst,
+ * and the mapping between 00008.log and [uuid3].log
+ * | +-- OPTIONS-000005
+ * | +-- MANIFEST-000008
+ * | +-- CURRENT
+ * | ...
+ * |
+ * +-- 9.zip
+ * +-- 8.zip
+ * ...
+ *
+ * Note the following.
+ * - Each [version].zip is a complete description of all the data and metadata needed to recover
+ * a RocksDB instance at the corresponding version. This is unlike the [version].delta files of
+ * HDFSBackedStateStore where previous delta files needs to be read to be recovered.
+ * - This is safe wrt speculatively executed tasks running concurrently in different executors
+ * as each task would upload a different copy of the generated immutable files and
+ * atomically update the [version].zip.
+ * - Immutable files are identified uniquely based on their file name and file size.
+ * - Immutable files can be reused only across adjacent checkpoints/versions.
+ * - This class is thread-safe. Specifically, it is safe to concurrently delete old files from a
+ * different thread than the task thread saving files.
+ *
+ * @param dfsRootDir Directory where the [version].zip files will be stored
+ * @param localTempDir Local directory for temporary work
+ * @param hadoopConf Hadoop configuration for talking to DFS
+ * @param loggingId Id that will be prepended in logs for isolating concurrent RocksDBs
+ */
+class RocksDBFileManager(
+ dfsRootDir: String,
+ localTempDir: File,
+ hadoopConf: Configuration,
+ loggingId: String = "")
+ extends Logging {
+
+ import RocksDBImmutableFile._
+
+ private val versionToRocksDBFiles = new ConcurrentHashMap[Long, Seq[RocksDBImmutableFile]]
+ private lazy val fm = CheckpointFileManager.create(new Path(dfsRootDir), hadoopConf)
+ private val fs = new Path(dfsRootDir).getFileSystem(hadoopConf)
+
+ /** Save all the files in given local checkpoint directory as a committed version in DFS */
+ def saveCheckpointToDfs(checkpointDir: File, version: Long, numKeys: Long): Unit = {
+ logFilesInDir(checkpointDir, s"Saving checkpoint files for version $version")
+ val allCheckpointFiles = listRocksDBFiles(checkpointDir)
+ val (localImmutableFiles, localOtherFiles) = allCheckpointFiles.partition(isImmutableFile)
Review comment:
Totally makes sense, thanks for the advice! Done in 3800c51.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] xuanyuanking commented on a change in pull request #32582: [WIP][SPARK-35436] RocksDBFileManager - save checkpoint to DFS
Posted by GitBox <gi...@apache.org>.
xuanyuanking commented on a change in pull request #32582:
URL: https://github.com/apache/spark/pull/32582#discussion_r634468675
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/state/RocksDBFileManager.scala
##########
@@ -0,0 +1,413 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.streaming.state
+
+import java.io.{File, FileInputStream, InputStream}
+import java.nio.charset.StandardCharsets.UTF_8
+import java.nio.file.Files
+import java.util.UUID
+import java.util.concurrent.ConcurrentHashMap
+import java.util.zip.{ZipEntry, ZipOutputStream}
+
+import scala.collection.JavaConverters._
+import scala.collection.Seq
+
+import com.fasterxml.jackson.annotation.JsonInclude.Include
+import com.fasterxml.jackson.databind.{DeserializationFeature, ObjectMapper}
+import com.fasterxml.jackson.module.scala.{DefaultScalaModule, ScalaObjectMapper}
+import org.apache.commons.io.{FilenameUtils, IOUtils}
+import org.apache.hadoop.conf.Configuration
+import org.apache.hadoop.fs.Path
+import org.json4s.NoTypeHints
+import org.json4s.jackson.Serialization
+
+import org.apache.spark.internal.Logging
+import org.apache.spark.sql.execution.streaming.CheckpointFileManager
+import org.apache.spark.util.Utils
+
+/**
+ * Class responsible for syncing RocksDB checkpoint files from local disk to DFS.
+ * Every version checkpoint is saved in specific directory structure that allows successive
+ * versions to reuse to SST data files. This allows each commit to be incremental, only new SST
+ * files generated by RocksDB will be uploaded. The directory structures on local disk and in DFS
+ * are as follows.
+ *
+ * Local checkpoint dir structure
+ * ------------------------------
+ * RocksDB generates a bunch of files in the local checkpoint directory. The most important among
+ * them are the SST files; they are the actual log structured data files. Rest of the files contain
+ * the metadata necessary for RocksDB to read the SST files and start from the checkpoint.
+ * Note that the SST files are hard links to files in the RocksDB's working directory, and therefore
+ * successive checkpoints can share some of the SST files. So these SST files have to be copied to
+ * DFS in shared directory such that different committed versions can save them.
+ *
+ * localCheckpointDir
+ * |
+ * +-- OPTIONS-000005
+ * +-- MANIFEST-000008
+ * +-- CURRENT
+ * +-- 00007.sst
+ * +-- 00011.sst
+ * +-- archive
+ * | +-- 00008.log
+ * | +-- 00013.log
+ * ...
+ *
+ *
+ * DFS directory structure after saving to DFS as version 10
+ * -----------------------------------------------------------
+ * The SST files are given unique file names and copied to the shared SST subdirectory. Every
+ * version maintains a mapping of local SST file name to the unique file name in DFS. This mapping
+ * is saved in a JSON file (named `metadata`), which is zipped along with other non-SST checkpoint
+ * files into a single file `[version].zip`.
+ *
+ * dfsRootDir
+ * |
+ * +-- SSTs
+ * | +-- 00007-[uuid1].sst
+ * | +-- 00011-[uuid2].sst
+ * +-- logs
+ * | +-- 00008-[uuid3].log
+ * | +-- 00013-[uuid4].log
+ * +-- 10.zip
+ * | +-- metadata <--- contains mapping between 00007.sst and [uuid1].sst
+ * | +-- OPTIONS-000005
+ * | +-- MANIFEST-000008
+ * | +-- CURRENT
+ * | ...
+ * |
+ * +-- 9.zip
+ * +-- 8.zip
+ * ...
+ *
+ * Note the following.
+ * - Each [version].zip is a complete description of all the data and metadata needed to recover
+ * a RocksDB instance at the corresponding version. This is unlike the [version].delta files of
+ * HDFSBackedStateStore where previous delta files needs to be read to be recovered.
+ * - This is safe wrt speculatively executed tasks running concurrently in different executors
+ * as each task would upload a different copy of the generated SST files and
+ * atomically update the [version].zip.
+ * - SST files are identified uniquely based on their file name and file size.
+ * - SST files can be reused only across adjacent checkpoints/versions.
+ * - This class is thread-safe. Specifically, it is safe to concurrently delete old files from a
+ * different thread than the task thread saving files.
+ *
+ * @param dfsRootDir Directory where the [version].zip files will be stored
+ * @param localTempDir Local directory for temporary work
+ * @param hadoopConf Hadoop configuration for talking to DFS
+ * @param loggingId Id that will be prepended in logs for isolating concurrent RocksDBs
+ */
+class RocksDBFileManager(
+ dfsRootDir: String,
+ localTempDir: File,
+ hadoopConf: Configuration,
+ loggingId: String = "")
+ extends Logging {
+
+ import RocksDBImmutableFile._
+
+ private val versionToRocksDBFiles = new ConcurrentHashMap[Long, Seq[RocksDBImmutableFile]]
+ private lazy val fm = CheckpointFileManager.create(new Path(dfsRootDir), hadoopConf)
+ private val fs = new Path(dfsRootDir).getFileSystem(hadoopConf)
+
+ /** Save all the files in given local checkpoint directory as a committed version in DFS */
+ def saveCheckpointToDfs(checkpointDir: File, version: Long, numKeys: Long): Unit = {
+ logFilesInDir(checkpointDir, s"Saving checkpoint files for version $version")
+ val allCheckpointFiles = listRocksDBFiles(checkpointDir)
+ val (localImmutableFiles, localOtherFiles) = allCheckpointFiles.partition(isImmutableFile)
+ val rocksDBFiles = saveImmutableFilesToDfs(version, checkpointDir, localImmutableFiles)
+ val metadata = RocksDBCheckpointMetadata(rocksDBFiles, numKeys)
+ rememberImmutableFiles(version, rocksDBFiles)
+ val metadataFile = localMetadataFile(checkpointDir)
+ metadata.writeToFile(metadataFile)
+ logInfo(s"Written metadata for version $version:\n${metadata.prettyJson}")
Review comment:
Per https://github.com/apache/spark/pull/32272#discussion_r624388116, we only use prettyJson in the log.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32582: [SPARK-35436][SS] RocksDBFileManager - save checkpoint to DFS
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32582:
URL: https://github.com/apache/spark/pull/32582#issuecomment-853862301
**[Test build #139285 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/139285/testReport)** for PR 32582 at commit [`f5a6104`](https://github.com/apache/spark/commit/f5a6104e6930dccad71db72bfc6ee8c47d9933da).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32582: [SPARK-35436][SS] RocksDBFileManager - save checkpoint to DFS
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32582:
URL: https://github.com/apache/spark/pull/32582#issuecomment-853783833
**[Test build #139285 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/139285/testReport)** for PR 32582 at commit [`f5a6104`](https://github.com/apache/spark/commit/f5a6104e6930dccad71db72bfc6ee8c47d9933da).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32582: [SPARK-35436][SS] RocksDBFileManager - save checkpoint to DFS
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32582:
URL: https://github.com/apache/spark/pull/32582#issuecomment-857337786
Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/44041/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #32582: [SPARK-35436][SS] RocksDBFileManager - save checkpoint to DFS
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #32582:
URL: https://github.com/apache/spark/pull/32582#issuecomment-850180715
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/139041/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32582: [SPARK-35436][SS] RocksDBFileManager - save checkpoint to DFS
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32582:
URL: https://github.com/apache/spark/pull/32582#issuecomment-854842906
**[Test build #139340 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/139340/testReport)** for PR 32582 at commit [`bdd9e8e`](https://github.com/apache/spark/commit/bdd9e8eb6d0cf5fd11def7ac3e630f2e2dae302a).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] xuanyuanking commented on a change in pull request #32582: [SPARK-35436][SS] RocksDBFileManager - save checkpoint to DFS
Posted by GitBox <gi...@apache.org>.
xuanyuanking commented on a change in pull request #32582:
URL: https://github.com/apache/spark/pull/32582#discussion_r645681881
##########
File path: sql/core/src/test/scala/org/apache/spark/sql/execution/streaming/state/RocksDBSuite.scala
##########
@@ -20,12 +20,88 @@ package org.apache.spark.sql.execution.streaming.state
import java.io._
import java.nio.charset.Charset
+import scala.language.implicitConversions
+
import org.apache.commons.io.FileUtils
+import org.apache.hadoop.conf.Configuration
import org.apache.spark._
+import org.apache.spark.sql.catalyst.util.quietly
+import org.apache.spark.sql.execution.streaming.CreateAtomicTestManager
+import org.apache.spark.sql.internal.SQLConf
+import org.apache.spark.util.Utils
class RocksDBSuite extends SparkFunSuite {
+ test("RocksDBFileManager: upload only new immutable files") {
+ withTempDir { dir =>
+ val dfsRootDir = dir.getAbsolutePath
+ val fileManager = new RocksDBFileManager(
+ dfsRootDir, Utils.createTempDir(), new Configuration)
+ val sstDir = s"$dfsRootDir/SSTs"
+ def numRemoteSSTFiles: Int = listFiles(sstDir).length
+ val logDir = s"$dfsRootDir/logs"
+ def numRemoteLogFiles: Int = listFiles(logDir).length
+
+ // Save a version of checkpoint files
+ val cpFiles1 = Seq(
+ "sst-file1.sst" -> 10,
+ "sst-file2.sst" -> 20,
+ "other-file1" -> 100,
+ "other-file2" -> 200,
+ "archive/00001.log" -> 1000,
+ "archive/00002.log" -> 2000
+ )
+ saveCheckpointFiles(fileManager, cpFiles1, version = 1, numKeys = 101)
+ assert(numRemoteSSTFiles == 2) // 2 sst files copied
+ assert(numRemoteLogFiles == 2) // 2 log files copied
+
+ // Save SAME version again with different checkpoint files and verify
+ val cpFiles1_ = Seq(
+ "sst-file1.sst" -> 10, // same SST file as before, should not get copied
+ "sst-file2.sst" -> 25, // new SST file with same name as before, but different length
+ "sst-file3.sst" -> 30, // new SST file
+ "other-file1" -> 100, // same non-SST file as before, should not get copied
+ "other-file2" -> 210, // new non-SST file with same name as before, but different length
+ "other-file3" -> 300, // new non-SST file
+ "archive/00001.log" -> 1000, // same log file as before, should not get copied
+ "archive/00002.log" -> 2500, // new log file with same name as before, but different length
+ "archive/00003.log" -> 3000 // new log file
+ )
+ saveCheckpointFiles(fileManager, cpFiles1_, version = 1, numKeys = 1001)
+ assert(numRemoteSSTFiles === 4, "shouldn't copy same files again") // 2 old + 2 new SST files
+ assert(numRemoteLogFiles === 4, "shouldn't copy same files again") // 2 old + 2 new log files
+
+ // Save another version and verify
+ val cpFiles2 = Seq(
+ "sst-file4.sst" -> 40,
+ "other-file4" -> 400,
+ "archive/00004.log" -> 4000
+ )
+ saveCheckpointFiles(fileManager, cpFiles2, version = 2, numKeys = 1501)
+ assert(numRemoteSSTFiles === 5) // 1 new file over earlier 4 files
+ assert(numRemoteLogFiles === 5) // 1 new file over earlier 4 files
+ }
+ }
+
+ test("RocksDBFileManager: error writing [version].zip cancels the output stream") {
+ quietly {
+ val hadoopConf = new Configuration()
+ hadoopConf.set(
+ SQLConf.STREAMING_CHECKPOINT_FILE_MANAGER_CLASS.parent.key,
+ classOf[CreateAtomicTestManager].getName)
+ val dfsRootDir = Utils.createTempDir().getAbsolutePath
+ val fileManager = new RocksDBFileManager(dfsRootDir, Utils.createTempDir(), hadoopConf)
+ val cpFiles = Seq("sst-file1.sst" -> 10, "sst-file2.sst" -> 20, "other-file1" -> 100)
+ CreateAtomicTestManager.shouldFailInCreateAtomic = true
+ CreateAtomicTestManager.cancelCalledInCreateAtomic = false
+ intercept[IOException] {
+ saveCheckpointFiles(fileManager, cpFiles, version = 1, numKeys = 101)
+ }
Review comment:
This is failed on purpose.
```
[info] java.io.IOException: Copy failed intentionally
[info] at org.apache.spark.sql.execution.streaming.CreateAtomicTestManager$$anon$3.close(CheckpointFileManagerSuite.scala:169)
```
This test aims to test the behavior that the `cancel` should be called no matter any error happens during `close`. See the comment and code here: https://github.com/apache/spark/pull/32582/files#diff-e3d3914d0398d61fdd299b1f8d3e869ec6a86e97606677c724969e421c9bf44eR222-R227
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32582: [SPARK-35436][SS] RocksDBFileManager - save checkpoint to DFS
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32582:
URL: https://github.com/apache/spark/pull/32582#issuecomment-854872515
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/43862/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HeartSaVioR commented on pull request #32582: [SPARK-35436][SS] RocksDBFileManager - save checkpoint to DFS
Posted by GitBox <gi...@apache.org>.
HeartSaVioR commented on pull request #32582:
URL: https://github.com/apache/spark/pull/32582#issuecomment-857294105
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #32582: [SPARK-35436][SS] RocksDBFileManager - save checkpoint to DFS
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #32582:
URL: https://github.com/apache/spark/pull/32582#issuecomment-854942920
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/139340/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] xuanyuanking commented on a change in pull request #32582: [SPARK-35436][SS] RocksDBFileManager - save checkpoint to DFS
Posted by GitBox <gi...@apache.org>.
xuanyuanking commented on a change in pull request #32582:
URL: https://github.com/apache/spark/pull/32582#discussion_r643851862
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/state/RocksDBFileManager.scala
##########
@@ -17,18 +17,270 @@
package org.apache.spark.sql.execution.streaming.state
-import java.io.File
+import java.io.{File, FileInputStream, InputStream}
import java.nio.charset.StandardCharsets.UTF_8
import java.nio.file.Files
+import java.util.UUID
+import java.util.concurrent.ConcurrentHashMap
+import java.util.zip.{ZipEntry, ZipOutputStream}
+import scala.collection.JavaConverters._
import scala.collection.Seq
import com.fasterxml.jackson.annotation.JsonInclude.Include
import com.fasterxml.jackson.databind.{DeserializationFeature, ObjectMapper}
import com.fasterxml.jackson.module.scala.{DefaultScalaModule, ScalaObjectMapper}
+import org.apache.commons.io.{FilenameUtils, IOUtils}
+import org.apache.hadoop.conf.Configuration
+import org.apache.hadoop.fs.Path
import org.json4s.NoTypeHints
import org.json4s.jackson.Serialization
+import org.apache.spark.internal.Logging
+import org.apache.spark.sql.execution.streaming.CheckpointFileManager
+import org.apache.spark.util.Utils
+
+/**
+ * Class responsible for syncing RocksDB checkpoint files from local disk to DFS.
+ * Every version checkpoint is saved in specific directory structure that allows successive
+ * versions to reuse to SST data files and archived log files. This allows each commit to be
+ * incremental, only new SST files and archived log files generated by RocksDB will be uploaded.
+ * The directory structures on local disk and in DFS are as follows.
+ *
+ * Local checkpoint dir structure
+ * ------------------------------
+ * RocksDB generates a bunch of files in the local checkpoint directory. The most important among
+ * them are the SST files; they are the actual log structured data files. Rest of the files contain
+ * the metadata necessary for RocksDB to read the SST files and start from the checkpoint.
+ * Note that the SST files are hard links to files in the RocksDB's working directory, and therefore
+ * successive checkpoints can share some of the SST files. So these SST files have to be copied to
+ * DFS in shared directory such that different committed versions can save them.
+ *
+ * We consider both SST files and archived log files as immutable files which can be shared between
+ * different checkpoints.
+ *
+ * localCheckpointDir
+ * |
+ * +-- OPTIONS-000005
+ * +-- MANIFEST-000008
+ * +-- CURRENT
+ * +-- 00007.sst
+ * +-- 00011.sst
+ * +-- archive
+ * | +-- 00008.log
+ * | +-- 00013.log
+ * ...
+ *
+ *
+ * DFS directory structure after saving to DFS as version 10
+ * -----------------------------------------------------------
+ * The SST and archived log files are given unique file names and copied to the shared subdirectory.
+ * Every version maintains a mapping of local immutable file name to the unique file name in DFS.
+ * This mapping is saved in a JSON file (named `metadata`), which is zipped along with other
+ * checkpoint files into a single file `[version].zip`.
+ *
+ * dfsRootDir
+ * |
+ * +-- SSTs
+ * | +-- 00007-[uuid1].sst
+ * | +-- 00011-[uuid2].sst
+ * +-- logs
+ * | +-- 00008-[uuid3].log
+ * | +-- 00013-[uuid4].log
+ * +-- 10.zip
+ * | +-- metadata <--- contains mapping between 00007.sst and [uuid1].sst,
+ * and the mapping between 00008.log and [uuid3].log
+ * | +-- OPTIONS-000005
+ * | +-- MANIFEST-000008
+ * | +-- CURRENT
+ * | ...
+ * |
+ * +-- 9.zip
+ * +-- 8.zip
+ * ...
+ *
+ * Note the following.
+ * - Each [version].zip is a complete description of all the data and metadata needed to recover
+ * a RocksDB instance at the corresponding version. This is unlike the [version].delta files of
+ * HDFSBackedStateStore where previous delta files needs to be read to be recovered.
+ * - This is safe wrt speculatively executed tasks running concurrently in different executors
+ * as each task would upload a different copy of the generated immutable files and
+ * atomically update the [version].zip.
+ * - Immutable files are identified uniquely based on their file name and file size.
+ * - Immutable files can be reused only across adjacent checkpoints/versions.
+ * - This class is thread-safe. Specifically, it is safe to concurrently delete old files from a
+ * different thread than the task thread saving files.
+ *
+ * @param dfsRootDir Directory where the [version].zip files will be stored
+ * @param localTempDir Local directory for temporary work
+ * @param hadoopConf Hadoop configuration for talking to DFS
+ * @param loggingId Id that will be prepended in logs for isolating concurrent RocksDBs
+ */
+class RocksDBFileManager(
+ dfsRootDir: String,
Review comment:
Thanks! Done in 3800c51
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #32582: [SPARK-35436][SS] RocksDBFileManager - save checkpoint to DFS
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #32582:
URL: https://github.com/apache/spark/pull/32582#issuecomment-852934454
**[Test build #139221 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/139221/testReport)** for PR 32582 at commit [`3800c51`](https://github.com/apache/spark/commit/3800c5173f056b6b63abf4865a6bdf9da16136d5).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32582: [SPARK-35436][SS] RocksDBFileManager - save checkpoint to DFS
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32582:
URL: https://github.com/apache/spark/pull/32582#issuecomment-852934454
**[Test build #139221 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/139221/testReport)** for PR 32582 at commit [`3800c51`](https://github.com/apache/spark/commit/3800c5173f056b6b63abf4865a6bdf9da16136d5).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #32582: [SPARK-35436][SS] RocksDBFileManager - save checkpoint to DFS
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #32582:
URL: https://github.com/apache/spark/pull/32582#issuecomment-857417824
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/139529/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #32582: [SPARK-35436][SS] RocksDBFileManager - save checkpoint to DFS
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #32582:
URL: https://github.com/apache/spark/pull/32582#issuecomment-852965676
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/43744/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] viirya commented on pull request #32582: [SPARK-35436][SS] RocksDBFileManager - save checkpoint to DFS
Posted by GitBox <gi...@apache.org>.
viirya commented on pull request #32582:
URL: https://github.com/apache/spark/pull/32582#issuecomment-853993479
Thank you @xuanyuanking. I will take another look today.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32582: [WIP][SPARK-35436] RocksDBFileManager - save checkpoint to DFS
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32582:
URL: https://github.com/apache/spark/pull/32582#issuecomment-843243715
**[Test build #138685 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/138685/testReport)** for PR 32582 at commit [`5e6bf13`](https://github.com/apache/spark/commit/5e6bf130ae1ae104a48020cd8e08c07b3968c4c0).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32582: [SPARK-35436][SS] RocksDBFileManager - save checkpoint to DFS
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32582:
URL: https://github.com/apache/spark/pull/32582#issuecomment-854927287
**[Test build #139340 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/139340/testReport)** for PR 32582 at commit [`bdd9e8e`](https://github.com/apache/spark/commit/bdd9e8eb6d0cf5fd11def7ac3e630f2e2dae302a).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] xuanyuanking commented on pull request #32582: [SPARK-35436][SS] RocksDBFileManager - save checkpoint to DFS
Posted by GitBox <gi...@apache.org>.
xuanyuanking commented on pull request #32582:
URL: https://github.com/apache/spark/pull/32582#issuecomment-852937628
Great thanks for the detailed review! All comments addressed. Please take another look when you have time.
Regarding the logs, agree on more discussion about lower down some flooded messages. I'll also revisit other logs we add. For this PR, 7 INFO logs are for recoding file operations and metrics.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] xuanyuanking commented on pull request #32582: [SPARK-35436][SS] RocksDBFileManager - save checkpoint to DFS
Posted by GitBox <gi...@apache.org>.
xuanyuanking commented on pull request #32582:
URL: https://github.com/apache/spark/pull/32582#issuecomment-857418906
Great thanks for the help! @HeartSaVioR
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] xuanyuanking commented on pull request #32582: [SPARK-35436][SS] RocksDBFileManager - save checkpoint to DFS
Posted by GitBox <gi...@apache.org>.
xuanyuanking commented on pull request #32582:
URL: https://github.com/apache/spark/pull/32582#issuecomment-854287887
Great thanks! @viirya
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] viirya commented on a change in pull request #32582: [SPARK-35436][SS] RocksDBFileManager - save checkpoint to DFS
Posted by GitBox <gi...@apache.org>.
viirya commented on a change in pull request #32582:
URL: https://github.com/apache/spark/pull/32582#discussion_r645323513
##########
File path: sql/core/src/test/scala/org/apache/spark/sql/execution/streaming/state/RocksDBSuite.scala
##########
@@ -54,4 +130,30 @@ class RocksDBSuite extends SparkFunSuite {
"""{"sstFiles":[{"localFileName":"00001.sst","dfsSstFileName":"00001-uuid.sst","sizeBytes":12345678901234}],"logFiles":[{"localFileName":"00001.log","dfsLogFileName":"00001-uuid.log","sizeBytes":12345678901234}],"numKeys":12345678901234}""")
// scalastyle:on line.size.limit
}
+
+ def generateFiles(dir: String, fileToLengths: Seq[(String, Int)]): Unit = {
+ fileToLengths.foreach { case (fileName, length) =>
+ val file = new File(dir, fileName)
+ FileUtils.write(file, "a" * length)
+ }
+ }
+
+ def saveCheckpointFiles(
+ fileManager: RocksDBFileManager,
+ fileToLengths: Seq[(String, Int)],
+ version: Int,
+ numKeys: Int): Unit = {
+ val checkpointDir = Utils.createTempDir().getAbsolutePath // local dir to create checkpoints
Review comment:
Isn't `checkpointDir` already be defined in `RocksDBFileManager`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] xuanyuanking commented on a change in pull request #32582: [SPARK-35436][SS] RocksDBFileManager - save checkpoint to DFS
Posted by GitBox <gi...@apache.org>.
xuanyuanking commented on a change in pull request #32582:
URL: https://github.com/apache/spark/pull/32582#discussion_r645667555
##########
File path: sql/core/src/test/scala/org/apache/spark/sql/execution/streaming/state/RocksDBSuite.scala
##########
@@ -20,12 +20,88 @@ package org.apache.spark.sql.execution.streaming.state
import java.io._
import java.nio.charset.Charset
+import scala.language.implicitConversions
+
import org.apache.commons.io.FileUtils
+import org.apache.hadoop.conf.Configuration
import org.apache.spark._
+import org.apache.spark.sql.catalyst.util.quietly
+import org.apache.spark.sql.execution.streaming.CreateAtomicTestManager
+import org.apache.spark.sql.internal.SQLConf
+import org.apache.spark.util.Utils
class RocksDBSuite extends SparkFunSuite {
+ test("RocksDBFileManager: upload only new immutable files") {
+ withTempDir { dir =>
+ val dfsRootDir = dir.getAbsolutePath
+ val fileManager = new RocksDBFileManager(
+ dfsRootDir, Utils.createTempDir(), new Configuration)
+ val sstDir = s"$dfsRootDir/SSTs"
+ def numRemoteSSTFiles: Int = listFiles(sstDir).length
+ val logDir = s"$dfsRootDir/logs"
+ def numRemoteLogFiles: Int = listFiles(logDir).length
+
+ // Save a version of checkpoint files
+ val cpFiles1 = Seq(
+ "sst-file1.sst" -> 10,
+ "sst-file2.sst" -> 20,
+ "other-file1" -> 100,
+ "other-file2" -> 200,
+ "archive/00001.log" -> 1000,
+ "archive/00002.log" -> 2000
+ )
+ saveCheckpointFiles(fileManager, cpFiles1, version = 1, numKeys = 101)
+ assert(numRemoteSSTFiles == 2) // 2 sst files copied
+ assert(numRemoteLogFiles == 2) // 2 log files copied
+
+ // Save SAME version again with different checkpoint files and verify
+ val cpFiles1_ = Seq(
+ "sst-file1.sst" -> 10, // same SST file as before, should not get copied
+ "sst-file2.sst" -> 25, // new SST file with same name as before, but different length
+ "sst-file3.sst" -> 30, // new SST file
+ "other-file1" -> 100, // same non-SST file as before, should not get copied
+ "other-file2" -> 210, // new non-SST file with same name as before, but different length
+ "other-file3" -> 300, // new non-SST file
+ "archive/00001.log" -> 1000, // same log file as before, should not get copied
+ "archive/00002.log" -> 2500, // new log file with same name as before, but different length
+ "archive/00003.log" -> 3000 // new log file
+ )
+ saveCheckpointFiles(fileManager, cpFiles1_, version = 1, numKeys = 1001)
+ assert(numRemoteSSTFiles === 4, "shouldn't copy same files again") // 2 old + 2 new SST files
+ assert(numRemoteLogFiles === 4, "shouldn't copy same files again") // 2 old + 2 new log files
+
+ // Save another version and verify
+ val cpFiles2 = Seq(
+ "sst-file4.sst" -> 40,
+ "other-file4" -> 400,
+ "archive/00004.log" -> 4000
+ )
+ saveCheckpointFiles(fileManager, cpFiles2, version = 2, numKeys = 1501)
+ assert(numRemoteSSTFiles === 5) // 1 new file over earlier 4 files
+ assert(numRemoteLogFiles === 5) // 1 new file over earlier 4 files
+ }
+ }
+
+ test("RocksDBFileManager: error writing [version].zip cancels the output stream") {
+ quietly {
+ val hadoopConf = new Configuration()
+ hadoopConf.set(
+ SQLConf.STREAMING_CHECKPOINT_FILE_MANAGER_CLASS.parent.key,
+ classOf[CreateAtomicTestManager].getName)
+ val dfsRootDir = Utils.createTempDir().getAbsolutePath
Review comment:
This test will be further extended and more temp dir will be added. So maybe we just have the root path using `withTempDir`? See the demo here: https://github.com/apache/spark/pull/32767/files#diff-dc6f9dfe11e76f890ff2986f866853bcac263027c82562f9a52f4672a5460826R37
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] xuanyuanking commented on a change in pull request #32582: [SPARK-35436][SS] RocksDBFileManager - save checkpoint to DFS
Posted by GitBox <gi...@apache.org>.
xuanyuanking commented on a change in pull request #32582:
URL: https://github.com/apache/spark/pull/32582#discussion_r647256039
##########
File path: sql/core/src/test/scala/org/apache/spark/sql/execution/streaming/state/RocksDBSuite.scala
##########
@@ -54,4 +130,30 @@ class RocksDBSuite extends SparkFunSuite {
"""{"sstFiles":[{"localFileName":"00001.sst","dfsSstFileName":"00001-uuid.sst","sizeBytes":12345678901234}],"logFiles":[{"localFileName":"00001.log","dfsLogFileName":"00001-uuid.log","sizeBytes":12345678901234}],"numKeys":12345678901234}""")
// scalastyle:on line.size.limit
}
+
+ def generateFiles(dir: String, fileToLengths: Seq[(String, Int)]): Unit = {
+ fileToLengths.foreach { case (fileName, length) =>
+ val file = new File(dir, fileName)
+ FileUtils.write(file, "a" * length)
+ }
+ }
+
+ def saveCheckpointFiles(
+ fileManager: RocksDBFileManager,
+ fileToLengths: Seq[(String, Int)],
+ version: Int,
+ numKeys: Int): Unit = {
+ val checkpointDir = Utils.createTempDir().getAbsolutePath // local dir to create checkpoints
Review comment:
Sorry just found I missed this comment.
Not the same one. The `localTempDir` is controlled by RocksDBFileManager. The lifetime of temp dir for `checkpointDir` is controlled by `RocksDBStateStoreProvider `.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #32582: [SPARK-35436][SS] RocksDBFileManager - save checkpoint to DFS
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #32582:
URL: https://github.com/apache/spark/pull/32582#issuecomment-853850149
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] xuanyuanking commented on a change in pull request #32582: [WIP][SPARK-35436] RocksDBFileManager - save checkpoint to DFS
Posted by GitBox <gi...@apache.org>.
xuanyuanking commented on a change in pull request #32582:
URL: https://github.com/apache/spark/pull/32582#discussion_r634466838
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/state/RocksDBFileManager.scala
##########
@@ -0,0 +1,413 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.streaming.state
+
+import java.io.{File, FileInputStream, InputStream}
+import java.nio.charset.StandardCharsets.UTF_8
+import java.nio.file.Files
+import java.util.UUID
+import java.util.concurrent.ConcurrentHashMap
+import java.util.zip.{ZipEntry, ZipOutputStream}
+
+import scala.collection.JavaConverters._
+import scala.collection.Seq
+
+import com.fasterxml.jackson.annotation.JsonInclude.Include
+import com.fasterxml.jackson.databind.{DeserializationFeature, ObjectMapper}
+import com.fasterxml.jackson.module.scala.{DefaultScalaModule, ScalaObjectMapper}
+import org.apache.commons.io.{FilenameUtils, IOUtils}
+import org.apache.hadoop.conf.Configuration
+import org.apache.hadoop.fs.Path
+import org.json4s.NoTypeHints
+import org.json4s.jackson.Serialization
+
+import org.apache.spark.internal.Logging
+import org.apache.spark.sql.execution.streaming.CheckpointFileManager
+import org.apache.spark.util.Utils
+
+/**
+ * Class responsible for syncing RocksDB checkpoint files from local disk to DFS.
+ * Every version checkpoint is saved in specific directory structure that allows successive
+ * versions to reuse to SST data files. This allows each commit to be incremental, only new SST
+ * files generated by RocksDB will be uploaded. The directory structures on local disk and in DFS
+ * are as follows.
+ *
+ * Local checkpoint dir structure
+ * ------------------------------
+ * RocksDB generates a bunch of files in the local checkpoint directory. The most important among
+ * them are the SST files; they are the actual log structured data files. Rest of the files contain
+ * the metadata necessary for RocksDB to read the SST files and start from the checkpoint.
+ * Note that the SST files are hard links to files in the RocksDB's working directory, and therefore
+ * successive checkpoints can share some of the SST files. So these SST files have to be copied to
+ * DFS in shared directory such that different committed versions can save them.
+ *
+ * localCheckpointDir
+ * |
+ * +-- OPTIONS-000005
+ * +-- MANIFEST-000008
+ * +-- CURRENT
+ * +-- 00007.sst
+ * +-- 00011.sst
+ * +-- archive
+ * | +-- 00008.log
+ * | +-- 00013.log
+ * ...
+ *
+ *
+ * DFS directory structure after saving to DFS as version 10
+ * -----------------------------------------------------------
+ * The SST files are given unique file names and copied to the shared SST subdirectory. Every
+ * version maintains a mapping of local SST file name to the unique file name in DFS. This mapping
+ * is saved in a JSON file (named `metadata`), which is zipped along with other non-SST checkpoint
+ * files into a single file `[version].zip`.
+ *
+ * dfsRootDir
+ * |
+ * +-- SSTs
+ * | +-- 00007-[uuid1].sst
+ * | +-- 00011-[uuid2].sst
+ * +-- logs
+ * | +-- 00008-[uuid3].log
+ * | +-- 00013-[uuid4].log
+ * +-- 10.zip
+ * | +-- metadata <--- contains mapping between 00007.sst and [uuid1].sst
+ * | +-- OPTIONS-000005
+ * | +-- MANIFEST-000008
+ * | +-- CURRENT
+ * | ...
+ * |
+ * +-- 9.zip
+ * +-- 8.zip
+ * ...
+ *
+ * Note the following.
+ * - Each [version].zip is a complete description of all the data and metadata needed to recover
+ * a RocksDB instance at the corresponding version. This is unlike the [version].delta files of
+ * HDFSBackedStateStore where previous delta files needs to be read to be recovered.
+ * - This is safe wrt speculatively executed tasks running concurrently in different executors
+ * as each task would upload a different copy of the generated SST files and
+ * atomically update the [version].zip.
+ * - SST files are identified uniquely based on their file name and file size.
+ * - SST files can be reused only across adjacent checkpoints/versions.
+ * - This class is thread-safe. Specifically, it is safe to concurrently delete old files from a
+ * different thread than the task thread saving files.
+ *
+ * @param dfsRootDir Directory where the [version].zip files will be stored
+ * @param localTempDir Local directory for temporary work
+ * @param hadoopConf Hadoop configuration for talking to DFS
+ * @param loggingId Id that will be prepended in logs for isolating concurrent RocksDBs
+ */
+class RocksDBFileManager(
+ dfsRootDir: String,
+ localTempDir: File,
+ hadoopConf: Configuration,
+ loggingId: String = "")
+ extends Logging {
+
+ import RocksDBImmutableFile._
+
+ private val versionToRocksDBFiles = new ConcurrentHashMap[Long, Seq[RocksDBImmutableFile]]
+ private lazy val fm = CheckpointFileManager.create(new Path(dfsRootDir), hadoopConf)
+ private val fs = new Path(dfsRootDir).getFileSystem(hadoopConf)
+
+ /** Save all the files in given local checkpoint directory as a committed version in DFS */
+ def saveCheckpointToDfs(checkpointDir: File, version: Long, numKeys: Long): Unit = {
+ logFilesInDir(checkpointDir, s"Saving checkpoint files for version $version")
+ val allCheckpointFiles = listRocksDBFiles(checkpointDir)
+ val (localImmutableFiles, localOtherFiles) = allCheckpointFiles.partition(isImmutableFile)
Review comment:
Per https://github.com/apache/spark/pull/32272#discussion_r632908535, we only generate the RocksDBLogFile object in the operation of saving checkpoint here.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32582: [SPARK-35436][SS] RocksDBFileManager - save checkpoint to DFS
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32582:
URL: https://github.com/apache/spark/pull/32582#issuecomment-857320972
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/44041/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32582: [SPARK-35436][SS] RocksDBFileManager - save checkpoint to DFS
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32582:
URL: https://github.com/apache/spark/pull/32582#issuecomment-857411039
**[Test build #139529 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/139529/testReport)** for PR 32582 at commit [`c05b140`](https://github.com/apache/spark/commit/c05b140f73f855a148ccedaf753841fb94a4ab1d).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #32582: [SPARK-35436][SS] RocksDBFileManager - save checkpoint to DFS
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #32582:
URL: https://github.com/apache/spark/pull/32582#issuecomment-853051027
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/139221/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #32582: [SPARK-35436][SS] RocksDBFileManager - save checkpoint to DFS
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #32582:
URL: https://github.com/apache/spark/pull/32582#issuecomment-853783833
**[Test build #139285 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/139285/testReport)** for PR 32582 at commit [`f5a6104`](https://github.com/apache/spark/commit/f5a6104e6930dccad71db72bfc6ee8c47d9933da).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] xuanyuanking commented on a change in pull request #32582: [SPARK-35436][SS] RocksDBFileManager - save checkpoint to DFS
Posted by GitBox <gi...@apache.org>.
xuanyuanking commented on a change in pull request #32582:
URL: https://github.com/apache/spark/pull/32582#discussion_r643852461
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/state/RocksDBFileManager.scala
##########
@@ -17,18 +17,270 @@
package org.apache.spark.sql.execution.streaming.state
-import java.io.File
+import java.io.{File, FileInputStream, InputStream}
import java.nio.charset.StandardCharsets.UTF_8
import java.nio.file.Files
+import java.util.UUID
+import java.util.concurrent.ConcurrentHashMap
+import java.util.zip.{ZipEntry, ZipOutputStream}
+import scala.collection.JavaConverters._
import scala.collection.Seq
import com.fasterxml.jackson.annotation.JsonInclude.Include
import com.fasterxml.jackson.databind.{DeserializationFeature, ObjectMapper}
import com.fasterxml.jackson.module.scala.{DefaultScalaModule, ScalaObjectMapper}
+import org.apache.commons.io.{FilenameUtils, IOUtils}
+import org.apache.hadoop.conf.Configuration
+import org.apache.hadoop.fs.Path
import org.json4s.NoTypeHints
import org.json4s.jackson.Serialization
+import org.apache.spark.internal.Logging
+import org.apache.spark.sql.execution.streaming.CheckpointFileManager
+import org.apache.spark.util.Utils
+
+/**
+ * Class responsible for syncing RocksDB checkpoint files from local disk to DFS.
+ * Every version checkpoint is saved in specific directory structure that allows successive
+ * versions to reuse to SST data files and archived log files. This allows each commit to be
+ * incremental, only new SST files and archived log files generated by RocksDB will be uploaded.
+ * The directory structures on local disk and in DFS are as follows.
+ *
+ * Local checkpoint dir structure
+ * ------------------------------
+ * RocksDB generates a bunch of files in the local checkpoint directory. The most important among
+ * them are the SST files; they are the actual log structured data files. Rest of the files contain
+ * the metadata necessary for RocksDB to read the SST files and start from the checkpoint.
+ * Note that the SST files are hard links to files in the RocksDB's working directory, and therefore
+ * successive checkpoints can share some of the SST files. So these SST files have to be copied to
+ * DFS in shared directory such that different committed versions can save them.
+ *
+ * We consider both SST files and archived log files as immutable files which can be shared between
+ * different checkpoints.
+ *
+ * localCheckpointDir
+ * |
+ * +-- OPTIONS-000005
+ * +-- MANIFEST-000008
+ * +-- CURRENT
+ * +-- 00007.sst
+ * +-- 00011.sst
+ * +-- archive
+ * | +-- 00008.log
+ * | +-- 00013.log
+ * ...
+ *
+ *
+ * DFS directory structure after saving to DFS as version 10
+ * -----------------------------------------------------------
+ * The SST and archived log files are given unique file names and copied to the shared subdirectory.
+ * Every version maintains a mapping of local immutable file name to the unique file name in DFS.
+ * This mapping is saved in a JSON file (named `metadata`), which is zipped along with other
+ * checkpoint files into a single file `[version].zip`.
+ *
+ * dfsRootDir
+ * |
+ * +-- SSTs
+ * | +-- 00007-[uuid1].sst
+ * | +-- 00011-[uuid2].sst
+ * +-- logs
+ * | +-- 00008-[uuid3].log
+ * | +-- 00013-[uuid4].log
+ * +-- 10.zip
+ * | +-- metadata <--- contains mapping between 00007.sst and [uuid1].sst,
+ * and the mapping between 00008.log and [uuid3].log
+ * | +-- OPTIONS-000005
+ * | +-- MANIFEST-000008
+ * | +-- CURRENT
+ * | ...
+ * |
+ * +-- 9.zip
+ * +-- 8.zip
+ * ...
+ *
+ * Note the following.
+ * - Each [version].zip is a complete description of all the data and metadata needed to recover
+ * a RocksDB instance at the corresponding version. This is unlike the [version].delta files of
+ * HDFSBackedStateStore where previous delta files needs to be read to be recovered.
+ * - This is safe wrt speculatively executed tasks running concurrently in different executors
+ * as each task would upload a different copy of the generated immutable files and
+ * atomically update the [version].zip.
+ * - Immutable files are identified uniquely based on their file name and file size.
+ * - Immutable files can be reused only across adjacent checkpoints/versions.
+ * - This class is thread-safe. Specifically, it is safe to concurrently delete old files from a
+ * different thread than the task thread saving files.
+ *
+ * @param dfsRootDir Directory where the [version].zip files will be stored
+ * @param localTempDir Local directory for temporary work
+ * @param hadoopConf Hadoop configuration for talking to DFS
+ * @param loggingId Id that will be prepended in logs for isolating concurrent RocksDBs
+ */
+class RocksDBFileManager(
+ dfsRootDir: String,
+ localTempDir: File,
+ hadoopConf: Configuration,
+ loggingId: String = "")
+ extends Logging {
+
+ import RocksDBImmutableFile._
+
+ private val versionToRocksDBFiles = new ConcurrentHashMap[Long, Seq[RocksDBImmutableFile]]
+ private lazy val fm = CheckpointFileManager.create(new Path(dfsRootDir), hadoopConf)
+ private val fs = new Path(dfsRootDir).getFileSystem(hadoopConf)
+
+ /** Save all the files in given local checkpoint directory as a committed version in DFS */
+ def saveCheckpointToDfs(checkpointDir: File, version: Long, numKeys: Long): Unit = {
+ logFilesInDir(checkpointDir, s"Saving checkpoint files for version $version")
+ val allCheckpointFiles = listRocksDBFiles(checkpointDir)
+ val (localImmutableFiles, localOtherFiles) = allCheckpointFiles.partition(isImmutableFile)
+ val rocksDBFiles = saveImmutableFilesToDfs(version, checkpointDir, localImmutableFiles)
+ val metadata = RocksDBCheckpointMetadata(rocksDBFiles, numKeys)
+ rememberImmutableFiles(version, rocksDBFiles)
+ val metadataFile = localMetadataFile(checkpointDir)
+ metadata.writeToFile(metadataFile)
+ logInfo(s"Written metadata for version $version:\n${metadata.prettyJson}")
+
+ if (version <= 0 && numKeys == 0) {
+ // If we're writing the initial version and there's no data, we have to explicitly initialize
+ // the root directory. Normally saveImmutableFilesToDfs will do this initialization, but
+ // when there's no data that method won't write any files, and zipToDfsFile uses the
+ // CheckpointFileManager.createAtomic API which doesn't auto-initialize parent directories.
+ fm.mkdirs(new Path(dfsRootDir))
+ }
+ zipToDfsFile(localOtherFiles :+ metadataFile, dfsBatchZipFile(version))
+ logInfo(s"Saved checkpoint file for version $version")
+ }
+
+ /** Save immutable files to DFS directory */
+ private def saveImmutableFilesToDfs(
+ version: Long,
+ localDir: File,
+ localFiles: Seq[File]): Seq[RocksDBImmutableFile] = {
+ // Get the immutable files used in previous versions, as some of those uploaded files can be
+ // reused for this version
+ logInfo(s"Saving RocksDB files to DFS for $version")
+ val prevFilesToSizes = versionToRocksDBFiles.values.asScala.flatten.map { f =>
+ f.localFileName -> f
+ }.toMap
+
+ var bytesCopied = 0L
+ var filesCopied = 0L
+ var filesReused = 0L
+
+ val immutableFiles = localFiles.map { localFile =>
+ prevFilesToSizes
+ .get(localFile.getName)
+ .filter(_.isSameFile(localFile))
+ .map { reusable =>
+ filesReused += 1
+ reusable
+ }.getOrElse {
+ val localFileName = localFile.getName
+ val dfsFileName = newDFSFileName(localFileName)
+ val dfsFile = dfsFilePath(dfsFileName)
+ // Note: The implementation of copyFromLocalFile() closes the output stream when there is
+ // any exception while copying. So this may generate partial files on DFS. But that is
+ // okay because until the main [version].zip file is written, those partial files are
+ // not going to be used at all. Eventually these files should get cleared.
+ fs.copyFromLocalFile(
+ new Path(localFile.getAbsoluteFile.toURI), dfsFile)
+ val localFileSize = localFile.length()
+ logInfo(s"Copied $localFile to $dfsFile - $localFileSize bytes")
+ filesCopied += 1
+ bytesCopied += localFileSize
+
+ RocksDBImmutableFile(localFile.getName, dfsFileName, localFileSize)
+ }
+ }
+ logInfo(s"Copied $filesCopied files ($bytesCopied bytes) from local to" +
Review comment:
Sure, done in 3800c51
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on pull request #32582: [WIP][SPARK-35436] RocksDBFileManager - save checkpoint to DFS
Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on pull request #32582:
URL: https://github.com/apache/spark/pull/32582#issuecomment-843287022
cc @viirya
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #32582: [SPARK-35436][SS] RocksDBFileManager - save checkpoint to DFS
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #32582:
URL: https://github.com/apache/spark/pull/32582#issuecomment-857342024
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/44041/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] xuanyuanking commented on pull request #32582: [SPARK-35436][SS] RocksDBFileManager - save checkpoint to DFS
Posted by GitBox <gi...@apache.org>.
xuanyuanking commented on pull request #32582:
URL: https://github.com/apache/spark/pull/32582#issuecomment-856883464
Gentle ping @viirya @HeartSaVioR Do you think this is ready to go?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] xuanyuanking commented on a change in pull request #32582: [SPARK-35436][SS] RocksDBFileManager - save checkpoint to DFS
Posted by GitBox <gi...@apache.org>.
xuanyuanking commented on a change in pull request #32582:
URL: https://github.com/apache/spark/pull/32582#discussion_r644679005
##########
File path: core/src/main/scala/org/apache/spark/util/Utils.scala
##########
@@ -1129,6 +1129,15 @@ private[spark] object Utils extends Logging {
s"${TimeUnit.NANOSECONDS.toMillis(System.nanoTime() - startTimeNs)} ms"
}
+ /**
+ * Lists files recursively.
+ */
+ def recursiveList(f: File): Array[File] = {
+ require(f.isDirectory)
+ val current = f.listFiles
+ current ++ current.filter(_.isDirectory).flatMap(recursiveList)
Review comment:
Sure, done in f5a6104
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] viirya commented on a change in pull request #32582: [SPARK-35436][SS] RocksDBFileManager - save checkpoint to DFS
Posted by GitBox <gi...@apache.org>.
viirya commented on a change in pull request #32582:
URL: https://github.com/apache/spark/pull/32582#discussion_r644473480
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/state/RocksDBFileManager.scala
##########
@@ -17,18 +17,265 @@
package org.apache.spark.sql.execution.streaming.state
-import java.io.File
+import java.io.{File, FileInputStream, InputStream}
import java.nio.charset.StandardCharsets.UTF_8
import java.nio.file.Files
+import java.util.UUID
+import java.util.concurrent.ConcurrentHashMap
+import java.util.zip.{ZipEntry, ZipOutputStream}
+import scala.collection.JavaConverters._
import scala.collection.Seq
import com.fasterxml.jackson.annotation.JsonInclude.Include
import com.fasterxml.jackson.databind.{DeserializationFeature, ObjectMapper}
import com.fasterxml.jackson.module.scala.{DefaultScalaModule, ScalaObjectMapper}
+import org.apache.commons.io.{FilenameUtils, IOUtils}
+import org.apache.hadoop.conf.Configuration
+import org.apache.hadoop.fs.Path
import org.json4s.NoTypeHints
import org.json4s.jackson.Serialization
+import org.apache.spark.internal.Logging
+import org.apache.spark.sql.execution.streaming.CheckpointFileManager
+import org.apache.spark.util.Utils
+
+/**
+ * Class responsible for syncing RocksDB checkpoint files from local disk to DFS.
+ * For each version, checkpoint is saved in specific directory structure that allows successive
+ * versions to reuse to SST data files and archived log files. This allows each commit to be
+ * incremental, only new SST files and archived log files generated by RocksDB will be uploaded.
+ * The directory structures on local disk and in DFS are as follows.
+ *
+ * Local checkpoint dir structure
+ * ------------------------------
+ * RocksDB generates a bunch of files in the local checkpoint directory. The most important among
+ * them are the SST files; they are the actual log structured data files. Rest of the files contain
+ * the metadata necessary for RocksDB to read the SST files and start from the checkpoint.
+ * Note that the SST files are hard links to files in the RocksDB's working directory, and therefore
+ * successive checkpoints can share some of the SST files. So these SST files have to be copied to
+ * DFS in shared directory such that different committed versions can save them.
+ *
+ * We consider both SST files and archived log files as immutable files which can be shared between
+ * different checkpoints.
+ *
+ * localCheckpointDir
+ * |
+ * +-- OPTIONS-000005
+ * +-- MANIFEST-000008
+ * +-- CURRENT
+ * +-- 00007.sst
+ * +-- 00011.sst
+ * +-- archive
+ * | +-- 00008.log
+ * | +-- 00013.log
+ * ...
+ *
+ *
+ * DFS directory structure after saving to DFS as version 10
+ * -----------------------------------------------------------
+ * The SST and archived log files are given unique file names and copied to the shared subdirectory.
+ * Every version maintains a mapping of local immutable file name to the unique file name in DFS.
+ * This mapping is saved in a JSON file (named `metadata`), which is zipped along with other
+ * checkpoint files into a single file `[version].zip`.
+ *
+ * dfsRootDir
+ * |
+ * +-- SSTs
+ * | +-- 00007-[uuid1].sst
+ * | +-- 00011-[uuid2].sst
+ * +-- logs
+ * | +-- 00008-[uuid3].log
+ * | +-- 00013-[uuid4].log
+ * +-- 10.zip
+ * | +-- metadata <--- contains mapping between 00007.sst and [uuid1].sst,
+ * and the mapping between 00008.log and [uuid3].log
+ * | +-- OPTIONS-000005
+ * | +-- MANIFEST-000008
+ * | +-- CURRENT
+ * | ...
+ * |
+ * +-- 9.zip
+ * +-- 8.zip
+ * ...
+ *
+ * Note the following.
+ * - Each [version].zip is a complete description of all the data and metadata needed to recover
+ * a RocksDB instance at the corresponding version. The SST files and log files are not included
+ * in the zip files, they can be shared cross different versions. This is unlike the
+ * [version].delta files of HDFSBackedStateStore where previous delta files needs to be read
+ * to be recovered.
+ * - This is safe wrt speculatively executed tasks running concurrently in different executors
+ * as each task would upload a different copy of the generated immutable files and
+ * atomically update the [version].zip.
+ * - Immutable files are identified uniquely based on their file name and file size.
+ * - Immutable files can be reused only across adjacent checkpoints/versions.
+ * - This class is thread-safe. Specifically, it is safe to concurrently delete old files from a
+ * different thread than the task thread saving files.
Review comment:
Can you talk more about there this claim (thread-safe) applies? Where we delete old files?
Do you mean `versionToRocksDBFiles`? When we prepare files for new version, there is another thread (maintenance thread) deleting old files?
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/state/RocksDBFileManager.scala
##########
@@ -17,18 +17,265 @@
package org.apache.spark.sql.execution.streaming.state
-import java.io.File
+import java.io.{File, FileInputStream, InputStream}
import java.nio.charset.StandardCharsets.UTF_8
import java.nio.file.Files
+import java.util.UUID
+import java.util.concurrent.ConcurrentHashMap
+import java.util.zip.{ZipEntry, ZipOutputStream}
+import scala.collection.JavaConverters._
import scala.collection.Seq
import com.fasterxml.jackson.annotation.JsonInclude.Include
import com.fasterxml.jackson.databind.{DeserializationFeature, ObjectMapper}
import com.fasterxml.jackson.module.scala.{DefaultScalaModule, ScalaObjectMapper}
+import org.apache.commons.io.{FilenameUtils, IOUtils}
+import org.apache.hadoop.conf.Configuration
+import org.apache.hadoop.fs.Path
import org.json4s.NoTypeHints
import org.json4s.jackson.Serialization
+import org.apache.spark.internal.Logging
+import org.apache.spark.sql.execution.streaming.CheckpointFileManager
+import org.apache.spark.util.Utils
+
+/**
+ * Class responsible for syncing RocksDB checkpoint files from local disk to DFS.
+ * For each version, checkpoint is saved in specific directory structure that allows successive
+ * versions to reuse to SST data files and archived log files. This allows each commit to be
+ * incremental, only new SST files and archived log files generated by RocksDB will be uploaded.
+ * The directory structures on local disk and in DFS are as follows.
+ *
+ * Local checkpoint dir structure
+ * ------------------------------
+ * RocksDB generates a bunch of files in the local checkpoint directory. The most important among
+ * them are the SST files; they are the actual log structured data files. Rest of the files contain
+ * the metadata necessary for RocksDB to read the SST files and start from the checkpoint.
+ * Note that the SST files are hard links to files in the RocksDB's working directory, and therefore
+ * successive checkpoints can share some of the SST files. So these SST files have to be copied to
+ * DFS in shared directory such that different committed versions can save them.
+ *
+ * We consider both SST files and archived log files as immutable files which can be shared between
+ * different checkpoints.
+ *
+ * localCheckpointDir
+ * |
+ * +-- OPTIONS-000005
+ * +-- MANIFEST-000008
+ * +-- CURRENT
+ * +-- 00007.sst
+ * +-- 00011.sst
+ * +-- archive
+ * | +-- 00008.log
+ * | +-- 00013.log
+ * ...
+ *
+ *
+ * DFS directory structure after saving to DFS as version 10
+ * -----------------------------------------------------------
+ * The SST and archived log files are given unique file names and copied to the shared subdirectory.
+ * Every version maintains a mapping of local immutable file name to the unique file name in DFS.
+ * This mapping is saved in a JSON file (named `metadata`), which is zipped along with other
+ * checkpoint files into a single file `[version].zip`.
+ *
+ * dfsRootDir
+ * |
+ * +-- SSTs
+ * | +-- 00007-[uuid1].sst
+ * | +-- 00011-[uuid2].sst
+ * +-- logs
+ * | +-- 00008-[uuid3].log
+ * | +-- 00013-[uuid4].log
+ * +-- 10.zip
+ * | +-- metadata <--- contains mapping between 00007.sst and [uuid1].sst,
+ * and the mapping between 00008.log and [uuid3].log
+ * | +-- OPTIONS-000005
+ * | +-- MANIFEST-000008
+ * | +-- CURRENT
+ * | ...
+ * |
+ * +-- 9.zip
+ * +-- 8.zip
+ * ...
+ *
+ * Note the following.
+ * - Each [version].zip is a complete description of all the data and metadata needed to recover
+ * a RocksDB instance at the corresponding version. The SST files and log files are not included
+ * in the zip files, they can be shared cross different versions. This is unlike the
+ * [version].delta files of HDFSBackedStateStore where previous delta files needs to be read
+ * to be recovered.
+ * - This is safe wrt speculatively executed tasks running concurrently in different executors
+ * as each task would upload a different copy of the generated immutable files and
+ * atomically update the [version].zip.
+ * - Immutable files are identified uniquely based on their file name and file size.
+ * - Immutable files can be reused only across adjacent checkpoints/versions.
+ * - This class is thread-safe. Specifically, it is safe to concurrently delete old files from a
+ * different thread than the task thread saving files.
+ *
+ * @param dfsRootDir Directory where the [version].zip files will be stored
+ * @param localTempDir Local directory for temporary work
+ * @param hadoopConf Hadoop configuration for talking to DFS
+ * @param loggingId Id that will be prepended in logs for isolating concurrent RocksDBs
+ */
+class RocksDBFileManager(
+ dfsRootDir: String,
+ localTempDir: File,
+ hadoopConf: Configuration,
+ loggingId: String = "")
+ extends Logging {
+
+ import RocksDBImmutableFile._
+
+ private val versionToRocksDBFiles = new ConcurrentHashMap[Long, Seq[RocksDBImmutableFile]]
+ private lazy val fm = CheckpointFileManager.create(new Path(dfsRootDir), hadoopConf)
+ private val fs = new Path(dfsRootDir).getFileSystem(hadoopConf)
+
+ /** Save all the files in given local checkpoint directory as a committed version in DFS */
+ def saveCheckpointToDfs(checkpointDir: File, version: Long, numKeys: Long): Unit = {
+ logFilesInDir(checkpointDir, s"Saving checkpoint files for version $version")
+ val (localImmutableFiles, localOtherFiles) = listRocksDBFiles(checkpointDir)
+ val rocksDBFiles = saveImmutableFilesToDfs(version, localImmutableFiles)
+ val metadata = RocksDBCheckpointMetadata(rocksDBFiles, numKeys)
+ val metadataFile = localMetadataFile(checkpointDir)
+ metadata.writeToFile(metadataFile)
+ logInfo(s"Written metadata for version $version:\n${metadata.prettyJson}")
+
+ if (version <= 0 && numKeys == 0) {
+ // If we're writing the initial version and there's no data, we have to explicitly initialize
+ // the root directory. Normally saveImmutableFilesToDfs will do this initialization, but
+ // when there's no data that method won't write any files, and zipToDfsFile uses the
+ // CheckpointFileManager.createAtomic API which doesn't auto-initialize parent directories.
+ fm.mkdirs(new Path(dfsRootDir))
Review comment:
Should we check if the dfs root dir already exists?
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/state/RocksDBFileManager.scala
##########
@@ -17,18 +17,265 @@
package org.apache.spark.sql.execution.streaming.state
-import java.io.File
+import java.io.{File, FileInputStream, InputStream}
import java.nio.charset.StandardCharsets.UTF_8
import java.nio.file.Files
+import java.util.UUID
+import java.util.concurrent.ConcurrentHashMap
+import java.util.zip.{ZipEntry, ZipOutputStream}
+import scala.collection.JavaConverters._
import scala.collection.Seq
import com.fasterxml.jackson.annotation.JsonInclude.Include
import com.fasterxml.jackson.databind.{DeserializationFeature, ObjectMapper}
import com.fasterxml.jackson.module.scala.{DefaultScalaModule, ScalaObjectMapper}
+import org.apache.commons.io.{FilenameUtils, IOUtils}
+import org.apache.hadoop.conf.Configuration
+import org.apache.hadoop.fs.Path
import org.json4s.NoTypeHints
import org.json4s.jackson.Serialization
+import org.apache.spark.internal.Logging
+import org.apache.spark.sql.execution.streaming.CheckpointFileManager
+import org.apache.spark.util.Utils
+
+/**
+ * Class responsible for syncing RocksDB checkpoint files from local disk to DFS.
+ * For each version, checkpoint is saved in specific directory structure that allows successive
+ * versions to reuse to SST data files and archived log files. This allows each commit to be
+ * incremental, only new SST files and archived log files generated by RocksDB will be uploaded.
+ * The directory structures on local disk and in DFS are as follows.
+ *
+ * Local checkpoint dir structure
+ * ------------------------------
+ * RocksDB generates a bunch of files in the local checkpoint directory. The most important among
+ * them are the SST files; they are the actual log structured data files. Rest of the files contain
+ * the metadata necessary for RocksDB to read the SST files and start from the checkpoint.
+ * Note that the SST files are hard links to files in the RocksDB's working directory, and therefore
+ * successive checkpoints can share some of the SST files. So these SST files have to be copied to
+ * DFS in shared directory such that different committed versions can save them.
+ *
+ * We consider both SST files and archived log files as immutable files which can be shared between
+ * different checkpoints.
+ *
+ * localCheckpointDir
+ * |
+ * +-- OPTIONS-000005
+ * +-- MANIFEST-000008
+ * +-- CURRENT
+ * +-- 00007.sst
+ * +-- 00011.sst
+ * +-- archive
+ * | +-- 00008.log
+ * | +-- 00013.log
+ * ...
+ *
+ *
+ * DFS directory structure after saving to DFS as version 10
+ * -----------------------------------------------------------
+ * The SST and archived log files are given unique file names and copied to the shared subdirectory.
+ * Every version maintains a mapping of local immutable file name to the unique file name in DFS.
+ * This mapping is saved in a JSON file (named `metadata`), which is zipped along with other
+ * checkpoint files into a single file `[version].zip`.
+ *
+ * dfsRootDir
+ * |
+ * +-- SSTs
+ * | +-- 00007-[uuid1].sst
+ * | +-- 00011-[uuid2].sst
+ * +-- logs
+ * | +-- 00008-[uuid3].log
+ * | +-- 00013-[uuid4].log
+ * +-- 10.zip
+ * | +-- metadata <--- contains mapping between 00007.sst and [uuid1].sst,
+ * and the mapping between 00008.log and [uuid3].log
+ * | +-- OPTIONS-000005
+ * | +-- MANIFEST-000008
+ * | +-- CURRENT
+ * | ...
+ * |
+ * +-- 9.zip
+ * +-- 8.zip
+ * ...
+ *
+ * Note the following.
+ * - Each [version].zip is a complete description of all the data and metadata needed to recover
+ * a RocksDB instance at the corresponding version. The SST files and log files are not included
+ * in the zip files, they can be shared cross different versions. This is unlike the
+ * [version].delta files of HDFSBackedStateStore where previous delta files needs to be read
+ * to be recovered.
+ * - This is safe wrt speculatively executed tasks running concurrently in different executors
+ * as each task would upload a different copy of the generated immutable files and
+ * atomically update the [version].zip.
+ * - Immutable files are identified uniquely based on their file name and file size.
+ * - Immutable files can be reused only across adjacent checkpoints/versions.
+ * - This class is thread-safe. Specifically, it is safe to concurrently delete old files from a
+ * different thread than the task thread saving files.
+ *
+ * @param dfsRootDir Directory where the [version].zip files will be stored
+ * @param localTempDir Local directory for temporary work
+ * @param hadoopConf Hadoop configuration for talking to DFS
+ * @param loggingId Id that will be prepended in logs for isolating concurrent RocksDBs
+ */
+class RocksDBFileManager(
+ dfsRootDir: String,
+ localTempDir: File,
+ hadoopConf: Configuration,
+ loggingId: String = "")
+ extends Logging {
+
+ import RocksDBImmutableFile._
+
+ private val versionToRocksDBFiles = new ConcurrentHashMap[Long, Seq[RocksDBImmutableFile]]
+ private lazy val fm = CheckpointFileManager.create(new Path(dfsRootDir), hadoopConf)
+ private val fs = new Path(dfsRootDir).getFileSystem(hadoopConf)
+
+ /** Save all the files in given local checkpoint directory as a committed version in DFS */
+ def saveCheckpointToDfs(checkpointDir: File, version: Long, numKeys: Long): Unit = {
+ logFilesInDir(checkpointDir, s"Saving checkpoint files for version $version")
+ val (localImmutableFiles, localOtherFiles) = listRocksDBFiles(checkpointDir)
+ val rocksDBFiles = saveImmutableFilesToDfs(version, localImmutableFiles)
+ val metadata = RocksDBCheckpointMetadata(rocksDBFiles, numKeys)
+ val metadataFile = localMetadataFile(checkpointDir)
+ metadata.writeToFile(metadataFile)
+ logInfo(s"Written metadata for version $version:\n${metadata.prettyJson}")
+
+ if (version <= 0 && numKeys == 0) {
+ // If we're writing the initial version and there's no data, we have to explicitly initialize
+ // the root directory. Normally saveImmutableFilesToDfs will do this initialization, but
+ // when there's no data that method won't write any files, and zipToDfsFile uses the
+ // CheckpointFileManager.createAtomic API which doesn't auto-initialize parent directories.
+ fm.mkdirs(new Path(dfsRootDir))
+ }
+ zipToDfsFile(localOtherFiles :+ metadataFile, dfsBatchZipFile(version))
+ logInfo(s"Saved checkpoint file for version $version")
+ }
+
+ /** Save immutable files to DFS directory */
+ private def saveImmutableFilesToDfs(
+ version: Long,
+ localFiles: Seq[File]): Seq[RocksDBImmutableFile] = {
+ // Get the immutable files used in previous versions, as some of those uploaded files can be
+ // reused for this version
+ logInfo(s"Saving RocksDB files to DFS for $version")
+ val prevFilesToSizes = versionToRocksDBFiles.values.asScala.flatten.map { f =>
+ f.localFileName -> f
+ }.toMap
+
+ var bytesCopied = 0L
+ var filesCopied = 0L
+ var filesReused = 0L
+
+ val immutableFiles = localFiles.map { localFile =>
+ prevFilesToSizes
+ .get(localFile.getName)
+ .filter(_.isSameFile(localFile))
+ .map { reusable =>
+ filesReused += 1
+ reusable
+ }.getOrElse {
+ val localFileName = localFile.getName
+ val dfsFileName = newDFSFileName(localFileName)
+ val dfsFile = dfsFilePath(dfsFileName)
+ // Note: The implementation of copyFromLocalFile() closes the output stream when there is
+ // any exception while copying. So this may generate partial files on DFS. But that is
+ // okay because until the main [version].zip file is written, those partial files are
+ // not going to be used at all. Eventually these files should get cleared.
+ fs.copyFromLocalFile(
+ new Path(localFile.getAbsoluteFile.toURI), dfsFile)
+ val localFileSize = localFile.length()
+ logInfo(s"Copied $localFile to $dfsFile - $localFileSize bytes")
+ filesCopied += 1
+ bytesCopied += localFileSize
+
+ RocksDBImmutableFile(localFile.getName, dfsFileName, localFileSize)
+ }
+ }
+ logInfo(s"Copied $filesCopied files ($bytesCopied bytes) from local to" +
+ s" DFS for version $version. $filesReused files reused without copying.")
+ versionToRocksDBFiles.put(version, immutableFiles)
+
+ immutableFiles
+ }
+
+ /**
+ * Compress files to a single zip file in DFS. Only the file names are embedded in the zip.
+ * Any error while writing will ensure that the file is not written.
+ */
+ private def zipToDfsFile(files: Seq[File], dfsZipFile: Path): Unit = {
+ lazy val filesStr = s"$dfsZipFile\n\t${files.mkString("\n\t")}"
+ var in: InputStream = null
+ val out = fm.createAtomic(dfsZipFile, overwriteIfPossible = true)
+ var totalBytes = 0L
+ val zout = new ZipOutputStream(out)
+ try {
+ files.foreach { file =>
+ zout.putNextEntry(new ZipEntry(file.getName))
+ in = new FileInputStream(file)
+ val bytes = IOUtils.copy(in, zout)
+ in.close()
+ zout.closeEntry()
+ totalBytes += bytes
+ }
+ zout.close() // so that any error in closing also cancels the output stream
+ logInfo(s"Zipped $totalBytes bytes (before compression) to $filesStr")
Review comment:
The zipped file is `dfsZipFile`, right? why `filesStr` here?
##########
File path: sql/core/src/test/scala/org/apache/spark/sql/execution/streaming/state/RocksDBSuite.scala
##########
@@ -20,12 +20,86 @@ package org.apache.spark.sql.execution.streaming.state
import java.io._
import java.nio.charset.Charset
+import scala.language.implicitConversions
+
import org.apache.commons.io.FileUtils
+import org.apache.hadoop.conf.Configuration
import org.apache.spark._
+import org.apache.spark.sql.catalyst.util.quietly
+import org.apache.spark.sql.execution.streaming.CreateAtomicTestManager
+import org.apache.spark.sql.internal.SQLConf
+import org.apache.spark.util.Utils
class RocksDBSuite extends SparkFunSuite {
+ test("RocksDBFileManager: upload only new immutable files") {
+ val dfsRootDir = Utils.createTempDir().getAbsolutePath
Review comment:
Wrap using `withTempDir`?
##########
File path: core/src/main/scala/org/apache/spark/util/Utils.scala
##########
@@ -1129,6 +1129,15 @@ private[spark] object Utils extends Logging {
s"${TimeUnit.NANOSECONDS.toMillis(System.nanoTime() - startTimeNs)} ms"
}
+ /**
+ * Lists files recursively.
+ */
+ def recursiveList(f: File): Array[File] = {
+ require(f.isDirectory)
+ val current = f.listFiles
+ current ++ current.filter(_.isDirectory).flatMap(recursiveList)
Review comment:
nit: I think this can be done easily using non-recursive approach?
##########
File path: sql/core/src/test/scala/org/apache/spark/sql/execution/streaming/state/RocksDBSuite.scala
##########
@@ -54,4 +130,30 @@ class RocksDBSuite extends SparkFunSuite {
"""{"sstFiles":[{"localFileName":"00001.sst","dfsSstFileName":"00001-uuid.sst","sizeBytes":12345678901234}],"logFiles":[{"localFileName":"00001.log","dfsLogFileName":"00001-uuid.log","sizeBytes":12345678901234}],"numKeys":12345678901234}""")
// scalastyle:on line.size.limit
}
+
+ def generateFiles(dir: String, fileToLengths: Seq[(String, Int)]): Unit = {
+ fileToLengths.foreach { case (fileName, length) =>
+ val file = new File(dir, fileName)
+ FileUtils.write(file, "a" * length)
+ }
+ }
+
+ def saveCheckpointFiles(
+ fileManager: RocksDBFileManager,
+ fileToLengths: Seq[(String, Int)],
+ version: Int,
+ numKeys: Int): Unit = {
+ val checkpointDir = Utils.createTempDir().getAbsolutePath // local dir to create checkpoints
Review comment:
Isn't `checkpointDir` already be defined in `RocksDBFileManager`?
##########
File path: sql/core/src/test/scala/org/apache/spark/sql/execution/streaming/state/RocksDBSuite.scala
##########
@@ -20,12 +20,88 @@ package org.apache.spark.sql.execution.streaming.state
import java.io._
import java.nio.charset.Charset
+import scala.language.implicitConversions
+
import org.apache.commons.io.FileUtils
+import org.apache.hadoop.conf.Configuration
import org.apache.spark._
+import org.apache.spark.sql.catalyst.util.quietly
+import org.apache.spark.sql.execution.streaming.CreateAtomicTestManager
+import org.apache.spark.sql.internal.SQLConf
+import org.apache.spark.util.Utils
class RocksDBSuite extends SparkFunSuite {
+ test("RocksDBFileManager: upload only new immutable files") {
+ withTempDir { dir =>
+ val dfsRootDir = dir.getAbsolutePath
+ val fileManager = new RocksDBFileManager(
+ dfsRootDir, Utils.createTempDir(), new Configuration)
+ val sstDir = s"$dfsRootDir/SSTs"
+ def numRemoteSSTFiles: Int = listFiles(sstDir).length
+ val logDir = s"$dfsRootDir/logs"
+ def numRemoteLogFiles: Int = listFiles(logDir).length
+
+ // Save a version of checkpoint files
+ val cpFiles1 = Seq(
+ "sst-file1.sst" -> 10,
+ "sst-file2.sst" -> 20,
+ "other-file1" -> 100,
+ "other-file2" -> 200,
+ "archive/00001.log" -> 1000,
+ "archive/00002.log" -> 2000
+ )
+ saveCheckpointFiles(fileManager, cpFiles1, version = 1, numKeys = 101)
+ assert(numRemoteSSTFiles == 2) // 2 sst files copied
+ assert(numRemoteLogFiles == 2) // 2 log files copied
+
+ // Save SAME version again with different checkpoint files and verify
+ val cpFiles1_ = Seq(
+ "sst-file1.sst" -> 10, // same SST file as before, should not get copied
+ "sst-file2.sst" -> 25, // new SST file with same name as before, but different length
+ "sst-file3.sst" -> 30, // new SST file
+ "other-file1" -> 100, // same non-SST file as before, should not get copied
+ "other-file2" -> 210, // new non-SST file with same name as before, but different length
+ "other-file3" -> 300, // new non-SST file
+ "archive/00001.log" -> 1000, // same log file as before, should not get copied
+ "archive/00002.log" -> 2500, // new log file with same name as before, but different length
+ "archive/00003.log" -> 3000 // new log file
+ )
+ saveCheckpointFiles(fileManager, cpFiles1_, version = 1, numKeys = 1001)
Review comment:
`version = 1`? In practice, will we save checkpoint files for same version? I.e. for same micro-batch?
##########
File path: sql/core/src/test/scala/org/apache/spark/sql/execution/streaming/state/RocksDBSuite.scala
##########
@@ -20,12 +20,88 @@ package org.apache.spark.sql.execution.streaming.state
import java.io._
import java.nio.charset.Charset
+import scala.language.implicitConversions
+
import org.apache.commons.io.FileUtils
+import org.apache.hadoop.conf.Configuration
import org.apache.spark._
+import org.apache.spark.sql.catalyst.util.quietly
+import org.apache.spark.sql.execution.streaming.CreateAtomicTestManager
+import org.apache.spark.sql.internal.SQLConf
+import org.apache.spark.util.Utils
class RocksDBSuite extends SparkFunSuite {
+ test("RocksDBFileManager: upload only new immutable files") {
+ withTempDir { dir =>
+ val dfsRootDir = dir.getAbsolutePath
+ val fileManager = new RocksDBFileManager(
+ dfsRootDir, Utils.createTempDir(), new Configuration)
+ val sstDir = s"$dfsRootDir/SSTs"
+ def numRemoteSSTFiles: Int = listFiles(sstDir).length
+ val logDir = s"$dfsRootDir/logs"
+ def numRemoteLogFiles: Int = listFiles(logDir).length
+
+ // Save a version of checkpoint files
+ val cpFiles1 = Seq(
+ "sst-file1.sst" -> 10,
+ "sst-file2.sst" -> 20,
+ "other-file1" -> 100,
+ "other-file2" -> 200,
+ "archive/00001.log" -> 1000,
+ "archive/00002.log" -> 2000
+ )
+ saveCheckpointFiles(fileManager, cpFiles1, version = 1, numKeys = 101)
+ assert(numRemoteSSTFiles == 2) // 2 sst files copied
+ assert(numRemoteLogFiles == 2) // 2 log files copied
+
+ // Save SAME version again with different checkpoint files and verify
+ val cpFiles1_ = Seq(
+ "sst-file1.sst" -> 10, // same SST file as before, should not get copied
+ "sst-file2.sst" -> 25, // new SST file with same name as before, but different length
+ "sst-file3.sst" -> 30, // new SST file
+ "other-file1" -> 100, // same non-SST file as before, should not get copied
+ "other-file2" -> 210, // new non-SST file with same name as before, but different length
+ "other-file3" -> 300, // new non-SST file
+ "archive/00001.log" -> 1000, // same log file as before, should not get copied
+ "archive/00002.log" -> 2500, // new log file with same name as before, but different length
+ "archive/00003.log" -> 3000 // new log file
+ )
+ saveCheckpointFiles(fileManager, cpFiles1_, version = 1, numKeys = 1001)
+ assert(numRemoteSSTFiles === 4, "shouldn't copy same files again") // 2 old + 2 new SST files
+ assert(numRemoteLogFiles === 4, "shouldn't copy same files again") // 2 old + 2 new log files
+
+ // Save another version and verify
+ val cpFiles2 = Seq(
+ "sst-file4.sst" -> 40,
+ "other-file4" -> 400,
+ "archive/00004.log" -> 4000
+ )
+ saveCheckpointFiles(fileManager, cpFiles2, version = 2, numKeys = 1501)
+ assert(numRemoteSSTFiles === 5) // 1 new file over earlier 4 files
+ assert(numRemoteLogFiles === 5) // 1 new file over earlier 4 files
+ }
+ }
+
+ test("RocksDBFileManager: error writing [version].zip cancels the output stream") {
+ quietly {
+ val hadoopConf = new Configuration()
+ hadoopConf.set(
+ SQLConf.STREAMING_CHECKPOINT_FILE_MANAGER_CLASS.parent.key,
+ classOf[CreateAtomicTestManager].getName)
+ val dfsRootDir = Utils.createTempDir().getAbsolutePath
Review comment:
Use `withTempDir` too?
##########
File path: sql/core/src/test/scala/org/apache/spark/sql/execution/streaming/state/RocksDBSuite.scala
##########
@@ -20,12 +20,88 @@ package org.apache.spark.sql.execution.streaming.state
import java.io._
import java.nio.charset.Charset
+import scala.language.implicitConversions
+
import org.apache.commons.io.FileUtils
+import org.apache.hadoop.conf.Configuration
import org.apache.spark._
+import org.apache.spark.sql.catalyst.util.quietly
+import org.apache.spark.sql.execution.streaming.CreateAtomicTestManager
+import org.apache.spark.sql.internal.SQLConf
+import org.apache.spark.util.Utils
class RocksDBSuite extends SparkFunSuite {
+ test("RocksDBFileManager: upload only new immutable files") {
+ withTempDir { dir =>
+ val dfsRootDir = dir.getAbsolutePath
+ val fileManager = new RocksDBFileManager(
+ dfsRootDir, Utils.createTempDir(), new Configuration)
+ val sstDir = s"$dfsRootDir/SSTs"
+ def numRemoteSSTFiles: Int = listFiles(sstDir).length
+ val logDir = s"$dfsRootDir/logs"
+ def numRemoteLogFiles: Int = listFiles(logDir).length
+
+ // Save a version of checkpoint files
+ val cpFiles1 = Seq(
+ "sst-file1.sst" -> 10,
+ "sst-file2.sst" -> 20,
+ "other-file1" -> 100,
+ "other-file2" -> 200,
+ "archive/00001.log" -> 1000,
+ "archive/00002.log" -> 2000
+ )
+ saveCheckpointFiles(fileManager, cpFiles1, version = 1, numKeys = 101)
+ assert(numRemoteSSTFiles == 2) // 2 sst files copied
+ assert(numRemoteLogFiles == 2) // 2 log files copied
+
+ // Save SAME version again with different checkpoint files and verify
+ val cpFiles1_ = Seq(
+ "sst-file1.sst" -> 10, // same SST file as before, should not get copied
+ "sst-file2.sst" -> 25, // new SST file with same name as before, but different length
+ "sst-file3.sst" -> 30, // new SST file
+ "other-file1" -> 100, // same non-SST file as before, should not get copied
+ "other-file2" -> 210, // new non-SST file with same name as before, but different length
+ "other-file3" -> 300, // new non-SST file
+ "archive/00001.log" -> 1000, // same log file as before, should not get copied
+ "archive/00002.log" -> 2500, // new log file with same name as before, but different length
+ "archive/00003.log" -> 3000 // new log file
+ )
+ saveCheckpointFiles(fileManager, cpFiles1_, version = 1, numKeys = 1001)
+ assert(numRemoteSSTFiles === 4, "shouldn't copy same files again") // 2 old + 2 new SST files
+ assert(numRemoteLogFiles === 4, "shouldn't copy same files again") // 2 old + 2 new log files
+
+ // Save another version and verify
+ val cpFiles2 = Seq(
+ "sst-file4.sst" -> 40,
+ "other-file4" -> 400,
+ "archive/00004.log" -> 4000
+ )
+ saveCheckpointFiles(fileManager, cpFiles2, version = 2, numKeys = 1501)
+ assert(numRemoteSSTFiles === 5) // 1 new file over earlier 4 files
+ assert(numRemoteLogFiles === 5) // 1 new file over earlier 4 files
+ }
+ }
+
+ test("RocksDBFileManager: error writing [version].zip cancels the output stream") {
+ quietly {
+ val hadoopConf = new Configuration()
+ hadoopConf.set(
+ SQLConf.STREAMING_CHECKPOINT_FILE_MANAGER_CLASS.parent.key,
+ classOf[CreateAtomicTestManager].getName)
+ val dfsRootDir = Utils.createTempDir().getAbsolutePath
+ val fileManager = new RocksDBFileManager(dfsRootDir, Utils.createTempDir(), hadoopConf)
+ val cpFiles = Seq("sst-file1.sst" -> 10, "sst-file2.sst" -> 20, "other-file1" -> 100)
+ CreateAtomicTestManager.shouldFailInCreateAtomic = true
+ CreateAtomicTestManager.cancelCalledInCreateAtomic = false
Review comment:
If `shouldFailInCreateAtomic` is true, `cancelCalledInCreateAtomic` will be set to false when `createAtomic` is called. We don't need to set it manually.
##########
File path: sql/core/src/test/scala/org/apache/spark/sql/execution/streaming/state/RocksDBSuite.scala
##########
@@ -20,12 +20,88 @@ package org.apache.spark.sql.execution.streaming.state
import java.io._
import java.nio.charset.Charset
+import scala.language.implicitConversions
+
import org.apache.commons.io.FileUtils
+import org.apache.hadoop.conf.Configuration
import org.apache.spark._
+import org.apache.spark.sql.catalyst.util.quietly
+import org.apache.spark.sql.execution.streaming.CreateAtomicTestManager
+import org.apache.spark.sql.internal.SQLConf
+import org.apache.spark.util.Utils
class RocksDBSuite extends SparkFunSuite {
+ test("RocksDBFileManager: upload only new immutable files") {
+ withTempDir { dir =>
+ val dfsRootDir = dir.getAbsolutePath
+ val fileManager = new RocksDBFileManager(
+ dfsRootDir, Utils.createTempDir(), new Configuration)
+ val sstDir = s"$dfsRootDir/SSTs"
+ def numRemoteSSTFiles: Int = listFiles(sstDir).length
+ val logDir = s"$dfsRootDir/logs"
+ def numRemoteLogFiles: Int = listFiles(logDir).length
+
+ // Save a version of checkpoint files
+ val cpFiles1 = Seq(
+ "sst-file1.sst" -> 10,
+ "sst-file2.sst" -> 20,
+ "other-file1" -> 100,
+ "other-file2" -> 200,
+ "archive/00001.log" -> 1000,
+ "archive/00002.log" -> 2000
+ )
+ saveCheckpointFiles(fileManager, cpFiles1, version = 1, numKeys = 101)
+ assert(numRemoteSSTFiles == 2) // 2 sst files copied
+ assert(numRemoteLogFiles == 2) // 2 log files copied
+
+ // Save SAME version again with different checkpoint files and verify
+ val cpFiles1_ = Seq(
+ "sst-file1.sst" -> 10, // same SST file as before, should not get copied
+ "sst-file2.sst" -> 25, // new SST file with same name as before, but different length
+ "sst-file3.sst" -> 30, // new SST file
+ "other-file1" -> 100, // same non-SST file as before, should not get copied
+ "other-file2" -> 210, // new non-SST file with same name as before, but different length
+ "other-file3" -> 300, // new non-SST file
+ "archive/00001.log" -> 1000, // same log file as before, should not get copied
+ "archive/00002.log" -> 2500, // new log file with same name as before, but different length
+ "archive/00003.log" -> 3000 // new log file
+ )
+ saveCheckpointFiles(fileManager, cpFiles1_, version = 1, numKeys = 1001)
+ assert(numRemoteSSTFiles === 4, "shouldn't copy same files again") // 2 old + 2 new SST files
+ assert(numRemoteLogFiles === 4, "shouldn't copy same files again") // 2 old + 2 new log files
+
+ // Save another version and verify
+ val cpFiles2 = Seq(
+ "sst-file4.sst" -> 40,
+ "other-file4" -> 400,
+ "archive/00004.log" -> 4000
+ )
+ saveCheckpointFiles(fileManager, cpFiles2, version = 2, numKeys = 1501)
+ assert(numRemoteSSTFiles === 5) // 1 new file over earlier 4 files
+ assert(numRemoteLogFiles === 5) // 1 new file over earlier 4 files
+ }
+ }
+
+ test("RocksDBFileManager: error writing [version].zip cancels the output stream") {
+ quietly {
+ val hadoopConf = new Configuration()
+ hadoopConf.set(
+ SQLConf.STREAMING_CHECKPOINT_FILE_MANAGER_CLASS.parent.key,
+ classOf[CreateAtomicTestManager].getName)
+ val dfsRootDir = Utils.createTempDir().getAbsolutePath
+ val fileManager = new RocksDBFileManager(dfsRootDir, Utils.createTempDir(), hadoopConf)
+ val cpFiles = Seq("sst-file1.sst" -> 10, "sst-file2.sst" -> 20, "other-file1" -> 100)
+ CreateAtomicTestManager.shouldFailInCreateAtomic = true
+ CreateAtomicTestManager.cancelCalledInCreateAtomic = false
+ intercept[IOException] {
+ saveCheckpointFiles(fileManager, cpFiles, version = 1, numKeys = 101)
+ }
Review comment:
I'm not sure why this will fail?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HeartSaVioR closed pull request #32582: [SPARK-35436][SS] RocksDBFileManager - save checkpoint to DFS
Posted by GitBox <gi...@apache.org>.
HeartSaVioR closed pull request #32582:
URL: https://github.com/apache/spark/pull/32582
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] xuanyuanking commented on a change in pull request #32582: [SPARK-35436][SS] RocksDBFileManager - save checkpoint to DFS
Posted by GitBox <gi...@apache.org>.
xuanyuanking commented on a change in pull request #32582:
URL: https://github.com/apache/spark/pull/32582#discussion_r645672037
##########
File path: sql/core/src/test/scala/org/apache/spark/sql/execution/streaming/state/RocksDBSuite.scala
##########
@@ -20,12 +20,88 @@ package org.apache.spark.sql.execution.streaming.state
import java.io._
import java.nio.charset.Charset
+import scala.language.implicitConversions
+
import org.apache.commons.io.FileUtils
+import org.apache.hadoop.conf.Configuration
import org.apache.spark._
+import org.apache.spark.sql.catalyst.util.quietly
+import org.apache.spark.sql.execution.streaming.CreateAtomicTestManager
+import org.apache.spark.sql.internal.SQLConf
+import org.apache.spark.util.Utils
class RocksDBSuite extends SparkFunSuite {
+ test("RocksDBFileManager: upload only new immutable files") {
+ withTempDir { dir =>
+ val dfsRootDir = dir.getAbsolutePath
+ val fileManager = new RocksDBFileManager(
+ dfsRootDir, Utils.createTempDir(), new Configuration)
+ val sstDir = s"$dfsRootDir/SSTs"
+ def numRemoteSSTFiles: Int = listFiles(sstDir).length
+ val logDir = s"$dfsRootDir/logs"
+ def numRemoteLogFiles: Int = listFiles(logDir).length
+
+ // Save a version of checkpoint files
+ val cpFiles1 = Seq(
+ "sst-file1.sst" -> 10,
+ "sst-file2.sst" -> 20,
+ "other-file1" -> 100,
+ "other-file2" -> 200,
+ "archive/00001.log" -> 1000,
+ "archive/00002.log" -> 2000
+ )
+ saveCheckpointFiles(fileManager, cpFiles1, version = 1, numKeys = 101)
+ assert(numRemoteSSTFiles == 2) // 2 sst files copied
+ assert(numRemoteLogFiles == 2) // 2 log files copied
+
+ // Save SAME version again with different checkpoint files and verify
+ val cpFiles1_ = Seq(
+ "sst-file1.sst" -> 10, // same SST file as before, should not get copied
+ "sst-file2.sst" -> 25, // new SST file with same name as before, but different length
+ "sst-file3.sst" -> 30, // new SST file
+ "other-file1" -> 100, // same non-SST file as before, should not get copied
+ "other-file2" -> 210, // new non-SST file with same name as before, but different length
+ "other-file3" -> 300, // new non-SST file
+ "archive/00001.log" -> 1000, // same log file as before, should not get copied
+ "archive/00002.log" -> 2500, // new log file with same name as before, but different length
+ "archive/00003.log" -> 3000 // new log file
+ )
+ saveCheckpointFiles(fileManager, cpFiles1_, version = 1, numKeys = 1001)
+ assert(numRemoteSSTFiles === 4, "shouldn't copy same files again") // 2 old + 2 new SST files
+ assert(numRemoteLogFiles === 4, "shouldn't copy same files again") // 2 old + 2 new log files
+
+ // Save another version and verify
+ val cpFiles2 = Seq(
+ "sst-file4.sst" -> 40,
+ "other-file4" -> 400,
+ "archive/00004.log" -> 4000
+ )
+ saveCheckpointFiles(fileManager, cpFiles2, version = 2, numKeys = 1501)
+ assert(numRemoteSSTFiles === 5) // 1 new file over earlier 4 files
+ assert(numRemoteLogFiles === 5) // 1 new file over earlier 4 files
+ }
+ }
+
+ test("RocksDBFileManager: error writing [version].zip cancels the output stream") {
+ quietly {
+ val hadoopConf = new Configuration()
+ hadoopConf.set(
+ SQLConf.STREAMING_CHECKPOINT_FILE_MANAGER_CLASS.parent.key,
+ classOf[CreateAtomicTestManager].getName)
+ val dfsRootDir = Utils.createTempDir().getAbsolutePath
+ val fileManager = new RocksDBFileManager(dfsRootDir, Utils.createTempDir(), hadoopConf)
+ val cpFiles = Seq("sst-file1.sst" -> 10, "sst-file2.sst" -> 20, "other-file1" -> 100)
+ CreateAtomicTestManager.shouldFailInCreateAtomic = true
+ CreateAtomicTestManager.cancelCalledInCreateAtomic = false
Review comment:
That's right. Just want to set it to `false` explicitly here since we have a `assert(cancelCalledInCreateAtomic)` in the end. Let me delete this one. Done in bdd9e8e.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32582: [SPARK-35436][SS] RocksDBFileManager - save checkpoint to DFS
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32582:
URL: https://github.com/apache/spark/pull/32582#issuecomment-853809800
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/43809/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #32582: [SPARK-35436][SS] RocksDBFileManager - save checkpoint to DFS
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #32582:
URL: https://github.com/apache/spark/pull/32582#issuecomment-853863870
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/139285/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #32582: [SPARK-35436][SS] RocksDBFileManager - save checkpoint to DFS
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #32582:
URL: https://github.com/apache/spark/pull/32582#issuecomment-857345167
**[Test build #139529 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/139529/testReport)** for PR 32582 at commit [`c05b140`](https://github.com/apache/spark/commit/c05b140f73f855a148ccedaf753841fb94a4ab1d).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] viirya commented on pull request #32582: [SPARK-35436][SS] RocksDBFileManager - save checkpoint to DFS
Posted by GitBox <gi...@apache.org>.
viirya commented on pull request #32582:
URL: https://github.com/apache/spark/pull/32582#issuecomment-850083083
Thanks @xuanyuanking. I will find some time to review this.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] viirya commented on a change in pull request #32582: [SPARK-35436][SS] RocksDBFileManager - save checkpoint to DFS
Posted by GitBox <gi...@apache.org>.
viirya commented on a change in pull request #32582:
URL: https://github.com/apache/spark/pull/32582#discussion_r645326621
##########
File path: sql/core/src/test/scala/org/apache/spark/sql/execution/streaming/state/RocksDBSuite.scala
##########
@@ -20,12 +20,88 @@ package org.apache.spark.sql.execution.streaming.state
import java.io._
import java.nio.charset.Charset
+import scala.language.implicitConversions
+
import org.apache.commons.io.FileUtils
+import org.apache.hadoop.conf.Configuration
import org.apache.spark._
+import org.apache.spark.sql.catalyst.util.quietly
+import org.apache.spark.sql.execution.streaming.CreateAtomicTestManager
+import org.apache.spark.sql.internal.SQLConf
+import org.apache.spark.util.Utils
class RocksDBSuite extends SparkFunSuite {
+ test("RocksDBFileManager: upload only new immutable files") {
+ withTempDir { dir =>
+ val dfsRootDir = dir.getAbsolutePath
+ val fileManager = new RocksDBFileManager(
+ dfsRootDir, Utils.createTempDir(), new Configuration)
+ val sstDir = s"$dfsRootDir/SSTs"
+ def numRemoteSSTFiles: Int = listFiles(sstDir).length
+ val logDir = s"$dfsRootDir/logs"
+ def numRemoteLogFiles: Int = listFiles(logDir).length
+
+ // Save a version of checkpoint files
+ val cpFiles1 = Seq(
+ "sst-file1.sst" -> 10,
+ "sst-file2.sst" -> 20,
+ "other-file1" -> 100,
+ "other-file2" -> 200,
+ "archive/00001.log" -> 1000,
+ "archive/00002.log" -> 2000
+ )
+ saveCheckpointFiles(fileManager, cpFiles1, version = 1, numKeys = 101)
+ assert(numRemoteSSTFiles == 2) // 2 sst files copied
+ assert(numRemoteLogFiles == 2) // 2 log files copied
+
+ // Save SAME version again with different checkpoint files and verify
+ val cpFiles1_ = Seq(
+ "sst-file1.sst" -> 10, // same SST file as before, should not get copied
+ "sst-file2.sst" -> 25, // new SST file with same name as before, but different length
+ "sst-file3.sst" -> 30, // new SST file
+ "other-file1" -> 100, // same non-SST file as before, should not get copied
+ "other-file2" -> 210, // new non-SST file with same name as before, but different length
+ "other-file3" -> 300, // new non-SST file
+ "archive/00001.log" -> 1000, // same log file as before, should not get copied
+ "archive/00002.log" -> 2500, // new log file with same name as before, but different length
+ "archive/00003.log" -> 3000 // new log file
+ )
+ saveCheckpointFiles(fileManager, cpFiles1_, version = 1, numKeys = 1001)
+ assert(numRemoteSSTFiles === 4, "shouldn't copy same files again") // 2 old + 2 new SST files
+ assert(numRemoteLogFiles === 4, "shouldn't copy same files again") // 2 old + 2 new log files
+
+ // Save another version and verify
+ val cpFiles2 = Seq(
+ "sst-file4.sst" -> 40,
+ "other-file4" -> 400,
+ "archive/00004.log" -> 4000
+ )
+ saveCheckpointFiles(fileManager, cpFiles2, version = 2, numKeys = 1501)
+ assert(numRemoteSSTFiles === 5) // 1 new file over earlier 4 files
+ assert(numRemoteLogFiles === 5) // 1 new file over earlier 4 files
+ }
+ }
+
+ test("RocksDBFileManager: error writing [version].zip cancels the output stream") {
+ quietly {
+ val hadoopConf = new Configuration()
+ hadoopConf.set(
+ SQLConf.STREAMING_CHECKPOINT_FILE_MANAGER_CLASS.parent.key,
+ classOf[CreateAtomicTestManager].getName)
+ val dfsRootDir = Utils.createTempDir().getAbsolutePath
+ val fileManager = new RocksDBFileManager(dfsRootDir, Utils.createTempDir(), hadoopConf)
+ val cpFiles = Seq("sst-file1.sst" -> 10, "sst-file2.sst" -> 20, "other-file1" -> 100)
+ CreateAtomicTestManager.shouldFailInCreateAtomic = true
+ CreateAtomicTestManager.cancelCalledInCreateAtomic = false
Review comment:
If `shouldFailInCreateAtomic` is true, `cancelCalledInCreateAtomic` will be set to false when `createAtomic` is called. We don't need to set it manually.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32582: [SPARK-35436][SS] RocksDBFileManager - save checkpoint to DFS
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32582:
URL: https://github.com/apache/spark/pull/32582#issuecomment-854890685
Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/43862/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HeartSaVioR commented on a change in pull request #32582: [SPARK-35436][SS] RocksDBFileManager - save checkpoint to DFS
Posted by GitBox <gi...@apache.org>.
HeartSaVioR commented on a change in pull request #32582:
URL: https://github.com/apache/spark/pull/32582#discussion_r642019968
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/state/RocksDBFileManager.scala
##########
@@ -17,18 +17,270 @@
package org.apache.spark.sql.execution.streaming.state
-import java.io.File
+import java.io.{File, FileInputStream, InputStream}
import java.nio.charset.StandardCharsets.UTF_8
import java.nio.file.Files
+import java.util.UUID
+import java.util.concurrent.ConcurrentHashMap
+import java.util.zip.{ZipEntry, ZipOutputStream}
+import scala.collection.JavaConverters._
import scala.collection.Seq
import com.fasterxml.jackson.annotation.JsonInclude.Include
import com.fasterxml.jackson.databind.{DeserializationFeature, ObjectMapper}
import com.fasterxml.jackson.module.scala.{DefaultScalaModule, ScalaObjectMapper}
+import org.apache.commons.io.{FilenameUtils, IOUtils}
+import org.apache.hadoop.conf.Configuration
+import org.apache.hadoop.fs.Path
import org.json4s.NoTypeHints
import org.json4s.jackson.Serialization
+import org.apache.spark.internal.Logging
+import org.apache.spark.sql.execution.streaming.CheckpointFileManager
+import org.apache.spark.util.Utils
+
+/**
+ * Class responsible for syncing RocksDB checkpoint files from local disk to DFS.
+ * Every version checkpoint is saved in specific directory structure that allows successive
+ * versions to reuse to SST data files and archived log files. This allows each commit to be
+ * incremental, only new SST files and archived log files generated by RocksDB will be uploaded.
+ * The directory structures on local disk and in DFS are as follows.
+ *
+ * Local checkpoint dir structure
+ * ------------------------------
+ * RocksDB generates a bunch of files in the local checkpoint directory. The most important among
+ * them are the SST files; they are the actual log structured data files. Rest of the files contain
+ * the metadata necessary for RocksDB to read the SST files and start from the checkpoint.
+ * Note that the SST files are hard links to files in the RocksDB's working directory, and therefore
+ * successive checkpoints can share some of the SST files. So these SST files have to be copied to
+ * DFS in shared directory such that different committed versions can save them.
+ *
+ * We consider both SST files and archived log files as immutable files which can be shared between
+ * different checkpoints.
+ *
+ * localCheckpointDir
+ * |
+ * +-- OPTIONS-000005
+ * +-- MANIFEST-000008
+ * +-- CURRENT
+ * +-- 00007.sst
+ * +-- 00011.sst
+ * +-- archive
+ * | +-- 00008.log
+ * | +-- 00013.log
+ * ...
+ *
+ *
+ * DFS directory structure after saving to DFS as version 10
+ * -----------------------------------------------------------
+ * The SST and archived log files are given unique file names and copied to the shared subdirectory.
+ * Every version maintains a mapping of local immutable file name to the unique file name in DFS.
+ * This mapping is saved in a JSON file (named `metadata`), which is zipped along with other
+ * checkpoint files into a single file `[version].zip`.
+ *
+ * dfsRootDir
+ * |
+ * +-- SSTs
+ * | +-- 00007-[uuid1].sst
+ * | +-- 00011-[uuid2].sst
+ * +-- logs
+ * | +-- 00008-[uuid3].log
+ * | +-- 00013-[uuid4].log
+ * +-- 10.zip
+ * | +-- metadata <--- contains mapping between 00007.sst and [uuid1].sst,
+ * and the mapping between 00008.log and [uuid3].log
+ * | +-- OPTIONS-000005
+ * | +-- MANIFEST-000008
+ * | +-- CURRENT
+ * | ...
+ * |
+ * +-- 9.zip
+ * +-- 8.zip
+ * ...
+ *
+ * Note the following.
+ * - Each [version].zip is a complete description of all the data and metadata needed to recover
+ * a RocksDB instance at the corresponding version. This is unlike the [version].delta files of
+ * HDFSBackedStateStore where previous delta files needs to be read to be recovered.
+ * - This is safe wrt speculatively executed tasks running concurrently in different executors
+ * as each task would upload a different copy of the generated immutable files and
+ * atomically update the [version].zip.
+ * - Immutable files are identified uniquely based on their file name and file size.
+ * - Immutable files can be reused only across adjacent checkpoints/versions.
+ * - This class is thread-safe. Specifically, it is safe to concurrently delete old files from a
+ * different thread than the task thread saving files.
+ *
+ * @param dfsRootDir Directory where the [version].zip files will be stored
+ * @param localTempDir Local directory for temporary work
+ * @param hadoopConf Hadoop configuration for talking to DFS
+ * @param loggingId Id that will be prepended in logs for isolating concurrent RocksDBs
+ */
+class RocksDBFileManager(
+ dfsRootDir: String,
Review comment:
nit: indentation looks off
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/state/RocksDBFileManager.scala
##########
@@ -17,18 +17,270 @@
package org.apache.spark.sql.execution.streaming.state
-import java.io.File
+import java.io.{File, FileInputStream, InputStream}
import java.nio.charset.StandardCharsets.UTF_8
import java.nio.file.Files
+import java.util.UUID
+import java.util.concurrent.ConcurrentHashMap
+import java.util.zip.{ZipEntry, ZipOutputStream}
+import scala.collection.JavaConverters._
import scala.collection.Seq
import com.fasterxml.jackson.annotation.JsonInclude.Include
import com.fasterxml.jackson.databind.{DeserializationFeature, ObjectMapper}
import com.fasterxml.jackson.module.scala.{DefaultScalaModule, ScalaObjectMapper}
+import org.apache.commons.io.{FilenameUtils, IOUtils}
+import org.apache.hadoop.conf.Configuration
+import org.apache.hadoop.fs.Path
import org.json4s.NoTypeHints
import org.json4s.jackson.Serialization
+import org.apache.spark.internal.Logging
+import org.apache.spark.sql.execution.streaming.CheckpointFileManager
+import org.apache.spark.util.Utils
+
+/**
+ * Class responsible for syncing RocksDB checkpoint files from local disk to DFS.
+ * Every version checkpoint is saved in specific directory structure that allows successive
+ * versions to reuse to SST data files and archived log files. This allows each commit to be
+ * incremental, only new SST files and archived log files generated by RocksDB will be uploaded.
+ * The directory structures on local disk and in DFS are as follows.
+ *
+ * Local checkpoint dir structure
+ * ------------------------------
+ * RocksDB generates a bunch of files in the local checkpoint directory. The most important among
+ * them are the SST files; they are the actual log structured data files. Rest of the files contain
+ * the metadata necessary for RocksDB to read the SST files and start from the checkpoint.
+ * Note that the SST files are hard links to files in the RocksDB's working directory, and therefore
+ * successive checkpoints can share some of the SST files. So these SST files have to be copied to
+ * DFS in shared directory such that different committed versions can save them.
+ *
+ * We consider both SST files and archived log files as immutable files which can be shared between
+ * different checkpoints.
+ *
+ * localCheckpointDir
+ * |
+ * +-- OPTIONS-000005
+ * +-- MANIFEST-000008
+ * +-- CURRENT
+ * +-- 00007.sst
+ * +-- 00011.sst
+ * +-- archive
+ * | +-- 00008.log
+ * | +-- 00013.log
+ * ...
+ *
+ *
+ * DFS directory structure after saving to DFS as version 10
+ * -----------------------------------------------------------
+ * The SST and archived log files are given unique file names and copied to the shared subdirectory.
+ * Every version maintains a mapping of local immutable file name to the unique file name in DFS.
+ * This mapping is saved in a JSON file (named `metadata`), which is zipped along with other
+ * checkpoint files into a single file `[version].zip`.
+ *
+ * dfsRootDir
+ * |
+ * +-- SSTs
+ * | +-- 00007-[uuid1].sst
+ * | +-- 00011-[uuid2].sst
+ * +-- logs
+ * | +-- 00008-[uuid3].log
+ * | +-- 00013-[uuid4].log
+ * +-- 10.zip
+ * | +-- metadata <--- contains mapping between 00007.sst and [uuid1].sst,
+ * and the mapping between 00008.log and [uuid3].log
+ * | +-- OPTIONS-000005
+ * | +-- MANIFEST-000008
+ * | +-- CURRENT
+ * | ...
+ * |
+ * +-- 9.zip
+ * +-- 8.zip
+ * ...
+ *
+ * Note the following.
+ * - Each [version].zip is a complete description of all the data and metadata needed to recover
+ * a RocksDB instance at the corresponding version. This is unlike the [version].delta files of
+ * HDFSBackedStateStore where previous delta files needs to be read to be recovered.
+ * - This is safe wrt speculatively executed tasks running concurrently in different executors
+ * as each task would upload a different copy of the generated immutable files and
+ * atomically update the [version].zip.
+ * - Immutable files are identified uniquely based on their file name and file size.
+ * - Immutable files can be reused only across adjacent checkpoints/versions.
+ * - This class is thread-safe. Specifically, it is safe to concurrently delete old files from a
+ * different thread than the task thread saving files.
+ *
+ * @param dfsRootDir Directory where the [version].zip files will be stored
+ * @param localTempDir Local directory for temporary work
+ * @param hadoopConf Hadoop configuration for talking to DFS
+ * @param loggingId Id that will be prepended in logs for isolating concurrent RocksDBs
+ */
+class RocksDBFileManager(
+ dfsRootDir: String,
+ localTempDir: File,
+ hadoopConf: Configuration,
+ loggingId: String = "")
+ extends Logging {
+
+ import RocksDBImmutableFile._
+
+ private val versionToRocksDBFiles = new ConcurrentHashMap[Long, Seq[RocksDBImmutableFile]]
+ private lazy val fm = CheckpointFileManager.create(new Path(dfsRootDir), hadoopConf)
+ private val fs = new Path(dfsRootDir).getFileSystem(hadoopConf)
+
+ /** Save all the files in given local checkpoint directory as a committed version in DFS */
+ def saveCheckpointToDfs(checkpointDir: File, version: Long, numKeys: Long): Unit = {
+ logFilesInDir(checkpointDir, s"Saving checkpoint files for version $version")
+ val allCheckpointFiles = listRocksDBFiles(checkpointDir)
+ val (localImmutableFiles, localOtherFiles) = allCheckpointFiles.partition(isImmutableFile)
+ val rocksDBFiles = saveImmutableFilesToDfs(version, checkpointDir, localImmutableFiles)
+ val metadata = RocksDBCheckpointMetadata(rocksDBFiles, numKeys)
+ rememberImmutableFiles(version, rocksDBFiles)
Review comment:
Unless there's some strong reason to put this here, personally I'd have this in `saveImmutableFilesToDfs` as immutable files in the map is used there, not here. The `saveImmutableFilesToDfs` method just needs to know and handle the map, except the case the manager is initialized with existing checkpoint.
And then `rememberImmutableFiles` method can be inlined - `saveImmutableFilesToDfs` method (and someone who modifies this method) should know about `versionToRocksDBFiles` so a new method for this to explain what the one-liner does is redundant.
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/state/RocksDBFileManager.scala
##########
@@ -17,18 +17,270 @@
package org.apache.spark.sql.execution.streaming.state
-import java.io.File
+import java.io.{File, FileInputStream, InputStream}
import java.nio.charset.StandardCharsets.UTF_8
import java.nio.file.Files
+import java.util.UUID
+import java.util.concurrent.ConcurrentHashMap
+import java.util.zip.{ZipEntry, ZipOutputStream}
+import scala.collection.JavaConverters._
import scala.collection.Seq
import com.fasterxml.jackson.annotation.JsonInclude.Include
import com.fasterxml.jackson.databind.{DeserializationFeature, ObjectMapper}
import com.fasterxml.jackson.module.scala.{DefaultScalaModule, ScalaObjectMapper}
+import org.apache.commons.io.{FilenameUtils, IOUtils}
+import org.apache.hadoop.conf.Configuration
+import org.apache.hadoop.fs.Path
import org.json4s.NoTypeHints
import org.json4s.jackson.Serialization
+import org.apache.spark.internal.Logging
+import org.apache.spark.sql.execution.streaming.CheckpointFileManager
+import org.apache.spark.util.Utils
+
+/**
+ * Class responsible for syncing RocksDB checkpoint files from local disk to DFS.
+ * Every version checkpoint is saved in specific directory structure that allows successive
+ * versions to reuse to SST data files and archived log files. This allows each commit to be
+ * incremental, only new SST files and archived log files generated by RocksDB will be uploaded.
+ * The directory structures on local disk and in DFS are as follows.
+ *
+ * Local checkpoint dir structure
+ * ------------------------------
+ * RocksDB generates a bunch of files in the local checkpoint directory. The most important among
+ * them are the SST files; they are the actual log structured data files. Rest of the files contain
+ * the metadata necessary for RocksDB to read the SST files and start from the checkpoint.
+ * Note that the SST files are hard links to files in the RocksDB's working directory, and therefore
+ * successive checkpoints can share some of the SST files. So these SST files have to be copied to
+ * DFS in shared directory such that different committed versions can save them.
+ *
+ * We consider both SST files and archived log files as immutable files which can be shared between
+ * different checkpoints.
+ *
+ * localCheckpointDir
+ * |
+ * +-- OPTIONS-000005
+ * +-- MANIFEST-000008
+ * +-- CURRENT
+ * +-- 00007.sst
+ * +-- 00011.sst
+ * +-- archive
+ * | +-- 00008.log
+ * | +-- 00013.log
+ * ...
+ *
+ *
+ * DFS directory structure after saving to DFS as version 10
+ * -----------------------------------------------------------
+ * The SST and archived log files are given unique file names and copied to the shared subdirectory.
+ * Every version maintains a mapping of local immutable file name to the unique file name in DFS.
+ * This mapping is saved in a JSON file (named `metadata`), which is zipped along with other
+ * checkpoint files into a single file `[version].zip`.
+ *
+ * dfsRootDir
+ * |
+ * +-- SSTs
+ * | +-- 00007-[uuid1].sst
+ * | +-- 00011-[uuid2].sst
+ * +-- logs
+ * | +-- 00008-[uuid3].log
+ * | +-- 00013-[uuid4].log
+ * +-- 10.zip
+ * | +-- metadata <--- contains mapping between 00007.sst and [uuid1].sst,
+ * and the mapping between 00008.log and [uuid3].log
+ * | +-- OPTIONS-000005
+ * | +-- MANIFEST-000008
+ * | +-- CURRENT
+ * | ...
+ * |
+ * +-- 9.zip
+ * +-- 8.zip
+ * ...
+ *
+ * Note the following.
+ * - Each [version].zip is a complete description of all the data and metadata needed to recover
+ * a RocksDB instance at the corresponding version. This is unlike the [version].delta files of
+ * HDFSBackedStateStore where previous delta files needs to be read to be recovered.
+ * - This is safe wrt speculatively executed tasks running concurrently in different executors
+ * as each task would upload a different copy of the generated immutable files and
+ * atomically update the [version].zip.
+ * - Immutable files are identified uniquely based on their file name and file size.
+ * - Immutable files can be reused only across adjacent checkpoints/versions.
+ * - This class is thread-safe. Specifically, it is safe to concurrently delete old files from a
+ * different thread than the task thread saving files.
+ *
+ * @param dfsRootDir Directory where the [version].zip files will be stored
+ * @param localTempDir Local directory for temporary work
+ * @param hadoopConf Hadoop configuration for talking to DFS
+ * @param loggingId Id that will be prepended in logs for isolating concurrent RocksDBs
+ */
+class RocksDBFileManager(
+ dfsRootDir: String,
+ localTempDir: File,
+ hadoopConf: Configuration,
+ loggingId: String = "")
+ extends Logging {
+
+ import RocksDBImmutableFile._
+
+ private val versionToRocksDBFiles = new ConcurrentHashMap[Long, Seq[RocksDBImmutableFile]]
Review comment:
Just to confirm, could I safely assume versionToRocksDBFiles will be loaded when RocksDBFileManager is initialized with existing checkpoint in further PR?
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/state/RocksDBFileManager.scala
##########
@@ -17,18 +17,270 @@
package org.apache.spark.sql.execution.streaming.state
-import java.io.File
+import java.io.{File, FileInputStream, InputStream}
import java.nio.charset.StandardCharsets.UTF_8
import java.nio.file.Files
+import java.util.UUID
+import java.util.concurrent.ConcurrentHashMap
+import java.util.zip.{ZipEntry, ZipOutputStream}
+import scala.collection.JavaConverters._
import scala.collection.Seq
import com.fasterxml.jackson.annotation.JsonInclude.Include
import com.fasterxml.jackson.databind.{DeserializationFeature, ObjectMapper}
import com.fasterxml.jackson.module.scala.{DefaultScalaModule, ScalaObjectMapper}
+import org.apache.commons.io.{FilenameUtils, IOUtils}
+import org.apache.hadoop.conf.Configuration
+import org.apache.hadoop.fs.Path
import org.json4s.NoTypeHints
import org.json4s.jackson.Serialization
+import org.apache.spark.internal.Logging
+import org.apache.spark.sql.execution.streaming.CheckpointFileManager
+import org.apache.spark.util.Utils
+
+/**
+ * Class responsible for syncing RocksDB checkpoint files from local disk to DFS.
+ * Every version checkpoint is saved in specific directory structure that allows successive
Review comment:
nit: Every version -> For each version,
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/state/RocksDBFileManager.scala
##########
@@ -17,18 +17,270 @@
package org.apache.spark.sql.execution.streaming.state
-import java.io.File
+import java.io.{File, FileInputStream, InputStream}
import java.nio.charset.StandardCharsets.UTF_8
import java.nio.file.Files
+import java.util.UUID
+import java.util.concurrent.ConcurrentHashMap
+import java.util.zip.{ZipEntry, ZipOutputStream}
+import scala.collection.JavaConverters._
import scala.collection.Seq
import com.fasterxml.jackson.annotation.JsonInclude.Include
import com.fasterxml.jackson.databind.{DeserializationFeature, ObjectMapper}
import com.fasterxml.jackson.module.scala.{DefaultScalaModule, ScalaObjectMapper}
+import org.apache.commons.io.{FilenameUtils, IOUtils}
+import org.apache.hadoop.conf.Configuration
+import org.apache.hadoop.fs.Path
import org.json4s.NoTypeHints
import org.json4s.jackson.Serialization
+import org.apache.spark.internal.Logging
+import org.apache.spark.sql.execution.streaming.CheckpointFileManager
+import org.apache.spark.util.Utils
+
+/**
+ * Class responsible for syncing RocksDB checkpoint files from local disk to DFS.
+ * Every version checkpoint is saved in specific directory structure that allows successive
+ * versions to reuse to SST data files and archived log files. This allows each commit to be
+ * incremental, only new SST files and archived log files generated by RocksDB will be uploaded.
+ * The directory structures on local disk and in DFS are as follows.
+ *
+ * Local checkpoint dir structure
+ * ------------------------------
+ * RocksDB generates a bunch of files in the local checkpoint directory. The most important among
+ * them are the SST files; they are the actual log structured data files. Rest of the files contain
+ * the metadata necessary for RocksDB to read the SST files and start from the checkpoint.
+ * Note that the SST files are hard links to files in the RocksDB's working directory, and therefore
+ * successive checkpoints can share some of the SST files. So these SST files have to be copied to
+ * DFS in shared directory such that different committed versions can save them.
+ *
+ * We consider both SST files and archived log files as immutable files which can be shared between
+ * different checkpoints.
+ *
+ * localCheckpointDir
+ * |
+ * +-- OPTIONS-000005
+ * +-- MANIFEST-000008
+ * +-- CURRENT
+ * +-- 00007.sst
+ * +-- 00011.sst
+ * +-- archive
+ * | +-- 00008.log
+ * | +-- 00013.log
+ * ...
+ *
+ *
+ * DFS directory structure after saving to DFS as version 10
+ * -----------------------------------------------------------
+ * The SST and archived log files are given unique file names and copied to the shared subdirectory.
+ * Every version maintains a mapping of local immutable file name to the unique file name in DFS.
+ * This mapping is saved in a JSON file (named `metadata`), which is zipped along with other
+ * checkpoint files into a single file `[version].zip`.
+ *
+ * dfsRootDir
+ * |
+ * +-- SSTs
+ * | +-- 00007-[uuid1].sst
+ * | +-- 00011-[uuid2].sst
+ * +-- logs
+ * | +-- 00008-[uuid3].log
+ * | +-- 00013-[uuid4].log
+ * +-- 10.zip
+ * | +-- metadata <--- contains mapping between 00007.sst and [uuid1].sst,
+ * and the mapping between 00008.log and [uuid3].log
+ * | +-- OPTIONS-000005
+ * | +-- MANIFEST-000008
+ * | +-- CURRENT
+ * | ...
+ * |
+ * +-- 9.zip
+ * +-- 8.zip
+ * ...
+ *
+ * Note the following.
+ * - Each [version].zip is a complete description of all the data and metadata needed to recover
+ * a RocksDB instance at the corresponding version. This is unlike the [version].delta files of
+ * HDFSBackedStateStore where previous delta files needs to be read to be recovered.
+ * - This is safe wrt speculatively executed tasks running concurrently in different executors
+ * as each task would upload a different copy of the generated immutable files and
+ * atomically update the [version].zip.
+ * - Immutable files are identified uniquely based on their file name and file size.
+ * - Immutable files can be reused only across adjacent checkpoints/versions.
+ * - This class is thread-safe. Specifically, it is safe to concurrently delete old files from a
+ * different thread than the task thread saving files.
+ *
+ * @param dfsRootDir Directory where the [version].zip files will be stored
+ * @param localTempDir Local directory for temporary work
+ * @param hadoopConf Hadoop configuration for talking to DFS
+ * @param loggingId Id that will be prepended in logs for isolating concurrent RocksDBs
+ */
+class RocksDBFileManager(
+ dfsRootDir: String,
+ localTempDir: File,
+ hadoopConf: Configuration,
+ loggingId: String = "")
+ extends Logging {
+
+ import RocksDBImmutableFile._
+
+ private val versionToRocksDBFiles = new ConcurrentHashMap[Long, Seq[RocksDBImmutableFile]]
+ private lazy val fm = CheckpointFileManager.create(new Path(dfsRootDir), hadoopConf)
+ private val fs = new Path(dfsRootDir).getFileSystem(hadoopConf)
+
+ /** Save all the files in given local checkpoint directory as a committed version in DFS */
+ def saveCheckpointToDfs(checkpointDir: File, version: Long, numKeys: Long): Unit = {
+ logFilesInDir(checkpointDir, s"Saving checkpoint files for version $version")
+ val allCheckpointFiles = listRocksDBFiles(checkpointDir)
+ val (localImmutableFiles, localOtherFiles) = allCheckpointFiles.partition(isImmutableFile)
+ val rocksDBFiles = saveImmutableFilesToDfs(version, checkpointDir, localImmutableFiles)
+ val metadata = RocksDBCheckpointMetadata(rocksDBFiles, numKeys)
+ rememberImmutableFiles(version, rocksDBFiles)
+ val metadataFile = localMetadataFile(checkpointDir)
+ metadata.writeToFile(metadataFile)
+ logInfo(s"Written metadata for version $version:\n${metadata.prettyJson}")
+
+ if (version <= 0 && numKeys == 0) {
+ // If we're writing the initial version and there's no data, we have to explicitly initialize
+ // the root directory. Normally saveImmutableFilesToDfs will do this initialization, but
+ // when there's no data that method won't write any files, and zipToDfsFile uses the
+ // CheckpointFileManager.createAtomic API which doesn't auto-initialize parent directories.
+ fm.mkdirs(new Path(dfsRootDir))
+ }
+ zipToDfsFile(localOtherFiles :+ metadataFile, dfsBatchZipFile(version))
+ logInfo(s"Saved checkpoint file for version $version")
+ }
+
+ /** Save immutable files to DFS directory */
+ private def saveImmutableFilesToDfs(
+ version: Long,
+ localDir: File,
+ localFiles: Seq[File]): Seq[RocksDBImmutableFile] = {
+ // Get the immutable files used in previous versions, as some of those uploaded files can be
+ // reused for this version
+ logInfo(s"Saving RocksDB files to DFS for $version")
+ val prevFilesToSizes = versionToRocksDBFiles.values.asScala.flatten.map { f =>
+ f.localFileName -> f
+ }.toMap
+
+ var bytesCopied = 0L
+ var filesCopied = 0L
+ var filesReused = 0L
+
+ val immutableFiles = localFiles.map { localFile =>
+ prevFilesToSizes
+ .get(localFile.getName)
+ .filter(_.isSameFile(localFile))
+ .map { reusable =>
+ filesReused += 1
+ reusable
+ }.getOrElse {
+ val localFileName = localFile.getName
+ val dfsFileName = newDFSFileName(localFileName)
+ val dfsFile = dfsFilePath(dfsFileName)
+ // Note: The implementation of copyFromLocalFile() closes the output stream when there is
+ // any exception while copying. So this may generate partial files on DFS. But that is
+ // okay because until the main [version].zip file is written, those partial files are
+ // not going to be used at all. Eventually these files should get cleared.
+ fs.copyFromLocalFile(
+ new Path(localFile.getAbsoluteFile.toURI), dfsFile)
+ val localFileSize = localFile.length()
+ logInfo(s"Copied $localFile to $dfsFile - $localFileSize bytes")
+ filesCopied += 1
+ bytesCopied += localFileSize
+
+ RocksDBImmutableFile(localFile.getName, dfsFileName, localFileSize)
+ }
+ }
+ logInfo(s"Copied $filesCopied files ($bytesCopied bytes) from local to" +
Review comment:
We count `filesReused` but not leaving the information here. Can we add it in the log message as well?
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/state/RocksDBFileManager.scala
##########
@@ -17,18 +17,270 @@
package org.apache.spark.sql.execution.streaming.state
-import java.io.File
+import java.io.{File, FileInputStream, InputStream}
import java.nio.charset.StandardCharsets.UTF_8
import java.nio.file.Files
+import java.util.UUID
+import java.util.concurrent.ConcurrentHashMap
+import java.util.zip.{ZipEntry, ZipOutputStream}
+import scala.collection.JavaConverters._
import scala.collection.Seq
import com.fasterxml.jackson.annotation.JsonInclude.Include
import com.fasterxml.jackson.databind.{DeserializationFeature, ObjectMapper}
import com.fasterxml.jackson.module.scala.{DefaultScalaModule, ScalaObjectMapper}
+import org.apache.commons.io.{FilenameUtils, IOUtils}
+import org.apache.hadoop.conf.Configuration
+import org.apache.hadoop.fs.Path
import org.json4s.NoTypeHints
import org.json4s.jackson.Serialization
+import org.apache.spark.internal.Logging
+import org.apache.spark.sql.execution.streaming.CheckpointFileManager
+import org.apache.spark.util.Utils
+
+/**
+ * Class responsible for syncing RocksDB checkpoint files from local disk to DFS.
+ * Every version checkpoint is saved in specific directory structure that allows successive
+ * versions to reuse to SST data files and archived log files. This allows each commit to be
+ * incremental, only new SST files and archived log files generated by RocksDB will be uploaded.
+ * The directory structures on local disk and in DFS are as follows.
+ *
+ * Local checkpoint dir structure
+ * ------------------------------
+ * RocksDB generates a bunch of files in the local checkpoint directory. The most important among
+ * them are the SST files; they are the actual log structured data files. Rest of the files contain
+ * the metadata necessary for RocksDB to read the SST files and start from the checkpoint.
+ * Note that the SST files are hard links to files in the RocksDB's working directory, and therefore
+ * successive checkpoints can share some of the SST files. So these SST files have to be copied to
+ * DFS in shared directory such that different committed versions can save them.
+ *
+ * We consider both SST files and archived log files as immutable files which can be shared between
+ * different checkpoints.
+ *
+ * localCheckpointDir
+ * |
+ * +-- OPTIONS-000005
+ * +-- MANIFEST-000008
+ * +-- CURRENT
+ * +-- 00007.sst
+ * +-- 00011.sst
+ * +-- archive
+ * | +-- 00008.log
+ * | +-- 00013.log
+ * ...
+ *
+ *
+ * DFS directory structure after saving to DFS as version 10
+ * -----------------------------------------------------------
+ * The SST and archived log files are given unique file names and copied to the shared subdirectory.
+ * Every version maintains a mapping of local immutable file name to the unique file name in DFS.
+ * This mapping is saved in a JSON file (named `metadata`), which is zipped along with other
+ * checkpoint files into a single file `[version].zip`.
+ *
+ * dfsRootDir
+ * |
+ * +-- SSTs
+ * | +-- 00007-[uuid1].sst
+ * | +-- 00011-[uuid2].sst
+ * +-- logs
+ * | +-- 00008-[uuid3].log
+ * | +-- 00013-[uuid4].log
+ * +-- 10.zip
+ * | +-- metadata <--- contains mapping between 00007.sst and [uuid1].sst,
+ * and the mapping between 00008.log and [uuid3].log
+ * | +-- OPTIONS-000005
+ * | +-- MANIFEST-000008
+ * | +-- CURRENT
+ * | ...
+ * |
+ * +-- 9.zip
+ * +-- 8.zip
+ * ...
+ *
+ * Note the following.
+ * - Each [version].zip is a complete description of all the data and metadata needed to recover
+ * a RocksDB instance at the corresponding version. This is unlike the [version].delta files of
+ * HDFSBackedStateStore where previous delta files needs to be read to be recovered.
+ * - This is safe wrt speculatively executed tasks running concurrently in different executors
+ * as each task would upload a different copy of the generated immutable files and
+ * atomically update the [version].zip.
+ * - Immutable files are identified uniquely based on their file name and file size.
+ * - Immutable files can be reused only across adjacent checkpoints/versions.
+ * - This class is thread-safe. Specifically, it is safe to concurrently delete old files from a
+ * different thread than the task thread saving files.
+ *
+ * @param dfsRootDir Directory where the [version].zip files will be stored
+ * @param localTempDir Local directory for temporary work
+ * @param hadoopConf Hadoop configuration for talking to DFS
+ * @param loggingId Id that will be prepended in logs for isolating concurrent RocksDBs
+ */
+class RocksDBFileManager(
+ dfsRootDir: String,
+ localTempDir: File,
+ hadoopConf: Configuration,
+ loggingId: String = "")
+ extends Logging {
+
+ import RocksDBImmutableFile._
+
+ private val versionToRocksDBFiles = new ConcurrentHashMap[Long, Seq[RocksDBImmutableFile]]
+ private lazy val fm = CheckpointFileManager.create(new Path(dfsRootDir), hadoopConf)
+ private val fs = new Path(dfsRootDir).getFileSystem(hadoopConf)
+
+ /** Save all the files in given local checkpoint directory as a committed version in DFS */
+ def saveCheckpointToDfs(checkpointDir: File, version: Long, numKeys: Long): Unit = {
+ logFilesInDir(checkpointDir, s"Saving checkpoint files for version $version")
+ val allCheckpointFiles = listRocksDBFiles(checkpointDir)
+ val (localImmutableFiles, localOtherFiles) = allCheckpointFiles.partition(isImmutableFile)
+ val rocksDBFiles = saveImmutableFilesToDfs(version, checkpointDir, localImmutableFiles)
+ val metadata = RocksDBCheckpointMetadata(rocksDBFiles, numKeys)
+ rememberImmutableFiles(version, rocksDBFiles)
+ val metadataFile = localMetadataFile(checkpointDir)
+ metadata.writeToFile(metadataFile)
+ logInfo(s"Written metadata for version $version:\n${metadata.prettyJson}")
+
+ if (version <= 0 && numKeys == 0) {
Review comment:
I'm confusing here; if I understand correctly, we use version as batch ID + 1 for HDFSBackedStateStoreProvider, so version 0 with numKeys > 0 doesn't make sense at least for now. (As we don't have a functionality of "initial state" for now.)
How it works for RocksDB state store provider? Would the version be same with batch ID?
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/state/RocksDBFileManager.scala
##########
@@ -17,18 +17,270 @@
package org.apache.spark.sql.execution.streaming.state
-import java.io.File
+import java.io.{File, FileInputStream, InputStream}
import java.nio.charset.StandardCharsets.UTF_8
import java.nio.file.Files
+import java.util.UUID
+import java.util.concurrent.ConcurrentHashMap
+import java.util.zip.{ZipEntry, ZipOutputStream}
+import scala.collection.JavaConverters._
import scala.collection.Seq
import com.fasterxml.jackson.annotation.JsonInclude.Include
import com.fasterxml.jackson.databind.{DeserializationFeature, ObjectMapper}
import com.fasterxml.jackson.module.scala.{DefaultScalaModule, ScalaObjectMapper}
+import org.apache.commons.io.{FilenameUtils, IOUtils}
+import org.apache.hadoop.conf.Configuration
+import org.apache.hadoop.fs.Path
import org.json4s.NoTypeHints
import org.json4s.jackson.Serialization
+import org.apache.spark.internal.Logging
+import org.apache.spark.sql.execution.streaming.CheckpointFileManager
+import org.apache.spark.util.Utils
+
+/**
+ * Class responsible for syncing RocksDB checkpoint files from local disk to DFS.
+ * Every version checkpoint is saved in specific directory structure that allows successive
+ * versions to reuse to SST data files and archived log files. This allows each commit to be
+ * incremental, only new SST files and archived log files generated by RocksDB will be uploaded.
+ * The directory structures on local disk and in DFS are as follows.
+ *
+ * Local checkpoint dir structure
+ * ------------------------------
+ * RocksDB generates a bunch of files in the local checkpoint directory. The most important among
+ * them are the SST files; they are the actual log structured data files. Rest of the files contain
+ * the metadata necessary for RocksDB to read the SST files and start from the checkpoint.
+ * Note that the SST files are hard links to files in the RocksDB's working directory, and therefore
+ * successive checkpoints can share some of the SST files. So these SST files have to be copied to
+ * DFS in shared directory such that different committed versions can save them.
+ *
+ * We consider both SST files and archived log files as immutable files which can be shared between
+ * different checkpoints.
+ *
+ * localCheckpointDir
+ * |
+ * +-- OPTIONS-000005
+ * +-- MANIFEST-000008
+ * +-- CURRENT
+ * +-- 00007.sst
+ * +-- 00011.sst
+ * +-- archive
+ * | +-- 00008.log
+ * | +-- 00013.log
+ * ...
+ *
+ *
+ * DFS directory structure after saving to DFS as version 10
+ * -----------------------------------------------------------
+ * The SST and archived log files are given unique file names and copied to the shared subdirectory.
+ * Every version maintains a mapping of local immutable file name to the unique file name in DFS.
+ * This mapping is saved in a JSON file (named `metadata`), which is zipped along with other
+ * checkpoint files into a single file `[version].zip`.
+ *
+ * dfsRootDir
+ * |
+ * +-- SSTs
+ * | +-- 00007-[uuid1].sst
+ * | +-- 00011-[uuid2].sst
+ * +-- logs
+ * | +-- 00008-[uuid3].log
+ * | +-- 00013-[uuid4].log
+ * +-- 10.zip
+ * | +-- metadata <--- contains mapping between 00007.sst and [uuid1].sst,
+ * and the mapping between 00008.log and [uuid3].log
+ * | +-- OPTIONS-000005
+ * | +-- MANIFEST-000008
+ * | +-- CURRENT
+ * | ...
+ * |
+ * +-- 9.zip
+ * +-- 8.zip
+ * ...
+ *
+ * Note the following.
+ * - Each [version].zip is a complete description of all the data and metadata needed to recover
+ * a RocksDB instance at the corresponding version. This is unlike the [version].delta files of
+ * HDFSBackedStateStore where previous delta files needs to be read to be recovered.
+ * - This is safe wrt speculatively executed tasks running concurrently in different executors
+ * as each task would upload a different copy of the generated immutable files and
+ * atomically update the [version].zip.
+ * - Immutable files are identified uniquely based on their file name and file size.
+ * - Immutable files can be reused only across adjacent checkpoints/versions.
+ * - This class is thread-safe. Specifically, it is safe to concurrently delete old files from a
+ * different thread than the task thread saving files.
+ *
+ * @param dfsRootDir Directory where the [version].zip files will be stored
+ * @param localTempDir Local directory for temporary work
+ * @param hadoopConf Hadoop configuration for talking to DFS
+ * @param loggingId Id that will be prepended in logs for isolating concurrent RocksDBs
+ */
+class RocksDBFileManager(
+ dfsRootDir: String,
+ localTempDir: File,
+ hadoopConf: Configuration,
+ loggingId: String = "")
+ extends Logging {
+
+ import RocksDBImmutableFile._
+
+ private val versionToRocksDBFiles = new ConcurrentHashMap[Long, Seq[RocksDBImmutableFile]]
+ private lazy val fm = CheckpointFileManager.create(new Path(dfsRootDir), hadoopConf)
+ private val fs = new Path(dfsRootDir).getFileSystem(hadoopConf)
+
+ /** Save all the files in given local checkpoint directory as a committed version in DFS */
+ def saveCheckpointToDfs(checkpointDir: File, version: Long, numKeys: Long): Unit = {
+ logFilesInDir(checkpointDir, s"Saving checkpoint files for version $version")
+ val allCheckpointFiles = listRocksDBFiles(checkpointDir)
+ val (localImmutableFiles, localOtherFiles) = allCheckpointFiles.partition(isImmutableFile)
Review comment:
How many archived log files we expect on each call? Because...
```
private def listRocksDBFiles(localDir: File): Seq[File] = {
val topLevelFiles = localDir.listFiles.filter(!_.isDirectory)
val archivedLogFiles =
Option(new File(localDir, LOG_FILES_LOCAL_SUBDIR).listFiles())
.getOrElse(Array[File]())
// To ignore .log.crc files
.filter(file => isLogFile(file.getName))
topLevelFiles ++ archivedLogFiles
}
```
the elements of `archivedLogFiles` are all true for `isArchivedLogFile`, so checking the immutability is unnecessary.
Only the elements of `topLevelFiles` need to be scanned to check immutability, and even `isArchivedLogFile` is not necessary for them as it will be always `false`.
Probably we could change `listRocksDBFiles` to also classify immutable files and mutable files. I guess it's OK to have multiple returns (with adding comment about the return value) as it's a private method.
```
private def listRocksDBFiles(localDir: File): (Seq[File], Seq[File]) = {
val topLevelFiles = localDir.listFiles.filter(!_.isDirectory)
val archivedLogFiles =
Option(new File(localDir, LOG_FILES_LOCAL_SUBDIR).listFiles())
.getOrElse(Array[File]())
// To ignore .log.crc files
.filter(file => isLogFile(file.getName))
val (topLevelSstFiles, topLevelOtherFiles) = topLevelFiles.partition(isSstFile)
(topLevelSstFiles ++ archivedLogFiles, topLevelOtherFiles)
}
```
Would it make sense?
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/state/RocksDBFileManager.scala
##########
@@ -17,18 +17,270 @@
package org.apache.spark.sql.execution.streaming.state
-import java.io.File
+import java.io.{File, FileInputStream, InputStream}
import java.nio.charset.StandardCharsets.UTF_8
import java.nio.file.Files
+import java.util.UUID
+import java.util.concurrent.ConcurrentHashMap
+import java.util.zip.{ZipEntry, ZipOutputStream}
+import scala.collection.JavaConverters._
import scala.collection.Seq
import com.fasterxml.jackson.annotation.JsonInclude.Include
import com.fasterxml.jackson.databind.{DeserializationFeature, ObjectMapper}
import com.fasterxml.jackson.module.scala.{DefaultScalaModule, ScalaObjectMapper}
+import org.apache.commons.io.{FilenameUtils, IOUtils}
+import org.apache.hadoop.conf.Configuration
+import org.apache.hadoop.fs.Path
import org.json4s.NoTypeHints
import org.json4s.jackson.Serialization
+import org.apache.spark.internal.Logging
+import org.apache.spark.sql.execution.streaming.CheckpointFileManager
+import org.apache.spark.util.Utils
+
+/**
+ * Class responsible for syncing RocksDB checkpoint files from local disk to DFS.
+ * Every version checkpoint is saved in specific directory structure that allows successive
+ * versions to reuse to SST data files and archived log files. This allows each commit to be
+ * incremental, only new SST files and archived log files generated by RocksDB will be uploaded.
+ * The directory structures on local disk and in DFS are as follows.
+ *
+ * Local checkpoint dir structure
+ * ------------------------------
+ * RocksDB generates a bunch of files in the local checkpoint directory. The most important among
+ * them are the SST files; they are the actual log structured data files. Rest of the files contain
+ * the metadata necessary for RocksDB to read the SST files and start from the checkpoint.
+ * Note that the SST files are hard links to files in the RocksDB's working directory, and therefore
+ * successive checkpoints can share some of the SST files. So these SST files have to be copied to
+ * DFS in shared directory such that different committed versions can save them.
+ *
+ * We consider both SST files and archived log files as immutable files which can be shared between
+ * different checkpoints.
+ *
+ * localCheckpointDir
+ * |
+ * +-- OPTIONS-000005
+ * +-- MANIFEST-000008
+ * +-- CURRENT
+ * +-- 00007.sst
+ * +-- 00011.sst
+ * +-- archive
+ * | +-- 00008.log
+ * | +-- 00013.log
+ * ...
+ *
+ *
+ * DFS directory structure after saving to DFS as version 10
+ * -----------------------------------------------------------
+ * The SST and archived log files are given unique file names and copied to the shared subdirectory.
+ * Every version maintains a mapping of local immutable file name to the unique file name in DFS.
+ * This mapping is saved in a JSON file (named `metadata`), which is zipped along with other
+ * checkpoint files into a single file `[version].zip`.
+ *
+ * dfsRootDir
+ * |
+ * +-- SSTs
+ * | +-- 00007-[uuid1].sst
+ * | +-- 00011-[uuid2].sst
+ * +-- logs
+ * | +-- 00008-[uuid3].log
+ * | +-- 00013-[uuid4].log
+ * +-- 10.zip
+ * | +-- metadata <--- contains mapping between 00007.sst and [uuid1].sst,
+ * and the mapping between 00008.log and [uuid3].log
+ * | +-- OPTIONS-000005
+ * | +-- MANIFEST-000008
+ * | +-- CURRENT
+ * | ...
+ * |
+ * +-- 9.zip
+ * +-- 8.zip
+ * ...
+ *
+ * Note the following.
+ * - Each [version].zip is a complete description of all the data and metadata needed to recover
+ * a RocksDB instance at the corresponding version. This is unlike the [version].delta files of
+ * HDFSBackedStateStore where previous delta files needs to be read to be recovered.
+ * - This is safe wrt speculatively executed tasks running concurrently in different executors
+ * as each task would upload a different copy of the generated immutable files and
+ * atomically update the [version].zip.
+ * - Immutable files are identified uniquely based on their file name and file size.
+ * - Immutable files can be reused only across adjacent checkpoints/versions.
+ * - This class is thread-safe. Specifically, it is safe to concurrently delete old files from a
+ * different thread than the task thread saving files.
+ *
+ * @param dfsRootDir Directory where the [version].zip files will be stored
+ * @param localTempDir Local directory for temporary work
+ * @param hadoopConf Hadoop configuration for talking to DFS
+ * @param loggingId Id that will be prepended in logs for isolating concurrent RocksDBs
+ */
+class RocksDBFileManager(
+ dfsRootDir: String,
+ localTempDir: File,
+ hadoopConf: Configuration,
+ loggingId: String = "")
+ extends Logging {
+
+ import RocksDBImmutableFile._
+
+ private val versionToRocksDBFiles = new ConcurrentHashMap[Long, Seq[RocksDBImmutableFile]]
+ private lazy val fm = CheckpointFileManager.create(new Path(dfsRootDir), hadoopConf)
+ private val fs = new Path(dfsRootDir).getFileSystem(hadoopConf)
+
+ /** Save all the files in given local checkpoint directory as a committed version in DFS */
+ def saveCheckpointToDfs(checkpointDir: File, version: Long, numKeys: Long): Unit = {
+ logFilesInDir(checkpointDir, s"Saving checkpoint files for version $version")
+ val allCheckpointFiles = listRocksDBFiles(checkpointDir)
+ val (localImmutableFiles, localOtherFiles) = allCheckpointFiles.partition(isImmutableFile)
+ val rocksDBFiles = saveImmutableFilesToDfs(version, checkpointDir, localImmutableFiles)
+ val metadata = RocksDBCheckpointMetadata(rocksDBFiles, numKeys)
+ rememberImmutableFiles(version, rocksDBFiles)
+ val metadataFile = localMetadataFile(checkpointDir)
+ metadata.writeToFile(metadataFile)
+ logInfo(s"Written metadata for version $version:\n${metadata.prettyJson}")
+
+ if (version <= 0 && numKeys == 0) {
+ // If we're writing the initial version and there's no data, we have to explicitly initialize
+ // the root directory. Normally saveImmutableFilesToDfs will do this initialization, but
+ // when there's no data that method won't write any files, and zipToDfsFile uses the
+ // CheckpointFileManager.createAtomic API which doesn't auto-initialize parent directories.
+ fm.mkdirs(new Path(dfsRootDir))
+ }
+ zipToDfsFile(localOtherFiles :+ metadataFile, dfsBatchZipFile(version))
+ logInfo(s"Saved checkpoint file for version $version")
+ }
+
+ /** Save immutable files to DFS directory */
+ private def saveImmutableFilesToDfs(
+ version: Long,
+ localDir: File,
Review comment:
nit: unused
##########
File path: sql/core/src/test/scala/org/apache/spark/sql/execution/streaming/state/RocksDBSuite.scala
##########
@@ -20,12 +20,86 @@ package org.apache.spark.sql.execution.streaming.state
import java.io._
import java.nio.charset.Charset
+import scala.language.implicitConversions
+
import org.apache.commons.io.FileUtils
+import org.apache.hadoop.conf.Configuration
import org.apache.spark._
+import org.apache.spark.sql.catalyst.util.quietly
+import org.apache.spark.sql.execution.streaming.CreateAtomicTestManager
+import org.apache.spark.sql.internal.SQLConf
+import org.apache.spark.util.Utils
class RocksDBSuite extends SparkFunSuite {
+ test("RocksDBFileManager: upload only new immutable files") {
+ val dfsRootDir = Utils.createTempDir().getAbsolutePath
+ val fileManager = new RocksDBFileManager(
+ dfsRootDir, Utils.createTempDir(), new Configuration)
+ val sstDir = s"$dfsRootDir/SSTs"
+ def numRemoteSSTFiles: Int = listFiles(sstDir).length
+ val logDir = s"$dfsRootDir/logs"
+ def numRemoteLogFiles: Int = listFiles(logDir).length
Review comment:
Shall we have some code to check other files & metadata as well? We don't seem to cover the test on existence of zip file and the content.
##########
File path: sql/core/src/test/scala/org/apache/spark/sql/execution/streaming/state/RocksDBSuite.scala
##########
@@ -54,4 +128,31 @@ class RocksDBSuite extends SparkFunSuite {
"""{"sstFiles":[{"localFileName":"00001.sst","dfsSstFileName":"00001-uuid.sst","sizeBytes":12345678901234}],"logFiles":[{"localFileName":"00001.log","dfsLogFileName":"00001-uuid.log","sizeBytes":12345678901234}],"numKeys":12345678901234}""")
// scalastyle:on line.size.limit
}
+
Review comment:
nit: empty two lines
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/state/RocksDBFileManager.scala
##########
@@ -17,18 +17,270 @@
package org.apache.spark.sql.execution.streaming.state
-import java.io.File
+import java.io.{File, FileInputStream, InputStream}
import java.nio.charset.StandardCharsets.UTF_8
import java.nio.file.Files
+import java.util.UUID
+import java.util.concurrent.ConcurrentHashMap
+import java.util.zip.{ZipEntry, ZipOutputStream}
+import scala.collection.JavaConverters._
import scala.collection.Seq
import com.fasterxml.jackson.annotation.JsonInclude.Include
import com.fasterxml.jackson.databind.{DeserializationFeature, ObjectMapper}
import com.fasterxml.jackson.module.scala.{DefaultScalaModule, ScalaObjectMapper}
+import org.apache.commons.io.{FilenameUtils, IOUtils}
+import org.apache.hadoop.conf.Configuration
+import org.apache.hadoop.fs.Path
import org.json4s.NoTypeHints
import org.json4s.jackson.Serialization
+import org.apache.spark.internal.Logging
+import org.apache.spark.sql.execution.streaming.CheckpointFileManager
+import org.apache.spark.util.Utils
+
+/**
+ * Class responsible for syncing RocksDB checkpoint files from local disk to DFS.
+ * Every version checkpoint is saved in specific directory structure that allows successive
+ * versions to reuse to SST data files and archived log files. This allows each commit to be
+ * incremental, only new SST files and archived log files generated by RocksDB will be uploaded.
+ * The directory structures on local disk and in DFS are as follows.
+ *
+ * Local checkpoint dir structure
+ * ------------------------------
+ * RocksDB generates a bunch of files in the local checkpoint directory. The most important among
+ * them are the SST files; they are the actual log structured data files. Rest of the files contain
+ * the metadata necessary for RocksDB to read the SST files and start from the checkpoint.
+ * Note that the SST files are hard links to files in the RocksDB's working directory, and therefore
+ * successive checkpoints can share some of the SST files. So these SST files have to be copied to
+ * DFS in shared directory such that different committed versions can save them.
+ *
+ * We consider both SST files and archived log files as immutable files which can be shared between
+ * different checkpoints.
+ *
+ * localCheckpointDir
+ * |
+ * +-- OPTIONS-000005
+ * +-- MANIFEST-000008
+ * +-- CURRENT
+ * +-- 00007.sst
+ * +-- 00011.sst
+ * +-- archive
+ * | +-- 00008.log
+ * | +-- 00013.log
+ * ...
+ *
+ *
+ * DFS directory structure after saving to DFS as version 10
+ * -----------------------------------------------------------
+ * The SST and archived log files are given unique file names and copied to the shared subdirectory.
+ * Every version maintains a mapping of local immutable file name to the unique file name in DFS.
+ * This mapping is saved in a JSON file (named `metadata`), which is zipped along with other
+ * checkpoint files into a single file `[version].zip`.
+ *
+ * dfsRootDir
+ * |
+ * +-- SSTs
+ * | +-- 00007-[uuid1].sst
+ * | +-- 00011-[uuid2].sst
+ * +-- logs
+ * | +-- 00008-[uuid3].log
+ * | +-- 00013-[uuid4].log
+ * +-- 10.zip
+ * | +-- metadata <--- contains mapping between 00007.sst and [uuid1].sst,
+ * and the mapping between 00008.log and [uuid3].log
+ * | +-- OPTIONS-000005
+ * | +-- MANIFEST-000008
+ * | +-- CURRENT
+ * | ...
+ * |
+ * +-- 9.zip
+ * +-- 8.zip
+ * ...
+ *
+ * Note the following.
+ * - Each [version].zip is a complete description of all the data and metadata needed to recover
Review comment:
It'd be nice to note here again the SST files and LOG files are not included to the zip file. I was mistaken as it contains all files including data as well, and concerned about the overhead of creating and sending zip file due to the size.
##########
File path: sql/core/src/test/scala/org/apache/spark/sql/execution/streaming/state/RocksDBSuite.scala
##########
@@ -54,4 +128,31 @@ class RocksDBSuite extends SparkFunSuite {
"""{"sstFiles":[{"localFileName":"00001.sst","dfsSstFileName":"00001-uuid.sst","sizeBytes":12345678901234}],"logFiles":[{"localFileName":"00001.log","dfsLogFileName":"00001-uuid.log","sizeBytes":12345678901234}],"numKeys":12345678901234}""")
// scalastyle:on line.size.limit
}
+
+
+ def generateFiles(dir: String, fileToLengths: Seq[(String, Int)]): Unit = {
+ fileToLengths.foreach { case (fileName, length) =>
+ val file = new File(dir, fileName)
+ FileUtils.write(file, "a" * length)
+ }
+ }
+
+ def saveCheckpointFiles(
+ fileManager: RocksDBFileManager,
+ fileToLengths: Seq[(String, Int)],
+ version: Int,
+ numKeys: Int): Unit = {
+ val checkpointDir = Utils.createTempDir().getAbsolutePath // local dir to create checkpoints
+ generateFiles(checkpointDir, fileToLengths)
+ fileManager.saveCheckpointToDfs(checkpointDir, version, numKeys)
+ }
+
+ implicit def toFile(path: String): File = new File(path)
Review comment:
With this, `def listFiles(file: String)` is probably unnecessary, as I expect implicit conversion happening when calling `listFiles(file: File)` with String.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32582: [SPARK-35436][SS] RocksDBFileManager - save checkpoint to DFS
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32582:
URL: https://github.com/apache/spark/pull/32582#issuecomment-857345167
**[Test build #139529 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/139529/testReport)** for PR 32582 at commit [`c05b140`](https://github.com/apache/spark/commit/c05b140f73f855a148ccedaf753841fb94a4ab1d).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32582: [SPARK-35436][SS] RocksDBFileManager - save checkpoint to DFS
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32582:
URL: https://github.com/apache/spark/pull/32582#issuecomment-857357944
**[Test build #139516 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/139516/testReport)** for PR 32582 at commit [`f86cea4`](https://github.com/apache/spark/commit/f86cea4fb9c75d3dc29ec83d6f05184431d8584e).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HeartSaVioR commented on a change in pull request #32582: [SPARK-35436][SS] RocksDBFileManager - save checkpoint to DFS
Posted by GitBox <gi...@apache.org>.
HeartSaVioR commented on a change in pull request #32582:
URL: https://github.com/apache/spark/pull/32582#discussion_r645312635
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/state/RocksDBFileManager.scala
##########
@@ -17,18 +17,270 @@
package org.apache.spark.sql.execution.streaming.state
-import java.io.File
+import java.io.{File, FileInputStream, InputStream}
import java.nio.charset.StandardCharsets.UTF_8
import java.nio.file.Files
+import java.util.UUID
+import java.util.concurrent.ConcurrentHashMap
+import java.util.zip.{ZipEntry, ZipOutputStream}
+import scala.collection.JavaConverters._
import scala.collection.Seq
import com.fasterxml.jackson.annotation.JsonInclude.Include
import com.fasterxml.jackson.databind.{DeserializationFeature, ObjectMapper}
import com.fasterxml.jackson.module.scala.{DefaultScalaModule, ScalaObjectMapper}
+import org.apache.commons.io.{FilenameUtils, IOUtils}
+import org.apache.hadoop.conf.Configuration
+import org.apache.hadoop.fs.Path
import org.json4s.NoTypeHints
import org.json4s.jackson.Serialization
+import org.apache.spark.internal.Logging
+import org.apache.spark.sql.execution.streaming.CheckpointFileManager
+import org.apache.spark.util.Utils
+
+/**
+ * Class responsible for syncing RocksDB checkpoint files from local disk to DFS.
+ * Every version checkpoint is saved in specific directory structure that allows successive
+ * versions to reuse to SST data files and archived log files. This allows each commit to be
+ * incremental, only new SST files and archived log files generated by RocksDB will be uploaded.
+ * The directory structures on local disk and in DFS are as follows.
+ *
+ * Local checkpoint dir structure
+ * ------------------------------
+ * RocksDB generates a bunch of files in the local checkpoint directory. The most important among
+ * them are the SST files; they are the actual log structured data files. Rest of the files contain
+ * the metadata necessary for RocksDB to read the SST files and start from the checkpoint.
+ * Note that the SST files are hard links to files in the RocksDB's working directory, and therefore
+ * successive checkpoints can share some of the SST files. So these SST files have to be copied to
+ * DFS in shared directory such that different committed versions can save them.
+ *
+ * We consider both SST files and archived log files as immutable files which can be shared between
+ * different checkpoints.
+ *
+ * localCheckpointDir
+ * |
+ * +-- OPTIONS-000005
+ * +-- MANIFEST-000008
+ * +-- CURRENT
+ * +-- 00007.sst
+ * +-- 00011.sst
+ * +-- archive
+ * | +-- 00008.log
+ * | +-- 00013.log
+ * ...
+ *
+ *
+ * DFS directory structure after saving to DFS as version 10
+ * -----------------------------------------------------------
+ * The SST and archived log files are given unique file names and copied to the shared subdirectory.
+ * Every version maintains a mapping of local immutable file name to the unique file name in DFS.
+ * This mapping is saved in a JSON file (named `metadata`), which is zipped along with other
+ * checkpoint files into a single file `[version].zip`.
+ *
+ * dfsRootDir
+ * |
+ * +-- SSTs
+ * | +-- 00007-[uuid1].sst
+ * | +-- 00011-[uuid2].sst
+ * +-- logs
+ * | +-- 00008-[uuid3].log
+ * | +-- 00013-[uuid4].log
+ * +-- 10.zip
+ * | +-- metadata <--- contains mapping between 00007.sst and [uuid1].sst,
+ * and the mapping between 00008.log and [uuid3].log
+ * | +-- OPTIONS-000005
+ * | +-- MANIFEST-000008
+ * | +-- CURRENT
+ * | ...
+ * |
+ * +-- 9.zip
+ * +-- 8.zip
+ * ...
+ *
+ * Note the following.
+ * - Each [version].zip is a complete description of all the data and metadata needed to recover
+ * a RocksDB instance at the corresponding version. This is unlike the [version].delta files of
+ * HDFSBackedStateStore where previous delta files needs to be read to be recovered.
+ * - This is safe wrt speculatively executed tasks running concurrently in different executors
+ * as each task would upload a different copy of the generated immutable files and
+ * atomically update the [version].zip.
+ * - Immutable files are identified uniquely based on their file name and file size.
+ * - Immutable files can be reused only across adjacent checkpoints/versions.
+ * - This class is thread-safe. Specifically, it is safe to concurrently delete old files from a
+ * different thread than the task thread saving files.
+ *
+ * @param dfsRootDir Directory where the [version].zip files will be stored
+ * @param localTempDir Local directory for temporary work
+ * @param hadoopConf Hadoop configuration for talking to DFS
+ * @param loggingId Id that will be prepended in logs for isolating concurrent RocksDBs
+ */
+class RocksDBFileManager(
+ dfsRootDir: String,
+ localTempDir: File,
+ hadoopConf: Configuration,
+ loggingId: String = "")
+ extends Logging {
+
+ import RocksDBImmutableFile._
+
+ private val versionToRocksDBFiles = new ConcurrentHashMap[Long, Seq[RocksDBImmutableFile]]
+ private lazy val fm = CheckpointFileManager.create(new Path(dfsRootDir), hadoopConf)
+ private val fs = new Path(dfsRootDir).getFileSystem(hadoopConf)
+
+ /** Save all the files in given local checkpoint directory as a committed version in DFS */
+ def saveCheckpointToDfs(checkpointDir: File, version: Long, numKeys: Long): Unit = {
+ logFilesInDir(checkpointDir, s"Saving checkpoint files for version $version")
+ val allCheckpointFiles = listRocksDBFiles(checkpointDir)
+ val (localImmutableFiles, localOtherFiles) = allCheckpointFiles.partition(isImmutableFile)
+ val rocksDBFiles = saveImmutableFilesToDfs(version, checkpointDir, localImmutableFiles)
+ val metadata = RocksDBCheckpointMetadata(rocksDBFiles, numKeys)
+ rememberImmutableFiles(version, rocksDBFiles)
+ val metadataFile = localMetadataFile(checkpointDir)
+ metadata.writeToFile(metadataFile)
+ logInfo(s"Written metadata for version $version:\n${metadata.prettyJson}")
+
+ if (version <= 0 && numKeys == 0) {
Review comment:
What about simply making sure the directory has been created in prior? We can add below in initialization and don't bother with handling specific case.
```
// create a root directory on initialization if necessary
if (!fm.exists(new Path(dfsRootDir))) {
fm.mkdirs(new Path(dfsRootDir))
}
```
If we'd like to avoid paying the cost on calling exist, let's keep the code and change the version to 1 as I met the case the version starts from 1 during test.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HeartSaVioR commented on a change in pull request #32582: [SPARK-35436][SS] RocksDBFileManager - save checkpoint to DFS
Posted by GitBox <gi...@apache.org>.
HeartSaVioR commented on a change in pull request #32582:
URL: https://github.com/apache/spark/pull/32582#discussion_r642018073
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/state/RocksDBFileManager.scala
##########
@@ -17,18 +17,270 @@
package org.apache.spark.sql.execution.streaming.state
-import java.io.File
+import java.io.{File, FileInputStream, InputStream}
import java.nio.charset.StandardCharsets.UTF_8
import java.nio.file.Files
+import java.util.UUID
+import java.util.concurrent.ConcurrentHashMap
+import java.util.zip.{ZipEntry, ZipOutputStream}
+import scala.collection.JavaConverters._
import scala.collection.Seq
import com.fasterxml.jackson.annotation.JsonInclude.Include
import com.fasterxml.jackson.databind.{DeserializationFeature, ObjectMapper}
import com.fasterxml.jackson.module.scala.{DefaultScalaModule, ScalaObjectMapper}
+import org.apache.commons.io.{FilenameUtils, IOUtils}
+import org.apache.hadoop.conf.Configuration
+import org.apache.hadoop.fs.Path
import org.json4s.NoTypeHints
import org.json4s.jackson.Serialization
+import org.apache.spark.internal.Logging
+import org.apache.spark.sql.execution.streaming.CheckpointFileManager
+import org.apache.spark.util.Utils
+
+/**
+ * Class responsible for syncing RocksDB checkpoint files from local disk to DFS.
+ * Every version checkpoint is saved in specific directory structure that allows successive
Review comment:
nit: Every version -> For each version,
or just , between version and checkpoint?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] xuanyuanking commented on a change in pull request #32582: [SPARK-35436][SS] RocksDBFileManager - save checkpoint to DFS
Posted by GitBox <gi...@apache.org>.
xuanyuanking commented on a change in pull request #32582:
URL: https://github.com/apache/spark/pull/32582#discussion_r643853022
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/state/RocksDBFileManager.scala
##########
@@ -17,18 +17,270 @@
package org.apache.spark.sql.execution.streaming.state
-import java.io.File
+import java.io.{File, FileInputStream, InputStream}
import java.nio.charset.StandardCharsets.UTF_8
import java.nio.file.Files
+import java.util.UUID
+import java.util.concurrent.ConcurrentHashMap
+import java.util.zip.{ZipEntry, ZipOutputStream}
+import scala.collection.JavaConverters._
import scala.collection.Seq
import com.fasterxml.jackson.annotation.JsonInclude.Include
import com.fasterxml.jackson.databind.{DeserializationFeature, ObjectMapper}
import com.fasterxml.jackson.module.scala.{DefaultScalaModule, ScalaObjectMapper}
+import org.apache.commons.io.{FilenameUtils, IOUtils}
+import org.apache.hadoop.conf.Configuration
+import org.apache.hadoop.fs.Path
import org.json4s.NoTypeHints
import org.json4s.jackson.Serialization
+import org.apache.spark.internal.Logging
+import org.apache.spark.sql.execution.streaming.CheckpointFileManager
+import org.apache.spark.util.Utils
+
+/**
+ * Class responsible for syncing RocksDB checkpoint files from local disk to DFS.
+ * Every version checkpoint is saved in specific directory structure that allows successive
+ * versions to reuse to SST data files and archived log files. This allows each commit to be
+ * incremental, only new SST files and archived log files generated by RocksDB will be uploaded.
+ * The directory structures on local disk and in DFS are as follows.
+ *
+ * Local checkpoint dir structure
+ * ------------------------------
+ * RocksDB generates a bunch of files in the local checkpoint directory. The most important among
+ * them are the SST files; they are the actual log structured data files. Rest of the files contain
+ * the metadata necessary for RocksDB to read the SST files and start from the checkpoint.
+ * Note that the SST files are hard links to files in the RocksDB's working directory, and therefore
+ * successive checkpoints can share some of the SST files. So these SST files have to be copied to
+ * DFS in shared directory such that different committed versions can save them.
+ *
+ * We consider both SST files and archived log files as immutable files which can be shared between
+ * different checkpoints.
+ *
+ * localCheckpointDir
+ * |
+ * +-- OPTIONS-000005
+ * +-- MANIFEST-000008
+ * +-- CURRENT
+ * +-- 00007.sst
+ * +-- 00011.sst
+ * +-- archive
+ * | +-- 00008.log
+ * | +-- 00013.log
+ * ...
+ *
+ *
+ * DFS directory structure after saving to DFS as version 10
+ * -----------------------------------------------------------
+ * The SST and archived log files are given unique file names and copied to the shared subdirectory.
+ * Every version maintains a mapping of local immutable file name to the unique file name in DFS.
+ * This mapping is saved in a JSON file (named `metadata`), which is zipped along with other
+ * checkpoint files into a single file `[version].zip`.
+ *
+ * dfsRootDir
+ * |
+ * +-- SSTs
+ * | +-- 00007-[uuid1].sst
+ * | +-- 00011-[uuid2].sst
+ * +-- logs
+ * | +-- 00008-[uuid3].log
+ * | +-- 00013-[uuid4].log
+ * +-- 10.zip
+ * | +-- metadata <--- contains mapping between 00007.sst and [uuid1].sst,
+ * and the mapping between 00008.log and [uuid3].log
+ * | +-- OPTIONS-000005
+ * | +-- MANIFEST-000008
+ * | +-- CURRENT
+ * | ...
+ * |
+ * +-- 9.zip
+ * +-- 8.zip
+ * ...
+ *
+ * Note the following.
+ * - Each [version].zip is a complete description of all the data and metadata needed to recover
Review comment:
Make sense, add the comment in 3800c51
##########
File path: sql/core/src/test/scala/org/apache/spark/sql/execution/streaming/state/RocksDBSuite.scala
##########
@@ -54,4 +128,31 @@ class RocksDBSuite extends SparkFunSuite {
"""{"sstFiles":[{"localFileName":"00001.sst","dfsSstFileName":"00001-uuid.sst","sizeBytes":12345678901234}],"logFiles":[{"localFileName":"00001.log","dfsLogFileName":"00001-uuid.log","sizeBytes":12345678901234}],"numKeys":12345678901234}""")
// scalastyle:on line.size.limit
}
+
Review comment:
Thanks, done in 3800c51
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #32582: [SPARK-35436][SS] RocksDBFileManager - save checkpoint to DFS
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #32582:
URL: https://github.com/apache/spark/pull/32582#issuecomment-857361732
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/139516/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] viirya commented on pull request #32582: [WIP][SPARK-35436] RocksDBFileManager - save checkpoint to DFS
Posted by GitBox <gi...@apache.org>.
viirya commented on pull request #32582:
URL: https://github.com/apache/spark/pull/32582#issuecomment-843377733
Thanks @dongjoon-hyun. I'll look at this.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] xuanyuanking commented on a change in pull request #32582: [SPARK-35436][SS] RocksDBFileManager - save checkpoint to DFS
Posted by GitBox <gi...@apache.org>.
xuanyuanking commented on a change in pull request #32582:
URL: https://github.com/apache/spark/pull/32582#discussion_r643851762
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/state/RocksDBFileManager.scala
##########
@@ -17,18 +17,270 @@
package org.apache.spark.sql.execution.streaming.state
-import java.io.File
+import java.io.{File, FileInputStream, InputStream}
import java.nio.charset.StandardCharsets.UTF_8
import java.nio.file.Files
+import java.util.UUID
+import java.util.concurrent.ConcurrentHashMap
+import java.util.zip.{ZipEntry, ZipOutputStream}
+import scala.collection.JavaConverters._
import scala.collection.Seq
import com.fasterxml.jackson.annotation.JsonInclude.Include
import com.fasterxml.jackson.databind.{DeserializationFeature, ObjectMapper}
import com.fasterxml.jackson.module.scala.{DefaultScalaModule, ScalaObjectMapper}
+import org.apache.commons.io.{FilenameUtils, IOUtils}
+import org.apache.hadoop.conf.Configuration
+import org.apache.hadoop.fs.Path
import org.json4s.NoTypeHints
import org.json4s.jackson.Serialization
+import org.apache.spark.internal.Logging
+import org.apache.spark.sql.execution.streaming.CheckpointFileManager
+import org.apache.spark.util.Utils
+
+/**
+ * Class responsible for syncing RocksDB checkpoint files from local disk to DFS.
+ * Every version checkpoint is saved in specific directory structure that allows successive
Review comment:
Thanks, changed to `For each version` in 3800c51
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] viirya commented on pull request #32582: [SPARK-35436][SS] RocksDBFileManager - save checkpoint to DFS
Posted by GitBox <gi...@apache.org>.
viirya commented on pull request #32582:
URL: https://github.com/apache/spark/pull/32582#issuecomment-857247663
Thanks @xuanyuanking for the work. Let's move forward to next related PR.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #32582: [SPARK-35436][SS] RocksDBFileManager - save checkpoint to DFS
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #32582:
URL: https://github.com/apache/spark/pull/32582#issuecomment-850180715
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/139041/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] xuanyuanking commented on a change in pull request #32582: [SPARK-35436][SS] RocksDBFileManager - save checkpoint to DFS
Posted by GitBox <gi...@apache.org>.
xuanyuanking commented on a change in pull request #32582:
URL: https://github.com/apache/spark/pull/32582#discussion_r645673106
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/state/RocksDBFileManager.scala
##########
@@ -17,18 +17,270 @@
package org.apache.spark.sql.execution.streaming.state
-import java.io.File
+import java.io.{File, FileInputStream, InputStream}
import java.nio.charset.StandardCharsets.UTF_8
import java.nio.file.Files
+import java.util.UUID
+import java.util.concurrent.ConcurrentHashMap
+import java.util.zip.{ZipEntry, ZipOutputStream}
+import scala.collection.JavaConverters._
import scala.collection.Seq
import com.fasterxml.jackson.annotation.JsonInclude.Include
import com.fasterxml.jackson.databind.{DeserializationFeature, ObjectMapper}
import com.fasterxml.jackson.module.scala.{DefaultScalaModule, ScalaObjectMapper}
+import org.apache.commons.io.{FilenameUtils, IOUtils}
+import org.apache.hadoop.conf.Configuration
+import org.apache.hadoop.fs.Path
import org.json4s.NoTypeHints
import org.json4s.jackson.Serialization
+import org.apache.spark.internal.Logging
+import org.apache.spark.sql.execution.streaming.CheckpointFileManager
+import org.apache.spark.util.Utils
+
+/**
+ * Class responsible for syncing RocksDB checkpoint files from local disk to DFS.
+ * Every version checkpoint is saved in specific directory structure that allows successive
+ * versions to reuse to SST data files and archived log files. This allows each commit to be
+ * incremental, only new SST files and archived log files generated by RocksDB will be uploaded.
+ * The directory structures on local disk and in DFS are as follows.
+ *
+ * Local checkpoint dir structure
+ * ------------------------------
+ * RocksDB generates a bunch of files in the local checkpoint directory. The most important among
+ * them are the SST files; they are the actual log structured data files. Rest of the files contain
+ * the metadata necessary for RocksDB to read the SST files and start from the checkpoint.
+ * Note that the SST files are hard links to files in the RocksDB's working directory, and therefore
+ * successive checkpoints can share some of the SST files. So these SST files have to be copied to
+ * DFS in shared directory such that different committed versions can save them.
+ *
+ * We consider both SST files and archived log files as immutable files which can be shared between
+ * different checkpoints.
+ *
+ * localCheckpointDir
+ * |
+ * +-- OPTIONS-000005
+ * +-- MANIFEST-000008
+ * +-- CURRENT
+ * +-- 00007.sst
+ * +-- 00011.sst
+ * +-- archive
+ * | +-- 00008.log
+ * | +-- 00013.log
+ * ...
+ *
+ *
+ * DFS directory structure after saving to DFS as version 10
+ * -----------------------------------------------------------
+ * The SST and archived log files are given unique file names and copied to the shared subdirectory.
+ * Every version maintains a mapping of local immutable file name to the unique file name in DFS.
+ * This mapping is saved in a JSON file (named `metadata`), which is zipped along with other
+ * checkpoint files into a single file `[version].zip`.
+ *
+ * dfsRootDir
+ * |
+ * +-- SSTs
+ * | +-- 00007-[uuid1].sst
+ * | +-- 00011-[uuid2].sst
+ * +-- logs
+ * | +-- 00008-[uuid3].log
+ * | +-- 00013-[uuid4].log
+ * +-- 10.zip
+ * | +-- metadata <--- contains mapping between 00007.sst and [uuid1].sst,
+ * and the mapping between 00008.log and [uuid3].log
+ * | +-- OPTIONS-000005
+ * | +-- MANIFEST-000008
+ * | +-- CURRENT
+ * | ...
+ * |
+ * +-- 9.zip
+ * +-- 8.zip
+ * ...
+ *
+ * Note the following.
+ * - Each [version].zip is a complete description of all the data and metadata needed to recover
+ * a RocksDB instance at the corresponding version. This is unlike the [version].delta files of
+ * HDFSBackedStateStore where previous delta files needs to be read to be recovered.
+ * - This is safe wrt speculatively executed tasks running concurrently in different executors
+ * as each task would upload a different copy of the generated immutable files and
+ * atomically update the [version].zip.
+ * - Immutable files are identified uniquely based on their file name and file size.
+ * - Immutable files can be reused only across adjacent checkpoints/versions.
+ * - This class is thread-safe. Specifically, it is safe to concurrently delete old files from a
+ * different thread than the task thread saving files.
+ *
+ * @param dfsRootDir Directory where the [version].zip files will be stored
+ * @param localTempDir Local directory for temporary work
+ * @param hadoopConf Hadoop configuration for talking to DFS
+ * @param loggingId Id that will be prepended in logs for isolating concurrent RocksDBs
+ */
+class RocksDBFileManager(
+ dfsRootDir: String,
+ localTempDir: File,
+ hadoopConf: Configuration,
+ loggingId: String = "")
+ extends Logging {
+
+ import RocksDBImmutableFile._
+
+ private val versionToRocksDBFiles = new ConcurrentHashMap[Long, Seq[RocksDBImmutableFile]]
+ private lazy val fm = CheckpointFileManager.create(new Path(dfsRootDir), hadoopConf)
+ private val fs = new Path(dfsRootDir).getFileSystem(hadoopConf)
+
+ /** Save all the files in given local checkpoint directory as a committed version in DFS */
+ def saveCheckpointToDfs(checkpointDir: File, version: Long, numKeys: Long): Unit = {
+ logFilesInDir(checkpointDir, s"Saving checkpoint files for version $version")
+ val allCheckpointFiles = listRocksDBFiles(checkpointDir)
+ val (localImmutableFiles, localOtherFiles) = allCheckpointFiles.partition(isImmutableFile)
+ val rocksDBFiles = saveImmutableFilesToDfs(version, checkpointDir, localImmutableFiles)
+ val metadata = RocksDBCheckpointMetadata(rocksDBFiles, numKeys)
+ rememberImmutableFiles(version, rocksDBFiles)
+ val metadataFile = localMetadataFile(checkpointDir)
+ metadata.writeToFile(metadataFile)
+ logInfo(s"Written metadata for version $version:\n${metadata.prettyJson}")
+
+ if (version <= 0 && numKeys == 0) {
Review comment:
Yeah, let's change the version to 1 for avoiding the extra cost of file operations. Will change it in the next commit.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #32582: [WIP][SPARK-35436] RocksDBFileManager - save checkpoint to DFS
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #32582:
URL: https://github.com/apache/spark/pull/32582#issuecomment-843399497
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/138685/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32582: [WIP][SPARK-35436] RocksDBFileManager - save checkpoint to DFS
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32582:
URL: https://github.com/apache/spark/pull/32582#issuecomment-843370665
Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/43206/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] xuanyuanking commented on a change in pull request #32582: [SPARK-35436][SS] RocksDBFileManager - save checkpoint to DFS
Posted by GitBox <gi...@apache.org>.
xuanyuanking commented on a change in pull request #32582:
URL: https://github.com/apache/spark/pull/32582#discussion_r644694390
##########
File path: sql/core/src/test/scala/org/apache/spark/sql/execution/streaming/state/RocksDBSuite.scala
##########
@@ -20,12 +20,86 @@ package org.apache.spark.sql.execution.streaming.state
import java.io._
import java.nio.charset.Charset
+import scala.language.implicitConversions
+
import org.apache.commons.io.FileUtils
+import org.apache.hadoop.conf.Configuration
import org.apache.spark._
+import org.apache.spark.sql.catalyst.util.quietly
+import org.apache.spark.sql.execution.streaming.CreateAtomicTestManager
+import org.apache.spark.sql.internal.SQLConf
+import org.apache.spark.util.Utils
class RocksDBSuite extends SparkFunSuite {
+ test("RocksDBFileManager: upload only new immutable files") {
+ val dfsRootDir = Utils.createTempDir().getAbsolutePath
+ val fileManager = new RocksDBFileManager(
+ dfsRootDir, Utils.createTempDir(), new Configuration)
+ val sstDir = s"$dfsRootDir/SSTs"
+ def numRemoteSSTFiles: Int = listFiles(sstDir).length
+ val logDir = s"$dfsRootDir/logs"
+ def numRemoteLogFiles: Int = listFiles(logDir).length
Review comment:
Referenced this comment to https://github.com/apache/spark/pull/32767/files#diff-dc6f9dfe11e76f890ff2986f866853bcac263027c82562f9a52f4672a5460826R79
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #32582: [SPARK-35436][SS] RocksDBFileManager - save checkpoint to DFS
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #32582:
URL: https://github.com/apache/spark/pull/32582#issuecomment-857417824
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/139529/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HeartSaVioR commented on pull request #32582: [SPARK-35436][SS] RocksDBFileManager - save checkpoint to DFS
Posted by GitBox <gi...@apache.org>.
HeartSaVioR commented on pull request #32582:
URL: https://github.com/apache/spark/pull/32582#issuecomment-857378260
Thanks @xuanyuanking for your contribution! I merged this to master branch.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HeartSaVioR commented on pull request #32582: [SPARK-35436][SS] RocksDBFileManager - save checkpoint to DFS
Posted by GitBox <gi...@apache.org>.
HeartSaVioR commented on pull request #32582:
URL: https://github.com/apache/spark/pull/32582#issuecomment-850958908
And I see lots of information regarding files will be printed out via INFO level. I don't know whether the information would make the log file being flooded, but I guess we want to leave the information for the first phase on ease of debugging. We could discuss to lower down the level if we are getting bothered from the log size.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] viirya commented on pull request #32582: [SPARK-35436][SS] RocksDBFileManager - save checkpoint to DFS
Posted by GitBox <gi...@apache.org>.
viirya commented on pull request #32582:
URL: https://github.com/apache/spark/pull/32582#issuecomment-853540267
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] xuanyuanking commented on a change in pull request #32582: [SPARK-35436][SS] RocksDBFileManager - save checkpoint to DFS
Posted by GitBox <gi...@apache.org>.
xuanyuanking commented on a change in pull request #32582:
URL: https://github.com/apache/spark/pull/32582#discussion_r644694390
##########
File path: sql/core/src/test/scala/org/apache/spark/sql/execution/streaming/state/RocksDBSuite.scala
##########
@@ -20,12 +20,86 @@ package org.apache.spark.sql.execution.streaming.state
import java.io._
import java.nio.charset.Charset
+import scala.language.implicitConversions
+
import org.apache.commons.io.FileUtils
+import org.apache.hadoop.conf.Configuration
import org.apache.spark._
+import org.apache.spark.sql.catalyst.util.quietly
+import org.apache.spark.sql.execution.streaming.CreateAtomicTestManager
+import org.apache.spark.sql.internal.SQLConf
+import org.apache.spark.util.Utils
class RocksDBSuite extends SparkFunSuite {
+ test("RocksDBFileManager: upload only new immutable files") {
+ val dfsRootDir = Utils.createTempDir().getAbsolutePath
+ val fileManager = new RocksDBFileManager(
+ dfsRootDir, Utils.createTempDir(), new Configuration)
+ val sstDir = s"$dfsRootDir/SSTs"
+ def numRemoteSSTFiles: Int = listFiles(sstDir).length
+ val logDir = s"$dfsRootDir/logs"
+ def numRemoteLogFiles: Int = listFiles(logDir).length
Review comment:
Referenced this comment to https://github.com/apache/spark/pull/32767/files#diff-dc6f9dfe11e76f890ff2986f866853bcac263027c82562f9a52f4672a5460826R97
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] viirya commented on a change in pull request #32582: [SPARK-35436][SS] RocksDBFileManager - save checkpoint to DFS
Posted by GitBox <gi...@apache.org>.
viirya commented on a change in pull request #32582:
URL: https://github.com/apache/spark/pull/32582#discussion_r644473480
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/state/RocksDBFileManager.scala
##########
@@ -17,18 +17,265 @@
package org.apache.spark.sql.execution.streaming.state
-import java.io.File
+import java.io.{File, FileInputStream, InputStream}
import java.nio.charset.StandardCharsets.UTF_8
import java.nio.file.Files
+import java.util.UUID
+import java.util.concurrent.ConcurrentHashMap
+import java.util.zip.{ZipEntry, ZipOutputStream}
+import scala.collection.JavaConverters._
import scala.collection.Seq
import com.fasterxml.jackson.annotation.JsonInclude.Include
import com.fasterxml.jackson.databind.{DeserializationFeature, ObjectMapper}
import com.fasterxml.jackson.module.scala.{DefaultScalaModule, ScalaObjectMapper}
+import org.apache.commons.io.{FilenameUtils, IOUtils}
+import org.apache.hadoop.conf.Configuration
+import org.apache.hadoop.fs.Path
import org.json4s.NoTypeHints
import org.json4s.jackson.Serialization
+import org.apache.spark.internal.Logging
+import org.apache.spark.sql.execution.streaming.CheckpointFileManager
+import org.apache.spark.util.Utils
+
+/**
+ * Class responsible for syncing RocksDB checkpoint files from local disk to DFS.
+ * For each version, checkpoint is saved in specific directory structure that allows successive
+ * versions to reuse to SST data files and archived log files. This allows each commit to be
+ * incremental, only new SST files and archived log files generated by RocksDB will be uploaded.
+ * The directory structures on local disk and in DFS are as follows.
+ *
+ * Local checkpoint dir structure
+ * ------------------------------
+ * RocksDB generates a bunch of files in the local checkpoint directory. The most important among
+ * them are the SST files; they are the actual log structured data files. Rest of the files contain
+ * the metadata necessary for RocksDB to read the SST files and start from the checkpoint.
+ * Note that the SST files are hard links to files in the RocksDB's working directory, and therefore
+ * successive checkpoints can share some of the SST files. So these SST files have to be copied to
+ * DFS in shared directory such that different committed versions can save them.
+ *
+ * We consider both SST files and archived log files as immutable files which can be shared between
+ * different checkpoints.
+ *
+ * localCheckpointDir
+ * |
+ * +-- OPTIONS-000005
+ * +-- MANIFEST-000008
+ * +-- CURRENT
+ * +-- 00007.sst
+ * +-- 00011.sst
+ * +-- archive
+ * | +-- 00008.log
+ * | +-- 00013.log
+ * ...
+ *
+ *
+ * DFS directory structure after saving to DFS as version 10
+ * -----------------------------------------------------------
+ * The SST and archived log files are given unique file names and copied to the shared subdirectory.
+ * Every version maintains a mapping of local immutable file name to the unique file name in DFS.
+ * This mapping is saved in a JSON file (named `metadata`), which is zipped along with other
+ * checkpoint files into a single file `[version].zip`.
+ *
+ * dfsRootDir
+ * |
+ * +-- SSTs
+ * | +-- 00007-[uuid1].sst
+ * | +-- 00011-[uuid2].sst
+ * +-- logs
+ * | +-- 00008-[uuid3].log
+ * | +-- 00013-[uuid4].log
+ * +-- 10.zip
+ * | +-- metadata <--- contains mapping between 00007.sst and [uuid1].sst,
+ * and the mapping between 00008.log and [uuid3].log
+ * | +-- OPTIONS-000005
+ * | +-- MANIFEST-000008
+ * | +-- CURRENT
+ * | ...
+ * |
+ * +-- 9.zip
+ * +-- 8.zip
+ * ...
+ *
+ * Note the following.
+ * - Each [version].zip is a complete description of all the data and metadata needed to recover
+ * a RocksDB instance at the corresponding version. The SST files and log files are not included
+ * in the zip files, they can be shared cross different versions. This is unlike the
+ * [version].delta files of HDFSBackedStateStore where previous delta files needs to be read
+ * to be recovered.
+ * - This is safe wrt speculatively executed tasks running concurrently in different executors
+ * as each task would upload a different copy of the generated immutable files and
+ * atomically update the [version].zip.
+ * - Immutable files are identified uniquely based on their file name and file size.
+ * - Immutable files can be reused only across adjacent checkpoints/versions.
+ * - This class is thread-safe. Specifically, it is safe to concurrently delete old files from a
+ * different thread than the task thread saving files.
Review comment:
Can you talk more about there this claim (thread-safe) applies? Where we delete old files?
Do you mean `versionToRocksDBFiles`? When we prepare files for new version, there is another thread (maintenance thread) deleting old files?
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/state/RocksDBFileManager.scala
##########
@@ -17,18 +17,265 @@
package org.apache.spark.sql.execution.streaming.state
-import java.io.File
+import java.io.{File, FileInputStream, InputStream}
import java.nio.charset.StandardCharsets.UTF_8
import java.nio.file.Files
+import java.util.UUID
+import java.util.concurrent.ConcurrentHashMap
+import java.util.zip.{ZipEntry, ZipOutputStream}
+import scala.collection.JavaConverters._
import scala.collection.Seq
import com.fasterxml.jackson.annotation.JsonInclude.Include
import com.fasterxml.jackson.databind.{DeserializationFeature, ObjectMapper}
import com.fasterxml.jackson.module.scala.{DefaultScalaModule, ScalaObjectMapper}
+import org.apache.commons.io.{FilenameUtils, IOUtils}
+import org.apache.hadoop.conf.Configuration
+import org.apache.hadoop.fs.Path
import org.json4s.NoTypeHints
import org.json4s.jackson.Serialization
+import org.apache.spark.internal.Logging
+import org.apache.spark.sql.execution.streaming.CheckpointFileManager
+import org.apache.spark.util.Utils
+
+/**
+ * Class responsible for syncing RocksDB checkpoint files from local disk to DFS.
+ * For each version, checkpoint is saved in specific directory structure that allows successive
+ * versions to reuse to SST data files and archived log files. This allows each commit to be
+ * incremental, only new SST files and archived log files generated by RocksDB will be uploaded.
+ * The directory structures on local disk and in DFS are as follows.
+ *
+ * Local checkpoint dir structure
+ * ------------------------------
+ * RocksDB generates a bunch of files in the local checkpoint directory. The most important among
+ * them are the SST files; they are the actual log structured data files. Rest of the files contain
+ * the metadata necessary for RocksDB to read the SST files and start from the checkpoint.
+ * Note that the SST files are hard links to files in the RocksDB's working directory, and therefore
+ * successive checkpoints can share some of the SST files. So these SST files have to be copied to
+ * DFS in shared directory such that different committed versions can save them.
+ *
+ * We consider both SST files and archived log files as immutable files which can be shared between
+ * different checkpoints.
+ *
+ * localCheckpointDir
+ * |
+ * +-- OPTIONS-000005
+ * +-- MANIFEST-000008
+ * +-- CURRENT
+ * +-- 00007.sst
+ * +-- 00011.sst
+ * +-- archive
+ * | +-- 00008.log
+ * | +-- 00013.log
+ * ...
+ *
+ *
+ * DFS directory structure after saving to DFS as version 10
+ * -----------------------------------------------------------
+ * The SST and archived log files are given unique file names and copied to the shared subdirectory.
+ * Every version maintains a mapping of local immutable file name to the unique file name in DFS.
+ * This mapping is saved in a JSON file (named `metadata`), which is zipped along with other
+ * checkpoint files into a single file `[version].zip`.
+ *
+ * dfsRootDir
+ * |
+ * +-- SSTs
+ * | +-- 00007-[uuid1].sst
+ * | +-- 00011-[uuid2].sst
+ * +-- logs
+ * | +-- 00008-[uuid3].log
+ * | +-- 00013-[uuid4].log
+ * +-- 10.zip
+ * | +-- metadata <--- contains mapping between 00007.sst and [uuid1].sst,
+ * and the mapping between 00008.log and [uuid3].log
+ * | +-- OPTIONS-000005
+ * | +-- MANIFEST-000008
+ * | +-- CURRENT
+ * | ...
+ * |
+ * +-- 9.zip
+ * +-- 8.zip
+ * ...
+ *
+ * Note the following.
+ * - Each [version].zip is a complete description of all the data and metadata needed to recover
+ * a RocksDB instance at the corresponding version. The SST files and log files are not included
+ * in the zip files, they can be shared cross different versions. This is unlike the
+ * [version].delta files of HDFSBackedStateStore where previous delta files needs to be read
+ * to be recovered.
+ * - This is safe wrt speculatively executed tasks running concurrently in different executors
+ * as each task would upload a different copy of the generated immutable files and
+ * atomically update the [version].zip.
+ * - Immutable files are identified uniquely based on their file name and file size.
+ * - Immutable files can be reused only across adjacent checkpoints/versions.
+ * - This class is thread-safe. Specifically, it is safe to concurrently delete old files from a
+ * different thread than the task thread saving files.
+ *
+ * @param dfsRootDir Directory where the [version].zip files will be stored
+ * @param localTempDir Local directory for temporary work
+ * @param hadoopConf Hadoop configuration for talking to DFS
+ * @param loggingId Id that will be prepended in logs for isolating concurrent RocksDBs
+ */
+class RocksDBFileManager(
+ dfsRootDir: String,
+ localTempDir: File,
+ hadoopConf: Configuration,
+ loggingId: String = "")
+ extends Logging {
+
+ import RocksDBImmutableFile._
+
+ private val versionToRocksDBFiles = new ConcurrentHashMap[Long, Seq[RocksDBImmutableFile]]
+ private lazy val fm = CheckpointFileManager.create(new Path(dfsRootDir), hadoopConf)
+ private val fs = new Path(dfsRootDir).getFileSystem(hadoopConf)
+
+ /** Save all the files in given local checkpoint directory as a committed version in DFS */
+ def saveCheckpointToDfs(checkpointDir: File, version: Long, numKeys: Long): Unit = {
+ logFilesInDir(checkpointDir, s"Saving checkpoint files for version $version")
+ val (localImmutableFiles, localOtherFiles) = listRocksDBFiles(checkpointDir)
+ val rocksDBFiles = saveImmutableFilesToDfs(version, localImmutableFiles)
+ val metadata = RocksDBCheckpointMetadata(rocksDBFiles, numKeys)
+ val metadataFile = localMetadataFile(checkpointDir)
+ metadata.writeToFile(metadataFile)
+ logInfo(s"Written metadata for version $version:\n${metadata.prettyJson}")
+
+ if (version <= 0 && numKeys == 0) {
+ // If we're writing the initial version and there's no data, we have to explicitly initialize
+ // the root directory. Normally saveImmutableFilesToDfs will do this initialization, but
+ // when there's no data that method won't write any files, and zipToDfsFile uses the
+ // CheckpointFileManager.createAtomic API which doesn't auto-initialize parent directories.
+ fm.mkdirs(new Path(dfsRootDir))
Review comment:
Should we check if the dfs root dir already exists?
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/state/RocksDBFileManager.scala
##########
@@ -17,18 +17,265 @@
package org.apache.spark.sql.execution.streaming.state
-import java.io.File
+import java.io.{File, FileInputStream, InputStream}
import java.nio.charset.StandardCharsets.UTF_8
import java.nio.file.Files
+import java.util.UUID
+import java.util.concurrent.ConcurrentHashMap
+import java.util.zip.{ZipEntry, ZipOutputStream}
+import scala.collection.JavaConverters._
import scala.collection.Seq
import com.fasterxml.jackson.annotation.JsonInclude.Include
import com.fasterxml.jackson.databind.{DeserializationFeature, ObjectMapper}
import com.fasterxml.jackson.module.scala.{DefaultScalaModule, ScalaObjectMapper}
+import org.apache.commons.io.{FilenameUtils, IOUtils}
+import org.apache.hadoop.conf.Configuration
+import org.apache.hadoop.fs.Path
import org.json4s.NoTypeHints
import org.json4s.jackson.Serialization
+import org.apache.spark.internal.Logging
+import org.apache.spark.sql.execution.streaming.CheckpointFileManager
+import org.apache.spark.util.Utils
+
+/**
+ * Class responsible for syncing RocksDB checkpoint files from local disk to DFS.
+ * For each version, checkpoint is saved in specific directory structure that allows successive
+ * versions to reuse to SST data files and archived log files. This allows each commit to be
+ * incremental, only new SST files and archived log files generated by RocksDB will be uploaded.
+ * The directory structures on local disk and in DFS are as follows.
+ *
+ * Local checkpoint dir structure
+ * ------------------------------
+ * RocksDB generates a bunch of files in the local checkpoint directory. The most important among
+ * them are the SST files; they are the actual log structured data files. Rest of the files contain
+ * the metadata necessary for RocksDB to read the SST files and start from the checkpoint.
+ * Note that the SST files are hard links to files in the RocksDB's working directory, and therefore
+ * successive checkpoints can share some of the SST files. So these SST files have to be copied to
+ * DFS in shared directory such that different committed versions can save them.
+ *
+ * We consider both SST files and archived log files as immutable files which can be shared between
+ * different checkpoints.
+ *
+ * localCheckpointDir
+ * |
+ * +-- OPTIONS-000005
+ * +-- MANIFEST-000008
+ * +-- CURRENT
+ * +-- 00007.sst
+ * +-- 00011.sst
+ * +-- archive
+ * | +-- 00008.log
+ * | +-- 00013.log
+ * ...
+ *
+ *
+ * DFS directory structure after saving to DFS as version 10
+ * -----------------------------------------------------------
+ * The SST and archived log files are given unique file names and copied to the shared subdirectory.
+ * Every version maintains a mapping of local immutable file name to the unique file name in DFS.
+ * This mapping is saved in a JSON file (named `metadata`), which is zipped along with other
+ * checkpoint files into a single file `[version].zip`.
+ *
+ * dfsRootDir
+ * |
+ * +-- SSTs
+ * | +-- 00007-[uuid1].sst
+ * | +-- 00011-[uuid2].sst
+ * +-- logs
+ * | +-- 00008-[uuid3].log
+ * | +-- 00013-[uuid4].log
+ * +-- 10.zip
+ * | +-- metadata <--- contains mapping between 00007.sst and [uuid1].sst,
+ * and the mapping between 00008.log and [uuid3].log
+ * | +-- OPTIONS-000005
+ * | +-- MANIFEST-000008
+ * | +-- CURRENT
+ * | ...
+ * |
+ * +-- 9.zip
+ * +-- 8.zip
+ * ...
+ *
+ * Note the following.
+ * - Each [version].zip is a complete description of all the data and metadata needed to recover
+ * a RocksDB instance at the corresponding version. The SST files and log files are not included
+ * in the zip files, they can be shared cross different versions. This is unlike the
+ * [version].delta files of HDFSBackedStateStore where previous delta files needs to be read
+ * to be recovered.
+ * - This is safe wrt speculatively executed tasks running concurrently in different executors
+ * as each task would upload a different copy of the generated immutable files and
+ * atomically update the [version].zip.
+ * - Immutable files are identified uniquely based on their file name and file size.
+ * - Immutable files can be reused only across adjacent checkpoints/versions.
+ * - This class is thread-safe. Specifically, it is safe to concurrently delete old files from a
+ * different thread than the task thread saving files.
+ *
+ * @param dfsRootDir Directory where the [version].zip files will be stored
+ * @param localTempDir Local directory for temporary work
+ * @param hadoopConf Hadoop configuration for talking to DFS
+ * @param loggingId Id that will be prepended in logs for isolating concurrent RocksDBs
+ */
+class RocksDBFileManager(
+ dfsRootDir: String,
+ localTempDir: File,
+ hadoopConf: Configuration,
+ loggingId: String = "")
+ extends Logging {
+
+ import RocksDBImmutableFile._
+
+ private val versionToRocksDBFiles = new ConcurrentHashMap[Long, Seq[RocksDBImmutableFile]]
+ private lazy val fm = CheckpointFileManager.create(new Path(dfsRootDir), hadoopConf)
+ private val fs = new Path(dfsRootDir).getFileSystem(hadoopConf)
+
+ /** Save all the files in given local checkpoint directory as a committed version in DFS */
+ def saveCheckpointToDfs(checkpointDir: File, version: Long, numKeys: Long): Unit = {
+ logFilesInDir(checkpointDir, s"Saving checkpoint files for version $version")
+ val (localImmutableFiles, localOtherFiles) = listRocksDBFiles(checkpointDir)
+ val rocksDBFiles = saveImmutableFilesToDfs(version, localImmutableFiles)
+ val metadata = RocksDBCheckpointMetadata(rocksDBFiles, numKeys)
+ val metadataFile = localMetadataFile(checkpointDir)
+ metadata.writeToFile(metadataFile)
+ logInfo(s"Written metadata for version $version:\n${metadata.prettyJson}")
+
+ if (version <= 0 && numKeys == 0) {
+ // If we're writing the initial version and there's no data, we have to explicitly initialize
+ // the root directory. Normally saveImmutableFilesToDfs will do this initialization, but
+ // when there's no data that method won't write any files, and zipToDfsFile uses the
+ // CheckpointFileManager.createAtomic API which doesn't auto-initialize parent directories.
+ fm.mkdirs(new Path(dfsRootDir))
+ }
+ zipToDfsFile(localOtherFiles :+ metadataFile, dfsBatchZipFile(version))
+ logInfo(s"Saved checkpoint file for version $version")
+ }
+
+ /** Save immutable files to DFS directory */
+ private def saveImmutableFilesToDfs(
+ version: Long,
+ localFiles: Seq[File]): Seq[RocksDBImmutableFile] = {
+ // Get the immutable files used in previous versions, as some of those uploaded files can be
+ // reused for this version
+ logInfo(s"Saving RocksDB files to DFS for $version")
+ val prevFilesToSizes = versionToRocksDBFiles.values.asScala.flatten.map { f =>
+ f.localFileName -> f
+ }.toMap
+
+ var bytesCopied = 0L
+ var filesCopied = 0L
+ var filesReused = 0L
+
+ val immutableFiles = localFiles.map { localFile =>
+ prevFilesToSizes
+ .get(localFile.getName)
+ .filter(_.isSameFile(localFile))
+ .map { reusable =>
+ filesReused += 1
+ reusable
+ }.getOrElse {
+ val localFileName = localFile.getName
+ val dfsFileName = newDFSFileName(localFileName)
+ val dfsFile = dfsFilePath(dfsFileName)
+ // Note: The implementation of copyFromLocalFile() closes the output stream when there is
+ // any exception while copying. So this may generate partial files on DFS. But that is
+ // okay because until the main [version].zip file is written, those partial files are
+ // not going to be used at all. Eventually these files should get cleared.
+ fs.copyFromLocalFile(
+ new Path(localFile.getAbsoluteFile.toURI), dfsFile)
+ val localFileSize = localFile.length()
+ logInfo(s"Copied $localFile to $dfsFile - $localFileSize bytes")
+ filesCopied += 1
+ bytesCopied += localFileSize
+
+ RocksDBImmutableFile(localFile.getName, dfsFileName, localFileSize)
+ }
+ }
+ logInfo(s"Copied $filesCopied files ($bytesCopied bytes) from local to" +
+ s" DFS for version $version. $filesReused files reused without copying.")
+ versionToRocksDBFiles.put(version, immutableFiles)
+
+ immutableFiles
+ }
+
+ /**
+ * Compress files to a single zip file in DFS. Only the file names are embedded in the zip.
+ * Any error while writing will ensure that the file is not written.
+ */
+ private def zipToDfsFile(files: Seq[File], dfsZipFile: Path): Unit = {
+ lazy val filesStr = s"$dfsZipFile\n\t${files.mkString("\n\t")}"
+ var in: InputStream = null
+ val out = fm.createAtomic(dfsZipFile, overwriteIfPossible = true)
+ var totalBytes = 0L
+ val zout = new ZipOutputStream(out)
+ try {
+ files.foreach { file =>
+ zout.putNextEntry(new ZipEntry(file.getName))
+ in = new FileInputStream(file)
+ val bytes = IOUtils.copy(in, zout)
+ in.close()
+ zout.closeEntry()
+ totalBytes += bytes
+ }
+ zout.close() // so that any error in closing also cancels the output stream
+ logInfo(s"Zipped $totalBytes bytes (before compression) to $filesStr")
Review comment:
The zipped file is `dfsZipFile`, right? why `filesStr` here?
##########
File path: sql/core/src/test/scala/org/apache/spark/sql/execution/streaming/state/RocksDBSuite.scala
##########
@@ -20,12 +20,86 @@ package org.apache.spark.sql.execution.streaming.state
import java.io._
import java.nio.charset.Charset
+import scala.language.implicitConversions
+
import org.apache.commons.io.FileUtils
+import org.apache.hadoop.conf.Configuration
import org.apache.spark._
+import org.apache.spark.sql.catalyst.util.quietly
+import org.apache.spark.sql.execution.streaming.CreateAtomicTestManager
+import org.apache.spark.sql.internal.SQLConf
+import org.apache.spark.util.Utils
class RocksDBSuite extends SparkFunSuite {
+ test("RocksDBFileManager: upload only new immutable files") {
+ val dfsRootDir = Utils.createTempDir().getAbsolutePath
Review comment:
Wrap using `withTempDir`?
##########
File path: core/src/main/scala/org/apache/spark/util/Utils.scala
##########
@@ -1129,6 +1129,15 @@ private[spark] object Utils extends Logging {
s"${TimeUnit.NANOSECONDS.toMillis(System.nanoTime() - startTimeNs)} ms"
}
+ /**
+ * Lists files recursively.
+ */
+ def recursiveList(f: File): Array[File] = {
+ require(f.isDirectory)
+ val current = f.listFiles
+ current ++ current.filter(_.isDirectory).flatMap(recursiveList)
Review comment:
nit: I think this can be done easily using non-recursive approach?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] viirya commented on pull request #32582: [SPARK-35436][SS] RocksDBFileManager - save checkpoint to DFS
Posted by GitBox <gi...@apache.org>.
viirya commented on pull request #32582:
URL: https://github.com/apache/spark/pull/32582#issuecomment-853540267
Sorry for late. I will take look this soon.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] xuanyuanking commented on a change in pull request #32582: [SPARK-35436][SS] RocksDBFileManager - save checkpoint to DFS
Posted by GitBox <gi...@apache.org>.
xuanyuanking commented on a change in pull request #32582:
URL: https://github.com/apache/spark/pull/32582#discussion_r643852798
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/state/RocksDBFileManager.scala
##########
@@ -17,18 +17,270 @@
package org.apache.spark.sql.execution.streaming.state
-import java.io.File
+import java.io.{File, FileInputStream, InputStream}
import java.nio.charset.StandardCharsets.UTF_8
import java.nio.file.Files
+import java.util.UUID
+import java.util.concurrent.ConcurrentHashMap
+import java.util.zip.{ZipEntry, ZipOutputStream}
+import scala.collection.JavaConverters._
import scala.collection.Seq
import com.fasterxml.jackson.annotation.JsonInclude.Include
import com.fasterxml.jackson.databind.{DeserializationFeature, ObjectMapper}
import com.fasterxml.jackson.module.scala.{DefaultScalaModule, ScalaObjectMapper}
+import org.apache.commons.io.{FilenameUtils, IOUtils}
+import org.apache.hadoop.conf.Configuration
+import org.apache.hadoop.fs.Path
import org.json4s.NoTypeHints
import org.json4s.jackson.Serialization
+import org.apache.spark.internal.Logging
+import org.apache.spark.sql.execution.streaming.CheckpointFileManager
+import org.apache.spark.util.Utils
+
+/**
+ * Class responsible for syncing RocksDB checkpoint files from local disk to DFS.
+ * Every version checkpoint is saved in specific directory structure that allows successive
+ * versions to reuse to SST data files and archived log files. This allows each commit to be
+ * incremental, only new SST files and archived log files generated by RocksDB will be uploaded.
+ * The directory structures on local disk and in DFS are as follows.
+ *
+ * Local checkpoint dir structure
+ * ------------------------------
+ * RocksDB generates a bunch of files in the local checkpoint directory. The most important among
+ * them are the SST files; they are the actual log structured data files. Rest of the files contain
+ * the metadata necessary for RocksDB to read the SST files and start from the checkpoint.
+ * Note that the SST files are hard links to files in the RocksDB's working directory, and therefore
+ * successive checkpoints can share some of the SST files. So these SST files have to be copied to
+ * DFS in shared directory such that different committed versions can save them.
+ *
+ * We consider both SST files and archived log files as immutable files which can be shared between
+ * different checkpoints.
+ *
+ * localCheckpointDir
+ * |
+ * +-- OPTIONS-000005
+ * +-- MANIFEST-000008
+ * +-- CURRENT
+ * +-- 00007.sst
+ * +-- 00011.sst
+ * +-- archive
+ * | +-- 00008.log
+ * | +-- 00013.log
+ * ...
+ *
+ *
+ * DFS directory structure after saving to DFS as version 10
+ * -----------------------------------------------------------
+ * The SST and archived log files are given unique file names and copied to the shared subdirectory.
+ * Every version maintains a mapping of local immutable file name to the unique file name in DFS.
+ * This mapping is saved in a JSON file (named `metadata`), which is zipped along with other
+ * checkpoint files into a single file `[version].zip`.
+ *
+ * dfsRootDir
+ * |
+ * +-- SSTs
+ * | +-- 00007-[uuid1].sst
+ * | +-- 00011-[uuid2].sst
+ * +-- logs
+ * | +-- 00008-[uuid3].log
+ * | +-- 00013-[uuid4].log
+ * +-- 10.zip
+ * | +-- metadata <--- contains mapping between 00007.sst and [uuid1].sst,
+ * and the mapping between 00008.log and [uuid3].log
+ * | +-- OPTIONS-000005
+ * | +-- MANIFEST-000008
+ * | +-- CURRENT
+ * | ...
+ * |
+ * +-- 9.zip
+ * +-- 8.zip
+ * ...
+ *
+ * Note the following.
+ * - Each [version].zip is a complete description of all the data and metadata needed to recover
+ * a RocksDB instance at the corresponding version. This is unlike the [version].delta files of
+ * HDFSBackedStateStore where previous delta files needs to be read to be recovered.
+ * - This is safe wrt speculatively executed tasks running concurrently in different executors
+ * as each task would upload a different copy of the generated immutable files and
+ * atomically update the [version].zip.
+ * - Immutable files are identified uniquely based on their file name and file size.
+ * - Immutable files can be reused only across adjacent checkpoints/versions.
+ * - This class is thread-safe. Specifically, it is safe to concurrently delete old files from a
+ * different thread than the task thread saving files.
+ *
+ * @param dfsRootDir Directory where the [version].zip files will be stored
+ * @param localTempDir Local directory for temporary work
+ * @param hadoopConf Hadoop configuration for talking to DFS
+ * @param loggingId Id that will be prepended in logs for isolating concurrent RocksDBs
+ */
+class RocksDBFileManager(
+ dfsRootDir: String,
+ localTempDir: File,
+ hadoopConf: Configuration,
+ loggingId: String = "")
+ extends Logging {
+
+ import RocksDBImmutableFile._
+
+ private val versionToRocksDBFiles = new ConcurrentHashMap[Long, Seq[RocksDBImmutableFile]]
+ private lazy val fm = CheckpointFileManager.create(new Path(dfsRootDir), hadoopConf)
+ private val fs = new Path(dfsRootDir).getFileSystem(hadoopConf)
+
+ /** Save all the files in given local checkpoint directory as a committed version in DFS */
+ def saveCheckpointToDfs(checkpointDir: File, version: Long, numKeys: Long): Unit = {
+ logFilesInDir(checkpointDir, s"Saving checkpoint files for version $version")
+ val allCheckpointFiles = listRocksDBFiles(checkpointDir)
+ val (localImmutableFiles, localOtherFiles) = allCheckpointFiles.partition(isImmutableFile)
+ val rocksDBFiles = saveImmutableFilesToDfs(version, checkpointDir, localImmutableFiles)
+ val metadata = RocksDBCheckpointMetadata(rocksDBFiles, numKeys)
+ rememberImmutableFiles(version, rocksDBFiles)
Review comment:
Agree, deleted the `rememberImmutableFiles` and made the corresponding changes in 3800c51
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32582: [SPARK-35436][SS] RocksDBFileManager - save checkpoint to DFS
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32582:
URL: https://github.com/apache/spark/pull/32582#issuecomment-857383789
Kubernetes integration test unable to build dist.
exiting with code: 1
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/44054/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32582: [WIP][SPARK-35436] RocksDBFileManager - save checkpoint to DFS
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32582:
URL: https://github.com/apache/spark/pull/32582#issuecomment-843397521
**[Test build #138685 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/138685/testReport)** for PR 32582 at commit [`5e6bf13`](https://github.com/apache/spark/commit/5e6bf130ae1ae104a48020cd8e08c07b3968c4c0).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds the following public classes _(experimental)_:
* `class RocksDBFileManager(`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #32582: [SPARK-35436][SS] RocksDBFileManager - save checkpoint to DFS
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #32582:
URL: https://github.com/apache/spark/pull/32582#issuecomment-850087350
**[Test build #139041 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/139041/testReport)** for PR 32582 at commit [`703f59f`](https://github.com/apache/spark/commit/703f59fd4055d68e5bf957e1dc4f17159256a65a).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #32582: [WIP][SPARK-35436] RocksDBFileManager - save checkpoint to DFS
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #32582:
URL: https://github.com/apache/spark/pull/32582#issuecomment-843398502
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/43206/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] xuanyuanking commented on a change in pull request #32582: [SPARK-35436][SS] RocksDBFileManager - save checkpoint to DFS
Posted by GitBox <gi...@apache.org>.
xuanyuanking commented on a change in pull request #32582:
URL: https://github.com/apache/spark/pull/32582#discussion_r644675514
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/state/RocksDBFileManager.scala
##########
@@ -17,18 +17,265 @@
package org.apache.spark.sql.execution.streaming.state
-import java.io.File
+import java.io.{File, FileInputStream, InputStream}
import java.nio.charset.StandardCharsets.UTF_8
import java.nio.file.Files
+import java.util.UUID
+import java.util.concurrent.ConcurrentHashMap
+import java.util.zip.{ZipEntry, ZipOutputStream}
+import scala.collection.JavaConverters._
import scala.collection.Seq
import com.fasterxml.jackson.annotation.JsonInclude.Include
import com.fasterxml.jackson.databind.{DeserializationFeature, ObjectMapper}
import com.fasterxml.jackson.module.scala.{DefaultScalaModule, ScalaObjectMapper}
+import org.apache.commons.io.{FilenameUtils, IOUtils}
+import org.apache.hadoop.conf.Configuration
+import org.apache.hadoop.fs.Path
import org.json4s.NoTypeHints
import org.json4s.jackson.Serialization
+import org.apache.spark.internal.Logging
+import org.apache.spark.sql.execution.streaming.CheckpointFileManager
+import org.apache.spark.util.Utils
+
+/**
+ * Class responsible for syncing RocksDB checkpoint files from local disk to DFS.
+ * For each version, checkpoint is saved in specific directory structure that allows successive
+ * versions to reuse to SST data files and archived log files. This allows each commit to be
+ * incremental, only new SST files and archived log files generated by RocksDB will be uploaded.
+ * The directory structures on local disk and in DFS are as follows.
+ *
+ * Local checkpoint dir structure
+ * ------------------------------
+ * RocksDB generates a bunch of files in the local checkpoint directory. The most important among
+ * them are the SST files; they are the actual log structured data files. Rest of the files contain
+ * the metadata necessary for RocksDB to read the SST files and start from the checkpoint.
+ * Note that the SST files are hard links to files in the RocksDB's working directory, and therefore
+ * successive checkpoints can share some of the SST files. So these SST files have to be copied to
+ * DFS in shared directory such that different committed versions can save them.
+ *
+ * We consider both SST files and archived log files as immutable files which can be shared between
+ * different checkpoints.
+ *
+ * localCheckpointDir
+ * |
+ * +-- OPTIONS-000005
+ * +-- MANIFEST-000008
+ * +-- CURRENT
+ * +-- 00007.sst
+ * +-- 00011.sst
+ * +-- archive
+ * | +-- 00008.log
+ * | +-- 00013.log
+ * ...
+ *
+ *
+ * DFS directory structure after saving to DFS as version 10
+ * -----------------------------------------------------------
+ * The SST and archived log files are given unique file names and copied to the shared subdirectory.
+ * Every version maintains a mapping of local immutable file name to the unique file name in DFS.
+ * This mapping is saved in a JSON file (named `metadata`), which is zipped along with other
+ * checkpoint files into a single file `[version].zip`.
+ *
+ * dfsRootDir
+ * |
+ * +-- SSTs
+ * | +-- 00007-[uuid1].sst
+ * | +-- 00011-[uuid2].sst
+ * +-- logs
+ * | +-- 00008-[uuid3].log
+ * | +-- 00013-[uuid4].log
+ * +-- 10.zip
+ * | +-- metadata <--- contains mapping between 00007.sst and [uuid1].sst,
+ * and the mapping between 00008.log and [uuid3].log
+ * | +-- OPTIONS-000005
+ * | +-- MANIFEST-000008
+ * | +-- CURRENT
+ * | ...
+ * |
+ * +-- 9.zip
+ * +-- 8.zip
+ * ...
+ *
+ * Note the following.
+ * - Each [version].zip is a complete description of all the data and metadata needed to recover
+ * a RocksDB instance at the corresponding version. The SST files and log files are not included
+ * in the zip files, they can be shared cross different versions. This is unlike the
+ * [version].delta files of HDFSBackedStateStore where previous delta files needs to be read
+ * to be recovered.
+ * - This is safe wrt speculatively executed tasks running concurrently in different executors
+ * as each task would upload a different copy of the generated immutable files and
+ * atomically update the [version].zip.
+ * - Immutable files are identified uniquely based on their file name and file size.
+ * - Immutable files can be reused only across adjacent checkpoints/versions.
+ * - This class is thread-safe. Specifically, it is safe to concurrently delete old files from a
+ * different thread than the task thread saving files.
Review comment:
Yes. It refers to delete the files contained in the old versions. Here's the description of the deleteOldVersions method of RocksDBFileManager, which will be called in RocksDBStateStoreProvider.doMaintenance. As we did before, I'll also refer this comment when the PR for delete path submitted.
```
* Delete old versions by deleting the associated version and SST files.
* At a high-level, this method finds which versions to delete, and which SST files that were
* last used in those versions. Its safe to delete these SST files because a SST file can
* be reused only in successive versions. Therefore, if a SST file F was last used in version
* V, then it wont be used in version V+1 or later, and if version V can be deleted, then
* F can safely be deleted as well.
```
##########
File path: core/src/main/scala/org/apache/spark/util/Utils.scala
##########
@@ -1129,6 +1129,15 @@ private[spark] object Utils extends Logging {
s"${TimeUnit.NANOSECONDS.toMillis(System.nanoTime() - startTimeNs)} ms"
}
+ /**
+ * Lists files recursively.
+ */
+ def recursiveList(f: File): Array[File] = {
+ require(f.isDirectory)
+ val current = f.listFiles
+ current ++ current.filter(_.isDirectory).flatMap(recursiveList)
Review comment:
Sure, done in f5a6104
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/state/RocksDBFileManager.scala
##########
@@ -17,18 +17,265 @@
package org.apache.spark.sql.execution.streaming.state
-import java.io.File
+import java.io.{File, FileInputStream, InputStream}
import java.nio.charset.StandardCharsets.UTF_8
import java.nio.file.Files
+import java.util.UUID
+import java.util.concurrent.ConcurrentHashMap
+import java.util.zip.{ZipEntry, ZipOutputStream}
+import scala.collection.JavaConverters._
import scala.collection.Seq
import com.fasterxml.jackson.annotation.JsonInclude.Include
import com.fasterxml.jackson.databind.{DeserializationFeature, ObjectMapper}
import com.fasterxml.jackson.module.scala.{DefaultScalaModule, ScalaObjectMapper}
+import org.apache.commons.io.{FilenameUtils, IOUtils}
+import org.apache.hadoop.conf.Configuration
+import org.apache.hadoop.fs.Path
import org.json4s.NoTypeHints
import org.json4s.jackson.Serialization
+import org.apache.spark.internal.Logging
+import org.apache.spark.sql.execution.streaming.CheckpointFileManager
+import org.apache.spark.util.Utils
+
+/**
+ * Class responsible for syncing RocksDB checkpoint files from local disk to DFS.
+ * For each version, checkpoint is saved in specific directory structure that allows successive
+ * versions to reuse to SST data files and archived log files. This allows each commit to be
+ * incremental, only new SST files and archived log files generated by RocksDB will be uploaded.
+ * The directory structures on local disk and in DFS are as follows.
+ *
+ * Local checkpoint dir structure
+ * ------------------------------
+ * RocksDB generates a bunch of files in the local checkpoint directory. The most important among
+ * them are the SST files; they are the actual log structured data files. Rest of the files contain
+ * the metadata necessary for RocksDB to read the SST files and start from the checkpoint.
+ * Note that the SST files are hard links to files in the RocksDB's working directory, and therefore
+ * successive checkpoints can share some of the SST files. So these SST files have to be copied to
+ * DFS in shared directory such that different committed versions can save them.
+ *
+ * We consider both SST files and archived log files as immutable files which can be shared between
+ * different checkpoints.
+ *
+ * localCheckpointDir
+ * |
+ * +-- OPTIONS-000005
+ * +-- MANIFEST-000008
+ * +-- CURRENT
+ * +-- 00007.sst
+ * +-- 00011.sst
+ * +-- archive
+ * | +-- 00008.log
+ * | +-- 00013.log
+ * ...
+ *
+ *
+ * DFS directory structure after saving to DFS as version 10
+ * -----------------------------------------------------------
+ * The SST and archived log files are given unique file names and copied to the shared subdirectory.
+ * Every version maintains a mapping of local immutable file name to the unique file name in DFS.
+ * This mapping is saved in a JSON file (named `metadata`), which is zipped along with other
+ * checkpoint files into a single file `[version].zip`.
+ *
+ * dfsRootDir
+ * |
+ * +-- SSTs
+ * | +-- 00007-[uuid1].sst
+ * | +-- 00011-[uuid2].sst
+ * +-- logs
+ * | +-- 00008-[uuid3].log
+ * | +-- 00013-[uuid4].log
+ * +-- 10.zip
+ * | +-- metadata <--- contains mapping between 00007.sst and [uuid1].sst,
+ * and the mapping between 00008.log and [uuid3].log
+ * | +-- OPTIONS-000005
+ * | +-- MANIFEST-000008
+ * | +-- CURRENT
+ * | ...
+ * |
+ * +-- 9.zip
+ * +-- 8.zip
+ * ...
+ *
+ * Note the following.
+ * - Each [version].zip is a complete description of all the data and metadata needed to recover
+ * a RocksDB instance at the corresponding version. The SST files and log files are not included
+ * in the zip files, they can be shared cross different versions. This is unlike the
+ * [version].delta files of HDFSBackedStateStore where previous delta files needs to be read
+ * to be recovered.
+ * - This is safe wrt speculatively executed tasks running concurrently in different executors
+ * as each task would upload a different copy of the generated immutable files and
+ * atomically update the [version].zip.
+ * - Immutable files are identified uniquely based on their file name and file size.
+ * - Immutable files can be reused only across adjacent checkpoints/versions.
+ * - This class is thread-safe. Specifically, it is safe to concurrently delete old files from a
+ * different thread than the task thread saving files.
+ *
+ * @param dfsRootDir Directory where the [version].zip files will be stored
+ * @param localTempDir Local directory for temporary work
+ * @param hadoopConf Hadoop configuration for talking to DFS
+ * @param loggingId Id that will be prepended in logs for isolating concurrent RocksDBs
+ */
+class RocksDBFileManager(
+ dfsRootDir: String,
+ localTempDir: File,
+ hadoopConf: Configuration,
+ loggingId: String = "")
+ extends Logging {
+
+ import RocksDBImmutableFile._
+
+ private val versionToRocksDBFiles = new ConcurrentHashMap[Long, Seq[RocksDBImmutableFile]]
+ private lazy val fm = CheckpointFileManager.create(new Path(dfsRootDir), hadoopConf)
+ private val fs = new Path(dfsRootDir).getFileSystem(hadoopConf)
+
+ /** Save all the files in given local checkpoint directory as a committed version in DFS */
+ def saveCheckpointToDfs(checkpointDir: File, version: Long, numKeys: Long): Unit = {
+ logFilesInDir(checkpointDir, s"Saving checkpoint files for version $version")
+ val (localImmutableFiles, localOtherFiles) = listRocksDBFiles(checkpointDir)
+ val rocksDBFiles = saveImmutableFilesToDfs(version, localImmutableFiles)
+ val metadata = RocksDBCheckpointMetadata(rocksDBFiles, numKeys)
+ val metadataFile = localMetadataFile(checkpointDir)
+ metadata.writeToFile(metadataFile)
+ logInfo(s"Written metadata for version $version:\n${metadata.prettyJson}")
+
+ if (version <= 0 && numKeys == 0) {
+ // If we're writing the initial version and there's no data, we have to explicitly initialize
+ // the root directory. Normally saveImmutableFilesToDfs will do this initialization, but
+ // when there's no data that method won't write any files, and zipToDfsFile uses the
+ // CheckpointFileManager.createAtomic API which doesn't auto-initialize parent directories.
+ fm.mkdirs(new Path(dfsRootDir))
Review comment:
Make sense, done in f5a6104
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/state/RocksDBFileManager.scala
##########
@@ -17,18 +17,265 @@
package org.apache.spark.sql.execution.streaming.state
-import java.io.File
+import java.io.{File, FileInputStream, InputStream}
import java.nio.charset.StandardCharsets.UTF_8
import java.nio.file.Files
+import java.util.UUID
+import java.util.concurrent.ConcurrentHashMap
+import java.util.zip.{ZipEntry, ZipOutputStream}
+import scala.collection.JavaConverters._
import scala.collection.Seq
import com.fasterxml.jackson.annotation.JsonInclude.Include
import com.fasterxml.jackson.databind.{DeserializationFeature, ObjectMapper}
import com.fasterxml.jackson.module.scala.{DefaultScalaModule, ScalaObjectMapper}
+import org.apache.commons.io.{FilenameUtils, IOUtils}
+import org.apache.hadoop.conf.Configuration
+import org.apache.hadoop.fs.Path
import org.json4s.NoTypeHints
import org.json4s.jackson.Serialization
+import org.apache.spark.internal.Logging
+import org.apache.spark.sql.execution.streaming.CheckpointFileManager
+import org.apache.spark.util.Utils
+
+/**
+ * Class responsible for syncing RocksDB checkpoint files from local disk to DFS.
+ * For each version, checkpoint is saved in specific directory structure that allows successive
+ * versions to reuse to SST data files and archived log files. This allows each commit to be
+ * incremental, only new SST files and archived log files generated by RocksDB will be uploaded.
+ * The directory structures on local disk and in DFS are as follows.
+ *
+ * Local checkpoint dir structure
+ * ------------------------------
+ * RocksDB generates a bunch of files in the local checkpoint directory. The most important among
+ * them are the SST files; they are the actual log structured data files. Rest of the files contain
+ * the metadata necessary for RocksDB to read the SST files and start from the checkpoint.
+ * Note that the SST files are hard links to files in the RocksDB's working directory, and therefore
+ * successive checkpoints can share some of the SST files. So these SST files have to be copied to
+ * DFS in shared directory such that different committed versions can save them.
+ *
+ * We consider both SST files and archived log files as immutable files which can be shared between
+ * different checkpoints.
+ *
+ * localCheckpointDir
+ * |
+ * +-- OPTIONS-000005
+ * +-- MANIFEST-000008
+ * +-- CURRENT
+ * +-- 00007.sst
+ * +-- 00011.sst
+ * +-- archive
+ * | +-- 00008.log
+ * | +-- 00013.log
+ * ...
+ *
+ *
+ * DFS directory structure after saving to DFS as version 10
+ * -----------------------------------------------------------
+ * The SST and archived log files are given unique file names and copied to the shared subdirectory.
+ * Every version maintains a mapping of local immutable file name to the unique file name in DFS.
+ * This mapping is saved in a JSON file (named `metadata`), which is zipped along with other
+ * checkpoint files into a single file `[version].zip`.
+ *
+ * dfsRootDir
+ * |
+ * +-- SSTs
+ * | +-- 00007-[uuid1].sst
+ * | +-- 00011-[uuid2].sst
+ * +-- logs
+ * | +-- 00008-[uuid3].log
+ * | +-- 00013-[uuid4].log
+ * +-- 10.zip
+ * | +-- metadata <--- contains mapping between 00007.sst and [uuid1].sst,
+ * and the mapping between 00008.log and [uuid3].log
+ * | +-- OPTIONS-000005
+ * | +-- MANIFEST-000008
+ * | +-- CURRENT
+ * | ...
+ * |
+ * +-- 9.zip
+ * +-- 8.zip
+ * ...
+ *
+ * Note the following.
+ * - Each [version].zip is a complete description of all the data and metadata needed to recover
+ * a RocksDB instance at the corresponding version. The SST files and log files are not included
+ * in the zip files, they can be shared cross different versions. This is unlike the
+ * [version].delta files of HDFSBackedStateStore where previous delta files needs to be read
+ * to be recovered.
+ * - This is safe wrt speculatively executed tasks running concurrently in different executors
+ * as each task would upload a different copy of the generated immutable files and
+ * atomically update the [version].zip.
+ * - Immutable files are identified uniquely based on their file name and file size.
+ * - Immutable files can be reused only across adjacent checkpoints/versions.
+ * - This class is thread-safe. Specifically, it is safe to concurrently delete old files from a
+ * different thread than the task thread saving files.
+ *
+ * @param dfsRootDir Directory where the [version].zip files will be stored
+ * @param localTempDir Local directory for temporary work
+ * @param hadoopConf Hadoop configuration for talking to DFS
+ * @param loggingId Id that will be prepended in logs for isolating concurrent RocksDBs
+ */
+class RocksDBFileManager(
+ dfsRootDir: String,
+ localTempDir: File,
+ hadoopConf: Configuration,
+ loggingId: String = "")
+ extends Logging {
+
+ import RocksDBImmutableFile._
+
+ private val versionToRocksDBFiles = new ConcurrentHashMap[Long, Seq[RocksDBImmutableFile]]
+ private lazy val fm = CheckpointFileManager.create(new Path(dfsRootDir), hadoopConf)
+ private val fs = new Path(dfsRootDir).getFileSystem(hadoopConf)
+
+ /** Save all the files in given local checkpoint directory as a committed version in DFS */
+ def saveCheckpointToDfs(checkpointDir: File, version: Long, numKeys: Long): Unit = {
+ logFilesInDir(checkpointDir, s"Saving checkpoint files for version $version")
+ val (localImmutableFiles, localOtherFiles) = listRocksDBFiles(checkpointDir)
+ val rocksDBFiles = saveImmutableFilesToDfs(version, localImmutableFiles)
+ val metadata = RocksDBCheckpointMetadata(rocksDBFiles, numKeys)
+ val metadataFile = localMetadataFile(checkpointDir)
+ metadata.writeToFile(metadataFile)
+ logInfo(s"Written metadata for version $version:\n${metadata.prettyJson}")
+
+ if (version <= 0 && numKeys == 0) {
+ // If we're writing the initial version and there's no data, we have to explicitly initialize
+ // the root directory. Normally saveImmutableFilesToDfs will do this initialization, but
+ // when there's no data that method won't write any files, and zipToDfsFile uses the
+ // CheckpointFileManager.createAtomic API which doesn't auto-initialize parent directories.
+ fm.mkdirs(new Path(dfsRootDir))
+ }
+ zipToDfsFile(localOtherFiles :+ metadataFile, dfsBatchZipFile(version))
+ logInfo(s"Saved checkpoint file for version $version")
+ }
+
+ /** Save immutable files to DFS directory */
+ private def saveImmutableFilesToDfs(
+ version: Long,
+ localFiles: Seq[File]): Seq[RocksDBImmutableFile] = {
+ // Get the immutable files used in previous versions, as some of those uploaded files can be
+ // reused for this version
+ logInfo(s"Saving RocksDB files to DFS for $version")
+ val prevFilesToSizes = versionToRocksDBFiles.values.asScala.flatten.map { f =>
+ f.localFileName -> f
+ }.toMap
+
+ var bytesCopied = 0L
+ var filesCopied = 0L
+ var filesReused = 0L
+
+ val immutableFiles = localFiles.map { localFile =>
+ prevFilesToSizes
+ .get(localFile.getName)
+ .filter(_.isSameFile(localFile))
+ .map { reusable =>
+ filesReused += 1
+ reusable
+ }.getOrElse {
+ val localFileName = localFile.getName
+ val dfsFileName = newDFSFileName(localFileName)
+ val dfsFile = dfsFilePath(dfsFileName)
+ // Note: The implementation of copyFromLocalFile() closes the output stream when there is
+ // any exception while copying. So this may generate partial files on DFS. But that is
+ // okay because until the main [version].zip file is written, those partial files are
+ // not going to be used at all. Eventually these files should get cleared.
+ fs.copyFromLocalFile(
+ new Path(localFile.getAbsoluteFile.toURI), dfsFile)
+ val localFileSize = localFile.length()
+ logInfo(s"Copied $localFile to $dfsFile - $localFileSize bytes")
+ filesCopied += 1
+ bytesCopied += localFileSize
+
+ RocksDBImmutableFile(localFile.getName, dfsFileName, localFileSize)
+ }
+ }
+ logInfo(s"Copied $filesCopied files ($bytesCopied bytes) from local to" +
+ s" DFS for version $version. $filesReused files reused without copying.")
+ versionToRocksDBFiles.put(version, immutableFiles)
+
+ immutableFiles
+ }
+
+ /**
+ * Compress files to a single zip file in DFS. Only the file names are embedded in the zip.
+ * Any error while writing will ensure that the file is not written.
+ */
+ private def zipToDfsFile(files: Seq[File], dfsZipFile: Path): Unit = {
+ lazy val filesStr = s"$dfsZipFile\n\t${files.mkString("\n\t")}"
+ var in: InputStream = null
+ val out = fm.createAtomic(dfsZipFile, overwriteIfPossible = true)
+ var totalBytes = 0L
+ val zout = new ZipOutputStream(out)
+ try {
+ files.foreach { file =>
+ zout.putNextEntry(new ZipEntry(file.getName))
+ in = new FileInputStream(file)
+ val bytes = IOUtils.copy(in, zout)
+ in.close()
+ zout.closeEntry()
+ totalBytes += bytes
+ }
+ zout.close() // so that any error in closing also cancels the output stream
+ logInfo(s"Zipped $totalBytes bytes (before compression) to $filesStr")
Review comment:
the `fileStr` contains the `dfsZipFile` name. The log format here is `${dfsZipFile} \n ${listing all the file names}`
##########
File path: sql/core/src/test/scala/org/apache/spark/sql/execution/streaming/state/RocksDBSuite.scala
##########
@@ -20,12 +20,86 @@ package org.apache.spark.sql.execution.streaming.state
import java.io._
import java.nio.charset.Charset
+import scala.language.implicitConversions
+
import org.apache.commons.io.FileUtils
+import org.apache.hadoop.conf.Configuration
import org.apache.spark._
+import org.apache.spark.sql.catalyst.util.quietly
+import org.apache.spark.sql.execution.streaming.CreateAtomicTestManager
+import org.apache.spark.sql.internal.SQLConf
+import org.apache.spark.util.Utils
class RocksDBSuite extends SparkFunSuite {
+ test("RocksDBFileManager: upload only new immutable files") {
+ val dfsRootDir = Utils.createTempDir().getAbsolutePath
Review comment:
Sure, done in f5a6104
##########
File path: sql/core/src/test/scala/org/apache/spark/sql/execution/streaming/state/RocksDBSuite.scala
##########
@@ -20,12 +20,86 @@ package org.apache.spark.sql.execution.streaming.state
import java.io._
import java.nio.charset.Charset
+import scala.language.implicitConversions
+
import org.apache.commons.io.FileUtils
+import org.apache.hadoop.conf.Configuration
import org.apache.spark._
+import org.apache.spark.sql.catalyst.util.quietly
+import org.apache.spark.sql.execution.streaming.CreateAtomicTestManager
+import org.apache.spark.sql.internal.SQLConf
+import org.apache.spark.util.Utils
class RocksDBSuite extends SparkFunSuite {
+ test("RocksDBFileManager: upload only new immutable files") {
+ val dfsRootDir = Utils.createTempDir().getAbsolutePath
+ val fileManager = new RocksDBFileManager(
+ dfsRootDir, Utils.createTempDir(), new Configuration)
+ val sstDir = s"$dfsRootDir/SSTs"
+ def numRemoteSSTFiles: Int = listFiles(sstDir).length
+ val logDir = s"$dfsRootDir/logs"
+ def numRemoteLogFiles: Int = listFiles(logDir).length
Review comment:
Referenced this comment to https://github.com/apache/spark/pull/32767/files#diff-dc6f9dfe11e76f890ff2986f866853bcac263027c82562f9a52f4672a5460826R97
##########
File path: sql/core/src/test/scala/org/apache/spark/sql/execution/streaming/state/RocksDBSuite.scala
##########
@@ -20,12 +20,86 @@ package org.apache.spark.sql.execution.streaming.state
import java.io._
import java.nio.charset.Charset
+import scala.language.implicitConversions
+
import org.apache.commons.io.FileUtils
+import org.apache.hadoop.conf.Configuration
import org.apache.spark._
+import org.apache.spark.sql.catalyst.util.quietly
+import org.apache.spark.sql.execution.streaming.CreateAtomicTestManager
+import org.apache.spark.sql.internal.SQLConf
+import org.apache.spark.util.Utils
class RocksDBSuite extends SparkFunSuite {
+ test("RocksDBFileManager: upload only new immutable files") {
+ val dfsRootDir = Utils.createTempDir().getAbsolutePath
+ val fileManager = new RocksDBFileManager(
+ dfsRootDir, Utils.createTempDir(), new Configuration)
+ val sstDir = s"$dfsRootDir/SSTs"
+ def numRemoteSSTFiles: Int = listFiles(sstDir).length
+ val logDir = s"$dfsRootDir/logs"
+ def numRemoteLogFiles: Int = listFiles(logDir).length
Review comment:
Referenced this comment to https://github.com/apache/spark/pull/32767/files#diff-dc6f9dfe11e76f890ff2986f866853bcac263027c82562f9a52f4672a5460826R79
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #32582: [SPARK-35436][SS] RocksDBFileManager - save checkpoint to DFS
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #32582:
URL: https://github.com/apache/spark/pull/32582#issuecomment-857301345
**[Test build #139516 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/139516/testReport)** for PR 32582 at commit [`f86cea4`](https://github.com/apache/spark/commit/f86cea4fb9c75d3dc29ec83d6f05184431d8584e).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #32582: [SPARK-35436][SS] RocksDBFileManager - save checkpoint to DFS
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #32582:
URL: https://github.com/apache/spark/pull/32582#issuecomment-853783833
**[Test build #139285 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/139285/testReport)** for PR 32582 at commit [`f5a6104`](https://github.com/apache/spark/commit/f5a6104e6930dccad71db72bfc6ee8c47d9933da).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] xuanyuanking commented on pull request #32582: [SPARK-35436] RocksDBFileManager - save checkpoint to DFS
Posted by GitBox <gi...@apache.org>.
xuanyuanking commented on pull request #32582:
URL: https://github.com/apache/spark/pull/32582#issuecomment-850079062
As we merged #32272, after rebasing and addressing the comment, this one is ready for review. cc @viirya and @HeartSaVioR
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #32582: [SPARK-35436][SS] RocksDBFileManager - save checkpoint to DFS
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #32582:
URL: https://github.com/apache/spark/pull/32582#issuecomment-854842906
**[Test build #139340 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/139340/testReport)** for PR 32582 at commit [`bdd9e8e`](https://github.com/apache/spark/commit/bdd9e8eb6d0cf5fd11def7ac3e630f2e2dae302a).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] xuanyuanking commented on a change in pull request #32582: [SPARK-35436][SS] RocksDBFileManager - save checkpoint to DFS
Posted by GitBox <gi...@apache.org>.
xuanyuanking commented on a change in pull request #32582:
URL: https://github.com/apache/spark/pull/32582#discussion_r645672037
##########
File path: sql/core/src/test/scala/org/apache/spark/sql/execution/streaming/state/RocksDBSuite.scala
##########
@@ -20,12 +20,88 @@ package org.apache.spark.sql.execution.streaming.state
import java.io._
import java.nio.charset.Charset
+import scala.language.implicitConversions
+
import org.apache.commons.io.FileUtils
+import org.apache.hadoop.conf.Configuration
import org.apache.spark._
+import org.apache.spark.sql.catalyst.util.quietly
+import org.apache.spark.sql.execution.streaming.CreateAtomicTestManager
+import org.apache.spark.sql.internal.SQLConf
+import org.apache.spark.util.Utils
class RocksDBSuite extends SparkFunSuite {
+ test("RocksDBFileManager: upload only new immutable files") {
+ withTempDir { dir =>
+ val dfsRootDir = dir.getAbsolutePath
+ val fileManager = new RocksDBFileManager(
+ dfsRootDir, Utils.createTempDir(), new Configuration)
+ val sstDir = s"$dfsRootDir/SSTs"
+ def numRemoteSSTFiles: Int = listFiles(sstDir).length
+ val logDir = s"$dfsRootDir/logs"
+ def numRemoteLogFiles: Int = listFiles(logDir).length
+
+ // Save a version of checkpoint files
+ val cpFiles1 = Seq(
+ "sst-file1.sst" -> 10,
+ "sst-file2.sst" -> 20,
+ "other-file1" -> 100,
+ "other-file2" -> 200,
+ "archive/00001.log" -> 1000,
+ "archive/00002.log" -> 2000
+ )
+ saveCheckpointFiles(fileManager, cpFiles1, version = 1, numKeys = 101)
+ assert(numRemoteSSTFiles == 2) // 2 sst files copied
+ assert(numRemoteLogFiles == 2) // 2 log files copied
+
+ // Save SAME version again with different checkpoint files and verify
+ val cpFiles1_ = Seq(
+ "sst-file1.sst" -> 10, // same SST file as before, should not get copied
+ "sst-file2.sst" -> 25, // new SST file with same name as before, but different length
+ "sst-file3.sst" -> 30, // new SST file
+ "other-file1" -> 100, // same non-SST file as before, should not get copied
+ "other-file2" -> 210, // new non-SST file with same name as before, but different length
+ "other-file3" -> 300, // new non-SST file
+ "archive/00001.log" -> 1000, // same log file as before, should not get copied
+ "archive/00002.log" -> 2500, // new log file with same name as before, but different length
+ "archive/00003.log" -> 3000 // new log file
+ )
+ saveCheckpointFiles(fileManager, cpFiles1_, version = 1, numKeys = 1001)
+ assert(numRemoteSSTFiles === 4, "shouldn't copy same files again") // 2 old + 2 new SST files
+ assert(numRemoteLogFiles === 4, "shouldn't copy same files again") // 2 old + 2 new log files
+
+ // Save another version and verify
+ val cpFiles2 = Seq(
+ "sst-file4.sst" -> 40,
+ "other-file4" -> 400,
+ "archive/00004.log" -> 4000
+ )
+ saveCheckpointFiles(fileManager, cpFiles2, version = 2, numKeys = 1501)
+ assert(numRemoteSSTFiles === 5) // 1 new file over earlier 4 files
+ assert(numRemoteLogFiles === 5) // 1 new file over earlier 4 files
+ }
+ }
+
+ test("RocksDBFileManager: error writing [version].zip cancels the output stream") {
+ quietly {
+ val hadoopConf = new Configuration()
+ hadoopConf.set(
+ SQLConf.STREAMING_CHECKPOINT_FILE_MANAGER_CLASS.parent.key,
+ classOf[CreateAtomicTestManager].getName)
+ val dfsRootDir = Utils.createTempDir().getAbsolutePath
+ val fileManager = new RocksDBFileManager(dfsRootDir, Utils.createTempDir(), hadoopConf)
+ val cpFiles = Seq("sst-file1.sst" -> 10, "sst-file2.sst" -> 20, "other-file1" -> 100)
+ CreateAtomicTestManager.shouldFailInCreateAtomic = true
+ CreateAtomicTestManager.cancelCalledInCreateAtomic = false
Review comment:
That's right. Just want to set it to `false` explicitly here since we have a `assert(cancelCalledInCreateAtomic)` in the end. Let me delete this one.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #32582: [SPARK-35436][SS] RocksDBFileManager - save checkpoint to DFS
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #32582:
URL: https://github.com/apache/spark/pull/32582#issuecomment-854942920
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/139340/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org