You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by "ericm-db (via GitHub)" <gi...@apache.org> on 2026/03/25 21:00:15 UTC
[PR] [SS] Integrate checkpoint V2 with auto-repair snapshot [spark]
ericm-db opened a new pull request, #55015:
URL: https://github.com/apache/spark/pull/55015
### What changes were proposed in this pull request?
This PR adds auto-repair snapshot support to the checkpoint V2 (state store checkpoint IDs) load path. Previously, auto-repair and V2 were completely disjoint: `loadWithCheckpointId` had no recovery logic for corrupt snapshots, and the auto-repair path (`loadSnapshotWithoutCheckpointId`) only handled V1 files without UUID awareness.
Changes:
- **RocksDBFileManager**: Added `getSnapshotVersionsAndUniqueIdsFromLineage()` which returns all lineage-matching snapshots sorted descending (the V2 equivalent of `getEligibleSnapshotsForVersion`). Refactored `getLatestSnapshotVersionAndUniqueIdFromLineage()` to delegate to it.
- **RocksDB**: Added `loadSnapshotWithCheckpointId()` which uses `AutoSnapshotLoader` with V2-specific callbacks that map version to uniqueId via a side-channel map. Changelog replay is included inside the load callback so corrupt changelogs also trigger fallback to the next older snapshot.
- **RocksDB**: Wrapped the snapshot load + changelog replay block in `loadWithCheckpointId()` with a try-catch that delegates to the new auto-repair method when enabled. Uses `getFullLineage()` to build the complete lineage chain (back to version 1) so that version 0 fallback with full changelog replay works correctly.
### Why are the changes needed?
Without this change, any corrupt or missing snapshot file in V2 mode causes a hard query failure with no recovery path. V1 mode already had auto-repair (falling back to older snapshots and replaying changelogs), but V2's `loadWithCheckpointId` bypassed that entirely. This is especially important because speculative execution can produce orphaned or incomplete snapshot files that V2 is designed to handle, but corruption of the "winning" snapshot had no fallback.
### Does this PR introduce _any_ user-facing change?
No. This is an internal improvement to fault tolerance. Queries using checkpoint V2 that previously would fail on corrupt snapshots will now automatically recover when `autoSnapshotRepair.enabled` is true (the production default).
### How was this patch tested?
Added integration test "Auto snapshot repair with checkpoint format V2" in `RocksDBSuite` covering:
- Single corrupt V2 snapshot: falls back to older snapshot in lineage
- All V2 snapshots corrupt: falls back to version 0 with full changelog replay
- Verified state correctness and `numSnapshotsAutoRepaired` metric after repair
Also verified existing tests pass:
- `AutoSnapshotLoaderSuite` (5/5)
- `RocksDBSuite` V1 auto-repair test
- `RocksDBStateStoreCheckpointFormatV2Suite` (24/24)
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Code (Claude Opus 4.6)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-56216][SS] Integrate checkpoint V2 with auto-repair snapshot [spark]
Posted by "ericm-db (via GitHub)" <gi...@apache.org>.
ericm-db commented on code in PR #55015:
URL: https://github.com/apache/spark/pull/55015#discussion_r3040700148
##########
sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/state/RocksDB.scala:
##########
@@ -720,13 +688,121 @@ class RocksDB(
override protected def getEligibleSnapshots(version: Long): Seq[Long] = {
fileManager.getEligibleSnapshotsForVersion(version)
}
+
+ override protected def getEligibleSnapshotsForRepair(version: Long): Seq[Long] = {
+ getEligibleSnapshots(version)
+ }
}
val (version, autoRepairCompleted) = snapshotLoader.loadSnapshot(versionToLoad)
performedSnapshotAutoRepair = autoRepairCompleted
version
}
+ /**
+ * V2-aware snapshot loading with auto-repair support.
+ * Uses lineage to discover eligible snapshots and falls back to older ones if corrupt.
+ * Changelog replay is included inside the load callback so that corrupt changelogs
+ * also trigger fallback to the next older snapshot.
+ *
+ * Sets [[performedSnapshotAutoRepair]] as a side effect if auto-repair was used.
+ *
+ * @param versionToLoad The target version to load
+ * @param stateStoreCkptId The checkpoint ID for the target version (for lineage enrichment)
+ * @param initialLineage The lineage for the target version (may be enriched by
+ * getEligibleSnapshotsForRepair during auto-repair)
+ * @return The lineage array to assign to currVersionLineage. During auto-repair,
+ * this may contain more entries than the initial sparse lineage.
+ */
+ private def loadSnapshotWithCheckpointId(
+ versionToLoad: Long,
+ stateStoreCkptId: Option[String],
+ initialLineage: Array[LineageItem]): Array[LineageItem] = {
+ // Local var (intentionally named differently from the class field currVersionLineage)
+ // so that getEligibleSnapshotsForRepair can enrich the lineage and getEligibleSnapshots
+ // (re-called after enrichment) sees the updated lineage.
+ var workingLineage = initialLineage
+ val allowAutoSnapshotRepair = if (enableChangelogCheckpointing) {
Review Comment:
I tried to make it a unified base class with a config, but it ended up being messier than simply duplicating the code here (as there were if-else checks throughout the shared method). I think it costs more than it saves here
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-56216][SS] Integrate checkpoint V2 with auto-repair snapshot [spark]
Posted by "micheal-o (via GitHub)" <gi...@apache.org>.
micheal-o commented on code in PR #55015:
URL: https://github.com/apache/spark/pull/55015#discussion_r3042480423
##########
sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/state/RocksDB.scala:
##########
@@ -490,70 +490,38 @@ class RocksDB(
// Handle normal checkpoint loading
closeDB(ignoreException = false)
- val (latestSnapshotVersion, latestSnapshotUniqueId) = {
- // Special handling when version is 0.
- // When loading the very first version (0), stateStoreCkptId does not need to be defined
- // because there won't be 0.changelog / 0.zip file created in RocksDB under v2.
- if (version == 0) {
- assert(stateStoreCkptId.isEmpty,
- "stateStoreCkptId should be empty when version is zero")
- (0L, None)
- // When there is a snapshot file, it is the ground truth, we can skip
- // reconstructing the lineage from changelog file.
- } else if (fileManager.existsSnapshotFile(version, stateStoreCkptId)) {
+ if (version == 0) {
+ // Version 0 is the empty initial state — load directly without
+ // DFS listing or auto-repair overhead.
+ assert(stateStoreCkptId.isEmpty,
+ "stateStoreCkptId should be empty when version is zero")
+ currVersionLineage = Array.empty
+ val metadata = fileManager.loadCheckpointFromDfs(0,
+ workingDir, rocksDBFileMapping, None)
+ loadedVersion = 0
+ lastSnapshotVersion = 0
+ fileManager.setMaxSeenVersion(0)
+ openLocalRocksDB(metadata)
+ reportSnapshotUploadToCoordinator(0)
+ } else {
+ // Build initial lineage. If the exact snapshot file exists on DFS, use a
+ // sparse lineage (just the target version) as an optimization to skip reading
+ // the changelog header. If auto-repair is needed later,
+ // getEligibleSnapshotsForRepair() will enrich this via getFullLineage().
+ if (fileManager.existsSnapshotFile(version, stateStoreCkptId)) {
currVersionLineage = Array(LineageItem(version, stateStoreCkptId.get))
- (version, stateStoreCkptId)
} else {
currVersionLineage = getLineageFromChangelogFile(version, stateStoreCkptId) :+
LineageItem(version, stateStoreCkptId.get)
currVersionLineage = currVersionLineage.sortBy(_.version)
-
- val latestSnapshotVersionsAndUniqueId =
- fileManager.getLatestSnapshotVersionAndUniqueIdFromLineage(currVersionLineage)
- latestSnapshotVersionsAndUniqueId match {
- case Some(pair) => (pair._1, Option(pair._2))
- case None if currVersionLineage.head.version == 1L =>
- logDebug(log"Cannot find latest snapshot based on lineage but first version " +
- log"is 1, use 0 as default. Lineage: ${MDC(LogKeys.LINEAGE, lineageManager)}")
- (0L, None)
- case _ =>
- throw QueryExecutionErrors.cannotFindBaseSnapshotCheckpoint(
- printLineageItems(currVersionLineage))
- }
}
+ // Delegate to AutoSnapshotLoader which handles both the happy path
+ // and auto-repair fallback to older snapshots.
+ currVersionLineage =
+ loadSnapshotWithCheckpointId(version, stateStoreCkptId, currVersionLineage)
}
-
- logInfo(log"Loaded latestSnapshotVersion: ${
- MDC(LogKeys.SNAPSHOT_VERSION, latestSnapshotVersion)}, latestSnapshotUniqueId: ${
- MDC(LogKeys.UUID, latestSnapshotUniqueId)}")
-
- val metadata = fileManager.loadCheckpointFromDfs(latestSnapshotVersion,
- workingDir, rocksDBFileMapping, latestSnapshotUniqueId)
- // version 0 is the empty initial state; loadCheckpointFromDfs skips DFS for it
loadedFromDfs = version > 0
- loadedVersion = latestSnapshotVersion
-
- // reset the last snapshot version to the latest available snapshot version
- lastSnapshotVersion = latestSnapshotVersion
lineageManager.resetLineage(currVersionLineage)
-
- // Initialize maxVersion upon successful load from DFS
- fileManager.setMaxSeenVersion(version)
-
- // Report this snapshot version to the coordinator
- reportSnapshotUploadToCoordinator(latestSnapshotVersion)
-
- openLocalRocksDB(metadata)
-
- if (loadedVersion != version) {
Review Comment:
keep changelog replay here. See my other comment below
##########
sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/state/RocksDB.scala:
##########
@@ -727,6 +697,119 @@ class RocksDB(
version
}
+ /**
+ * V2-aware snapshot loading with auto-repair support.
+ * Uses lineage to discover eligible snapshots and falls back to older ones if corrupt.
+ *
+ * Sets [[performedSnapshotAutoRepair]] as a side effect if auto-repair was used.
+ *
+ * @param versionToLoad The target version to load
+ * @param stateStoreCkptId The checkpoint ID for the target version (for lineage enrichment)
+ * @param initialLineage The lineage for the target version (may be enriched by
+ * getAdditionalEligibleSnapshots during auto-repair)
+ * @return The lineage array to assign to currVersionLineage. During auto-repair,
+ * this may contain more entries than the initial sparse lineage.
+ */
+ private def loadSnapshotWithCheckpointId(
+ versionToLoad: Long,
+ stateStoreCkptId: Option[String],
+ initialLineage: Array[LineageItem]): Array[LineageItem] = {
+ // Local var (intentionally named differently from the class field currVersionLineage)
+ // so that getAdditionalEligibleSnapshots can enrich the lineage and
+ // getEligibleSnapshots (re-called after enrichment) sees the updated lineage.
+ var workingLineage = initialLineage
+ val allowAutoSnapshotRepair = if (enableChangelogCheckpointing) {
+ conf.stateStoreConf.autoSnapshotRepairEnabled
+ } else {
+ false
+ }
+
+ val snapshotLoader = new AutoSnapshotLoader(
+ allowAutoSnapshotRepair,
+ conf.stateStoreConf.autoSnapshotRepairNumFailuresBeforeActivating,
+ conf.stateStoreConf.autoSnapshotRepairMaxChangeFileReplay,
+ loggingId) {
+ override protected def beforeLoad(): Unit = closeDB(ignoreException = false)
+
+ override protected def loadSnapshotFromCheckpoint(
+ snapshotVersion: Long, uniqueId: Option[String]): Unit = {
+ // Validate that the working lineage covers all versions needed to replay
+ // from this snapshot to versionToLoad. If not, throw so AutoSnapshotLoader
+ // treats this as a snapshot load failure and tries the next snapshot.
+ // This guards against falling back to a snapshot (e.g., version 0) when
+ // the lineage was not enriched (e.g., getFullLineage failed).
+ if (snapshotVersion != versionToLoad) {
+ val neededVersions = ((snapshotVersion + 1) to versionToLoad).toSet
+ val availableVersions = workingLineage.collect {
+ case i if i.version > snapshotVersion && i.version <= versionToLoad => i.version
+ }.toSet
+ if (availableVersions != neededVersions) {
+ throw new IllegalStateException(
+ s"Cannot replay changelogs from snapshot $snapshotVersion to $versionToLoad: " +
+ s"lineage is incomplete, missing versions " +
+ s"${(neededVersions diff availableVersions).toSeq.sorted.mkString(",")}")
+ }
+ }
+
+ val remoteMetaData = fileManager.loadCheckpointFromDfs(snapshotVersion,
+ workingDir, rocksDBFileMapping, uniqueId)
+
+ loadedVersion = snapshotVersion
+ fileManager.setMaxSeenVersion(versionToLoad)
+ openLocalRocksDB(remoteMetaData)
+ lastSnapshotVersion = snapshotVersion
+ reportSnapshotUploadToCoordinator(snapshotVersion)
+ }
+
+ override protected def onLoadSnapshotFromCheckpointFailure(): Unit = {
+ loadedVersion = -1
+ }
+
+ override protected def getEligibleSnapshots(
+ version: Long): Seq[(Long, Option[String])] = {
+ fileManager.getSnapshotVersionsAndUniqueIdsFromLineage(workingLineage)
+ .map { case (ver, id) => (ver, Option(id)) }
+ }
+
+ override protected def getAdditionalEligibleSnapshots(
+ version: Long): Seq[(Long, Option[String])] = {
+ // Build lineage back to (versionToLoad - maxChangeFileReplay) so that
+ // auto-repair can discover older snapshots in the lineage. We don't walk
+ // all the way to version 1 since snapshots beyond maxChangeFileReplay
+ // would be filtered out anyway.
+ if (stateStoreCkptId.isDefined) {
+ try {
+ val startVer = math.max(1L,
+ versionToLoad - conf.stateStoreConf.autoSnapshotRepairMaxChangeFileReplay)
+ workingLineage = getFullLineage(startVer, versionToLoad, stateStoreCkptId)
+ } catch {
+ case NonFatal(e) =>
+ logWarning(log"Failed to enrich lineage via getFullLineage during " +
+ log"auto-repair for version ${MDC(LogKeys.VERSION_NUM, versionToLoad)}. " +
+ log"Falling back to existing lineage; repair options may be limited.", e)
+ }
+ }
+ // Re-discover eligible snapshots from the (potentially enriched) lineage
+ getEligibleSnapshots(version)
+ }
+ }
+
+ val (snapshotVersion, autoRepairCompleted) = snapshotLoader.loadSnapshot(versionToLoad)
+ performedSnapshotAutoRepair = autoRepairCompleted
+
+ // Replay changelogs from loaded snapshot to target version
+ if (snapshotVersion != versionToLoad) {
Review Comment:
Don't replay changelog here. The name of this func is `loadSnapshot...`. Basically just for loading snapshot. Keep the changelog replay where it was previously.
##########
sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/state/AutoSnapshotLoader.scala:
##########
@@ -90,16 +114,16 @@ abstract class AutoSnapshotLoader(
beforeLoad() // if this fails, then we should fail
try {
// try to load the first eligible snapshot
- loadSnapshotFromCheckpoint(firstEligibleSnapshot)
- loadedSnapshot = Some(firstEligibleSnapshot)
+ loadSnapshotFromCheckpoint(firstEligibleSnapshot._1, firstEligibleSnapshot._2)
+ loadedSnapshot = Some(firstEligibleSnapshot._1)
} catch {
// Swallow only if auto snapshot repair is enabled
// If auto snapshot repair is not enabled, we should fail immediately
case NonFatal(e) if autoSnapshotRepairEnabled =>
onLoadSnapshotFromCheckpointFailure()
numFailuresForFirstSnapshot += 1
logError(log"Failed to load snapshot version " +
- log"${MDC(VERSION_NUM, firstEligibleSnapshot)}, " +
+ log"${MDC(VERSION_NUM, firstEligibleSnapshot._1)}, " +
Review Comment:
nit: include the checkptId in the log
##########
sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/state/RocksDB.scala:
##########
@@ -720,13 +688,121 @@ class RocksDB(
override protected def getEligibleSnapshots(version: Long): Seq[Long] = {
fileManager.getEligibleSnapshotsForVersion(version)
}
+
+ override protected def getEligibleSnapshotsForRepair(version: Long): Seq[Long] = {
+ getEligibleSnapshots(version)
+ }
}
val (version, autoRepairCompleted) = snapshotLoader.loadSnapshot(versionToLoad)
performedSnapshotAutoRepair = autoRepairCompleted
version
}
+ /**
+ * V2-aware snapshot loading with auto-repair support.
+ * Uses lineage to discover eligible snapshots and falls back to older ones if corrupt.
+ * Changelog replay is included inside the load callback so that corrupt changelogs
+ * also trigger fallback to the next older snapshot.
+ *
+ * Sets [[performedSnapshotAutoRepair]] as a side effect if auto-repair was used.
+ *
+ * @param versionToLoad The target version to load
+ * @param stateStoreCkptId The checkpoint ID for the target version (for lineage enrichment)
+ * @param initialLineage The lineage for the target version (may be enriched by
+ * getEligibleSnapshotsForRepair during auto-repair)
+ * @return The lineage array to assign to currVersionLineage. During auto-repair,
+ * this may contain more entries than the initial sparse lineage.
+ */
+ private def loadSnapshotWithCheckpointId(
+ versionToLoad: Long,
+ stateStoreCkptId: Option[String],
+ initialLineage: Array[LineageItem]): Array[LineageItem] = {
+ // Local var (intentionally named differently from the class field currVersionLineage)
+ // so that getEligibleSnapshotsForRepair can enrich the lineage and getEligibleSnapshots
+ // (re-called after enrichment) sees the updated lineage.
+ var workingLineage = initialLineage
+ val allowAutoSnapshotRepair = if (enableChangelogCheckpointing) {
Review Comment:
@ericm-db I was refering to the `allowAutoSnapshotRepair` conf check here
##########
sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/state/AutoSnapshotLoader.scala:
##########
@@ -114,38 +138,41 @@ abstract class AutoSnapshotLoader(
// we would only get here if auto snapshot repair is enabled
assert(autoSnapshotRepairEnabled)
- val remainingEligibleSnapshots = if (eligibleSnapshots.length > 1) {
- // skip the first snapshot, since we already tried it
- eligibleSnapshots.tail
- } else {
- // no more snapshots to try
- Seq.empty
- }
+ // Merge initial eligible snapshots with any additional ones discovered during
+ // repair (e.g., V2 enriches a sparse lineage to discover more snapshots).
+ // Deduplicate by version and filter out the first snapshot we already tried.
+ val remainingEligibleSnapshots =
+ (eligibleSnapshots ++ getAdditionalEligibleSnapshots(versionToLoad)
+ :+ (0L, Option.empty[String]))
+ .distinctBy(_._1)
+ .sortBy(_._1)(Ordering[Long].reverse)
+ .filter(_._1 != firstEligibleSnapshot._1)
// select remaining snapshots that are within the maxChangeFileReplay limit
val selectedRemainingSnapshots = remainingEligibleSnapshots.filter(
- s => versionToLoad - s <= maxChangeFileReplay)
+ s => versionToLoad - s._1 <= maxChangeFileReplay)
logInfo(log"Attempting to auto repair snapshot by skipping " +
- log"snapshot version ${MDC(VERSION_NUM, firstEligibleSnapshot)} " +
+ log"snapshot version ${MDC(VERSION_NUM, firstEligibleSnapshot._1)} " +
log"and trying to load with one of the selected snapshots " +
- log"${MDC(VERSION_NUM, selectedRemainingSnapshots)}, out of eligible snapshots " +
- log"${MDC(VERSION_NUM, remainingEligibleSnapshots)}. " +
+ log"${MDC(VERSION_NUM, selectedRemainingSnapshots.map(_._1))}, " +
+ log"out of eligible snapshots " +
+ log"${MDC(VERSION_NUM, remainingEligibleSnapshots.map(_._1))}. " +
log"maxChangeFileReplay: ${MDC(VERSION_NUM, maxChangeFileReplay)}")
// Now try to load using any of the selected snapshots,
// remember they are sorted in descending order
- for (snapshotVersion <- selectedRemainingSnapshots if loadedSnapshot.isEmpty) {
+ for (snapshot <- selectedRemainingSnapshots if loadedSnapshot.isEmpty) {
beforeLoad() // if this fails, then we should fail
try {
- loadSnapshotFromCheckpoint(snapshotVersion)
- loadedSnapshot = Some(snapshotVersion)
+ loadSnapshotFromCheckpoint(snapshot._1, snapshot._2)
+ loadedSnapshot = Some(snapshot._1)
logInfo(log"Successfully loaded snapshot version " +
- log"${MDC(VERSION_NUM, snapshotVersion)}. Repair complete.")
+ log"${MDC(VERSION_NUM, snapshot._1)}. Repair complete.")
Review Comment:
nit: include the checkptId in the log
##########
sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/state/AutoSnapshotLoader.scala:
##########
@@ -54,15 +54,39 @@ abstract class AutoSnapshotLoader(
/**
* Attempt to load the specified snapshot version from the checkpoint directory.
* Should throw an exception if the snapshot is corrupt.
+ * @param snapshotVersion The snapshot version to load
+ * @param uniqueId The unique checkpoint ID for V2, or None for V1
* @note Must support loading version 0
* */
- protected def loadSnapshotFromCheckpoint(snapshotVersion: Long): Unit
+ protected def loadSnapshotFromCheckpoint(
+ snapshotVersion: Long, uniqueId: Option[String]): Unit
/** Called when load fails, to do any necessary cleanup/variable reset */
protected def onLoadSnapshotFromCheckpointFailure(): Unit
- /** Get a list of eligible snapshot versions in the checkpoint directory that can be loaded */
- protected def getEligibleSnapshots(versionToLoad: Long): Seq[Long]
+ /**
+ * Get a list of eligible snapshot (version, uniqueId) pairs in the checkpoint directory
+ * that can be loaded. For V1, uniqueId is None.
+ */
+ protected def getEligibleSnapshots(
+ versionToLoad: Long): Seq[(Long, Option[String])]
+
+ /**
+ * Returns additional eligible snapshots discovered during auto-repair.
+ * Called once when the first snapshot load fails and auto-repair begins.
+ *
+ * V2 overrides this to enrich a sparse lineage (via getFullLineage) and return
+ * newly-discovered snapshots. The results are merged with the initial eligible
+ * snapshots (duplicates are removed by version).
+ *
+ * Default: no additional snapshots, which avoids redundant DFS listings for
+ * V1 and HDFS callers that have no additional snapshots to discover.
Review Comment:
nit: we can remove this default comment
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-56216][SS] Integrate checkpoint V2 with auto-repair snapshot [spark]
Posted by "anishshri-db (via GitHub)" <gi...@apache.org>.
anishshri-db commented on code in PR #55015:
URL: https://github.com/apache/spark/pull/55015#discussion_r3031441191
##########
sql/core/src/test/scala/org/apache/spark/sql/execution/streaming/state/RocksDBSuite.scala:
##########
@@ -4166,6 +4162,341 @@ class RocksDBSuite extends AlsoTestWithRocksDBFeatures with SharedSparkSession
}
}
+ testWithChangelogCheckpointingEnabled(
+ "Auto snapshot repair with checkpoint format V2") {
Review Comment:
How we ensure that the auto repair is based on the right lineage ?
##########
sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/state/AutoSnapshotLoader.scala:
##########
@@ -64,6 +64,24 @@ abstract class AutoSnapshotLoader(
/** Get a list of eligible snapshot versions in the checkpoint directory that can be loaded */
protected def getEligibleSnapshots(versionToLoad: Long): Seq[Long]
+ /**
+ * Called once when the first snapshot load fails and auto-repair is about to begin.
+ * Returns the eligible snapshots to use for the repair loop. Subclasses can override
+ * this to return an enriched set of snapshots (e.g., by building a fuller lineage
+ * that was initially sparse for optimization).
+ *
+ * The default delegates to [[getEligibleSnapshots]], which means it re-calls that method
+ * rather than reusing the initial list. This is intentional: V2 overrides this method to
+ * enrich the lineage first, then re-discovers snapshots from the enriched lineage.
+ * For V1 callers (which do not override), the re-call is benign since
+ * [[getEligibleSnapshots]] returns a consistent result from DFS listing.
+ *
+ * @param versionToLoad The version being loaded
+ * @return Eligible snapshot versions for repair
+ */
+ protected def getEligibleSnapshotsForRepair(versionToLoad: Long): Seq[Long] =
+ getEligibleSnapshots(versionToLoad)
Review Comment:
Could we keep this abstract similar to other methods ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-56216][SS] Integrate checkpoint V2 with auto-repair snapshot [spark]
Posted by "anishshri-db (via GitHub)" <gi...@apache.org>.
anishshri-db commented on code in PR #55015:
URL: https://github.com/apache/spark/pull/55015#discussion_r3031449374
##########
sql/core/src/test/scala/org/apache/spark/sql/execution/streaming/state/RocksDBSuite.scala:
##########
@@ -4117,10 +4117,6 @@ class RocksDBSuite extends AlsoTestWithRocksDBFeatures with SharedSparkSession
db.doMaintenance() // upload snapshot 4.zip
}
- def corruptFile(file: File): Unit =
Review Comment:
intentional ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-56216][SS] Integrate checkpoint V2 with auto-repair snapshot [spark]
Posted by "ericm-db (via GitHub)" <gi...@apache.org>.
ericm-db commented on code in PR #55015:
URL: https://github.com/apache/spark/pull/55015#discussion_r3034084943
##########
sql/core/src/test/scala/org/apache/spark/sql/execution/streaming/state/RocksDBSuite.scala:
##########
@@ -4166,6 +4162,341 @@ class RocksDBSuite extends AlsoTestWithRocksDBFeatures with SharedSparkSession
}
}
+ testWithChangelogCheckpointingEnabled(
+ "Auto snapshot repair with checkpoint format V2") {
Review Comment:
getSnapshotVersionsAndUniqueIdsFromLineage checks the snapshot files against the lineage, so only snapshots whose (version, uniqueId) pair matches get considered. A snapshot that has a different uniqueId that comes from a different task attempt is never picked up.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-56216][SS] Integrate checkpoint V2 with auto-repair snapshot [spark]
Posted by "ericm-db (via GitHub)" <gi...@apache.org>.
ericm-db commented on code in PR #55015:
URL: https://github.com/apache/spark/pull/55015#discussion_r3034076595
##########
sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/state/RocksDBFileManager.scala:
##########
@@ -455,14 +455,14 @@ class RocksDBFileManager(
/**
* Based on the ground truth lineage loaded from changelog file (lineage), this function
* does file listing to find all snapshot (version, uniqueId) pairs, and finds
- * the ground truth latest snapshot (version, uniqueId) the db instance needs to load.
+ * all snapshots that match the lineage, sorted by version descending.
*
* @param lineage The ground truth lineage loaded from changelog file, sorted by id
- * @return The ground truth latest snapshot (version, uniqueId) the db instance needs to load,
- * when the return value is None it means ther is no such snapshot found.
+ * @return All snapshot (version, uniqueId) pairs matching the lineage, sorted descending
+ * by version. Empty if no matching snapshots found.
*/
- def getLatestSnapshotVersionAndUniqueIdFromLineage(
- lineage: Array[LineageItem]): Option[(Long, String)] = {
+ def getSnapshotVersionsAndUniqueIdsFromLineage(
+ lineage: Array[LineageItem]): Seq[(Long, String)] = {
Review Comment:
It's used in RocksDB.loadSnapshotWithCheckpointId() inside the getEligibleSnapshots() override of AutoSnapshotLoader. It finds all snapshots on DFS that match the working lineage and populates the eligibleSnapshotUniqueIds so that loadSnapshotFromCheckpoint knows which uniqueId to use for each snapshot version.
getLatestSnapshotVersionAndUniqueIdFromLineage() also calls this method and returns the head of the list
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-56216][SS] Integrate checkpoint V2 with auto-repair snapshot [spark]
Posted by "ericm-db (via GitHub)" <gi...@apache.org>.
ericm-db commented on code in PR #55015:
URL: https://github.com/apache/spark/pull/55015#discussion_r3034076595
##########
sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/state/RocksDBFileManager.scala:
##########
@@ -455,14 +455,14 @@ class RocksDBFileManager(
/**
* Based on the ground truth lineage loaded from changelog file (lineage), this function
* does file listing to find all snapshot (version, uniqueId) pairs, and finds
- * the ground truth latest snapshot (version, uniqueId) the db instance needs to load.
+ * all snapshots that match the lineage, sorted by version descending.
*
* @param lineage The ground truth lineage loaded from changelog file, sorted by id
- * @return The ground truth latest snapshot (version, uniqueId) the db instance needs to load,
- * when the return value is None it means ther is no such snapshot found.
+ * @return All snapshot (version, uniqueId) pairs matching the lineage, sorted descending
+ * by version. Empty if no matching snapshots found.
*/
- def getLatestSnapshotVersionAndUniqueIdFromLineage(
- lineage: Array[LineageItem]): Option[(Long, String)] = {
+ def getSnapshotVersionsAndUniqueIdsFromLineage(
+ lineage: Array[LineageItem]): Seq[(Long, String)] = {
Review Comment:
It's used in RocksDB.loadSnapshotWithCheckpointId() inside the getEligibleSnapshots() override of AutoSnapshotLoader. It finds all snapshots on DFS that match the working lineage and populates the eligibleSnapshotUniqueIds so that loadSnapshotFromCheckpoint knows which uniqueId to use for each snapshot version. getLatestSnapshotVersionAndUniqueIdFromLineage() also now delegates to it via .headOption.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-56216][SS] Integrate checkpoint V2 with auto-repair snapshot [spark]
Posted by "anishshri-db (via GitHub)" <gi...@apache.org>.
anishshri-db closed pull request #55015: [SPARK-56216][SS] Integrate checkpoint V2 with auto-repair snapshot
URL: https://github.com/apache/spark/pull/55015
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-56216][SS] Integrate checkpoint V2 with auto-repair snapshot [spark]
Posted by "ericm-db (via GitHub)" <gi...@apache.org>.
ericm-db commented on code in PR #55015:
URL: https://github.com/apache/spark/pull/55015#discussion_r3034084943
##########
sql/core/src/test/scala/org/apache/spark/sql/execution/streaming/state/RocksDBSuite.scala:
##########
@@ -4166,6 +4162,341 @@ class RocksDBSuite extends AlsoTestWithRocksDBFeatures with SharedSparkSession
}
}
+ testWithChangelogCheckpointingEnabled(
+ "Auto snapshot repair with checkpoint format V2") {
Review Comment:
getSnapshotVersionsAndUniqueIdsFromLineage cross-references DFS snapshot files against the lineage, so only snapshots whose (version, uniqueId) pair matches get considered. A snapshot that has a different uniqueId that comes from a different task attempt is never picked up.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-56216][SS] Integrate checkpoint V2 with auto-repair snapshot [spark]
Posted by "micheal-o (via GitHub)" <gi...@apache.org>.
micheal-o commented on code in PR #55015:
URL: https://github.com/apache/spark/pull/55015#discussion_r3035250139
##########
sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/state/AutoSnapshotLoader.scala:
##########
@@ -64,6 +64,18 @@ abstract class AutoSnapshotLoader(
/** Get a list of eligible snapshot versions in the checkpoint directory that can be loaded */
protected def getEligibleSnapshots(versionToLoad: Long): Seq[Long]
+ /**
+ * Called once when the first snapshot load fails and auto-repair is about to begin.
+ * Returns the eligible snapshots to use for the repair loop.
+ *
+ * V2 overrides this to enrich a sparse lineage (via getFullLineage) before
+ * re-discovering snapshots. V1 can simply delegate to [[getEligibleSnapshots]].
+ *
+ * @param versionToLoad The version being loaded
+ * @return Eligible snapshot versions for repair
+ */
+ protected def getEligibleSnapshotsForRepair(versionToLoad: Long): Seq[Long]
Review Comment:
`getAdditionalEligibleSnapshots` instead? and fix the func description
##########
sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/state/HDFSBackedStateStoreProvider.scala:
##########
@@ -756,6 +756,10 @@ private[sql] class HDFSBackedStateStoreProvider extends StateStoreProvider with
// Combine the two sets of versions, so we can check both during load
(snapshotVersions ++ cachedVersions).distinct
}
+
+ override protected def getEligibleSnapshotsForRepair(versionToLoad: Long): Seq[Long] = {
+ getEligibleSnapshots(versionToLoad)
Review Comment:
We are basically doing storage LIST twice when it is not really needed. That isn't good.
##########
sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/state/RocksDB.scala:
##########
@@ -720,13 +688,121 @@ class RocksDB(
override protected def getEligibleSnapshots(version: Long): Seq[Long] = {
fileManager.getEligibleSnapshotsForVersion(version)
}
+
+ override protected def getEligibleSnapshotsForRepair(version: Long): Seq[Long] = {
+ getEligibleSnapshots(version)
+ }
}
val (version, autoRepairCompleted) = snapshotLoader.loadSnapshot(versionToLoad)
performedSnapshotAutoRepair = autoRepairCompleted
version
}
+ /**
+ * V2-aware snapshot loading with auto-repair support.
+ * Uses lineage to discover eligible snapshots and falls back to older ones if corrupt.
+ * Changelog replay is included inside the load callback so that corrupt changelogs
+ * also trigger fallback to the next older snapshot.
+ *
+ * Sets [[performedSnapshotAutoRepair]] as a side effect if auto-repair was used.
+ *
+ * @param versionToLoad The target version to load
+ * @param stateStoreCkptId The checkpoint ID for the target version (for lineage enrichment)
+ * @param initialLineage The lineage for the target version (may be enriched by
+ * getEligibleSnapshotsForRepair during auto-repair)
+ * @return The lineage array to assign to currVersionLineage. During auto-repair,
+ * this may contain more entries than the initial sparse lineage.
+ */
+ private def loadSnapshotWithCheckpointId(
+ versionToLoad: Long,
+ stateStoreCkptId: Option[String],
+ initialLineage: Array[LineageItem]): Array[LineageItem] = {
+ // Local var (intentionally named differently from the class field currVersionLineage)
+ // so that getEligibleSnapshotsForRepair can enrich the lineage and getEligibleSnapshots
+ // (re-called after enrichment) sees the updated lineage.
+ var workingLineage = initialLineage
+ val allowAutoSnapshotRepair = if (enableChangelogCheckpointing) {
+ conf.stateStoreConf.autoSnapshotRepairEnabled
+ } else {
+ false
+ }
+
+ // Side-channel map: version -> uniqueId, populated by getEligibleSnapshots,
+ // consumed by loadSnapshotFromCheckpoint. These two methods are tightly coupled:
+ // getEligibleSnapshots must always be called before loadSnapshotFromCheckpoint
+ // for a given snapshot version to ensure the uniqueId is available.
+ val eligibleSnapshotUniqueIds = mutable.Map[Long, String]()
+
+ val snapshotLoader = new AutoSnapshotLoader(
+ allowAutoSnapshotRepair,
+ conf.stateStoreConf.autoSnapshotRepairNumFailuresBeforeActivating,
+ conf.stateStoreConf.autoSnapshotRepairMaxChangeFileReplay,
+ loggingId) {
+ override protected def beforeLoad(): Unit = closeDB(ignoreException = false)
+
+ override protected def loadSnapshotFromCheckpoint(snapshotVersion: Long): Unit = {
+ val uniqueId = eligibleSnapshotUniqueIds.get(snapshotVersion)
Review Comment:
This side channeling of uniqueId is not a clean approach. Today in rocksdb our methods that support version also support checkpointId. So why not do the same for the AutoSnapshotLoader. Pass in the uniqueId. default to None when not specified. Will make the implementation simpler
##########
sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/state/RocksDB.scala:
##########
@@ -720,13 +688,121 @@ class RocksDB(
override protected def getEligibleSnapshots(version: Long): Seq[Long] = {
fileManager.getEligibleSnapshotsForVersion(version)
}
+
+ override protected def getEligibleSnapshotsForRepair(version: Long): Seq[Long] = {
+ getEligibleSnapshots(version)
+ }
}
val (version, autoRepairCompleted) = snapshotLoader.loadSnapshot(versionToLoad)
performedSnapshotAutoRepair = autoRepairCompleted
version
}
+ /**
+ * V2-aware snapshot loading with auto-repair support.
+ * Uses lineage to discover eligible snapshots and falls back to older ones if corrupt.
+ * Changelog replay is included inside the load callback so that corrupt changelogs
+ * also trigger fallback to the next older snapshot.
+ *
+ * Sets [[performedSnapshotAutoRepair]] as a side effect if auto-repair was used.
+ *
+ * @param versionToLoad The target version to load
+ * @param stateStoreCkptId The checkpoint ID for the target version (for lineage enrichment)
+ * @param initialLineage The lineage for the target version (may be enriched by
+ * getEligibleSnapshotsForRepair during auto-repair)
+ * @return The lineage array to assign to currVersionLineage. During auto-repair,
+ * this may contain more entries than the initial sparse lineage.
+ */
+ private def loadSnapshotWithCheckpointId(
+ versionToLoad: Long,
+ stateStoreCkptId: Option[String],
+ initialLineage: Array[LineageItem]): Array[LineageItem] = {
+ // Local var (intentionally named differently from the class field currVersionLineage)
+ // so that getEligibleSnapshotsForRepair can enrich the lineage and getEligibleSnapshots
+ // (re-called after enrichment) sees the updated lineage.
+ var workingLineage = initialLineage
+ val allowAutoSnapshotRepair = if (enableChangelogCheckpointing) {
+ conf.stateStoreConf.autoSnapshotRepairEnabled
+ } else {
+ false
+ }
+
+ // Side-channel map: version -> uniqueId, populated by getEligibleSnapshots,
+ // consumed by loadSnapshotFromCheckpoint. These two methods are tightly coupled:
+ // getEligibleSnapshots must always be called before loadSnapshotFromCheckpoint
+ // for a given snapshot version to ensure the uniqueId is available.
+ val eligibleSnapshotUniqueIds = mutable.Map[Long, String]()
+
+ val snapshotLoader = new AutoSnapshotLoader(
+ allowAutoSnapshotRepair,
+ conf.stateStoreConf.autoSnapshotRepairNumFailuresBeforeActivating,
+ conf.stateStoreConf.autoSnapshotRepairMaxChangeFileReplay,
+ loggingId) {
+ override protected def beforeLoad(): Unit = closeDB(ignoreException = false)
+
+ override protected def loadSnapshotFromCheckpoint(snapshotVersion: Long): Unit = {
+ val uniqueId = eligibleSnapshotUniqueIds.get(snapshotVersion)
+ val remoteMetaData = fileManager.loadCheckpointFromDfs(snapshotVersion,
+ workingDir, rocksDBFileMapping, uniqueId)
+
+ loadedVersion = snapshotVersion
+ fileManager.setMaxSeenVersion(versionToLoad)
+ openLocalRocksDB(remoteMetaData)
+ lastSnapshotVersion = snapshotVersion
+ reportSnapshotUploadToCoordinator(snapshotVersion)
+
+ // Replay changelogs from snapshot to target version
+ if (snapshotVersion != versionToLoad) {
+ val versionsAndUniqueIds = workingLineage.collect {
+ case i if i.version > snapshotVersion && i.version <= versionToLoad =>
+ (i.version, Option(i.checkpointUniqueId))
+ }
+ replayChangelog(versionsAndUniqueIds)
+ loadedVersion = versionToLoad
+ }
+
+ }
+
+ override protected def onLoadSnapshotFromCheckpointFailure(): Unit = {
+ loadedVersion = -1
+ }
+
+ override protected def getEligibleSnapshots(version: Long): Seq[Long] = {
+ val snapshotsInLineage =
+ fileManager.getSnapshotVersionsAndUniqueIdsFromLineage(workingLineage)
+ eligibleSnapshotUniqueIds.clear()
+ snapshotsInLineage.foreach { case (ver, id) =>
+ eligibleSnapshotUniqueIds.put(ver, id)
+ }
+ snapshotsInLineage.map(_._1)
+ }
+
+ override protected def getEligibleSnapshotsForRepair(version: Long): Seq[Long] = {
+ // Build the full lineage from version 1 to the target version so that
+ // auto-repair can fall back to any snapshot (including version 0 with
+ // full changelog replay). getFullLineage walks backward through
+ // changelog file headers to reconstruct the complete chain.
+ if (stateStoreCkptId.isDefined) {
+ try {
+ workingLineage = getFullLineage(1, versionToLoad, stateStoreCkptId)
Review Comment:
Getting lineage from batch 1 could be very expensive, and we might end up not using all due to `autoSnapshotRepairMaxChangeFileReplay`. So why not do starting from `versionToLoad - autoSnapshotRepairMaxChangeFileReplay`? What do you think?
##########
sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/state/RocksDB.scala:
##########
@@ -720,13 +688,121 @@ class RocksDB(
override protected def getEligibleSnapshots(version: Long): Seq[Long] = {
fileManager.getEligibleSnapshotsForVersion(version)
}
+
+ override protected def getEligibleSnapshotsForRepair(version: Long): Seq[Long] = {
+ getEligibleSnapshots(version)
Review Comment:
same here, unnecessary additional storage LIST
##########
sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/state/RocksDB.scala:
##########
@@ -720,13 +688,121 @@ class RocksDB(
override protected def getEligibleSnapshots(version: Long): Seq[Long] = {
fileManager.getEligibleSnapshotsForVersion(version)
}
+
+ override protected def getEligibleSnapshotsForRepair(version: Long): Seq[Long] = {
+ getEligibleSnapshots(version)
+ }
}
val (version, autoRepairCompleted) = snapshotLoader.loadSnapshot(versionToLoad)
performedSnapshotAutoRepair = autoRepairCompleted
version
}
+ /**
+ * V2-aware snapshot loading with auto-repair support.
+ * Uses lineage to discover eligible snapshots and falls back to older ones if corrupt.
+ * Changelog replay is included inside the load callback so that corrupt changelogs
+ * also trigger fallback to the next older snapshot.
+ *
+ * Sets [[performedSnapshotAutoRepair]] as a side effect if auto-repair was used.
+ *
+ * @param versionToLoad The target version to load
+ * @param stateStoreCkptId The checkpoint ID for the target version (for lineage enrichment)
+ * @param initialLineage The lineage for the target version (may be enriched by
+ * getEligibleSnapshotsForRepair during auto-repair)
+ * @return The lineage array to assign to currVersionLineage. During auto-repair,
+ * this may contain more entries than the initial sparse lineage.
+ */
+ private def loadSnapshotWithCheckpointId(
+ versionToLoad: Long,
+ stateStoreCkptId: Option[String],
+ initialLineage: Array[LineageItem]): Array[LineageItem] = {
+ // Local var (intentionally named differently from the class field currVersionLineage)
+ // so that getEligibleSnapshotsForRepair can enrich the lineage and getEligibleSnapshots
+ // (re-called after enrichment) sees the updated lineage.
+ var workingLineage = initialLineage
+ val allowAutoSnapshotRepair = if (enableChangelogCheckpointing) {
Review Comment:
No need to duplicate this here. Just make it common with the other one
##########
sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/state/RocksDB.scala:
##########
@@ -720,13 +688,121 @@ class RocksDB(
override protected def getEligibleSnapshots(version: Long): Seq[Long] = {
fileManager.getEligibleSnapshotsForVersion(version)
}
+
+ override protected def getEligibleSnapshotsForRepair(version: Long): Seq[Long] = {
+ getEligibleSnapshots(version)
+ }
}
val (version, autoRepairCompleted) = snapshotLoader.loadSnapshot(versionToLoad)
performedSnapshotAutoRepair = autoRepairCompleted
version
}
+ /**
+ * V2-aware snapshot loading with auto-repair support.
+ * Uses lineage to discover eligible snapshots and falls back to older ones if corrupt.
+ * Changelog replay is included inside the load callback so that corrupt changelogs
+ * also trigger fallback to the next older snapshot.
+ *
+ * Sets [[performedSnapshotAutoRepair]] as a side effect if auto-repair was used.
+ *
+ * @param versionToLoad The target version to load
+ * @param stateStoreCkptId The checkpoint ID for the target version (for lineage enrichment)
+ * @param initialLineage The lineage for the target version (may be enriched by
+ * getEligibleSnapshotsForRepair during auto-repair)
+ * @return The lineage array to assign to currVersionLineage. During auto-repair,
+ * this may contain more entries than the initial sparse lineage.
+ */
+ private def loadSnapshotWithCheckpointId(
+ versionToLoad: Long,
+ stateStoreCkptId: Option[String],
+ initialLineage: Array[LineageItem]): Array[LineageItem] = {
+ // Local var (intentionally named differently from the class field currVersionLineage)
+ // so that getEligibleSnapshotsForRepair can enrich the lineage and getEligibleSnapshots
+ // (re-called after enrichment) sees the updated lineage.
+ var workingLineage = initialLineage
+ val allowAutoSnapshotRepair = if (enableChangelogCheckpointing) {
+ conf.stateStoreConf.autoSnapshotRepairEnabled
+ } else {
+ false
+ }
+
+ // Side-channel map: version -> uniqueId, populated by getEligibleSnapshots,
+ // consumed by loadSnapshotFromCheckpoint. These two methods are tightly coupled:
+ // getEligibleSnapshots must always be called before loadSnapshotFromCheckpoint
+ // for a given snapshot version to ensure the uniqueId is available.
+ val eligibleSnapshotUniqueIds = mutable.Map[Long, String]()
+
+ val snapshotLoader = new AutoSnapshotLoader(
+ allowAutoSnapshotRepair,
+ conf.stateStoreConf.autoSnapshotRepairNumFailuresBeforeActivating,
+ conf.stateStoreConf.autoSnapshotRepairMaxChangeFileReplay,
+ loggingId) {
+ override protected def beforeLoad(): Unit = closeDB(ignoreException = false)
+
+ override protected def loadSnapshotFromCheckpoint(snapshotVersion: Long): Unit = {
+ val uniqueId = eligibleSnapshotUniqueIds.get(snapshotVersion)
+ val remoteMetaData = fileManager.loadCheckpointFromDfs(snapshotVersion,
+ workingDir, rocksDBFileMapping, uniqueId)
+
+ loadedVersion = snapshotVersion
+ fileManager.setMaxSeenVersion(versionToLoad)
+ openLocalRocksDB(remoteMetaData)
+ lastSnapshotVersion = snapshotVersion
+ reportSnapshotUploadToCoordinator(snapshotVersion)
+
+ // Replay changelogs from snapshot to target version
+ if (snapshotVersion != versionToLoad) {
Review Comment:
Why are we doing changelog replay within it? We can't recover from changelog corruption right? So i don't see the point?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-56216][SS] Integrate checkpoint V2 with auto-repair snapshot [spark]
Posted by "anishshri-db (via GitHub)" <gi...@apache.org>.
anishshri-db commented on code in PR #55015:
URL: https://github.com/apache/spark/pull/55015#discussion_r3031450350
##########
sql/core/src/test/scala/org/apache/spark/sql/execution/streaming/state/RocksDBSuite.scala:
##########
@@ -4166,6 +4162,341 @@ class RocksDBSuite extends AlsoTestWithRocksDBFeatures with SharedSparkSession
}
}
+ testWithChangelogCheckpointingEnabled(
+ "Auto snapshot repair with checkpoint format V2") {
+ withSQLConf(
+ SQLConf.STREAMING_CHECKPOINT_FILE_CHECKSUM_ENABLED.key -> false.toString,
+ SQLConf.STATE_STORE_MIN_DELTAS_FOR_SNAPSHOT.key -> "2"
+ ) {
+ withTempDir { dir =>
+ val remoteDir = dir.getCanonicalPath
+ val versionToUniqueId = mutable.Map[Long, String]()
+
+ // Build up state: 4 versions with snapshots at versions 2 and 4
+ withDB(remoteDir, enableStateStoreCheckpointIds = true,
+ versionToUniqueId = versionToUniqueId) { db =>
+ db.load(0)
+ db.put("a", "0")
+ db.commit()
+ assert(db.metricsOpt.get.numSnapshotsAutoRepaired == 0)
+
+ db.load(1)
+ db.put("b", "1")
+ db.commit() // snapshot is created
+ assert(db.metricsOpt.get.numSnapshotsAutoRepaired == 0)
+ db.doMaintenance() // upload snapshot 2_{uuid}.zip
+
+ db.load(2)
+ db.put("c", "2")
+ db.commit()
+ assert(db.metricsOpt.get.numSnapshotsAutoRepaired == 0)
+
+ db.load(3)
+ db.put("d", "3")
+ db.commit() // snapshot is created
+ assert(db.metricsOpt.get.numSnapshotsAutoRepaired == 0)
+ db.doMaintenance() // upload snapshot 4_{uuid}.zip
+ }
+
+ // Corrupt the latest V2 snapshot (version 4)
+ val uuid4 = versionToUniqueId(4)
+ corruptFile(new File(remoteDir, s"4_${uuid4}.zip"))
+
+ // Without auto-repair, this should fail
+ withDB(remoteDir, enableStateStoreCheckpointIds = true,
+ versionToUniqueId = versionToUniqueId) { db =>
+ val ex = intercept[java.nio.file.NoSuchFileException] {
+ db.load(4)
+ }
+ assert(ex.getMessage.contains("/metadata"))
+ }
+
+ // With auto-repair enabled, should succeed by falling back to snapshot at version 2
+ withSQLConf(SQLConf.STATE_STORE_AUTO_SNAPSHOT_REPAIR_ENABLED.key -> true.toString,
+ SQLConf.STATE_STORE_AUTO_SNAPSHOT_REPAIR_NUM_FAILURES_BEFORE_ACTIVATING.key -> "1",
+ SQLConf.STATE_STORE_AUTO_SNAPSHOT_REPAIR_MAX_CHANGE_FILE_REPLAY.key -> "5"
+ ) {
Review Comment:
nit: move to line above
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-56216][SS] Integrate checkpoint V2 with auto-repair snapshot [spark]
Posted by "anishshri-db (via GitHub)" <gi...@apache.org>.
anishshri-db commented on code in PR #55015:
URL: https://github.com/apache/spark/pull/55015#discussion_r3031448941
##########
sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/state/RocksDBFileManager.scala:
##########
@@ -455,14 +455,14 @@ class RocksDBFileManager(
/**
* Based on the ground truth lineage loaded from changelog file (lineage), this function
* does file listing to find all snapshot (version, uniqueId) pairs, and finds
- * the ground truth latest snapshot (version, uniqueId) the db instance needs to load.
+ * all snapshots that match the lineage, sorted by version descending.
*
* @param lineage The ground truth lineage loaded from changelog file, sorted by id
- * @return The ground truth latest snapshot (version, uniqueId) the db instance needs to load,
- * when the return value is None it means ther is no such snapshot found.
+ * @return All snapshot (version, uniqueId) pairs matching the lineage, sorted descending
+ * by version. Empty if no matching snapshots found.
*/
- def getLatestSnapshotVersionAndUniqueIdFromLineage(
- lineage: Array[LineageItem]): Option[(Long, String)] = {
+ def getSnapshotVersionsAndUniqueIdsFromLineage(
+ lineage: Array[LineageItem]): Seq[(Long, String)] = {
Review Comment:
Where is this lineage used ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-56216][SS] Integrate checkpoint V2 with auto-repair snapshot [spark]
Posted by "anishshri-db (via GitHub)" <gi...@apache.org>.
anishshri-db commented on code in PR #55015:
URL: https://github.com/apache/spark/pull/55015#discussion_r3031451090
##########
sql/core/src/test/scala/org/apache/spark/sql/execution/streaming/state/RocksDBSuite.scala:
##########
@@ -4166,6 +4162,341 @@ class RocksDBSuite extends AlsoTestWithRocksDBFeatures with SharedSparkSession
}
}
+ testWithChangelogCheckpointingEnabled(
+ "Auto snapshot repair with checkpoint format V2") {
+ withSQLConf(
+ SQLConf.STREAMING_CHECKPOINT_FILE_CHECKSUM_ENABLED.key -> false.toString,
+ SQLConf.STATE_STORE_MIN_DELTAS_FOR_SNAPSHOT.key -> "2"
+ ) {
+ withTempDir { dir =>
+ val remoteDir = dir.getCanonicalPath
+ val versionToUniqueId = mutable.Map[Long, String]()
+
+ // Build up state: 4 versions with snapshots at versions 2 and 4
+ withDB(remoteDir, enableStateStoreCheckpointIds = true,
+ versionToUniqueId = versionToUniqueId) { db =>
+ db.load(0)
+ db.put("a", "0")
+ db.commit()
+ assert(db.metricsOpt.get.numSnapshotsAutoRepaired == 0)
+
+ db.load(1)
+ db.put("b", "1")
+ db.commit() // snapshot is created
+ assert(db.metricsOpt.get.numSnapshotsAutoRepaired == 0)
+ db.doMaintenance() // upload snapshot 2_{uuid}.zip
+
+ db.load(2)
+ db.put("c", "2")
+ db.commit()
+ assert(db.metricsOpt.get.numSnapshotsAutoRepaired == 0)
+
+ db.load(3)
+ db.put("d", "3")
+ db.commit() // snapshot is created
+ assert(db.metricsOpt.get.numSnapshotsAutoRepaired == 0)
+ db.doMaintenance() // upload snapshot 4_{uuid}.zip
+ }
+
+ // Corrupt the latest V2 snapshot (version 4)
+ val uuid4 = versionToUniqueId(4)
+ corruptFile(new File(remoteDir, s"4_${uuid4}.zip"))
+
+ // Without auto-repair, this should fail
+ withDB(remoteDir, enableStateStoreCheckpointIds = true,
+ versionToUniqueId = versionToUniqueId) { db =>
+ val ex = intercept[java.nio.file.NoSuchFileException] {
+ db.load(4)
+ }
+ assert(ex.getMessage.contains("/metadata"))
+ }
+
+ // With auto-repair enabled, should succeed by falling back to snapshot at version 2
+ withSQLConf(SQLConf.STATE_STORE_AUTO_SNAPSHOT_REPAIR_ENABLED.key -> true.toString,
+ SQLConf.STATE_STORE_AUTO_SNAPSHOT_REPAIR_NUM_FAILURES_BEFORE_ACTIVATING.key -> "1",
+ SQLConf.STATE_STORE_AUTO_SNAPSHOT_REPAIR_MAX_CHANGE_FILE_REPLAY.key -> "5"
+ ) {
+ withDB(remoteDir, enableStateStoreCheckpointIds = true,
+ versionToUniqueId = versionToUniqueId) { db =>
+ db.load(4)
+ assert(toStr(db.get("a")) == "0")
+ assert(toStr(db.get("b")) == "1")
+ assert(toStr(db.get("c")) == "2")
+ assert(toStr(db.get("d")) == "3")
+ db.put("e", "4")
+ db.commit()
+ assert(db.metricsOpt.get.numSnapshotsAutoRepaired == 1)
+ assert(db.metricsOpt.get.loadedFromDfs === 1)
+ db.doMaintenance() // upload new snapshot
+ }
+
+ // Corrupt all V2 snapshot files - should fall back to version 0 + replay
+ val uuid2 = versionToUniqueId(2)
+ val uuid5 = versionToUniqueId(5)
+ corruptFile(new File(remoteDir, s"2_${uuid2}.zip"))
+ corruptFile(new File(remoteDir, s"5_${uuid5}.zip"))
+
+ withDB(remoteDir, enableStateStoreCheckpointIds = true,
+ versionToUniqueId = versionToUniqueId) { db =>
+ db.load(5)
+ assert(toStr(db.get("b")) == "1")
+ db.commit()
+ assert(db.metricsOpt.get.numSnapshotsAutoRepaired == 1)
+ assert(db.metricsOpt.get.loadedFromDfs === 1)
+ }
+ }
+ }
+ }
+ }
+
+ testWithChangelogCheckpointingEnabled(
+ "V2 auto-repair respects maxChangeFileReplay limit") {
+ withSQLConf(
+ SQLConf.STREAMING_CHECKPOINT_FILE_CHECKSUM_ENABLED.key -> false.toString,
+ SQLConf.STATE_STORE_MIN_DELTAS_FOR_SNAPSHOT.key -> "2",
+ SQLConf.STATE_STORE_AUTO_SNAPSHOT_REPAIR_ENABLED.key -> true.toString,
+ SQLConf.STATE_STORE_AUTO_SNAPSHOT_REPAIR_NUM_FAILURES_BEFORE_ACTIVATING.key -> "1"
+ ) {
Review Comment:
nit: move to line above ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SS] Integrate checkpoint V2 with auto-repair snapshot [spark]
Posted by "dongjoon-hyun (via GitHub)" <gi...@apache.org>.
dongjoon-hyun commented on PR #55015:
URL: https://github.com/apache/spark/pull/55015#issuecomment-4129753091
Thank you, @ericm-db . Please use a valid JIRA ID and convert to the normal PR.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org