You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by "ForVic (via GitHub)" <gi...@apache.org> on 2026/10/22 03:07:01 UTC

[PR] [SPARK-56093][CORE][HISTORY] Allow AppStatus to be cached and reused by the history server [spark]

ForVic opened a new pull request, #59696:
URL: https://github.com/apache/spark/pull/59696

   ### What changes were proposed in this pull request?
   
   This is a continuation of #54878, which was closed by the stale bot. It is rebased on the current master and addresses the review comments from that PR.
   
   When a Spark application completes, it writes out its AppStatus — the materialized state built by the `AppStatusListener` — as protobuf files to a shared path. When the History Server loads that application, it restores the UI from that snapshot instead of replaying the event log through a `ReplayListenerBus` and `AppStatusListener`. If there is no usable snapshot, the History Server replays the event log as before.
   
   - `spark.history.snapshot.enabled` / `spark.history.snapshot.path` must be set both for applications and for the History Server. Applications only write snapshots when `spark.eventLog.enabled` is true, since the History Server discovers applications through their event logs.
   - A snapshot is a data directory of length-delimited protobuf records (one file per KVStore entity type, including SQL, streaming, Thrift server and Connect UI data) plus a versioned manifest. The manifest is written under a temporary name and renamed into place, so readers only ever see a complete snapshot.
   - Snapshots are written with the same file permissions as event logs, and the application environment is redacted the same way it is in the event log.
   - Corrupt snapshots are deleted by the History Server and the UI falls back to replay. Snapshots that fail for other reasons (transient storage errors, or a snapshot written with a different schema/store version) are kept.
   - When the History Server cleaner is enabled, snapshots are deleted together with their event logs. Snapshots of unknown applications older than `spark.history.fs.cleaner.maxAge` are also deleted.
   
   Changes since #54878, addressing the review comments:
   - Log invalid snapshot imports as warnings, and log the duration when a snapshot write fails.
   - Validate record sizes against the remaining bytes in an entity file before allocating, so a corrupt size prefix fails as an invalid snapshot instead of an `OutOfMemoryError`.
   - Add `spark.history.snapshot.numWriteThreads` (default 1) to write entity files in parallel.
   - Turn `HistorySnapshotStore` into a class that reuses a single Hadoop `Configuration` and `FileSystem`.
   - Add snapshot retention tied to the event log cleaner, plus cleanup of orphaned snapshots.
   
   Other changes found while preparing this PR:
   - Redact the application environment and use event log file permissions for snapshots.
   - Reject manifests that reference paths outside of their snapshot directory.
   - Record the AppStatus store version in the manifest, and only delete snapshots that are actually corrupt.
   - Publish manifests atomically via rename, and clean up data directories left behind by failed writes.
   - Include Thrift server and Connect UI data in snapshots, and flush live SQL executions before the snapshot is written.
   - Remove the unconditional flush at application end, which changed behavior even when snapshots were disabled.
   - Remove the snapshot write metrics, since the driver metrics system is stopped before the status store is closed and they would never be reported.
   - Use structured logging, the `4.4.0` config version, and document the configs in `monitoring.md`.
   
   ### Why are the changes needed?
   
   The Spark History Server can be slow to load applications with large event logs, due to the deserialization overhead of replaying them. Replay repeats the exact work the driver already did while serving the live UI from the `AppStatusListener`'s state in the KVStore.
   
   ### Does this PR introduce _any_ user-facing change?
   
   Yes. It adds three configs, `spark.history.snapshot.enabled`, `spark.history.snapshot.path` and `spark.history.snapshot.numWriteThreads`, which are disabled by default. When enabled, the History Server UI loads faster for completed applications. We saw p90 History Server load times improve by about 10x and p99 by about 30x, with larger gains for larger event logs.
   
   ### How was this patch tested?
   
   New unit tests in `FsHistoryProviderSuite`, `AppStatusListenerSuite`, `HistoryServerDiskManagerSuite` and `SQLAppStatusListenerSuite`. They cover snapshot round trips, corrupt and incompatible snapshots, path validation, redaction, parallel writes and cleaner retention. An earlier version of this change also ran internally at scale across multiple History Server instances for several days.
   
   ### Was this patch authored or co-authored using generative AI tooling?
   
   Yes.
   Generated-by: GPT-5.4, Claude Opus 5.5
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [PR] [SPARK-56093][CORE][HISTORY] Allow AppStatus to be cached and reused by the history server [spark]

Posted by "sunchao (via GitHub)" <gi...@apache.org>.
sunchao commented on code in PR #59696:
URL: https://github.com/apache/spark/pull/59696#discussion_r4344405658


##########
core/src/main/scala/org/apache/spark/deploy/history/FsHistoryProvider.scala:
##########
@@ -1687,7 +1704,52 @@ private[history] class FsHistoryProvider(conf: SparkConf, clock: Clock)
     KVUtils.open(newStorePath, metadata, conf, live = false)
   }
 
-  private def createInMemoryStore(attempt: AttemptInfoWrapper): KVStore = {
+  private def createDiskStoreFromSnapshot(
+      dm: HistoryServerDiskManager,
+      appId: String,
+      attempt: AttemptInfoWrapper,
+      metadata: AppStatusStoreMetadata,
+      manifestPath: Path): KVStore = {
+    val lease = dm.leaseExact(snapshotStore.snapshotSize(manifestPath))
+    val startNs = System.nanoTime()
+    val newStorePath = try {
+      Utils.tryWithResource(KVUtils.open(lease.tmpPath, metadata, conf, live = false)) { store =>
+        snapshotStore.restoreSnapshot(store, manifestPath)
+      }
+      lease.commit(appId, attempt.info.attemptId)
+    } catch {
+      case e: Exception =>
+        lease.rollback()
+        throw e
+    }
+    val durationMs = TimeUnit.NANOSECONDS.toMillis(System.nanoTime() - startNs)
+    logInfo(log"Restored history snapshot ${MDC(PATH, manifestPath)} for " +
+      log"${MDC(APP_ID, appId)} into disk store in ${MDC(TOTAL_TIME, durationMs)} ms")
+
+    KVUtils.open(newStorePath, metadata, conf, live = false)
+  }
+
+  private def createInMemoryStore(appId: String, attempt: AttemptInfoWrapper): KVStore = {
+    snapshotStore.findSnapshot(appId, attempt.info.attemptId).foreach { snapshot =>
+      val store = new InMemoryStore()
+      store.setMetadata(new AppStatusStoreMetadata(AppStatusStore.CURRENT_VERSION))
+      val startNs = System.nanoTime()
+      try {
+        snapshotStore.restoreSnapshot(store, snapshot)
+        val durationMs = TimeUnit.NANOSECONDS.toMillis(System.nanoTime() - startNs)
+        logInfo(log"Restored history snapshot ${MDC(PATH, snapshot)} for " +
+          log"${MDC(APP_ID, appId)} into in-memory store in ${MDC(TOTAL_TIME, durationMs)} ms")
+        return store

Review Comment:
   [P2] Preserve streaming history when the live UI is disabled
   
   For an application with `spark.ui.enabled=false`, `SharedState.streamingQueryStatusListener` is never installed, although the core status store and this snapshot are still created. Streaming query events are still written to the event log, and normal replay installs `StreamingQueryStatusListener` through the History Server plugin, so those applications currently have Structured Streaming history. Returning the snapshot here (and in the disk path) skips that reconstruction: the snapshot has no streaming rows, and `setupUI()` omits the tab. A focused producer/replay probe showed `listener=false`, `eventLogged=true`, `liveRows=0`, and `replayRows=1`. Please collect streaming status when snapshots are enabled regardless of live-UI settings, or retain replay when that state was not captured.



##########
core/src/main/scala/org/apache/spark/deploy/history/HistorySnapshotStore.scala:
##########
@@ -0,0 +1,483 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.deploy.history
+
+import java.io.{BufferedInputStream, BufferedOutputStream, DataInputStream, DataOutputStream, IOException}
+import java.net.URLEncoder
+import java.nio.charset.StandardCharsets
+import java.util.{Properties, UUID}
+
+import scala.util.control.NonFatal
+
+import org.apache.hadoop.conf.Configuration
+import org.apache.hadoop.fs.{FileSystem, Path}
+
+import org.apache.spark.SparkConf
+import org.apache.spark.deploy.SparkHadoopUtil
+import org.apache.spark.internal.Logging
+import org.apache.spark.internal.LogKeys._
+import org.apache.spark.internal.config.History._
+import org.apache.spark.status._
+import org.apache.spark.status.api.v1
+import org.apache.spark.status.protobuf.KVStoreProtobufSerializer
+import org.apache.spark.util.Utils
+import org.apache.spark.util.kvstore.KVStore
+
+/**
+ * Writes and restores protobuf snapshots of an application's status store under
+ * `spark.history.snapshot.path`, so the History Server can load a completed application without
+ * replaying its event log.
+ */
+private[spark] class HistorySnapshotStore(conf: SparkConf, hadoopConf: Configuration)
+  extends Logging {
+
+  import HistorySnapshotStore._
+
+  def this(conf: SparkConf) = this(conf, SparkHadoopUtil.get.newConfiguration(conf))
+
+  private val configuredRoot: Option[Path] =
+    if (isEnabled(conf)) conf.get(SNAPSHOT_PATH).map(new Path(_)) else None
+
+  // Resolved lazily so a misconfigured snapshot path does not affect application startup.
+  private lazy val fs: FileSystem = configuredRoot.get.getFileSystem(hadoopConf)
+
+  private lazy val rootPath: Option[Path] = configuredRoot.map(fs.makeQualified)
+
+  private val serializer = new KVStoreProtobufSerializer()
+
+  /**
+   * Writes a versioned history snapshot for an application attempt and returns the manifest path.
+   *
+   * The snapshot is published under the configured snapshot root using a fresh manifest and data
+   * directory so readers can safely continue using the previous published snapshot while a new one
+   * is being written.
+   */
+  def writeSnapshot(store: KVStore, appId: String, attemptId: Option[String]): Option[Path] = {
+    snapshotDir(appId, attemptId).map { attemptDir =>
+      fs.mkdirs(attemptDir)
+      val snapshotId = newSnapshotId()
+      val dataDir = new Path(attemptDir, s"$SNAPSHOT_DIR_PREFIX$snapshotId")
+      val manifestPath = new Path(attemptDir, s"$MANIFEST_FILE_PREFIX$snapshotId.properties")
+
+      try {
+        fs.mkdirs(dataDir)
+        val entries = writeEntityFiles(store, dataDir, appId)
+
+        val manifest = new Properties()
+        manifest.setProperty("schemaVersion", SNAPSHOT_SCHEMA_VERSION.toString)
+        manifest.setProperty("snapshotDir", dataDir.getName)
+        manifest.setProperty("classNames", entries.map(_.className).mkString(","))
+        entries.foreach { entry =>
+          manifest.setProperty(s"${entry.className}.file", entry.fileName)
+        }
+
+        Utils.tryWithResource(fs.create(manifestPath, false)) { out =>
+          manifest.store(out, "spark history snapshot")
+        }
+        // Only older manifests are deleted, so readers always see either the old snapshot or the
+        // newly published one.
+        cleanupStaleSnapshots(attemptDir, manifestPath)
+        manifestPath
+      } catch {
+        case NonFatal(e) =>
+          deletePathQuietly(manifestPath, recursive = false)
+          deletePathQuietly(dataDir, recursive = true)
+          throw e
+      }
+    }
+  }
+
+  /**
+   * Returns the latest published history snapshot manifest for an application attempt, if any.
+   *
+   * When multiple manifests exist, the latest versioned manifest is treated as the active
+   * published snapshot.
+   */
+  def findSnapshot(appId: String, attemptId: Option[String]): Option[Path] = {
+    try {
+      snapshotDir(appId, attemptId).flatMap(latestManifest)
+    } catch {
+      case NonFatal(e) =>
+        logWarning(log"Failed to look up history snapshot for ${MDC(APP_ID, appId)}.", e)
+        None
+    }
+  }
+
+  /**
+   * Loads a published history snapshot into the provided KVStore.
+   *
+   * The manifest controls which entity files are restored and where the snapshot data directory is
+   * located relative to the manifest path.
+   */
+  def restoreSnapshot(store: KVStore, manifestPath: Path): Unit = {
+    val manifest = loadManifest(manifestPath)
+    val schemaVersion = manifest.getProperty("schemaVersion", "-1").toInt
+    require(schemaVersion == SNAPSHOT_SCHEMA_VERSION,
+      s"Unsupported history snapshot version $schemaVersion at $manifestPath")
+
+    val dataRoot = snapshotDataRoot(manifestPath, manifest)
+    manifestEntries(manifestPath, manifest).foreach { entry =>
+      loadSnapshotClass(manifestPath, entry.className).foreach { klass =>
+        readEntityFile(store, klass, new Path(dataRoot, entry.fileName))
+      }
+    }
+  }
+
+  /**
+   * Returns the total on-storage size of a published history snapshot, including its manifest.
+   *
+   * This is used by History Server disk-store allocation so the target local store can reserve
+   * enough space before restoring a snapshot.
+   */
+  def snapshotSize(manifestPath: Path): Long = {
+    val manifest = loadManifest(manifestPath)
+    val root = snapshotDataRoot(manifestPath, manifest)
+    manifestEntries(manifestPath, manifest)
+      .foldLeft(fs.getFileStatus(manifestPath).getLen) { (total, entry) =>
+        total + fs.getFileStatus(new Path(root, entry.fileName)).getLen
+      }
+  }
+
+  /**
+   * Deletes all snapshots of an application attempt. The History Server calls this when it cleans
+   * up the attempt's event log, so snapshots do not outlive their event logs.
+   */
+  def deleteSnapshots(appId: String, attemptId: Option[String]): Unit = {
+    Utils.tryLogNonFatalError {
+      snapshotDir(appId, attemptId).foreach { attemptDir =>
+        deletePathQuietly(attemptDir, recursive = true)
+        val appDir = attemptDir.getParent
+        if (fs.exists(appDir) && fs.listStatus(appDir).isEmpty) {
+          fs.delete(appDir, false)
+        }
+      }
+    }
+  }
+
+  private def snapshotDir(appId: String, attemptId: Option[String]): Option[Path] = {
+    rootPath.map { root =>
+      new Path(new Path(root, encode(appId)), encode(attemptId.getOrElse("_default_")))

Review Comment:
   [P1] Constrain snapshot cleanup to the intended attempt directory
   
   `URLEncoder` leaves `.` and `..` unchanged, and Hadoop `Path` normalizes them: an ordinary app ID with attempt ID `..` resolves to the entire snapshot root, while app ID `..` with attempt ID `.` resolves to its parent. These identifiers are copied directly from event logs by `AppListingListener`, and the cleaner calls `deleteSnapshots()` even when that application never wrote a snapshot. Its recursive delete can therefore remove other applications' snapshots or unrelated data with the History Server's permissions. Please reject these components and validate containment before using the path, especially for deletion. A focused Hadoop Path probe confirmed both resolutions; the existing disk-cache path's sanitization and canonical-parent check keep the same inputs contained.



##########
core/src/main/scala/org/apache/spark/deploy/history/HistorySnapshotStore.scala:
##########
@@ -0,0 +1,483 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.deploy.history
+
+import java.io.{BufferedInputStream, BufferedOutputStream, DataInputStream, DataOutputStream, IOException}
+import java.net.URLEncoder
+import java.nio.charset.StandardCharsets
+import java.util.{Properties, UUID}
+
+import scala.util.control.NonFatal
+
+import org.apache.hadoop.conf.Configuration
+import org.apache.hadoop.fs.{FileSystem, Path}
+
+import org.apache.spark.SparkConf
+import org.apache.spark.deploy.SparkHadoopUtil
+import org.apache.spark.internal.Logging
+import org.apache.spark.internal.LogKeys._
+import org.apache.spark.internal.config.History._
+import org.apache.spark.status._
+import org.apache.spark.status.api.v1
+import org.apache.spark.status.protobuf.KVStoreProtobufSerializer
+import org.apache.spark.util.Utils
+import org.apache.spark.util.kvstore.KVStore
+
+/**
+ * Writes and restores protobuf snapshots of an application's status store under
+ * `spark.history.snapshot.path`, so the History Server can load a completed application without
+ * replaying its event log.
+ */
+private[spark] class HistorySnapshotStore(conf: SparkConf, hadoopConf: Configuration)
+  extends Logging {
+
+  import HistorySnapshotStore._
+
+  def this(conf: SparkConf) = this(conf, SparkHadoopUtil.get.newConfiguration(conf))
+
+  private val configuredRoot: Option[Path] =
+    if (isEnabled(conf)) conf.get(SNAPSHOT_PATH).map(new Path(_)) else None
+
+  // Resolved lazily so a misconfigured snapshot path does not affect application startup.
+  private lazy val fs: FileSystem = configuredRoot.get.getFileSystem(hadoopConf)
+
+  private lazy val rootPath: Option[Path] = configuredRoot.map(fs.makeQualified)
+
+  private val serializer = new KVStoreProtobufSerializer()
+
+  /**
+   * Writes a versioned history snapshot for an application attempt and returns the manifest path.
+   *
+   * The snapshot is published under the configured snapshot root using a fresh manifest and data
+   * directory so readers can safely continue using the previous published snapshot while a new one
+   * is being written.
+   */
+  def writeSnapshot(store: KVStore, appId: String, attemptId: Option[String]): Option[Path] = {
+    snapshotDir(appId, attemptId).map { attemptDir =>
+      fs.mkdirs(attemptDir)
+      val snapshotId = newSnapshotId()
+      val dataDir = new Path(attemptDir, s"$SNAPSHOT_DIR_PREFIX$snapshotId")
+      val manifestPath = new Path(attemptDir, s"$MANIFEST_FILE_PREFIX$snapshotId.properties")
+
+      try {
+        fs.mkdirs(dataDir)
+        val entries = writeEntityFiles(store, dataDir, appId)
+
+        val manifest = new Properties()
+        manifest.setProperty("schemaVersion", SNAPSHOT_SCHEMA_VERSION.toString)
+        manifest.setProperty("snapshotDir", dataDir.getName)
+        manifest.setProperty("classNames", entries.map(_.className).mkString(","))
+        entries.foreach { entry =>
+          manifest.setProperty(s"${entry.className}.file", entry.fileName)
+        }
+
+        Utils.tryWithResource(fs.create(manifestPath, false)) { out =>
+          manifest.store(out, "spark history snapshot")
+        }
+        // Only older manifests are deleted, so readers always see either the old snapshot or the
+        // newly published one.
+        cleanupStaleSnapshots(attemptDir, manifestPath)
+        manifestPath
+      } catch {
+        case NonFatal(e) =>
+          deletePathQuietly(manifestPath, recursive = false)
+          deletePathQuietly(dataDir, recursive = true)
+          throw e
+      }
+    }
+  }
+
+  /**
+   * Returns the latest published history snapshot manifest for an application attempt, if any.
+   *
+   * When multiple manifests exist, the latest versioned manifest is treated as the active
+   * published snapshot.
+   */
+  def findSnapshot(appId: String, attemptId: Option[String]): Option[Path] = {
+    try {
+      snapshotDir(appId, attemptId).flatMap(latestManifest)
+    } catch {
+      case NonFatal(e) =>
+        logWarning(log"Failed to look up history snapshot for ${MDC(APP_ID, appId)}.", e)
+        None
+    }
+  }
+
+  /**
+   * Loads a published history snapshot into the provided KVStore.
+   *
+   * The manifest controls which entity files are restored and where the snapshot data directory is
+   * located relative to the manifest path.
+   */
+  def restoreSnapshot(store: KVStore, manifestPath: Path): Unit = {
+    val manifest = loadManifest(manifestPath)
+    val schemaVersion = manifest.getProperty("schemaVersion", "-1").toInt
+    require(schemaVersion == SNAPSHOT_SCHEMA_VERSION,
+      s"Unsupported history snapshot version $schemaVersion at $manifestPath")
+
+    val dataRoot = snapshotDataRoot(manifestPath, manifest)
+    manifestEntries(manifestPath, manifest).foreach { entry =>
+      loadSnapshotClass(manifestPath, entry.className).foreach { klass =>
+        readEntityFile(store, klass, new Path(dataRoot, entry.fileName))
+      }
+    }
+  }
+
+  /**
+   * Returns the total on-storage size of a published history snapshot, including its manifest.
+   *
+   * This is used by History Server disk-store allocation so the target local store can reserve
+   * enough space before restoring a snapshot.
+   */
+  def snapshotSize(manifestPath: Path): Long = {
+    val manifest = loadManifest(manifestPath)
+    val root = snapshotDataRoot(manifestPath, manifest)
+    manifestEntries(manifestPath, manifest)
+      .foldLeft(fs.getFileStatus(manifestPath).getLen) { (total, entry) =>
+        total + fs.getFileStatus(new Path(root, entry.fileName)).getLen
+      }
+  }
+
+  /**
+   * Deletes all snapshots of an application attempt. The History Server calls this when it cleans
+   * up the attempt's event log, so snapshots do not outlive their event logs.
+   */
+  def deleteSnapshots(appId: String, attemptId: Option[String]): Unit = {
+    Utils.tryLogNonFatalError {
+      snapshotDir(appId, attemptId).foreach { attemptDir =>
+        deletePathQuietly(attemptDir, recursive = true)
+        val appDir = attemptDir.getParent
+        if (fs.exists(appDir) && fs.listStatus(appDir).isEmpty) {
+          fs.delete(appDir, false)
+        }
+      }
+    }
+  }
+
+  private def snapshotDir(appId: String, attemptId: Option[String]): Option[Path] = {
+    rootPath.map { root =>
+      new Path(new Path(root, encode(appId)), encode(attemptId.getOrElse("_default_")))
+    }
+  }
+
+  /** Reads and parses a snapshot manifest file. */
+  private def loadManifest(manifestPath: Path): Properties = {
+    val properties = new Properties()
+    Utils.tryWithResource(fs.open(manifestPath)) { in =>
+      properties.load(in)
+    }
+    properties
+  }
+
+  /** Returns the newest published manifest under a snapshot root, ordered by manifest name. */
+  private def latestManifest(rootPath: Path): Option[Path] = {
+    if (!fs.exists(rootPath)) {
+      None
+    } else {
+      fs.listStatus(rootPath).map(_.getPath).filter(isVersionedManifest).sortBy(_.getName)
+        .lastOption
+    }
+  }
+
+  /** Removes stale manifests and data only when this writer still owns the latest publication. */
+  private def cleanupStaleSnapshots(rootPath: Path, keepManifestPath: Path): Unit = {
+    try {
+      // A concurrent writer may have published a newer manifest after this write completed.
+      // In that case, avoid deleting anything and let the newer publication own cleanup.
+      if (latestManifest(rootPath).exists(_ != keepManifestPath)) {
+        logWarning(log"Skipping cleanup for history snapshot root ${MDC(PATH, rootPath)} " +
+          log"because another manifest became active while publishing " +
+          log"${MDC(PATHS, keepManifestPath)}.")
+      } else {
+        val manifests = fs.listStatus(rootPath).map(_.getPath).toSeq
+        staleManifestsForCleanup(manifests, keepManifestPath).foreach(deleteSnapshot)
+      }
+    } catch {
+      case NonFatal(e) =>
+        logWarning(log"Failed to clean up stale history snapshots under ${MDC(PATH, rootPath)}.",
+          e)
+    }
+  }
+
+  /** Deletes a manifest and its associated snapshot artifacts, tolerating partial corruption. */
+  private def deleteSnapshot(manifestPath: Path): Unit = {
+    try {
+      deletePathQuietly(snapshotDataRoot(manifestPath, loadManifest(manifestPath)),
+        recursive = true)
+    } catch {
+      case NonFatal(e) =>
+        logWarning(log"Failed to load history snapshot manifest at " +
+          log"${MDC(PATH, manifestPath)} for deletion.", e)
+    }
+    deletePathQuietly(manifestPath, recursive = false)
+  }
+
+  private def deletePathQuietly(path: Path, recursive: Boolean): Unit = {
+    Utils.tryLogNonFatalError {
+      if (fs.exists(path)) {
+        fs.delete(path, recursive)
+      }
+    }
+  }
+
+  /** Writes one file per entity type into the snapshot data directory. */
+  private def writeEntityFiles(
+      store: KVStore,
+      dataDir: Path,
+      appId: String): Seq[SnapshotEntry] = {
+    snapshotEntityClasses.zipWithIndex.flatMap { case (klass, idx) =>
+      val fileName = f"$idx%03d-${klass.getSimpleName}.pb"
+      val path = new Path(dataDir, fileName)
+      if (writeEntityFile(store, klass, path, appId) > 0L) {
+        Some(SnapshotEntry(klass.getName, fileName))
+      } else {
+        fs.delete(path, false)
+        None
+      }
+    }
+  }
+
+  /** Writes all records for one entity type into a single snapshot file. */
+  private def writeEntityFile(
+      store: KVStore,
+      klass: Class[_],
+      path: Path,
+      appId: String): Long = {
+    singletonEntity(store, klass, appId)
+      .map { value =>
+        writeValuesFile(path, Iterator.single(redact(value)))
+      }
+      .getOrElse {
+        Utils.tryWithResource(store.view(klass.asInstanceOf[Class[AnyRef]]).closeableIterator()) {
+          it =>
+            writeValuesFile(path,
+              Iterator.continually(it).takeWhile(_.hasNext).map(_.next().asInstanceOf[AnyRef]))
+        }
+      }
+  }
+
+  /** Serializes a stream of KVStore values using length-delimited protobuf records. */
+  private def writeValuesFile(path: Path, values: Iterator[AnyRef]): Long = {
+    var count = 0L
+    Utils.tryWithResource(new DataOutputStream(new BufferedOutputStream(fs.create(path, true)))) {

Review Comment:
   [P2] Preserve the event log's access restrictions on snapshot files
   
   On a shared filesystem with the default Hadoop umask, these `fs.create` calls produce `0644` files and the new directories are `0755`. The existing event-log writer explicitly sets `0660`, consistent with the shared-directory security guidance. Enabling snapshots therefore makes another copy of application details, SQL text, and task information readable by filesystem users who cannot read the corresponding event log. A Hadoop filesystem probe produced `0644` for the snapshot call versus `0660` for the event-log control. Please apply the event-log permission policy to the snapshot artifacts before publishing them; deferring this changes the protection of the same application data.



##########
core/src/main/scala/org/apache/spark/status/AppStatusListener.scala:
##########
@@ -103,12 +108,18 @@ private[spark] class AppStatusListener(
   }
 
   kvstore.onFlush {
-    if (!live) {
+    if (!live || historySnapshotStore.isDefined) {
       val now = System.nanoTime()
       flush(update(_, now))
     }
   }
 
+  historySnapshotStore.foreach { snapshotStore =>
+    kvstore.onClose {
+      writeHistorySnapshot(snapshotStore)

Review Comment:
   [P2] Flush remaining live SQL metrics before writing the snapshot
   
   The core listener now flushes live state, but `SQLAppStatusListener.onFlush` still aggregates remaining executions only when `!live`. If the SparkContext stops while a SQL execution is active, the application-end event can be recorded before the SQL-execution-end event, and the stopped listener bus drops that later event. Replay preserves the partial metrics by flushing with `live=false`; this snapshot instead copies a SQL record with null `metricValues`, restores successfully, and shows empty metrics. With identical execution-start, driver-metric-update (`42`), and application-end events, a focused probe retained `42` after replay but returned an empty metric map after live-store close. Please flush the SQL listener's remaining state before publication or fall back to replay for this case.



##########
core/src/main/scala/org/apache/spark/deploy/history/FsHistoryProvider.scala:
##########
@@ -1687,7 +1704,52 @@ private[history] class FsHistoryProvider(conf: SparkConf, clock: Clock)
     KVUtils.open(newStorePath, metadata, conf, live = false)
   }
 
-  private def createInMemoryStore(attempt: AttemptInfoWrapper): KVStore = {
+  private def createDiskStoreFromSnapshot(
+      dm: HistoryServerDiskManager,
+      appId: String,
+      attempt: AttemptInfoWrapper,
+      metadata: AppStatusStoreMetadata,
+      manifestPath: Path): KVStore = {
+    val lease = dm.leaseExact(snapshotStore.snapshotSize(manifestPath))
+    val startNs = System.nanoTime()
+    val newStorePath = try {
+      Utils.tryWithResource(KVUtils.open(lease.tmpPath, metadata, conf, live = false)) { store =>
+        snapshotStore.restoreSnapshot(store, manifestPath)
+      }
+      lease.commit(appId, attempt.info.attemptId)

Review Comment:
   [P2] Validate required application data before accepting the restored store
   
   If the `ApplicationInfoWrapper` entity file is truncated to zero bytes, `restoreSnapshot()` returns successfully without restoring an application record. This path then commits the unusable store; the memory path likewise returns it. `getAppUI()` subsequently constructs `HistoryAppStatusStore`, whose constructor immediately calls `applicationInfo()` and throws `NoSuchElementException` outside the snapshot fallback handler. The UI fails despite a valid event log, and the disk path caches the failure for later requests. An isolated snapshot round trip confirmed that the valid control loads, whereas the empty entity is accepted and the next application-info read fails. Please validate the expected application/attempt before committing or returning either store, so this corruption triggers replay.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [PR] [SPARK-56093][CORE][HISTORY] Allow AppStatus to be cached and reused by the history server [spark]

Posted by "ForVic (via GitHub)" <gi...@apache.org>.
ForVic commented on code in PR #59696:
URL: https://github.com/apache/spark/pull/59696#discussion_r4345546984


##########
core/src/main/scala/org/apache/spark/deploy/history/HistorySnapshotStore.scala:
##########
@@ -0,0 +1,483 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.deploy.history
+
+import java.io.{BufferedInputStream, BufferedOutputStream, DataInputStream, DataOutputStream, IOException}
+import java.net.URLEncoder
+import java.nio.charset.StandardCharsets
+import java.util.{Properties, UUID}
+
+import scala.util.control.NonFatal
+
+import org.apache.hadoop.conf.Configuration
+import org.apache.hadoop.fs.{FileSystem, Path}
+
+import org.apache.spark.SparkConf
+import org.apache.spark.deploy.SparkHadoopUtil
+import org.apache.spark.internal.Logging
+import org.apache.spark.internal.LogKeys._
+import org.apache.spark.internal.config.History._
+import org.apache.spark.status._
+import org.apache.spark.status.api.v1
+import org.apache.spark.status.protobuf.KVStoreProtobufSerializer
+import org.apache.spark.util.Utils
+import org.apache.spark.util.kvstore.KVStore
+
+/**
+ * Writes and restores protobuf snapshots of an application's status store under
+ * `spark.history.snapshot.path`, so the History Server can load a completed application without
+ * replaying its event log.
+ */
+private[spark] class HistorySnapshotStore(conf: SparkConf, hadoopConf: Configuration)
+  extends Logging {
+
+  import HistorySnapshotStore._
+
+  def this(conf: SparkConf) = this(conf, SparkHadoopUtil.get.newConfiguration(conf))
+
+  private val configuredRoot: Option[Path] =
+    if (isEnabled(conf)) conf.get(SNAPSHOT_PATH).map(new Path(_)) else None
+
+  // Resolved lazily so a misconfigured snapshot path does not affect application startup.
+  private lazy val fs: FileSystem = configuredRoot.get.getFileSystem(hadoopConf)
+
+  private lazy val rootPath: Option[Path] = configuredRoot.map(fs.makeQualified)
+
+  private val serializer = new KVStoreProtobufSerializer()
+
+  /**
+   * Writes a versioned history snapshot for an application attempt and returns the manifest path.
+   *
+   * The snapshot is published under the configured snapshot root using a fresh manifest and data
+   * directory so readers can safely continue using the previous published snapshot while a new one
+   * is being written.
+   */
+  def writeSnapshot(store: KVStore, appId: String, attemptId: Option[String]): Option[Path] = {
+    snapshotDir(appId, attemptId).map { attemptDir =>
+      fs.mkdirs(attemptDir)
+      val snapshotId = newSnapshotId()
+      val dataDir = new Path(attemptDir, s"$SNAPSHOT_DIR_PREFIX$snapshotId")
+      val manifestPath = new Path(attemptDir, s"$MANIFEST_FILE_PREFIX$snapshotId.properties")
+
+      try {
+        fs.mkdirs(dataDir)
+        val entries = writeEntityFiles(store, dataDir, appId)
+
+        val manifest = new Properties()
+        manifest.setProperty("schemaVersion", SNAPSHOT_SCHEMA_VERSION.toString)
+        manifest.setProperty("snapshotDir", dataDir.getName)
+        manifest.setProperty("classNames", entries.map(_.className).mkString(","))
+        entries.foreach { entry =>
+          manifest.setProperty(s"${entry.className}.file", entry.fileName)
+        }
+
+        Utils.tryWithResource(fs.create(manifestPath, false)) { out =>
+          manifest.store(out, "spark history snapshot")
+        }
+        // Only older manifests are deleted, so readers always see either the old snapshot or the
+        // newly published one.
+        cleanupStaleSnapshots(attemptDir, manifestPath)
+        manifestPath
+      } catch {
+        case NonFatal(e) =>
+          deletePathQuietly(manifestPath, recursive = false)
+          deletePathQuietly(dataDir, recursive = true)
+          throw e
+      }
+    }
+  }
+
+  /**
+   * Returns the latest published history snapshot manifest for an application attempt, if any.
+   *
+   * When multiple manifests exist, the latest versioned manifest is treated as the active
+   * published snapshot.
+   */
+  def findSnapshot(appId: String, attemptId: Option[String]): Option[Path] = {
+    try {
+      snapshotDir(appId, attemptId).flatMap(latestManifest)
+    } catch {
+      case NonFatal(e) =>
+        logWarning(log"Failed to look up history snapshot for ${MDC(APP_ID, appId)}.", e)
+        None
+    }
+  }
+
+  /**
+   * Loads a published history snapshot into the provided KVStore.
+   *
+   * The manifest controls which entity files are restored and where the snapshot data directory is
+   * located relative to the manifest path.
+   */
+  def restoreSnapshot(store: KVStore, manifestPath: Path): Unit = {
+    val manifest = loadManifest(manifestPath)
+    val schemaVersion = manifest.getProperty("schemaVersion", "-1").toInt
+    require(schemaVersion == SNAPSHOT_SCHEMA_VERSION,
+      s"Unsupported history snapshot version $schemaVersion at $manifestPath")
+
+    val dataRoot = snapshotDataRoot(manifestPath, manifest)
+    manifestEntries(manifestPath, manifest).foreach { entry =>
+      loadSnapshotClass(manifestPath, entry.className).foreach { klass =>
+        readEntityFile(store, klass, new Path(dataRoot, entry.fileName))
+      }
+    }
+  }
+
+  /**
+   * Returns the total on-storage size of a published history snapshot, including its manifest.
+   *
+   * This is used by History Server disk-store allocation so the target local store can reserve
+   * enough space before restoring a snapshot.
+   */
+  def snapshotSize(manifestPath: Path): Long = {
+    val manifest = loadManifest(manifestPath)
+    val root = snapshotDataRoot(manifestPath, manifest)
+    manifestEntries(manifestPath, manifest)
+      .foldLeft(fs.getFileStatus(manifestPath).getLen) { (total, entry) =>
+        total + fs.getFileStatus(new Path(root, entry.fileName)).getLen
+      }
+  }
+
+  /**
+   * Deletes all snapshots of an application attempt. The History Server calls this when it cleans
+   * up the attempt's event log, so snapshots do not outlive their event logs.
+   */
+  def deleteSnapshots(appId: String, attemptId: Option[String]): Unit = {
+    Utils.tryLogNonFatalError {
+      snapshotDir(appId, attemptId).foreach { attemptDir =>
+        deletePathQuietly(attemptDir, recursive = true)
+        val appDir = attemptDir.getParent
+        if (fs.exists(appDir) && fs.listStatus(appDir).isEmpty) {
+          fs.delete(appDir, false)
+        }
+      }
+    }
+  }
+
+  private def snapshotDir(appId: String, attemptId: Option[String]): Option[Path] = {
+    rootPath.map { root =>
+      new Path(new Path(root, encode(appId)), encode(attemptId.getOrElse("_default_")))

Review Comment:
   Good catch, thanks. The snapshot dir is now validated to be exactly `<root>/<app>/<attempt>` (same approach as `HistoryServerDiskManager.appStorePath`), so `.`/`..` IDs are rejected before any read or delete. Added a test with both cases. Fixed in e7eb0e1093fa.



##########
core/src/main/scala/org/apache/spark/deploy/history/HistorySnapshotStore.scala:
##########
@@ -0,0 +1,483 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.deploy.history
+
+import java.io.{BufferedInputStream, BufferedOutputStream, DataInputStream, DataOutputStream, IOException}
+import java.net.URLEncoder
+import java.nio.charset.StandardCharsets
+import java.util.{Properties, UUID}
+
+import scala.util.control.NonFatal
+
+import org.apache.hadoop.conf.Configuration
+import org.apache.hadoop.fs.{FileSystem, Path}
+
+import org.apache.spark.SparkConf
+import org.apache.spark.deploy.SparkHadoopUtil
+import org.apache.spark.internal.Logging
+import org.apache.spark.internal.LogKeys._
+import org.apache.spark.internal.config.History._
+import org.apache.spark.status._
+import org.apache.spark.status.api.v1
+import org.apache.spark.status.protobuf.KVStoreProtobufSerializer
+import org.apache.spark.util.Utils
+import org.apache.spark.util.kvstore.KVStore
+
+/**
+ * Writes and restores protobuf snapshots of an application's status store under
+ * `spark.history.snapshot.path`, so the History Server can load a completed application without
+ * replaying its event log.
+ */
+private[spark] class HistorySnapshotStore(conf: SparkConf, hadoopConf: Configuration)
+  extends Logging {
+
+  import HistorySnapshotStore._
+
+  def this(conf: SparkConf) = this(conf, SparkHadoopUtil.get.newConfiguration(conf))
+
+  private val configuredRoot: Option[Path] =
+    if (isEnabled(conf)) conf.get(SNAPSHOT_PATH).map(new Path(_)) else None
+
+  // Resolved lazily so a misconfigured snapshot path does not affect application startup.
+  private lazy val fs: FileSystem = configuredRoot.get.getFileSystem(hadoopConf)
+
+  private lazy val rootPath: Option[Path] = configuredRoot.map(fs.makeQualified)
+
+  private val serializer = new KVStoreProtobufSerializer()
+
+  /**
+   * Writes a versioned history snapshot for an application attempt and returns the manifest path.
+   *
+   * The snapshot is published under the configured snapshot root using a fresh manifest and data
+   * directory so readers can safely continue using the previous published snapshot while a new one
+   * is being written.
+   */
+  def writeSnapshot(store: KVStore, appId: String, attemptId: Option[String]): Option[Path] = {
+    snapshotDir(appId, attemptId).map { attemptDir =>
+      fs.mkdirs(attemptDir)
+      val snapshotId = newSnapshotId()
+      val dataDir = new Path(attemptDir, s"$SNAPSHOT_DIR_PREFIX$snapshotId")
+      val manifestPath = new Path(attemptDir, s"$MANIFEST_FILE_PREFIX$snapshotId.properties")
+
+      try {
+        fs.mkdirs(dataDir)
+        val entries = writeEntityFiles(store, dataDir, appId)
+
+        val manifest = new Properties()
+        manifest.setProperty("schemaVersion", SNAPSHOT_SCHEMA_VERSION.toString)
+        manifest.setProperty("snapshotDir", dataDir.getName)
+        manifest.setProperty("classNames", entries.map(_.className).mkString(","))
+        entries.foreach { entry =>
+          manifest.setProperty(s"${entry.className}.file", entry.fileName)
+        }
+
+        Utils.tryWithResource(fs.create(manifestPath, false)) { out =>
+          manifest.store(out, "spark history snapshot")
+        }
+        // Only older manifests are deleted, so readers always see either the old snapshot or the
+        // newly published one.
+        cleanupStaleSnapshots(attemptDir, manifestPath)
+        manifestPath
+      } catch {
+        case NonFatal(e) =>
+          deletePathQuietly(manifestPath, recursive = false)
+          deletePathQuietly(dataDir, recursive = true)
+          throw e
+      }
+    }
+  }
+
+  /**
+   * Returns the latest published history snapshot manifest for an application attempt, if any.
+   *
+   * When multiple manifests exist, the latest versioned manifest is treated as the active
+   * published snapshot.
+   */
+  def findSnapshot(appId: String, attemptId: Option[String]): Option[Path] = {
+    try {
+      snapshotDir(appId, attemptId).flatMap(latestManifest)
+    } catch {
+      case NonFatal(e) =>
+        logWarning(log"Failed to look up history snapshot for ${MDC(APP_ID, appId)}.", e)
+        None
+    }
+  }
+
+  /**
+   * Loads a published history snapshot into the provided KVStore.
+   *
+   * The manifest controls which entity files are restored and where the snapshot data directory is
+   * located relative to the manifest path.
+   */
+  def restoreSnapshot(store: KVStore, manifestPath: Path): Unit = {
+    val manifest = loadManifest(manifestPath)
+    val schemaVersion = manifest.getProperty("schemaVersion", "-1").toInt
+    require(schemaVersion == SNAPSHOT_SCHEMA_VERSION,
+      s"Unsupported history snapshot version $schemaVersion at $manifestPath")
+
+    val dataRoot = snapshotDataRoot(manifestPath, manifest)
+    manifestEntries(manifestPath, manifest).foreach { entry =>
+      loadSnapshotClass(manifestPath, entry.className).foreach { klass =>
+        readEntityFile(store, klass, new Path(dataRoot, entry.fileName))
+      }
+    }
+  }
+
+  /**
+   * Returns the total on-storage size of a published history snapshot, including its manifest.
+   *
+   * This is used by History Server disk-store allocation so the target local store can reserve
+   * enough space before restoring a snapshot.
+   */
+  def snapshotSize(manifestPath: Path): Long = {
+    val manifest = loadManifest(manifestPath)
+    val root = snapshotDataRoot(manifestPath, manifest)
+    manifestEntries(manifestPath, manifest)
+      .foldLeft(fs.getFileStatus(manifestPath).getLen) { (total, entry) =>
+        total + fs.getFileStatus(new Path(root, entry.fileName)).getLen
+      }
+  }
+
+  /**
+   * Deletes all snapshots of an application attempt. The History Server calls this when it cleans
+   * up the attempt's event log, so snapshots do not outlive their event logs.
+   */
+  def deleteSnapshots(appId: String, attemptId: Option[String]): Unit = {
+    Utils.tryLogNonFatalError {
+      snapshotDir(appId, attemptId).foreach { attemptDir =>
+        deletePathQuietly(attemptDir, recursive = true)
+        val appDir = attemptDir.getParent
+        if (fs.exists(appDir) && fs.listStatus(appDir).isEmpty) {
+          fs.delete(appDir, false)
+        }
+      }
+    }
+  }
+
+  private def snapshotDir(appId: String, attemptId: Option[String]): Option[Path] = {
+    rootPath.map { root =>
+      new Path(new Path(root, encode(appId)), encode(attemptId.getOrElse("_default_")))
+    }
+  }
+
+  /** Reads and parses a snapshot manifest file. */
+  private def loadManifest(manifestPath: Path): Properties = {
+    val properties = new Properties()
+    Utils.tryWithResource(fs.open(manifestPath)) { in =>
+      properties.load(in)
+    }
+    properties
+  }
+
+  /** Returns the newest published manifest under a snapshot root, ordered by manifest name. */
+  private def latestManifest(rootPath: Path): Option[Path] = {
+    if (!fs.exists(rootPath)) {
+      None
+    } else {
+      fs.listStatus(rootPath).map(_.getPath).filter(isVersionedManifest).sortBy(_.getName)
+        .lastOption
+    }
+  }
+
+  /** Removes stale manifests and data only when this writer still owns the latest publication. */
+  private def cleanupStaleSnapshots(rootPath: Path, keepManifestPath: Path): Unit = {
+    try {
+      // A concurrent writer may have published a newer manifest after this write completed.
+      // In that case, avoid deleting anything and let the newer publication own cleanup.
+      if (latestManifest(rootPath).exists(_ != keepManifestPath)) {
+        logWarning(log"Skipping cleanup for history snapshot root ${MDC(PATH, rootPath)} " +
+          log"because another manifest became active while publishing " +
+          log"${MDC(PATHS, keepManifestPath)}.")
+      } else {
+        val manifests = fs.listStatus(rootPath).map(_.getPath).toSeq
+        staleManifestsForCleanup(manifests, keepManifestPath).foreach(deleteSnapshot)
+      }
+    } catch {
+      case NonFatal(e) =>
+        logWarning(log"Failed to clean up stale history snapshots under ${MDC(PATH, rootPath)}.",
+          e)
+    }
+  }
+
+  /** Deletes a manifest and its associated snapshot artifacts, tolerating partial corruption. */
+  private def deleteSnapshot(manifestPath: Path): Unit = {
+    try {
+      deletePathQuietly(snapshotDataRoot(manifestPath, loadManifest(manifestPath)),
+        recursive = true)
+    } catch {
+      case NonFatal(e) =>
+        logWarning(log"Failed to load history snapshot manifest at " +
+          log"${MDC(PATH, manifestPath)} for deletion.", e)
+    }
+    deletePathQuietly(manifestPath, recursive = false)
+  }
+
+  private def deletePathQuietly(path: Path, recursive: Boolean): Unit = {
+    Utils.tryLogNonFatalError {
+      if (fs.exists(path)) {
+        fs.delete(path, recursive)
+      }
+    }
+  }
+
+  /** Writes one file per entity type into the snapshot data directory. */
+  private def writeEntityFiles(
+      store: KVStore,
+      dataDir: Path,
+      appId: String): Seq[SnapshotEntry] = {
+    snapshotEntityClasses.zipWithIndex.flatMap { case (klass, idx) =>
+      val fileName = f"$idx%03d-${klass.getSimpleName}.pb"
+      val path = new Path(dataDir, fileName)
+      if (writeEntityFile(store, klass, path, appId) > 0L) {
+        Some(SnapshotEntry(klass.getName, fileName))
+      } else {
+        fs.delete(path, false)
+        None
+      }
+    }
+  }
+
+  /** Writes all records for one entity type into a single snapshot file. */
+  private def writeEntityFile(
+      store: KVStore,
+      klass: Class[_],
+      path: Path,
+      appId: String): Long = {
+    singletonEntity(store, klass, appId)
+      .map { value =>
+        writeValuesFile(path, Iterator.single(redact(value)))
+      }
+      .getOrElse {
+        Utils.tryWithResource(store.view(klass.asInstanceOf[Class[AnyRef]]).closeableIterator()) {
+          it =>
+            writeValuesFile(path,
+              Iterator.continually(it).takeWhile(_.hasNext).map(_.next().asInstanceOf[AnyRef]))
+        }
+      }
+  }
+
+  /** Serializes a stream of KVStore values using length-delimited protobuf records. */
+  private def writeValuesFile(path: Path, values: Iterator[AnyRef]): Long = {
+    var count = 0L
+    Utils.tryWithResource(new DataOutputStream(new BufferedOutputStream(fs.create(path, true)))) {

Review Comment:
   Agreed, this shouldn't be deferred. Snapshot dirs/files now use `LOG_FOLDER_PERMISSIONS`/`LOG_FILE_PERMISSIONS`, same as the event log writer. Added a test. Fixed in e7eb0e1093fa.



##########
core/src/main/scala/org/apache/spark/deploy/history/FsHistoryProvider.scala:
##########
@@ -1687,7 +1704,52 @@ private[history] class FsHistoryProvider(conf: SparkConf, clock: Clock)
     KVUtils.open(newStorePath, metadata, conf, live = false)
   }
 
-  private def createInMemoryStore(attempt: AttemptInfoWrapper): KVStore = {
+  private def createDiskStoreFromSnapshot(
+      dm: HistoryServerDiskManager,
+      appId: String,
+      attempt: AttemptInfoWrapper,
+      metadata: AppStatusStoreMetadata,
+      manifestPath: Path): KVStore = {
+    val lease = dm.leaseExact(snapshotStore.snapshotSize(manifestPath))
+    val startNs = System.nanoTime()
+    val newStorePath = try {
+      Utils.tryWithResource(KVUtils.open(lease.tmpPath, metadata, conf, live = false)) { store =>
+        snapshotStore.restoreSnapshot(store, manifestPath)
+      }
+      lease.commit(appId, attempt.info.attemptId)

Review Comment:
   Thanks. Both restore paths now verify the expected app/attempt is present before committing or returning the store, so this falls back to replay. Extended the corrupt-snapshot test to cover an empty entity file. Fixed in e7eb0e1093fa.



##########
core/src/main/scala/org/apache/spark/status/AppStatusListener.scala:
##########
@@ -103,12 +108,18 @@ private[spark] class AppStatusListener(
   }
 
   kvstore.onFlush {
-    if (!live) {
+    if (!live || historySnapshotStore.isDefined) {
       val now = System.nanoTime()
       flush(update(_, now))
     }
   }
 
+  historySnapshotStore.foreach { snapshotStore =>
+    kvstore.onClose {
+      writeHistorySnapshot(snapshotStore)

Review Comment:
   Fixed. `SQLAppStatusListener` now flushes live executions on close when snapshots are enabled; flush triggers run before the close trigger that writes the snapshot. Added a test. Fixed in e7eb0e1093fa.



##########
core/src/main/scala/org/apache/spark/deploy/history/FsHistoryProvider.scala:
##########
@@ -1687,7 +1704,52 @@ private[history] class FsHistoryProvider(conf: SparkConf, clock: Clock)
     KVUtils.open(newStorePath, metadata, conf, live = false)
   }
 
-  private def createInMemoryStore(attempt: AttemptInfoWrapper): KVStore = {
+  private def createDiskStoreFromSnapshot(
+      dm: HistoryServerDiskManager,
+      appId: String,
+      attempt: AttemptInfoWrapper,
+      metadata: AppStatusStoreMetadata,
+      manifestPath: Path): KVStore = {
+    val lease = dm.leaseExact(snapshotStore.snapshotSize(manifestPath))
+    val startNs = System.nanoTime()
+    val newStorePath = try {
+      Utils.tryWithResource(KVUtils.open(lease.tmpPath, metadata, conf, live = false)) { store =>
+        snapshotStore.restoreSnapshot(store, manifestPath)
+      }
+      lease.commit(appId, attempt.info.attemptId)
+    } catch {
+      case e: Exception =>
+        lease.rollback()
+        throw e
+    }
+    val durationMs = TimeUnit.NANOSECONDS.toMillis(System.nanoTime() - startNs)
+    logInfo(log"Restored history snapshot ${MDC(PATH, manifestPath)} for " +
+      log"${MDC(APP_ID, appId)} into disk store in ${MDC(TOTAL_TIME, durationMs)} ms")
+
+    KVUtils.open(newStorePath, metadata, conf, live = false)
+  }
+
+  private def createInMemoryStore(appId: String, attempt: AttemptInfoWrapper): KVStore = {
+    snapshotStore.findSnapshot(appId, attempt.info.attemptId).foreach { snapshot =>
+      val store = new InMemoryStore()
+      store.setMetadata(new AppStatusStoreMetadata(AppStatusStore.CURRENT_VERSION))
+      val startNs = System.nanoTime()
+      try {
+        snapshotStore.restoreSnapshot(store, snapshot)
+        val durationMs = TimeUnit.NANOSECONDS.toMillis(System.nanoTime() - startNs)
+        logInfo(log"Restored history snapshot ${MDC(PATH, snapshot)} for " +
+          log"${MDC(APP_ID, appId)} into in-memory store in ${MDC(TOTAL_TIME, durationMs)} ms")
+        return store

Review Comment:
   Good find. When snapshots are enabled, `SharedState` now installs `StreamingQueryStatusListener` regardless of `spark.ui.enabled` / `spark.sql.streaming.ui.enabled`, matching what replay reconstructs. Added a test. Fixed in e7eb0e1093fa.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [PR] [SPARK-56093][CORE][HISTORY] Allow AppStatus to be cached and reused by the history server [spark]

Posted by "ForVic (via GitHub)" <gi...@apache.org>.
ForVic commented on code in PR #59696:
URL: https://github.com/apache/spark/pull/59696#discussion_r4355703077


##########
core/src/main/scala/org/apache/spark/deploy/history/FsHistoryProvider.scala:
##########
@@ -1687,7 +1704,73 @@ private[history] class FsHistoryProvider(conf: SparkConf, clock: Clock)
     KVUtils.open(newStorePath, metadata, conf, live = false)
   }
 
-  private def createInMemoryStore(attempt: AttemptInfoWrapper): KVStore = {
+  /**
+   * Restores a history snapshot and checks that it contains the expected application attempt, so
+   * an incomplete snapshot falls back to replay instead of producing a store the UI cannot load.
+   */
+  private def restoreSnapshot(
+      store: KVStore,
+      snapshot: Path,
+      appId: String,
+      attempt: AttemptInfoWrapper): Unit = {
+    snapshotStore.restoreSnapshot(store, snapshot)
+    val restored = try {
+      Some(store.read(classOf[org.apache.spark.status.ApplicationInfoWrapper], appId).info)
+    } catch {
+      case _: NoSuchElementException => None
+    }
+    if (!restored.exists(_.attempts.exists(_.attemptId == attempt.info.attemptId))) {
+      throw new IOException(
+        s"History snapshot $snapshot does not contain app $appId / ${attempt.info.attemptId}")
+    }
+  }
+
+  private def createDiskStoreFromSnapshot(
+      dm: HistoryServerDiskManager,
+      appId: String,
+      attempt: AttemptInfoWrapper,
+      metadata: AppStatusStoreMetadata,
+      manifestPath: Path): KVStore = {
+    val lease = dm.leaseExact(snapshotStore.snapshotSize(manifestPath))
+    val startNs = System.nanoTime()
+    val newStorePath = try {
+      Utils.tryWithResource(KVUtils.open(lease.tmpPath, metadata, conf, live = false)) { store =>
+        restoreSnapshot(store, manifestPath, appId, attempt)
+      }
+      lease.commit(appId, attempt.info.attemptId)
+    } catch {
+      case e: Exception =>
+        lease.rollback()
+        throw e
+    }
+    val durationMs = TimeUnit.NANOSECONDS.toMillis(System.nanoTime() - startNs)
+    logInfo(log"Restored history snapshot ${MDC(PATH, manifestPath)} for " +
+      log"${MDC(APP_ID, appId)} into disk store in ${MDC(TOTAL_TIME, durationMs)} ms")
+
+    KVUtils.open(newStorePath, metadata, conf, live = false)
+  }
+
+  private def createInMemoryStore(appId: String, attempt: AttemptInfoWrapper): KVStore = {
+    snapshotStore.findSnapshot(appId, attempt.info.attemptId).foreach { snapshot =>
+      val store = new InMemoryStore()
+      store.setMetadata(new AppStatusStoreMetadata(AppStatusStore.CURRENT_VERSION))
+      val startNs = System.nanoTime()
+      try {
+        restoreSnapshot(store, snapshot, appId, attempt)
+        val durationMs = TimeUnit.NANOSECONDS.toMillis(System.nanoTime() - startNs)
+        logInfo(log"Restored history snapshot ${MDC(PATH, snapshot)} for " +
+          log"${MDC(APP_ID, appId)} into in-memory store in ${MDC(TOTAL_TIME, durationMs)} ms")
+        return store

Review Comment:
   Good catch, fixed in 39137513c45f. Snapshots now record the app's retention limits, and if the History Server is set to keep more than the snapshot captured, it replays the event log instead. Added your `retainedJobs=1` vs 1000 case as a test.



##########
core/src/main/scala/org/apache/spark/deploy/history/HistorySnapshotStore.scala:
##########
@@ -0,0 +1,511 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.deploy.history
+
+import java.io.{BufferedInputStream, BufferedOutputStream, DataInputStream, DataOutputStream, IOException}
+import java.net.URLEncoder
+import java.nio.charset.StandardCharsets
+import java.util.{Properties, UUID}
+
+import scala.util.control.NonFatal
+
+import org.apache.hadoop.conf.Configuration
+import org.apache.hadoop.fs.{FileSystem, FSDataOutputStream, Path}
+
+import org.apache.spark.SparkConf
+import org.apache.spark.deploy.SparkHadoopUtil
+import org.apache.spark.internal.Logging
+import org.apache.spark.internal.LogKeys._
+import org.apache.spark.internal.config.History._
+import org.apache.spark.status._
+import org.apache.spark.status.api.v1
+import org.apache.spark.status.protobuf.KVStoreProtobufSerializer
+import org.apache.spark.util.Utils
+import org.apache.spark.util.kvstore.KVStore
+
+/**
+ * Writes and restores protobuf snapshots of an application's status store under
+ * `spark.history.snapshot.path`, so the History Server can load a completed application without
+ * replaying its event log. Snapshot files use the same permissions as event logs.
+ */
+private[spark] class HistorySnapshotStore(conf: SparkConf, hadoopConf: Configuration)
+  extends Logging {
+
+  import HistorySnapshotStore._
+
+  def this(conf: SparkConf) = this(conf, SparkHadoopUtil.get.newConfiguration(conf))
+
+  private val configuredRoot: Option[Path] =
+    if (isEnabled(conf)) conf.get(SNAPSHOT_PATH).map(new Path(_)) else None
+
+  // Resolved lazily so a misconfigured snapshot path does not affect application startup.
+  private lazy val fs: FileSystem = configuredRoot.get.getFileSystem(hadoopConf)
+
+  private lazy val rootPath: Option[Path] = configuredRoot.map(fs.makeQualified)
+
+  private val serializer = new KVStoreProtobufSerializer()
+
+  /**
+   * Writes a versioned history snapshot for an application attempt and returns the manifest path.
+   *
+   * The snapshot is published under the configured snapshot root using a fresh manifest and data
+   * directory so readers can safely continue using the previous published snapshot while a new one
+   * is being written.
+   */
+  def writeSnapshot(store: KVStore, appId: String, attemptId: Option[String]): Option[Path] = {
+    snapshotDir(appId, attemptId).map { attemptDir =>
+      mkdirs(attemptDir.getParent)
+      mkdirs(attemptDir)
+      val snapshotId = newSnapshotId()
+      val dataDir = new Path(attemptDir, s"$SNAPSHOT_DIR_PREFIX$snapshotId")
+      val manifestPath = new Path(attemptDir, s"$MANIFEST_FILE_PREFIX$snapshotId.properties")
+
+      try {
+        mkdirs(dataDir)
+        val entries = writeEntityFiles(store, dataDir, appId)
+
+        val manifest = new Properties()
+        manifest.setProperty("schemaVersion", SNAPSHOT_SCHEMA_VERSION.toString)
+        manifest.setProperty("snapshotDir", dataDir.getName)
+        manifest.setProperty("classNames", entries.map(_.className).mkString(","))
+        entries.foreach { entry =>
+          manifest.setProperty(s"${entry.className}.file", entry.fileName)
+        }
+
+        Utils.tryWithResource(create(manifestPath, overwrite = false)) { out =>
+          manifest.store(out, "spark history snapshot")
+        }
+        // Only older manifests are deleted, so readers always see either the old snapshot or the
+        // newly published one.
+        cleanupStaleSnapshots(attemptDir, manifestPath)
+        manifestPath
+      } catch {
+        case NonFatal(e) =>
+          deletePathQuietly(manifestPath, recursive = false)
+          deletePathQuietly(dataDir, recursive = true)
+          throw e
+      }
+    }
+  }
+
+  /**
+   * Returns the latest published history snapshot manifest for an application attempt, if any.
+   *
+   * When multiple manifests exist, the latest versioned manifest is treated as the active
+   * published snapshot.
+   */
+  def findSnapshot(appId: String, attemptId: Option[String]): Option[Path] = {
+    try {
+      snapshotDir(appId, attemptId).flatMap(latestManifest)
+    } catch {
+      case NonFatal(e) =>
+        logWarning(log"Failed to look up history snapshot for ${MDC(APP_ID, appId)}.", e)
+        None
+    }
+  }
+
+  /**
+   * Loads a published history snapshot into the provided KVStore.
+   *
+   * The manifest controls which entity files are restored and where the snapshot data directory is
+   * located relative to the manifest path.
+   */
+  def restoreSnapshot(store: KVStore, manifestPath: Path): Unit = {
+    val manifest = loadManifest(manifestPath)
+    val schemaVersion = manifest.getProperty("schemaVersion", "-1").toInt
+    require(schemaVersion == SNAPSHOT_SCHEMA_VERSION,
+      s"Unsupported history snapshot version $schemaVersion at $manifestPath")
+
+    val dataRoot = snapshotDataRoot(manifestPath, manifest)
+    manifestEntries(manifestPath, manifest).foreach { entry =>
+      loadSnapshotClass(manifestPath, entry.className).foreach { klass =>
+        readEntityFile(store, klass, new Path(dataRoot, entry.fileName))
+      }
+    }
+  }
+
+  /**
+   * Returns the total on-storage size of a published history snapshot, including its manifest.
+   *
+   * This is used by History Server disk-store allocation so the target local store can reserve
+   * enough space before restoring a snapshot.
+   */
+  def snapshotSize(manifestPath: Path): Long = {
+    val manifest = loadManifest(manifestPath)
+    val root = snapshotDataRoot(manifestPath, manifest)
+    manifestEntries(manifestPath, manifest)
+      .foldLeft(fs.getFileStatus(manifestPath).getLen) { (total, entry) =>
+        total + fs.getFileStatus(new Path(root, entry.fileName)).getLen
+      }
+  }
+
+  /**
+   * Deletes all snapshots of an application attempt. The History Server calls this when it cleans
+   * up the attempt's event log, so snapshots do not outlive their event logs.
+   */
+  def deleteSnapshots(appId: String, attemptId: Option[String]): Unit = {
+    Utils.tryLogNonFatalError {
+      snapshotDir(appId, attemptId).foreach { attemptDir =>
+        deletePathQuietly(attemptDir, recursive = true)
+        val appDir = attemptDir.getParent
+        if (fs.exists(appDir) && fs.listStatus(appDir).isEmpty) {
+          fs.delete(appDir, false)
+        }
+      }
+    }
+  }
+
+  /**
+   * Returns the snapshot directory of an application attempt. The IDs come from event logs, and
+   * Hadoop normalizes `.` and `..` components, so the result is checked to stay within the root.
+   */
+  private def snapshotDir(appId: String, attemptId: Option[String]): Option[Path] = {
+    rootPath.map { root =>
+      val dir = new Path(new Path(root, encode(appId)), encode(attemptId.getOrElse("_default_")))
+      if (dir.getParent == null || dir.getParent.getParent != root) {
+        throw new IllegalArgumentException(
+          s"Snapshot path for app $appId / $attemptId escapes the snapshot directory")
+      }
+      dir
+    }
+  }
+
+  /** Reads and parses a snapshot manifest file. */
+  private def loadManifest(manifestPath: Path): Properties = {
+    val properties = new Properties()
+    Utils.tryWithResource(fs.open(manifestPath)) { in =>
+      properties.load(in)
+    }
+    properties
+  }
+
+  /** Returns the newest published manifest under a snapshot root, ordered by manifest name. */
+  private def latestManifest(rootPath: Path): Option[Path] = {
+    if (!fs.exists(rootPath)) {
+      None
+    } else {
+      fs.listStatus(rootPath).map(_.getPath).filter(isVersionedManifest).sortBy(_.getName)
+        .lastOption
+    }
+  }
+
+  /** Removes stale manifests and data only when this writer still owns the latest publication. */
+  private def cleanupStaleSnapshots(rootPath: Path, keepManifestPath: Path): Unit = {
+    try {
+      // A concurrent writer may have published a newer manifest after this write completed.
+      // In that case, avoid deleting anything and let the newer publication own cleanup.
+      if (latestManifest(rootPath).exists(_ != keepManifestPath)) {
+        logWarning(log"Skipping cleanup for history snapshot root ${MDC(PATH, rootPath)} " +
+          log"because another manifest became active while publishing " +
+          log"${MDC(PATHS, keepManifestPath)}.")
+      } else {
+        val manifests = fs.listStatus(rootPath).map(_.getPath).toSeq
+        staleManifestsForCleanup(manifests, keepManifestPath).foreach(deleteSnapshot)
+      }
+    } catch {
+      case NonFatal(e) =>
+        logWarning(log"Failed to clean up stale history snapshots under ${MDC(PATH, rootPath)}.",
+          e)
+    }
+  }
+
+  /** Deletes a manifest and its associated snapshot artifacts, tolerating partial corruption. */
+  private def deleteSnapshot(manifestPath: Path): Unit = {
+    try {
+      deletePathQuietly(snapshotDataRoot(manifestPath, loadManifest(manifestPath)),
+        recursive = true)
+    } catch {
+      case NonFatal(e) =>
+        logWarning(log"Failed to load history snapshot manifest at " +
+          log"${MDC(PATH, manifestPath)} for deletion.", e)
+    }
+    deletePathQuietly(manifestPath, recursive = false)
+  }
+
+  private def mkdirs(dir: Path): Unit = {
+    if (!FileSystem.mkdirs(fs, dir, EventLogFileWriter.LOG_FOLDER_PERMISSIONS)) {
+      throw new IOException(s"Failed to create history snapshot directory $dir")
+    }
+  }
+
+  private def create(path: Path, overwrite: Boolean): FSDataOutputStream = {
+    val out = fs.create(path, overwrite)
+    try {
+      fs.setPermission(path, EventLogFileWriter.LOG_FILE_PERMISSIONS)
+    } catch {
+      case NonFatal(e) =>
+        Utils.closeQuietly(out)
+        throw e
+    }
+    out
+  }
+
+  private def deletePathQuietly(path: Path, recursive: Boolean): Unit = {
+    Utils.tryLogNonFatalError {
+      if (fs.exists(path)) {
+        fs.delete(path, recursive)
+      }
+    }
+  }
+
+  /** Writes one file per entity type into the snapshot data directory. */
+  private def writeEntityFiles(
+      store: KVStore,
+      dataDir: Path,
+      appId: String): Seq[SnapshotEntry] = {
+    snapshotEntityClasses.zipWithIndex.flatMap { case (klass, idx) =>
+      val fileName = f"$idx%03d-${klass.getSimpleName}.pb"
+      val path = new Path(dataDir, fileName)
+      if (writeEntityFile(store, klass, path, appId) > 0L) {
+        Some(SnapshotEntry(klass.getName, fileName))
+      } else {
+        fs.delete(path, false)
+        None
+      }
+    }
+  }
+
+  /** Writes all records for one entity type into a single snapshot file. */
+  private def writeEntityFile(
+      store: KVStore,
+      klass: Class[_],
+      path: Path,
+      appId: String): Long = {
+    singletonEntity(store, klass, appId)
+      .map { value =>
+        writeValuesFile(path, Iterator.single(redact(value)))
+      }
+      .getOrElse {
+        Utils.tryWithResource(store.view(klass.asInstanceOf[Class[AnyRef]]).closeableIterator()) {
+          it =>
+            writeValuesFile(path,
+              Iterator.continually(it).takeWhile(_.hasNext).map(_.next().asInstanceOf[AnyRef]))
+        }
+      }
+  }
+
+  /** Serializes a stream of KVStore values using length-delimited protobuf records. */
+  private def writeValuesFile(path: Path, values: Iterator[AnyRef]): Long = {
+    var count = 0L
+    Utils.tryWithResource(
+        new DataOutputStream(new BufferedOutputStream(create(path, overwrite = true)))) { out =>
+      while (values.hasNext) {
+        val bytes = serializer.serialize(values.next())
+        out.writeInt(bytes.length)
+        out.write(bytes)
+        count += 1
+      }
+    }
+    count
+  }
+
+  /**
+   * Redacts the application environment like the event log does. The live store keeps it
+   * unredacted, but snapshots are written to shared storage and served by the History Server.
+   */
+  private def redact(value: AnyRef): AnyRef = value match {
+    case env: ApplicationEnvironmentInfoWrapper =>
+      val info = env.info
+      new ApplicationEnvironmentInfoWrapper(new v1.ApplicationEnvironmentInfo(
+        info.runtime,
+        Utils.redact(conf, info.sparkProperties),
+        Utils.redact(conf, info.hadoopProperties),
+        Utils.redact(conf, info.systemProperties),
+        Utils.redact(conf, info.metricsProperties),
+        info.classpathEntries,
+        info.resourceProfiles))
+    case other => other
+  }
+
+  /** Returns singleton snapshot entities whose keys are not discovered through a KVStore view. */
+  private def singletonEntity(store: KVStore, klass: Class[_], appId: String): Option[AnyRef] = {
+    def read(key: Any): Option[AnyRef] = {
+      try {
+        Some(store.read(klass.asInstanceOf[Class[AnyRef]], key))
+      } catch {
+        case _: NoSuchElementException => None
+      }
+    }
+
+    if (klass == classOf[org.apache.spark.status.ApplicationInfoWrapper]) {
+      read(appId)
+    } else if (klass == classOf[ApplicationEnvironmentInfoWrapper]) {
+      read(classOf[ApplicationEnvironmentInfoWrapper].getName)
+    } else if (klass == classOf[AppSummary]) {
+      read(classOf[AppSummary].getName)
+    } else {
+      None
+    }
+  }
+
+  /**
+   * Restores one entity file by replaying its length-delimited protobuf records into the store.
+   * Record sizes are checked against the remaining file length before allocating, so a corrupt
+   * size fails with an IOException rather than an OutOfMemoryError.
+   */
+  private def readEntityFile(store: KVStore, klass: Class[_], path: Path): Unit = {
+    var remaining = fs.getFileStatus(path).getLen
+    Utils.tryWithResource(new DataInputStream(new BufferedInputStream(fs.open(path)))) { in =>
+      while (remaining > 0) {
+        val size = in.readInt()
+        remaining -= Integer.BYTES
+        if (size < 0 || size > remaining) {
+          throw new IOException(s"Invalid record size $size in snapshot file $path")
+        }
+        val bytes = new Array[Byte](size)

Review Comment:
   Agreed, fixed in 39137513c45f. Added `spark.history.snapshot.maxRecordSize` (default 256m, max 512m, matching `eventLog.maxLineLength`). Records over the limit now fail with an `IOException` and fall back to replay instead of OOMing.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [PR] [SPARK-56093][CORE][HISTORY] Allow AppStatus to be cached and reused by the history server [spark]

Posted by "ForVic (via GitHub)" <gi...@apache.org>.
ForVic commented on PR #59696:
URL: https://github.com/apache/spark/pull/59696#issuecomment-6332025122

   cc @sunchao finally getting around to getting this one pushed through, addressed comments from original pr


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [PR] [SPARK-56093][CORE][HISTORY] Allow AppStatus to be cached and reused by the history server [spark]

Posted by "ForVic (via GitHub)" <gi...@apache.org>.
ForVic commented on PR #59696:
URL: https://github.com/apache/spark/pull/59696#issuecomment-6335168185

   thanks for the quick review @sunchao, addressed all comments


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [PR] [SPARK-56093][CORE][HISTORY] Allow AppStatus to be cached and reused by the history server [spark]

Posted by "sunchao (via GitHub)" <gi...@apache.org>.
sunchao commented on code in PR #59696:
URL: https://github.com/apache/spark/pull/59696#discussion_r4353739195


##########
core/src/main/scala/org/apache/spark/status/AppStatusListener.scala:
##########
@@ -103,12 +108,18 @@ private[spark] class AppStatusListener(
   }
 
   kvstore.onFlush {
-    if (!live) {
+    if (!live || historySnapshotStore.isDefined) {
       val now = System.nanoTime()
       flush(update(_, now))
     }
   }
 
+  historySnapshotStore.foreach { snapshotStore =>
+    kvstore.onClose {
+      writeHistorySnapshot(snapshotStore)

Review Comment:
   Verified fixed in e7eb0e10. The live+snapshots probe now persists and restores the partial metric value 42; live without snapshots keeps its prior behavior, and both replay controls retain 42. The current-head SQL regressions also pass. Resolving.



##########
core/src/main/scala/org/apache/spark/deploy/history/FsHistoryProvider.scala:
##########
@@ -1687,7 +1704,52 @@ private[history] class FsHistoryProvider(conf: SparkConf, clock: Clock)
     KVUtils.open(newStorePath, metadata, conf, live = false)
   }
 
-  private def createInMemoryStore(attempt: AttemptInfoWrapper): KVStore = {
+  private def createDiskStoreFromSnapshot(
+      dm: HistoryServerDiskManager,
+      appId: String,
+      attempt: AttemptInfoWrapper,
+      metadata: AppStatusStoreMetadata,
+      manifestPath: Path): KVStore = {
+    val lease = dm.leaseExact(snapshotStore.snapshotSize(manifestPath))
+    val startNs = System.nanoTime()
+    val newStorePath = try {
+      Utils.tryWithResource(KVUtils.open(lease.tmpPath, metadata, conf, live = false)) { store =>
+        snapshotStore.restoreSnapshot(store, manifestPath)
+      }
+      lease.commit(appId, attempt.info.attemptId)
+    } catch {
+      case e: Exception =>
+        lease.rollback()
+        throw e
+    }
+    val durationMs = TimeUnit.NANOSECONDS.toMillis(System.nanoTime() - startNs)
+    logInfo(log"Restored history snapshot ${MDC(PATH, manifestPath)} for " +
+      log"${MDC(APP_ID, appId)} into disk store in ${MDC(TOTAL_TIME, durationMs)} ms")
+
+    KVUtils.open(newStorePath, metadata, conf, live = false)
+  }
+
+  private def createInMemoryStore(appId: String, attempt: AttemptInfoWrapper): KVStore = {
+    snapshotStore.findSnapshot(appId, attempt.info.attemptId).foreach { snapshot =>
+      val store = new InMemoryStore()
+      store.setMetadata(new AppStatusStoreMetadata(AppStatusStore.CURRENT_VERSION))
+      val startNs = System.nanoTime()
+      try {
+        snapshotStore.restoreSnapshot(store, snapshot)
+        val durationMs = TimeUnit.NANOSECONDS.toMillis(System.nanoTime() - startNs)
+        logInfo(log"Restored history snapshot ${MDC(PATH, snapshot)} for " +
+          log"${MDC(APP_ID, appId)} into in-memory store in ${MDC(TOTAL_TIME, durationMs)} ms")
+        return store

Review Comment:
   Verified fixed in e7eb0e10. SharedState now supplies the streaming status listener when snapshots are enabled even if either live-UI setting prevents the tab, and StreamingQueryManager registers it before events are posted. The new UI-disabled SharedState test passes in current-head CI. Resolving.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [PR] [SPARK-56093][CORE][HISTORY] Allow AppStatus to be cached and reused by the history server [spark]

Posted by "ForVic (via GitHub)" <gi...@apache.org>.
ForVic commented on PR #59696:
URL: https://github.com/apache/spark/pull/59696#issuecomment-6356445475

   addressed all comments thanks @sunchao 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [PR] [SPARK-56093][CORE][HISTORY] Allow AppStatus to be cached and reused by the history server [spark]

Posted by "sunchao (via GitHub)" <gi...@apache.org>.
sunchao commented on code in PR #59696:
URL: https://github.com/apache/spark/pull/59696#discussion_r4353737328


##########
core/src/main/scala/org/apache/spark/deploy/history/FsHistoryProvider.scala:
##########
@@ -1687,7 +1704,73 @@ private[history] class FsHistoryProvider(conf: SparkConf, clock: Clock)
     KVUtils.open(newStorePath, metadata, conf, live = false)
   }
 
-  private def createInMemoryStore(attempt: AttemptInfoWrapper): KVStore = {
+  /**
+   * Restores a history snapshot and checks that it contains the expected application attempt, so
+   * an incomplete snapshot falls back to replay instead of producing a store the UI cannot load.
+   */
+  private def restoreSnapshot(
+      store: KVStore,
+      snapshot: Path,
+      appId: String,
+      attempt: AttemptInfoWrapper): Unit = {
+    snapshotStore.restoreSnapshot(store, snapshot)
+    val restored = try {
+      Some(store.read(classOf[org.apache.spark.status.ApplicationInfoWrapper], appId).info)
+    } catch {
+      case _: NoSuchElementException => None
+    }
+    if (!restored.exists(_.attempts.exists(_.attemptId == attempt.info.attemptId))) {
+      throw new IOException(
+        s"History snapshot $snapshot does not contain app $appId / ${attempt.info.attemptId}")
+    }
+  }
+
+  private def createDiskStoreFromSnapshot(
+      dm: HistoryServerDiskManager,
+      appId: String,
+      attempt: AttemptInfoWrapper,
+      metadata: AppStatusStoreMetadata,
+      manifestPath: Path): KVStore = {
+    val lease = dm.leaseExact(snapshotStore.snapshotSize(manifestPath))
+    val startNs = System.nanoTime()
+    val newStorePath = try {
+      Utils.tryWithResource(KVUtils.open(lease.tmpPath, metadata, conf, live = false)) { store =>
+        restoreSnapshot(store, manifestPath, appId, attempt)
+      }
+      lease.commit(appId, attempt.info.attemptId)
+    } catch {
+      case e: Exception =>
+        lease.rollback()
+        throw e
+    }
+    val durationMs = TimeUnit.NANOSECONDS.toMillis(System.nanoTime() - startNs)
+    logInfo(log"Restored history snapshot ${MDC(PATH, manifestPath)} for " +
+      log"${MDC(APP_ID, appId)} into disk store in ${MDC(TOTAL_TIME, durationMs)} ms")
+
+    KVUtils.open(newStorePath, metadata, conf, live = false)
+  }
+
+  private def createInMemoryStore(appId: String, attempt: AttemptInfoWrapper): KVStore = {
+    snapshotStore.findSnapshot(appId, attempt.info.attemptId).foreach { snapshot =>
+      val store = new InMemoryStore()
+      store.setMetadata(new AppStatusStoreMetadata(AppStatusStore.CURRENT_VERSION))
+      val startNs = System.nanoTime()
+      try {
+        restoreSnapshot(store, snapshot, appId, attempt)
+        val durationMs = TimeUnit.NANOSECONDS.toMillis(System.nanoTime() - startNs)
+        logInfo(log"Restored history snapshot ${MDC(PATH, snapshot)} for " +
+          log"${MDC(APP_ID, appId)} into in-memory store in ${MDC(TOTAL_TIME, durationMs)} ms")
+        return store

Review Comment:
   [P2] Check retention settings before preferring the driver snapshot
   
   The snapshot contains the driver's already-pruned status store, but this return bypasses replay without considering the History Server's retention settings (the disk path does the same). For example, with driver `spark.ui.retainedJobs=1` and History Server `spark.ui.retainedJobs=1000`, three completed jobs produce one restored job instead of the three that replay returns. A focused write/restore-versus-replay control reproduced exactly that difference. Increasing retention and restarting the server therefore no longer retrieves the additional history promised in `docs/monitoring.md`, although it remains in the event log. Please record/check the capture limits and fall back to replay when the snapshot cannot satisfy the server's requested retention.



##########
core/src/main/scala/org/apache/spark/deploy/history/HistorySnapshotStore.scala:
##########
@@ -0,0 +1,511 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.deploy.history
+
+import java.io.{BufferedInputStream, BufferedOutputStream, DataInputStream, DataOutputStream, IOException}
+import java.net.URLEncoder
+import java.nio.charset.StandardCharsets
+import java.util.{Properties, UUID}
+
+import scala.util.control.NonFatal
+
+import org.apache.hadoop.conf.Configuration
+import org.apache.hadoop.fs.{FileSystem, FSDataOutputStream, Path}
+
+import org.apache.spark.SparkConf
+import org.apache.spark.deploy.SparkHadoopUtil
+import org.apache.spark.internal.Logging
+import org.apache.spark.internal.LogKeys._
+import org.apache.spark.internal.config.History._
+import org.apache.spark.status._
+import org.apache.spark.status.api.v1
+import org.apache.spark.status.protobuf.KVStoreProtobufSerializer
+import org.apache.spark.util.Utils
+import org.apache.spark.util.kvstore.KVStore
+
+/**
+ * Writes and restores protobuf snapshots of an application's status store under
+ * `spark.history.snapshot.path`, so the History Server can load a completed application without
+ * replaying its event log. Snapshot files use the same permissions as event logs.
+ */
+private[spark] class HistorySnapshotStore(conf: SparkConf, hadoopConf: Configuration)
+  extends Logging {
+
+  import HistorySnapshotStore._
+
+  def this(conf: SparkConf) = this(conf, SparkHadoopUtil.get.newConfiguration(conf))
+
+  private val configuredRoot: Option[Path] =
+    if (isEnabled(conf)) conf.get(SNAPSHOT_PATH).map(new Path(_)) else None
+
+  // Resolved lazily so a misconfigured snapshot path does not affect application startup.
+  private lazy val fs: FileSystem = configuredRoot.get.getFileSystem(hadoopConf)
+
+  private lazy val rootPath: Option[Path] = configuredRoot.map(fs.makeQualified)
+
+  private val serializer = new KVStoreProtobufSerializer()
+
+  /**
+   * Writes a versioned history snapshot for an application attempt and returns the manifest path.
+   *
+   * The snapshot is published under the configured snapshot root using a fresh manifest and data
+   * directory so readers can safely continue using the previous published snapshot while a new one
+   * is being written.
+   */
+  def writeSnapshot(store: KVStore, appId: String, attemptId: Option[String]): Option[Path] = {
+    snapshotDir(appId, attemptId).map { attemptDir =>
+      mkdirs(attemptDir.getParent)
+      mkdirs(attemptDir)
+      val snapshotId = newSnapshotId()
+      val dataDir = new Path(attemptDir, s"$SNAPSHOT_DIR_PREFIX$snapshotId")
+      val manifestPath = new Path(attemptDir, s"$MANIFEST_FILE_PREFIX$snapshotId.properties")
+
+      try {
+        mkdirs(dataDir)
+        val entries = writeEntityFiles(store, dataDir, appId)
+
+        val manifest = new Properties()
+        manifest.setProperty("schemaVersion", SNAPSHOT_SCHEMA_VERSION.toString)
+        manifest.setProperty("snapshotDir", dataDir.getName)
+        manifest.setProperty("classNames", entries.map(_.className).mkString(","))
+        entries.foreach { entry =>
+          manifest.setProperty(s"${entry.className}.file", entry.fileName)
+        }
+
+        Utils.tryWithResource(create(manifestPath, overwrite = false)) { out =>
+          manifest.store(out, "spark history snapshot")
+        }
+        // Only older manifests are deleted, so readers always see either the old snapshot or the
+        // newly published one.
+        cleanupStaleSnapshots(attemptDir, manifestPath)
+        manifestPath
+      } catch {
+        case NonFatal(e) =>
+          deletePathQuietly(manifestPath, recursive = false)
+          deletePathQuietly(dataDir, recursive = true)
+          throw e
+      }
+    }
+  }
+
+  /**
+   * Returns the latest published history snapshot manifest for an application attempt, if any.
+   *
+   * When multiple manifests exist, the latest versioned manifest is treated as the active
+   * published snapshot.
+   */
+  def findSnapshot(appId: String, attemptId: Option[String]): Option[Path] = {
+    try {
+      snapshotDir(appId, attemptId).flatMap(latestManifest)
+    } catch {
+      case NonFatal(e) =>
+        logWarning(log"Failed to look up history snapshot for ${MDC(APP_ID, appId)}.", e)
+        None
+    }
+  }
+
+  /**
+   * Loads a published history snapshot into the provided KVStore.
+   *
+   * The manifest controls which entity files are restored and where the snapshot data directory is
+   * located relative to the manifest path.
+   */
+  def restoreSnapshot(store: KVStore, manifestPath: Path): Unit = {
+    val manifest = loadManifest(manifestPath)
+    val schemaVersion = manifest.getProperty("schemaVersion", "-1").toInt
+    require(schemaVersion == SNAPSHOT_SCHEMA_VERSION,
+      s"Unsupported history snapshot version $schemaVersion at $manifestPath")
+
+    val dataRoot = snapshotDataRoot(manifestPath, manifest)
+    manifestEntries(manifestPath, manifest).foreach { entry =>
+      loadSnapshotClass(manifestPath, entry.className).foreach { klass =>
+        readEntityFile(store, klass, new Path(dataRoot, entry.fileName))
+      }
+    }
+  }
+
+  /**
+   * Returns the total on-storage size of a published history snapshot, including its manifest.
+   *
+   * This is used by History Server disk-store allocation so the target local store can reserve
+   * enough space before restoring a snapshot.
+   */
+  def snapshotSize(manifestPath: Path): Long = {
+    val manifest = loadManifest(manifestPath)
+    val root = snapshotDataRoot(manifestPath, manifest)
+    manifestEntries(manifestPath, manifest)
+      .foldLeft(fs.getFileStatus(manifestPath).getLen) { (total, entry) =>
+        total + fs.getFileStatus(new Path(root, entry.fileName)).getLen
+      }
+  }
+
+  /**
+   * Deletes all snapshots of an application attempt. The History Server calls this when it cleans
+   * up the attempt's event log, so snapshots do not outlive their event logs.
+   */
+  def deleteSnapshots(appId: String, attemptId: Option[String]): Unit = {
+    Utils.tryLogNonFatalError {
+      snapshotDir(appId, attemptId).foreach { attemptDir =>
+        deletePathQuietly(attemptDir, recursive = true)
+        val appDir = attemptDir.getParent
+        if (fs.exists(appDir) && fs.listStatus(appDir).isEmpty) {
+          fs.delete(appDir, false)
+        }
+      }
+    }
+  }
+
+  /**
+   * Returns the snapshot directory of an application attempt. The IDs come from event logs, and
+   * Hadoop normalizes `.` and `..` components, so the result is checked to stay within the root.
+   */
+  private def snapshotDir(appId: String, attemptId: Option[String]): Option[Path] = {
+    rootPath.map { root =>
+      val dir = new Path(new Path(root, encode(appId)), encode(attemptId.getOrElse("_default_")))
+      if (dir.getParent == null || dir.getParent.getParent != root) {
+        throw new IllegalArgumentException(
+          s"Snapshot path for app $appId / $attemptId escapes the snapshot directory")
+      }
+      dir
+    }
+  }
+
+  /** Reads and parses a snapshot manifest file. */
+  private def loadManifest(manifestPath: Path): Properties = {
+    val properties = new Properties()
+    Utils.tryWithResource(fs.open(manifestPath)) { in =>
+      properties.load(in)
+    }
+    properties
+  }
+
+  /** Returns the newest published manifest under a snapshot root, ordered by manifest name. */
+  private def latestManifest(rootPath: Path): Option[Path] = {
+    if (!fs.exists(rootPath)) {
+      None
+    } else {
+      fs.listStatus(rootPath).map(_.getPath).filter(isVersionedManifest).sortBy(_.getName)
+        .lastOption
+    }
+  }
+
+  /** Removes stale manifests and data only when this writer still owns the latest publication. */
+  private def cleanupStaleSnapshots(rootPath: Path, keepManifestPath: Path): Unit = {
+    try {
+      // A concurrent writer may have published a newer manifest after this write completed.
+      // In that case, avoid deleting anything and let the newer publication own cleanup.
+      if (latestManifest(rootPath).exists(_ != keepManifestPath)) {
+        logWarning(log"Skipping cleanup for history snapshot root ${MDC(PATH, rootPath)} " +
+          log"because another manifest became active while publishing " +
+          log"${MDC(PATHS, keepManifestPath)}.")
+      } else {
+        val manifests = fs.listStatus(rootPath).map(_.getPath).toSeq
+        staleManifestsForCleanup(manifests, keepManifestPath).foreach(deleteSnapshot)
+      }
+    } catch {
+      case NonFatal(e) =>
+        logWarning(log"Failed to clean up stale history snapshots under ${MDC(PATH, rootPath)}.",
+          e)
+    }
+  }
+
+  /** Deletes a manifest and its associated snapshot artifacts, tolerating partial corruption. */
+  private def deleteSnapshot(manifestPath: Path): Unit = {
+    try {
+      deletePathQuietly(snapshotDataRoot(manifestPath, loadManifest(manifestPath)),
+        recursive = true)
+    } catch {
+      case NonFatal(e) =>
+        logWarning(log"Failed to load history snapshot manifest at " +
+          log"${MDC(PATH, manifestPath)} for deletion.", e)
+    }
+    deletePathQuietly(manifestPath, recursive = false)
+  }
+
+  private def mkdirs(dir: Path): Unit = {
+    if (!FileSystem.mkdirs(fs, dir, EventLogFileWriter.LOG_FOLDER_PERMISSIONS)) {
+      throw new IOException(s"Failed to create history snapshot directory $dir")
+    }
+  }
+
+  private def create(path: Path, overwrite: Boolean): FSDataOutputStream = {
+    val out = fs.create(path, overwrite)
+    try {
+      fs.setPermission(path, EventLogFileWriter.LOG_FILE_PERMISSIONS)
+    } catch {
+      case NonFatal(e) =>
+        Utils.closeQuietly(out)
+        throw e
+    }
+    out
+  }
+
+  private def deletePathQuietly(path: Path, recursive: Boolean): Unit = {
+    Utils.tryLogNonFatalError {
+      if (fs.exists(path)) {
+        fs.delete(path, recursive)
+      }
+    }
+  }
+
+  /** Writes one file per entity type into the snapshot data directory. */
+  private def writeEntityFiles(
+      store: KVStore,
+      dataDir: Path,
+      appId: String): Seq[SnapshotEntry] = {
+    snapshotEntityClasses.zipWithIndex.flatMap { case (klass, idx) =>
+      val fileName = f"$idx%03d-${klass.getSimpleName}.pb"
+      val path = new Path(dataDir, fileName)
+      if (writeEntityFile(store, klass, path, appId) > 0L) {
+        Some(SnapshotEntry(klass.getName, fileName))
+      } else {
+        fs.delete(path, false)
+        None
+      }
+    }
+  }
+
+  /** Writes all records for one entity type into a single snapshot file. */
+  private def writeEntityFile(
+      store: KVStore,
+      klass: Class[_],
+      path: Path,
+      appId: String): Long = {
+    singletonEntity(store, klass, appId)
+      .map { value =>
+        writeValuesFile(path, Iterator.single(redact(value)))
+      }
+      .getOrElse {
+        Utils.tryWithResource(store.view(klass.asInstanceOf[Class[AnyRef]]).closeableIterator()) {
+          it =>
+            writeValuesFile(path,
+              Iterator.continually(it).takeWhile(_.hasNext).map(_.next().asInstanceOf[AnyRef]))
+        }
+      }
+  }
+
+  /** Serializes a stream of KVStore values using length-delimited protobuf records. */
+  private def writeValuesFile(path: Path, values: Iterator[AnyRef]): Long = {
+    var count = 0L
+    Utils.tryWithResource(
+        new DataOutputStream(new BufferedOutputStream(create(path, overwrite = true)))) { out =>
+      while (values.hasNext) {
+        val bytes = serializer.serialize(values.next())
+        out.writeInt(bytes.length)
+        out.write(bytes)
+        count += 1
+      }
+    }
+    count
+  }
+
+  /**
+   * Redacts the application environment like the event log does. The live store keeps it
+   * unredacted, but snapshots are written to shared storage and served by the History Server.
+   */
+  private def redact(value: AnyRef): AnyRef = value match {
+    case env: ApplicationEnvironmentInfoWrapper =>
+      val info = env.info
+      new ApplicationEnvironmentInfoWrapper(new v1.ApplicationEnvironmentInfo(
+        info.runtime,
+        Utils.redact(conf, info.sparkProperties),
+        Utils.redact(conf, info.hadoopProperties),
+        Utils.redact(conf, info.systemProperties),
+        Utils.redact(conf, info.metricsProperties),
+        info.classpathEntries,
+        info.resourceProfiles))
+    case other => other
+  }
+
+  /** Returns singleton snapshot entities whose keys are not discovered through a KVStore view. */
+  private def singletonEntity(store: KVStore, klass: Class[_], appId: String): Option[AnyRef] = {
+    def read(key: Any): Option[AnyRef] = {
+      try {
+        Some(store.read(klass.asInstanceOf[Class[AnyRef]], key))
+      } catch {
+        case _: NoSuchElementException => None
+      }
+    }
+
+    if (klass == classOf[org.apache.spark.status.ApplicationInfoWrapper]) {
+      read(appId)
+    } else if (klass == classOf[ApplicationEnvironmentInfoWrapper]) {
+      read(classOf[ApplicationEnvironmentInfoWrapper].getName)
+    } else if (klass == classOf[AppSummary]) {
+      read(classOf[AppSummary].getName)
+    } else {
+      None
+    }
+  }
+
+  /**
+   * Restores one entity file by replaying its length-delimited protobuf records into the store.
+   * Record sizes are checked against the remaining file length before allocating, so a corrupt
+   * size fails with an IOException rather than an OutOfMemoryError.
+   */
+  private def readEntityFile(store: KVStore, klass: Class[_], path: Path): Unit = {
+    var remaining = fs.getFileStatus(path).getLen
+    Utils.tryWithResource(new DataInputStream(new BufferedInputStream(fs.open(path)))) { in =>
+      while (remaining > 0) {
+        val size = in.readInt()
+        remaining -= Integer.BYTES
+        if (size < 0 || size > remaining) {
+          throw new IOException(s"Invalid record size $size in snapshot file $path")
+        }
+        val bytes = new Array[Byte](size)

Review Comment:
   [P2] Bound record allocation independently of the entity-file length
   
   Checking against `remaining` only protects small files. If an entity file has at least `Int.MaxValue` bytes remaining, a corrupt record length of `Int.MaxValue` passes this guard and reaches `new Array[Byte](size)`, exceeding the JVM array-size limit. The resulting `OutOfMemoryError` escapes both provider `Exception` handlers, so the snapshot does not fall back to the valid event log. Smaller oversized lengths can likewise exceed the available heap. Existing event-log replay already has a configurable per-line limit with a hard cap; snapshot loading needs an independent supported per-record bound before allocating, with an `IOException` for oversize records. This finding is from the allocation and exception paths; I did not run a memory-exhaustion test.



##########
core/src/main/scala/org/apache/spark/deploy/history/HistorySnapshotStore.scala:
##########
@@ -0,0 +1,483 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.deploy.history
+
+import java.io.{BufferedInputStream, BufferedOutputStream, DataInputStream, DataOutputStream, IOException}
+import java.net.URLEncoder
+import java.nio.charset.StandardCharsets
+import java.util.{Properties, UUID}
+
+import scala.util.control.NonFatal
+
+import org.apache.hadoop.conf.Configuration
+import org.apache.hadoop.fs.{FileSystem, Path}
+
+import org.apache.spark.SparkConf
+import org.apache.spark.deploy.SparkHadoopUtil
+import org.apache.spark.internal.Logging
+import org.apache.spark.internal.LogKeys._
+import org.apache.spark.internal.config.History._
+import org.apache.spark.status._
+import org.apache.spark.status.api.v1
+import org.apache.spark.status.protobuf.KVStoreProtobufSerializer
+import org.apache.spark.util.Utils
+import org.apache.spark.util.kvstore.KVStore
+
+/**
+ * Writes and restores protobuf snapshots of an application's status store under
+ * `spark.history.snapshot.path`, so the History Server can load a completed application without
+ * replaying its event log.
+ */
+private[spark] class HistorySnapshotStore(conf: SparkConf, hadoopConf: Configuration)
+  extends Logging {
+
+  import HistorySnapshotStore._
+
+  def this(conf: SparkConf) = this(conf, SparkHadoopUtil.get.newConfiguration(conf))
+
+  private val configuredRoot: Option[Path] =
+    if (isEnabled(conf)) conf.get(SNAPSHOT_PATH).map(new Path(_)) else None
+
+  // Resolved lazily so a misconfigured snapshot path does not affect application startup.
+  private lazy val fs: FileSystem = configuredRoot.get.getFileSystem(hadoopConf)
+
+  private lazy val rootPath: Option[Path] = configuredRoot.map(fs.makeQualified)
+
+  private val serializer = new KVStoreProtobufSerializer()
+
+  /**
+   * Writes a versioned history snapshot for an application attempt and returns the manifest path.
+   *
+   * The snapshot is published under the configured snapshot root using a fresh manifest and data
+   * directory so readers can safely continue using the previous published snapshot while a new one
+   * is being written.
+   */
+  def writeSnapshot(store: KVStore, appId: String, attemptId: Option[String]): Option[Path] = {
+    snapshotDir(appId, attemptId).map { attemptDir =>
+      fs.mkdirs(attemptDir)
+      val snapshotId = newSnapshotId()
+      val dataDir = new Path(attemptDir, s"$SNAPSHOT_DIR_PREFIX$snapshotId")
+      val manifestPath = new Path(attemptDir, s"$MANIFEST_FILE_PREFIX$snapshotId.properties")
+
+      try {
+        fs.mkdirs(dataDir)
+        val entries = writeEntityFiles(store, dataDir, appId)
+
+        val manifest = new Properties()
+        manifest.setProperty("schemaVersion", SNAPSHOT_SCHEMA_VERSION.toString)
+        manifest.setProperty("snapshotDir", dataDir.getName)
+        manifest.setProperty("classNames", entries.map(_.className).mkString(","))
+        entries.foreach { entry =>
+          manifest.setProperty(s"${entry.className}.file", entry.fileName)
+        }
+
+        Utils.tryWithResource(fs.create(manifestPath, false)) { out =>
+          manifest.store(out, "spark history snapshot")
+        }
+        // Only older manifests are deleted, so readers always see either the old snapshot or the
+        // newly published one.
+        cleanupStaleSnapshots(attemptDir, manifestPath)
+        manifestPath
+      } catch {
+        case NonFatal(e) =>
+          deletePathQuietly(manifestPath, recursive = false)
+          deletePathQuietly(dataDir, recursive = true)
+          throw e
+      }
+    }
+  }
+
+  /**
+   * Returns the latest published history snapshot manifest for an application attempt, if any.
+   *
+   * When multiple manifests exist, the latest versioned manifest is treated as the active
+   * published snapshot.
+   */
+  def findSnapshot(appId: String, attemptId: Option[String]): Option[Path] = {
+    try {
+      snapshotDir(appId, attemptId).flatMap(latestManifest)
+    } catch {
+      case NonFatal(e) =>
+        logWarning(log"Failed to look up history snapshot for ${MDC(APP_ID, appId)}.", e)
+        None
+    }
+  }
+
+  /**
+   * Loads a published history snapshot into the provided KVStore.
+   *
+   * The manifest controls which entity files are restored and where the snapshot data directory is
+   * located relative to the manifest path.
+   */
+  def restoreSnapshot(store: KVStore, manifestPath: Path): Unit = {
+    val manifest = loadManifest(manifestPath)
+    val schemaVersion = manifest.getProperty("schemaVersion", "-1").toInt
+    require(schemaVersion == SNAPSHOT_SCHEMA_VERSION,
+      s"Unsupported history snapshot version $schemaVersion at $manifestPath")
+
+    val dataRoot = snapshotDataRoot(manifestPath, manifest)
+    manifestEntries(manifestPath, manifest).foreach { entry =>
+      loadSnapshotClass(manifestPath, entry.className).foreach { klass =>
+        readEntityFile(store, klass, new Path(dataRoot, entry.fileName))
+      }
+    }
+  }
+
+  /**
+   * Returns the total on-storage size of a published history snapshot, including its manifest.
+   *
+   * This is used by History Server disk-store allocation so the target local store can reserve
+   * enough space before restoring a snapshot.
+   */
+  def snapshotSize(manifestPath: Path): Long = {
+    val manifest = loadManifest(manifestPath)
+    val root = snapshotDataRoot(manifestPath, manifest)
+    manifestEntries(manifestPath, manifest)
+      .foldLeft(fs.getFileStatus(manifestPath).getLen) { (total, entry) =>
+        total + fs.getFileStatus(new Path(root, entry.fileName)).getLen
+      }
+  }
+
+  /**
+   * Deletes all snapshots of an application attempt. The History Server calls this when it cleans
+   * up the attempt's event log, so snapshots do not outlive their event logs.
+   */
+  def deleteSnapshots(appId: String, attemptId: Option[String]): Unit = {
+    Utils.tryLogNonFatalError {
+      snapshotDir(appId, attemptId).foreach { attemptDir =>
+        deletePathQuietly(attemptDir, recursive = true)
+        val appDir = attemptDir.getParent
+        if (fs.exists(appDir) && fs.listStatus(appDir).isEmpty) {
+          fs.delete(appDir, false)
+        }
+      }
+    }
+  }
+
+  private def snapshotDir(appId: String, attemptId: Option[String]): Option[Path] = {
+    rootPath.map { root =>
+      new Path(new Path(root, encode(appId)), encode(attemptId.getOrElse("_default_")))

Review Comment:
   Verified fixed in e7eb0e10. The current normalized-path check rejects the dot-component cases; an isolated current-source probe preserved both the unrelated marker and the other application snapshot. The new current-head CI regression also passes. Resolving.



##########
core/src/main/scala/org/apache/spark/deploy/history/HistorySnapshotStore.scala:
##########
@@ -0,0 +1,483 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.deploy.history
+
+import java.io.{BufferedInputStream, BufferedOutputStream, DataInputStream, DataOutputStream, IOException}
+import java.net.URLEncoder
+import java.nio.charset.StandardCharsets
+import java.util.{Properties, UUID}
+
+import scala.util.control.NonFatal
+
+import org.apache.hadoop.conf.Configuration
+import org.apache.hadoop.fs.{FileSystem, Path}
+
+import org.apache.spark.SparkConf
+import org.apache.spark.deploy.SparkHadoopUtil
+import org.apache.spark.internal.Logging
+import org.apache.spark.internal.LogKeys._
+import org.apache.spark.internal.config.History._
+import org.apache.spark.status._
+import org.apache.spark.status.api.v1
+import org.apache.spark.status.protobuf.KVStoreProtobufSerializer
+import org.apache.spark.util.Utils
+import org.apache.spark.util.kvstore.KVStore
+
+/**
+ * Writes and restores protobuf snapshots of an application's status store under
+ * `spark.history.snapshot.path`, so the History Server can load a completed application without
+ * replaying its event log.
+ */
+private[spark] class HistorySnapshotStore(conf: SparkConf, hadoopConf: Configuration)
+  extends Logging {
+
+  import HistorySnapshotStore._
+
+  def this(conf: SparkConf) = this(conf, SparkHadoopUtil.get.newConfiguration(conf))
+
+  private val configuredRoot: Option[Path] =
+    if (isEnabled(conf)) conf.get(SNAPSHOT_PATH).map(new Path(_)) else None
+
+  // Resolved lazily so a misconfigured snapshot path does not affect application startup.
+  private lazy val fs: FileSystem = configuredRoot.get.getFileSystem(hadoopConf)
+
+  private lazy val rootPath: Option[Path] = configuredRoot.map(fs.makeQualified)
+
+  private val serializer = new KVStoreProtobufSerializer()
+
+  /**
+   * Writes a versioned history snapshot for an application attempt and returns the manifest path.
+   *
+   * The snapshot is published under the configured snapshot root using a fresh manifest and data
+   * directory so readers can safely continue using the previous published snapshot while a new one
+   * is being written.
+   */
+  def writeSnapshot(store: KVStore, appId: String, attemptId: Option[String]): Option[Path] = {
+    snapshotDir(appId, attemptId).map { attemptDir =>
+      fs.mkdirs(attemptDir)
+      val snapshotId = newSnapshotId()
+      val dataDir = new Path(attemptDir, s"$SNAPSHOT_DIR_PREFIX$snapshotId")
+      val manifestPath = new Path(attemptDir, s"$MANIFEST_FILE_PREFIX$snapshotId.properties")
+
+      try {
+        fs.mkdirs(dataDir)
+        val entries = writeEntityFiles(store, dataDir, appId)
+
+        val manifest = new Properties()
+        manifest.setProperty("schemaVersion", SNAPSHOT_SCHEMA_VERSION.toString)
+        manifest.setProperty("snapshotDir", dataDir.getName)
+        manifest.setProperty("classNames", entries.map(_.className).mkString(","))
+        entries.foreach { entry =>
+          manifest.setProperty(s"${entry.className}.file", entry.fileName)
+        }
+
+        Utils.tryWithResource(fs.create(manifestPath, false)) { out =>
+          manifest.store(out, "spark history snapshot")
+        }
+        // Only older manifests are deleted, so readers always see either the old snapshot or the
+        // newly published one.
+        cleanupStaleSnapshots(attemptDir, manifestPath)
+        manifestPath
+      } catch {
+        case NonFatal(e) =>
+          deletePathQuietly(manifestPath, recursive = false)
+          deletePathQuietly(dataDir, recursive = true)
+          throw e
+      }
+    }
+  }
+
+  /**
+   * Returns the latest published history snapshot manifest for an application attempt, if any.
+   *
+   * When multiple manifests exist, the latest versioned manifest is treated as the active
+   * published snapshot.
+   */
+  def findSnapshot(appId: String, attemptId: Option[String]): Option[Path] = {
+    try {
+      snapshotDir(appId, attemptId).flatMap(latestManifest)
+    } catch {
+      case NonFatal(e) =>
+        logWarning(log"Failed to look up history snapshot for ${MDC(APP_ID, appId)}.", e)
+        None
+    }
+  }
+
+  /**
+   * Loads a published history snapshot into the provided KVStore.
+   *
+   * The manifest controls which entity files are restored and where the snapshot data directory is
+   * located relative to the manifest path.
+   */
+  def restoreSnapshot(store: KVStore, manifestPath: Path): Unit = {
+    val manifest = loadManifest(manifestPath)
+    val schemaVersion = manifest.getProperty("schemaVersion", "-1").toInt
+    require(schemaVersion == SNAPSHOT_SCHEMA_VERSION,
+      s"Unsupported history snapshot version $schemaVersion at $manifestPath")
+
+    val dataRoot = snapshotDataRoot(manifestPath, manifest)
+    manifestEntries(manifestPath, manifest).foreach { entry =>
+      loadSnapshotClass(manifestPath, entry.className).foreach { klass =>
+        readEntityFile(store, klass, new Path(dataRoot, entry.fileName))
+      }
+    }
+  }
+
+  /**
+   * Returns the total on-storage size of a published history snapshot, including its manifest.
+   *
+   * This is used by History Server disk-store allocation so the target local store can reserve
+   * enough space before restoring a snapshot.
+   */
+  def snapshotSize(manifestPath: Path): Long = {
+    val manifest = loadManifest(manifestPath)
+    val root = snapshotDataRoot(manifestPath, manifest)
+    manifestEntries(manifestPath, manifest)
+      .foldLeft(fs.getFileStatus(manifestPath).getLen) { (total, entry) =>
+        total + fs.getFileStatus(new Path(root, entry.fileName)).getLen
+      }
+  }
+
+  /**
+   * Deletes all snapshots of an application attempt. The History Server calls this when it cleans
+   * up the attempt's event log, so snapshots do not outlive their event logs.
+   */
+  def deleteSnapshots(appId: String, attemptId: Option[String]): Unit = {
+    Utils.tryLogNonFatalError {
+      snapshotDir(appId, attemptId).foreach { attemptDir =>
+        deletePathQuietly(attemptDir, recursive = true)
+        val appDir = attemptDir.getParent
+        if (fs.exists(appDir) && fs.listStatus(appDir).isEmpty) {
+          fs.delete(appDir, false)
+        }
+      }
+    }
+  }
+
+  private def snapshotDir(appId: String, attemptId: Option[String]): Option[Path] = {
+    rootPath.map { root =>
+      new Path(new Path(root, encode(appId)), encode(attemptId.getOrElse("_default_")))
+    }
+  }
+
+  /** Reads and parses a snapshot manifest file. */
+  private def loadManifest(manifestPath: Path): Properties = {
+    val properties = new Properties()
+    Utils.tryWithResource(fs.open(manifestPath)) { in =>
+      properties.load(in)
+    }
+    properties
+  }
+
+  /** Returns the newest published manifest under a snapshot root, ordered by manifest name. */
+  private def latestManifest(rootPath: Path): Option[Path] = {
+    if (!fs.exists(rootPath)) {
+      None
+    } else {
+      fs.listStatus(rootPath).map(_.getPath).filter(isVersionedManifest).sortBy(_.getName)
+        .lastOption
+    }
+  }
+
+  /** Removes stale manifests and data only when this writer still owns the latest publication. */
+  private def cleanupStaleSnapshots(rootPath: Path, keepManifestPath: Path): Unit = {
+    try {
+      // A concurrent writer may have published a newer manifest after this write completed.
+      // In that case, avoid deleting anything and let the newer publication own cleanup.
+      if (latestManifest(rootPath).exists(_ != keepManifestPath)) {
+        logWarning(log"Skipping cleanup for history snapshot root ${MDC(PATH, rootPath)} " +
+          log"because another manifest became active while publishing " +
+          log"${MDC(PATHS, keepManifestPath)}.")
+      } else {
+        val manifests = fs.listStatus(rootPath).map(_.getPath).toSeq
+        staleManifestsForCleanup(manifests, keepManifestPath).foreach(deleteSnapshot)
+      }
+    } catch {
+      case NonFatal(e) =>
+        logWarning(log"Failed to clean up stale history snapshots under ${MDC(PATH, rootPath)}.",
+          e)
+    }
+  }
+
+  /** Deletes a manifest and its associated snapshot artifacts, tolerating partial corruption. */
+  private def deleteSnapshot(manifestPath: Path): Unit = {
+    try {
+      deletePathQuietly(snapshotDataRoot(manifestPath, loadManifest(manifestPath)),
+        recursive = true)
+    } catch {
+      case NonFatal(e) =>
+        logWarning(log"Failed to load history snapshot manifest at " +
+          log"${MDC(PATH, manifestPath)} for deletion.", e)
+    }
+    deletePathQuietly(manifestPath, recursive = false)
+  }
+
+  private def deletePathQuietly(path: Path, recursive: Boolean): Unit = {
+    Utils.tryLogNonFatalError {
+      if (fs.exists(path)) {
+        fs.delete(path, recursive)
+      }
+    }
+  }
+
+  /** Writes one file per entity type into the snapshot data directory. */
+  private def writeEntityFiles(
+      store: KVStore,
+      dataDir: Path,
+      appId: String): Seq[SnapshotEntry] = {
+    snapshotEntityClasses.zipWithIndex.flatMap { case (klass, idx) =>
+      val fileName = f"$idx%03d-${klass.getSimpleName}.pb"
+      val path = new Path(dataDir, fileName)
+      if (writeEntityFile(store, klass, path, appId) > 0L) {
+        Some(SnapshotEntry(klass.getName, fileName))
+      } else {
+        fs.delete(path, false)
+        None
+      }
+    }
+  }
+
+  /** Writes all records for one entity type into a single snapshot file. */
+  private def writeEntityFile(
+      store: KVStore,
+      klass: Class[_],
+      path: Path,
+      appId: String): Long = {
+    singletonEntity(store, klass, appId)
+      .map { value =>
+        writeValuesFile(path, Iterator.single(redact(value)))
+      }
+      .getOrElse {
+        Utils.tryWithResource(store.view(klass.asInstanceOf[Class[AnyRef]]).closeableIterator()) {
+          it =>
+            writeValuesFile(path,
+              Iterator.continually(it).takeWhile(_.hasNext).map(_.next().asInstanceOf[AnyRef]))
+        }
+      }
+  }
+
+  /** Serializes a stream of KVStore values using length-delimited protobuf records. */
+  private def writeValuesFile(path: Path, values: Iterator[AnyRef]): Long = {
+    var count = 0L
+    Utils.tryWithResource(new DataOutputStream(new BufferedOutputStream(fs.create(path, true)))) {

Review Comment:
   Verified fixed in e7eb0e10. Current-source filesystem checks produced 0770 for the owned directories and 0660 for the manifest/entity files, and the new permission regression passed in current-head CI. Resolving.



##########
core/src/main/scala/org/apache/spark/deploy/history/FsHistoryProvider.scala:
##########
@@ -1687,7 +1704,52 @@ private[history] class FsHistoryProvider(conf: SparkConf, clock: Clock)
     KVUtils.open(newStorePath, metadata, conf, live = false)
   }
 
-  private def createInMemoryStore(attempt: AttemptInfoWrapper): KVStore = {
+  private def createDiskStoreFromSnapshot(
+      dm: HistoryServerDiskManager,
+      appId: String,
+      attempt: AttemptInfoWrapper,
+      metadata: AppStatusStoreMetadata,
+      manifestPath: Path): KVStore = {
+    val lease = dm.leaseExact(snapshotStore.snapshotSize(manifestPath))
+    val startNs = System.nanoTime()
+    val newStorePath = try {
+      Utils.tryWithResource(KVUtils.open(lease.tmpPath, metadata, conf, live = false)) { store =>
+        snapshotStore.restoreSnapshot(store, manifestPath)
+      }
+      lease.commit(appId, attempt.info.attemptId)

Review Comment:
   Verified fixed in e7eb0e10. Both restore paths validate the expected app/attempt inside the fallback boundary, before disk commit or memory-store return. Current-head CI passes the empty and partially truncated entity cases with memory and disk stores on both backends. Resolving.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org