[GitHub] [spark] mridulm commented on a diff in pull request #38567: [SPARK-41054][UI][CORE] Support RocksDB as KVStore in live UI

[GitBox <gi...@apache.org>]

mridulm commented on code in PR #38567:
URL: https://github.com/apache/spark/pull/38567#discussion_r1027618122


##########
core/src/main/scala/org/apache/spark/status/AppStatusStore.scala:
##########
@@ -769,7 +772,14 @@ private[spark] object AppStatusStore {
   def createLiveStore(
       conf: SparkConf,
       appStatusSource: Option[AppStatusSource] = None): AppStatusStore = {
-    val store = new ElementTrackingStore(new InMemoryStore(), conf)
+    val storePath = conf.get(LIVE_UI_LOCAL_STORE_DIR).map(new File(_))
+    // For the disk-based KV store of live UI, let's simply make it ROCKSDB only for now,
+    // instead of supporting both LevelDB and RocksDB. RocksDB is built based on LevelDB with
+    // improvements on writes and reads. Furthermore, we can reuse the RocksDBFileManager in
+    // streaming for replicating the local RocksDB file to DFS. The replication in DFS can be
+    // used for Spark history server.

Review Comment:
   nit: Can we remove the part about replication for now ? When we add support for it, we can suitably modify the comment.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org