[GitHub] [spark] cloud-fan commented on a diff in pull request #39202: [SPARK-41685][UI] Support Protobuf serializer for the KVStore in History server

[GitBox <gi...@apache.org>]

cloud-fan commented on code in PR #39202:
URL: https://github.com/apache/spark/pull/39202#discussion_r1058030669


##########
core/src/main/scala/org/apache/spark/internal/config/History.scala:
##########
@@ -79,6 +79,21 @@ private[spark] object History {
     .stringConf
     .createOptional
 
+  object LocalStoreSerializer extends Enumeration {
+    val JSON, PROTOBUF = Value
+  }
+
+  val LOCAL_STORE_SERIALIZER = ConfigBuilder("spark.history.store.serializer")
+    .doc("Serializer for writing/reading in-memory UI objects to/from disk-based KV Store; " +
+      "JSON or PROTOBUF. JSON serializer is the only choice before Spark 3.4.0, thus it is the " +
+      "default value. PROTOBUF serializer is fast and compact, and it is the default " +
+      "serializer for disk-based KV store of live UI.")

Review Comment:
   I'm a bit confused about this config. SHS only reads the rocksdb files, but not write them. So it should support whatever format of the rocksdb values. Is there a way to encode the format in rocksdb as well?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org