You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by "HeartSaVioR (via GitHub)" <gi...@apache.org> on 2023/07/19 05:38:50 UTC

[GitHub] [spark] HeartSaVioR commented on a diff in pull request #42066: [SPARK-44480][SS] Use thread pool to perform maintenance activity for hdfs/rocksdb state store providers

HeartSaVioR commented on code in PR #42066:
URL: https://github.com/apache/spark/pull/42066#discussion_r1267562812


##########
sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala:
##########
@@ -1852,6 +1852,24 @@ object SQLConf {
       .createWithDefault(
         "org.apache.spark.sql.execution.streaming.state.HDFSBackedStateStoreProvider")
 
+  val ENABLE_STATE_STORE_MAINTENANCE_THREAD_POOL =
+    buildConf("spark.sql.streaming.stateStore.enableStateStoreMaintenanceThreadPool")
+      .internal()
+      .doc("Whether background maintenance tasks are executed by single thread or thread pool")

Review Comment:
   Shall we provide the context to inform the reason the old mechanism is set to default? Something along the line (please refine the sentence): the thread pool is consuming more resource than single thread, which might affect the performance of the query. Enable this if you observe the issue with the maintenance task.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org