[GitHub] [spark] viirya commented on a change in pull request #23670: [SPARK-26601][SQL] Make broadcast-exchange thread pool configurable

[GitBox <gi...@apache.org>]

viirya commented on a change in pull request #23670: [SPARK-26601][SQL] Make broadcast-exchange thread pool configurable
URL: https://github.com/apache/spark/pull/23670#discussion_r283082708
 
 

 ##########
 File path: sql/catalyst/src/main/scala/org/apache/spark/sql/internal/StaticSQLConf.scala
 ##########
 @@ -132,6 +132,19 @@ object StaticSQLConf {
       .intConf
       .createWithDefault(1000)
 
+  val BROADCAST_EXCHANGE_MAX_THREAD_THREASHOLD =
+    buildStaticConf("spark.sql.broadcastExchange.maxThreadThreshold")
+      .internal()
+      .doc("The maximum degree of parallelism to fetch and broadcast the table. " +
+        "If we encounter memory issue like frequently full GC or OOM when broadcast table " +
+        "we can decrease this number in order to reduce memory usage. " +
+        "Notice the number should be carefully chosen since decreasing parallelism might " +
+        "cause longer waiting for other broadcasting. Also, increasing parallelism may " +
+        "cause memory problem.")
+      .intConf
+      .checkValue(thres => thres > 0 && thres <= 128, "The threshold should be positive.")
 
 Review comment:
   There is a maximum limit (128) for this config.  The error message doesn't reflect that. Once an invalid value like 200 is set, it can't figure out what goes wrong with it.
   

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org