[GitHub] [spark] dongjoon-hyun commented on a diff in pull request #38958: [SPARK-41433][CONNECT] Make Max Arrow BatchSize configurable

[GitBox <gi...@apache.org>]

dongjoon-hyun commented on code in PR #38958:
URL: https://github.com/apache/spark/pull/38958#discussion_r1041991912


##########
connector/connect/server/src/main/scala/org/apache/spark/sql/connect/config/Connect.scala:
##########
@@ -34,4 +35,12 @@ private[spark] object Connect {
       .version("3.4.0")
       .stringConf
       .createOptional
+
+  val CONNECT_GRPC_ARROW_MAX_BATCH_SIZE =
+    ConfigBuilder("spark.connect.grpc.arrow.maxBatchSize")
+      .doc("When using Apache Arrow, limit the maximum size of one arrow batch that " +
+        "can be sent from server side to client side.")

Review Comment:
   BTW, do we need to mention that we use only 70% of this variable for now? Otherwise, it seems that `spark.connect.grpc.arrow.maxBatchSize` could be misleading, @zhengruifeng .



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org