You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by GitBox <gi...@apache.org> on 2022/12/07 05:06:36 UTC
[GitHub] [spark] zhengruifeng opened a new pull request, #38958: [SPARK-41433][CONNECT] Make Max Arrow BatchSize configurable
zhengruifeng opened a new pull request, #38958:
URL: https://github.com/apache/spark/pull/38958
### What changes were proposed in this pull request?
Make Max Arrow BatchSize configurable
### Why are the changes needed?
make batchsize configurable
### Does this PR introduce _any_ user-facing change?
yes, one new configration
### How was this patch tested?
existing tests
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on a diff in pull request #38958: [SPARK-41433][CONNECT] Make Max Arrow BatchSize configurable
Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on code in PR #38958:
URL: https://github.com/apache/spark/pull/38958#discussion_r1041991912
##########
connector/connect/server/src/main/scala/org/apache/spark/sql/connect/config/Connect.scala:
##########
@@ -34,4 +35,12 @@ private[spark] object Connect {
.version("3.4.0")
.stringConf
.createOptional
+
+ val CONNECT_GRPC_ARROW_MAX_BATCH_SIZE =
+ ConfigBuilder("spark.connect.grpc.arrow.maxBatchSize")
+ .doc("When using Apache Arrow, limit the maximum size of one arrow batch that " +
+ "can be sent from server side to client side.")
Review Comment:
BTW, do we need to mention that we use only 70% of this variable for now? Otherwise, it seems that `spark.connect.grpc.arrow.maxBatchSize` could be misleading, @zhengruifeng .
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] zhengruifeng commented on pull request #38958: [SPARK-41433][CONNECT] Make Max Arrow BatchSize configurable
Posted by GitBox <gi...@apache.org>.
zhengruifeng commented on PR #38958:
URL: https://github.com/apache/spark/pull/38958#issuecomment-1340581564
merged into master
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] zhengruifeng commented on a diff in pull request #38958: [SPARK-41433][CONNECT] Make Max Arrow BatchSize configurable
Posted by GitBox <gi...@apache.org>.
zhengruifeng commented on code in PR #38958:
URL: https://github.com/apache/spark/pull/38958#discussion_r1042025366
##########
connector/connect/server/src/main/scala/org/apache/spark/sql/connect/config/Connect.scala:
##########
@@ -34,4 +35,12 @@ private[spark] object Connect {
.version("3.4.0")
.stringConf
.createOptional
+
+ val CONNECT_GRPC_ARROW_MAX_BATCH_SIZE =
+ ConfigBuilder("spark.connect.grpc.arrow.maxBatchSize")
+ .doc("When using Apache Arrow, limit the maximum size of one arrow batch that " +
+ "can be sent from server side to client side.")
Review Comment:
got it, will update this doc. Thanks!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] zhengruifeng commented on pull request #38958: [SPARK-41433][CONNECT] Make Max Arrow BatchSize configurable
Posted by GitBox <gi...@apache.org>.
zhengruifeng commented on PR #38958:
URL: https://github.com/apache/spark/pull/38958#issuecomment-1340494395
cc @grundprinzip @hvanhovell
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] amaliujia commented on a diff in pull request #38958: [SPARK-41433][CONNECT] Make Max Arrow BatchSize configurable
Posted by GitBox <gi...@apache.org>.
amaliujia commented on code in PR #38958:
URL: https://github.com/apache/spark/pull/38958#discussion_r1042549501
##########
connector/connect/server/src/main/scala/org/apache/spark/sql/connect/config/Connect.scala:
##########
@@ -34,4 +35,12 @@ private[spark] object Connect {
.version("3.4.0")
.stringConf
.createOptional
+
+ val CONNECT_GRPC_ARROW_MAX_BATCH_SIZE =
+ ConfigBuilder("spark.connect.grpc.arrow.maxBatchSize")
+ .doc("When using Apache Arrow, limit the maximum size of one arrow batch that " +
+ "can be sent from server side to client side.")
Review Comment:
nice catch!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] zhengruifeng closed pull request #38958: [SPARK-41433][CONNECT] Make Max Arrow BatchSize configurable
Posted by GitBox <gi...@apache.org>.
zhengruifeng closed pull request #38958: [SPARK-41433][CONNECT] Make Max Arrow BatchSize configurable
URL: https://github.com/apache/spark/pull/38958
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] amaliujia commented on pull request #38958: [SPARK-41433][CONNECT] Make Max Arrow BatchSize configurable
Posted by GitBox <gi...@apache.org>.
amaliujia commented on PR #38958:
URL: https://github.com/apache/spark/pull/38958#issuecomment-1341398873
+1 late LGTM
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org