You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by "dongjoon-hyun (via GitHub)" <gi...@apache.org> on 2026/09/03 04:11:34 UTC
[PR] [SPARK-59191] Support `grpc_max_message_size` in connection string [spark-connect-swift]
dongjoon-hyun opened a new pull request, #505:
URL: https://github.com/apache/spark-connect-swift/pull/505
### What changes were proposed in this pull request?
This PR proposes to support the `grpc_max_message_size` parameter in the Spark
Connect connection string.
- `SparkConnectClient.init(remote:)` parses it now. Previously it fell through to
the `default` branch and only printed `Unknown parameter: ...`, leaving the
existing `URIParams.PARAM_GRPC_MAX_MESSAGE_SIZE` constant unused.
- A non-positive or non-integer value throws `SparkConnectError.InvalidArgument`,
like the other invalid inputs in the same initializer.
- The value is applied to the gRPC transport through a default `MethodConfig`
(an empty service name applies to all services and methods) in a `ServiceConfig`,
at both places creating the transport: `SparkConnectClient.withGPRC` and
`DataFrame.withGPRC`. Both `maxRequestMessageBytes` and `maxResponseMessageBytes`
are set, matching PySpark which sets both `grpc.max_send_message_length` and
`grpc.max_receive_message_length`.
When the parameter is absent, an empty `ServiceConfig` is used and the behavior is
unchanged (gRPC's 4MiB default).
### Why are the changes needed?
`grpc_max_message_size` is supported by the PySpark and Scala clients and is used to
avoid hitting the default gRPC message size limit when collecting large results.
The Swift client silently dropped it, so users had no way to raise the limit.
### Does this PR introduce _any_ user-facing change?
Yes. `sc://localhost:15002/;grpc_max_message_size=268435456` is now honored instead
of ignored, and an invalid value such as `grpc_max_message_size=abc` now throws
`SparkConnectError.InvalidArgument` instead of being ignored.
### How was this patch tested?
Pass the CIs with the newly added test cases in `SparkConnectClientTests`.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Opus 5
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-59191] Support `grpc_max_message_size` in connection string [spark-connect-swift]
Posted by "dongjoon-hyun (via GitHub)" <gi...@apache.org>.
dongjoon-hyun commented on PR #505:
URL: https://github.com/apache/spark-connect-swift/pull/505#issuecomment-5520410645
Thank you, @viirya . 😄
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-59191] Support `grpc_max_message_size` in connection string [spark-connect-swift]
Posted by "dongjoon-hyun (via GitHub)" <gi...@apache.org>.
dongjoon-hyun commented on PR #505:
URL: https://github.com/apache/spark-connect-swift/pull/505#issuecomment-5520449481
Merged to main
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-59191] Support `grpc_max_message_size` in connection string [spark-connect-swift]
Posted by "dongjoon-hyun (via GitHub)" <gi...@apache.org>.
dongjoon-hyun closed pull request #505: [SPARK-59191] Support `grpc_max_message_size` in connection string
URL: https://github.com/apache/spark-connect-swift/pull/505
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-59191] Support `grpc_max_message_size` in connection string [spark-connect-swift]
Posted by "dongjoon-hyun (via GitHub)" <gi...@apache.org>.
dongjoon-hyun commented on PR #505:
URL: https://github.com/apache/spark-connect-swift/pull/505#issuecomment-5520441549
Merged to main
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org