You are viewing a plain text version of this content. The canonical link for it is here.

Posted to github@datafusion.apache.org by "andygrove (via GitHub)" <gi...@apache.org> on 2026/07/24 16:42:53 UTC

[PR] feat: bump default gRPC max message size 16 MiB -> 128 MiB [datafusion-ballista]

andygrove opened a new pull request, #2173:
URL: https://github.com/apache/datafusion-ballista/pull/2173

   ## Summary
   
   At SF1000 several TPC-H plans (Q11, Q21, Q22) encode above the 16 MiB default gRPC message-size ceiling and fail with:
   
   ```
   Status { code: OutOfRange, message: "Error, decoded message length too large:
           found ~22 MB, the limit is: 16777216 bytes" }
   ```
   
   Raising the runtime flags on scheduler/executor pods (`--grpc-server-max-*-message-size=...`) only lifts the server-side *receive* limits. Several paths still use `GrpcClientConfig::default()` (16 MiB) unconditionally — most notably the scheduler-to-executor gRPC client in `ExecutorManager::new`:
   
   https://github.com/apache/datafusion-ballista/blob/main/ballista/scheduler/src/state/executor_manager.rs#L88
   
   So task-assignment RPCs with large plans reject even when the servers have been raised, unless the user knows to also configure a config-producer override. Same story in `flight_proxy_service.rs` and the executor's client pool `Default` lookups.
   
   This PR bumps every default that fed a 16 MiB ceiling to **128 MiB**.
   
   Context: I hit this in [#2168](https://github.com/apache/datafusion-ballista/issues/2168) while running SF1000 TPC-H and got stuck for a while before realizing the scheduler-side flags don't cover this path.
   
   ## What changed
   
   - `ballista/core/src/config.rs` — `BALLISTA_CLIENT_GRPC_MAX_MESSAGE_SIZE` default (the SessionConfig-driven client path used by `distributed_query`).
   - `ballista/core/src/utils.rs` — `GrpcClientConfig::default()` `max_message_size` (used by scheduler-to-executor, flight proxy, executor client pool default lookups).
   - `ballista/scheduler/src/config.rs` and `ballista/executor/src/config.rs` — both CLI flags (`--grpc-server-max-{decoding,encoding}-message-size`) and the matching struct defaults.
   - `ballista/executor/src/executor_process.rs` — matching `ExecutorProcessConfig` defaults.
   - Help strings updated.
   
   The value is a **ceiling, not a preallocation**, so bumping it has no runtime cost for messages below the old limit.
   
   ## New test
   
   Adds a round-trip test that `ballista.client.grpc_max_message_size`, set on a `SessionConfig` via `options_mut().set(...)`, reaches `BallistaConfig::grpc_client_max_message_size()` unchanged. That path (options.set → ExtensionOptions → get_usize_setting) had no existing coverage, and it's the one Python clients rely on via their `cluster_config` overrides.
   
   ## Test plan
   
   - [x] `cargo test --workspace --lib` — all green (all pre-existing suites + 2 new/updated).
   - [x] `cargo fmt --all`.
   - [x] `cargo clippy --workspace --all-targets -- -D warnings` clean.
   - [ ] Reviewer sanity-check: any other places I missed that hard-code 16 MiB in a *default*? (Non-default hard-coded values in examples/ were left alone.)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org

Re: [PR] feat: bump default gRPC max message size 16 MiB -> 128 MiB [datafusion-ballista]

Posted by "milenkovicm (via GitHub)" <gi...@apache.org>.

milenkovicm commented on PR #2173:
URL: https://github.com/apache/datafusion-ballista/pull/2173#issuecomment-5073741400

   There is also outstanding pr which helps reducing message size #1853


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org

Re: [PR] feat: bump default gRPC max message size 16 MiB -> 128 MiB [datafusion-ballista]

Posted by "martin-g (via GitHub)" <gi...@apache.org>.

martin-g commented on PR #2173:
URL: https://github.com/apache/datafusion-ballista/pull/2173#issuecomment-5102078950

   I wonder whether 128MiB is too much ?!
   Having a lower value led to an optimisation like https://github.com/apache/datafusion-ballista/pull/1853. 
   With a big value like 128MiB it will be harder to notice that some message(s) became too heavy.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org

Re: [PR] feat: make gRPC max message size consistently configurable [datafusion-ballista]

Posted by "andygrove (via GitHub)" <gi...@apache.org>.

andygrove commented on PR #2173:
URL: https://github.com/apache/datafusion-ballista/pull/2173#issuecomment-5133044191

Thanks both, I've reworked this based on your feedback and updated the title and description.

@martin-g fair point about losing the signal that messages are getting heavy. I've dropped the default bump entirely, so everything stays at 16 MiB and optimisations like #1853 still get noticed when a message grows too big. The PR is now only about making the ceiling configurable where it wasn't, so operators who genuinely need a bigger one can raise it. I've also taken your MiB wording suggestions, just with the value left at 16MiB.

@phillipleblanc good catch, and it turned out to be worse than "not wired". `ExecutorManager` was computing `grpc_client_config.max_message_size` and then dropping it on the floor, because `get_client` built the client with a plain `ExecutorGrpcClient::new(connection)`. It now applies both `max_encoding_message_size` and `max_decoding_message_size`, with a comment explaining that these are tonic codec settings so `create_grpc_client_endpoint` cannot apply them. I checked every other client construction in the tree (`executor_process.rs`, `executor_server.rs`, `standalone.rs`, `flight_proxy_service.rs`, `client.rs`) and they all set the limits already, so this path was the only inconsistent one.

The net diff against main is now purely additive. One new scheduler flag, `--grpc-client-max-message-size`, defaulting to the 16 MiB the scheduler already used, plus tests that the flag reaches the client config and that an `override_config_producer` still takes precedence over it.

--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org

Re: [PR] feat: make gRPC max message size consistently configurable [datafusion-ballista]

Posted by "andygrove (via GitHub)" <gi...@apache.org>.

andygrove merged PR #2173:
URL: https://github.com/apache/datafusion-ballista/pull/2173


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org

Re: [PR] feat: make gRPC max message size consistently configurable [datafusion-ballista]

Posted by "andygrove (via GitHub)" <gi...@apache.org>.

andygrove commented on code in PR #2173:
URL: https://github.com/apache/datafusion-ballista/pull/2173#discussion_r3684122076


##########
ballista/executor/src/config.rs:
##########
@@ -169,17 +169,17 @@ pub struct Config {
         help = "Tracing log rotation policy."
     )]
     pub log_rotation_policy: ballista_core::config::LogRotationPolicy,
-    /// Maximum size of incoming gRPC messages in bytes (default: 16MB).
+    /// Maximum size of incoming gRPC messages in bytes (default: 128MB).

Review Comment:
   Applied, with the value left at 16MiB now that the default bump is gone.



##########
ballista/executor/src/config.rs:
##########
@@ -169,17 +169,17 @@ pub struct Config {
         help = "Tracing log rotation policy."
     )]
     pub log_rotation_policy: ballista_core::config::LogRotationPolicy,
-    /// Maximum size of incoming gRPC messages in bytes (default: 16MB).
+    /// Maximum size of incoming gRPC messages in bytes (default: 128MB).
     #[arg(
         long,
-        default_value_t = 16777216,
+        default_value_t = 134217728,
         help = "The maximum size of a decoded message at the grpc server side."
     )]
     pub grpc_server_max_decoding_message_size: u32,
-    /// Maximum size of outgoing gRPC messages in bytes (default: 16MB).
+    /// Maximum size of outgoing gRPC messages in bytes (default: 128MB).

Review Comment:
   Applied, with the value left at 16MiB now that the default bump is gone.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org

Re: [PR] feat: bump default gRPC max message size 16 MiB -> 128 MiB [datafusion-ballista]

Posted by "martin-g (via GitHub)" <gi...@apache.org>.

martin-g commented on code in PR #2173:
URL: https://github.com/apache/datafusion-ballista/pull/2173#discussion_r3655487400


##########
ballista/executor/src/config.rs:
##########
@@ -169,17 +169,17 @@ pub struct Config {
         help = "Tracing log rotation policy."
     )]
     pub log_rotation_policy: ballista_core::config::LogRotationPolicy,
-    /// Maximum size of incoming gRPC messages in bytes (default: 16MB).
+    /// Maximum size of incoming gRPC messages in bytes (default: 128MB).

Review Comment:
   ```suggestion
       /// Maximum size of incoming gRPC messages in bytes (default: 128MiB).
   ```



##########
ballista/executor/src/config.rs:
##########
@@ -169,17 +169,17 @@ pub struct Config {
         help = "Tracing log rotation policy."
     )]
     pub log_rotation_policy: ballista_core::config::LogRotationPolicy,
-    /// Maximum size of incoming gRPC messages in bytes (default: 16MB).
+    /// Maximum size of incoming gRPC messages in bytes (default: 128MB).
     #[arg(
         long,
-        default_value_t = 16777216,
+        default_value_t = 134217728,
         help = "The maximum size of a decoded message at the grpc server side."
     )]
     pub grpc_server_max_decoding_message_size: u32,
-    /// Maximum size of outgoing gRPC messages in bytes (default: 16MB).
+    /// Maximum size of outgoing gRPC messages in bytes (default: 128MB).

Review Comment:
   ```suggestion
       /// Maximum size of outgoing gRPC messages in bytes (default: 128MiB).
   ```



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org