You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@datafusion.apache.org by "milenkovicm (via GitHub)" <gi...@apache.org> on 2026/05/25 20:52:08 UTC
[I] make sort shuffle respect `datafusion.optimizer.hash_join_single_partition_threshold` [datafusion-ballista]
milenkovicm opened a new issue, #1770:
URL: https://github.com/apache/datafusion-ballista/issues/1770
**Is your feature request related to a problem or challenge? Please describe what you are trying to do.**
static job planner uses `ballista.optimizer.broadcast_join_threshold_bytes` configuration value to decide if shuffle should be broadcast (`CollectLeft`), this configuration value diverges from datafusion configurations:
- `datafusion.optimizer.hash_join_single_partition_threshold_rows`
- `datafusion.optimizer.hash_join_single_partition_threshold`
which serve same purpose.
**Describe the solution you'd like**
I would propose to change static planner to to respect same configuration values like datafusion `datafusion.optimizer.hash_join_single_partition_threshold_rows` & `datafusion.optimizer.hash_join_single_partition_threshold`
AQE implementation (#1752) uses the same values as datafusion so it would be good if we can have both planners (and datafusion) aligned
**Describe alternatives you've considered**
align both planners to `ballista.optimizer.broadcast_join_threshold_bytes` but that would diverge from datafusion configuration, which is not ideal as ballista tends to align with datafusion as much as possible
**Additional context**
Add any other context or screenshots about the feature request here.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] make sort shuffle respect `datafusion.optimizer.hash_join_single_partition_threshold` [datafusion-ballista]
Posted by "milenkovicm (via GitHub)" <gi...@apache.org>.
milenkovicm commented on issue #1770:
URL: https://github.com/apache/datafusion-ballista/issues/1770#issuecomment-4537161929
cc @andygrove i hope you approve this configuration change for broadcast planning
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] make sort shuffle respect `datafusion.optimizer.hash_join_single_partition_threshold` [datafusion-ballista]
Posted by "milenkovicm (via GitHub)" <gi...@apache.org>.
milenkovicm commented on issue #1770:
URL: https://github.com/apache/datafusion-ballista/issues/1770#issuecomment-4561491710
Hi @ramieali after a bit of consideration I'm not sure if this would be good first time issue
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] make sort shuffle respect `datafusion.optimizer.hash_join_single_partition_threshold` [datafusion-ballista]
Posted by "ramieali (via GitHub)" <gi...@apache.org>.
ramieali commented on issue #1770:
URL: https://github.com/apache/datafusion-ballista/issues/1770#issuecomment-4558910936
Hi @milenkovicm , i would like to work on this issue if it's still up for grabs
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] make sort shuffle respect `datafusion.optimizer.hash_join_single_partition_threshold` [datafusion-ballista]
Posted by "ramieali (via GitHub)" <gi...@apache.org>.
ramieali commented on issue #1770:
URL: https://github.com/apache/datafusion-ballista/issues/1770#issuecomment-4577985751
thanks for the heads up! if i am not underscoping it then i think a simple refactor would be changing planner config
https://github.com/apache/datafusion-ballista/blob/65ec1c479acf9fe7acb19a0780bc0a6100739b9c/ballista/scheduler/src/planner.rs#L280-L286
to `let threshold_bytes = config.optimizer.hash_join_single_partition_threshold;`
and flag ballista-specific config with deprecated attribute
```
#[deprecated(note = "use datafusion.optimizer.hash_join_single_partition_threshold instead")]
fn with_ballista_broadcast_join_threshold_bytes(self, threshold_bytes: usize)
-> Self;
```
and maybe address one of the examples that references it
https://github.com/apache/datafusion-ballista/blob/65ec1c479acf9fe7acb19a0780bc0a6100739b9c/examples/examples/standalone-broadcast-join.rs#L132
happy to open a PR if that's the right scope
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Static job planner to respect `datafusion....hash_join_single_partition_threshold` for broadcast decision [datafusion-ballista]
Posted by "milenkovicm (via GitHub)" <gi...@apache.org>.
milenkovicm commented on issue #1770:
URL: https://github.com/apache/datafusion-ballista/issues/1770#issuecomment-4581947985
changing title as it was super confusing and incorrect
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org