You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@datafusion.apache.org by "gene-bordegaray (via GitHub)" <gi...@apache.org> on 2026/07/11 13:47:47 UTC
[I] Allow co-partitioned `Partitioning::Range` inputs for symmetric hash joins [datafusion]
gene-bordegaray opened a new issue, #23479:
URL: https://github.com/apache/datafusion/issues/23479
Related:
- [#22395](https://github.com/apache/datafusion/issues/22395)
- [#21992](https://github.com/apache/datafusion/issues/21992)
- [#23184](https://github.com/apache/datafusion/pull/23184)
- [#23451](https://github.com/apache/datafusion/issues/23451)
- [#23266](https://github.com/apache/datafusion/issues/23266)
## Is your feature request related to a problem or challenge?
#23451 makes partitioned `SymmetricHashJoinExec` require co-partitioned children, but compatible range-partitioned streams still take the conservative hash-repartition path. A partitioned symmetric hash join can process matching partition indexes locally when the inputs have compatible range layouts.
## Example
```text
SymmetricHashJoinExec: mode=Partitioned, join_type=Full, on=[(range_key@0, range_key@0)]
StreamingTableExec: output_partitioning=Range([range_key@0 ASC], [(10), (20), (30)], 4)
StreamingTableExec: output_partitioning=Range([range_key@0 ASC], [(10), (20), (30)], 4)
```
The compatible range inputs are already co-partitioned by partition index. The optimizer should not insert hash repartitions.
## Describe the solution you'd like
Opt partitioned `SymmetricHashJoinExec` into the temporary compatible-range satisfaction path and provide a streaming test source that exposes range partitioning.
Covered join types:
- Supported `SymmetricHashJoinExec` join types in `StreamJoinPartitionMode::Partitioned`.
## Acceptance criteria
- Compatible range/range inputs avoid hash repartitioning in partitioned mode.
- Incompatible split points, partition counts, sort options, or key expressions still trigger repartitioning.
- Tests cover inner and full joins, including matched and unmatched rows.
- The streaming test source exposes compatible and incompatible range layouts.
## Additional context
This should follow the multi-child requirement model added in #23451 and precede the general consolidation in #23266.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Allow co-partitioned `Partitioning::Range` inputs for symmetric hash joins [datafusion]
Posted by "Udaytaneja (via GitHub)" <gi...@apache.org>.
Udaytaneja commented on issue #23479:
URL: https://github.com/apache/datafusion/issues/23479#issuecomment-4947212485
Hi mantainers, I would like to work on this one . Please assign this to me
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Allow co-partitioned `Partitioning::Range` inputs for symmetric hash joins [datafusion]
Posted by "Rich-T-kid (via GitHub)" <gi...@apache.org>.
Rich-T-kid commented on issue #23479:
URL: https://github.com/apache/datafusion/issues/23479#issuecomment-4947976309
> Hi mantainers, I would like to work on this one . Please assign this to me
@Udaytaneja u can take an issue your interested in by commenting
> take
on the issue and the GitHub bot will assign it to you 👍
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Allow co-partitioned `Partitioning::Range` inputs for symmetric hash joins [datafusion]
Posted by "gabotechs (via GitHub)" <gi...@apache.org>.
gabotechs closed issue #23479: Allow co-partitioned `Partitioning::Range` inputs for symmetric hash joins
URL: https://github.com/apache/datafusion/issues/23479
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Allow co-partitioned `Partitioning::Range` inputs for symmetric hash joins [datafusion]
Posted by "Udaytaneja (via GitHub)" <gi...@apache.org>.
Udaytaneja commented on issue #23479:
URL: https://github.com/apache/datafusion/issues/23479#issuecomment-4948020550
take
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org