You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@datafusion.apache.org by "mithuncy (via GitHub)" <gi...@apache.org> on 2026/07/14 17:03:53 UTC
[I] Add config-matrix tests in enforce_distribution.rs for range-satisfaction settings [datafusion]
mithuncy opened a new issue, #23572:
URL: https://github.com/apache/datafusion/issues/23572
Related:
- EPIC: #22395
- Windows: #23289 (PR #23416, where this follow-up was requested)
- Aggregations: #23239
- Joins: #23184
- PartitionedTopK: #23290 (PR #23355)
## Is your feature request related to a problem or challenge?
Each operator that opts into range satisfaction currently re-tests the same shared config machinery (`datafusion.optimizer.subset_repartition_threshold`, `datafusion.optimizer.preserve_file_partitions`, `datafusion.execution.target_partitions`) through per-operator blocks in `range_partitioning.slt`. The machinery is shared, so the file grows the same permutations for every operator.
This was raised in the #23416 review ([discussion](https://github.com/apache/datafusion/pull/23416#discussion_r3564136172)): keep `range_partitioning.slt` lean per operator, and cover the config permutations once, colocated with the decision logic.
## Describe the solution you'd like
A table-driven test in `datafusion/core/tests/physical_optimizer/enforce_distribution.rs` asserting reuse (no `RepartitionExec` inserted; the existing Range partitioning satisfies the requirement) vs. hash repartition (`RepartitionExec: Hash` inserted) across combinations of:
- key match: exact / subset / incompatible
- `subset_repartition_threshold`: met / not met
- `preserve_file_partitions`: met / not met
- `target_partitions`: equal to / greater than input partition count
Behavior verified while developing #23416 that the matrix must pin:
- exact key, threshold not met, equal partition count → reuse (repartitioning gains no parallelism)
- exact key, threshold not met, target > count, preserve not met → hash repartition
- exact key, threshold not met, target > count, preserve met → reuse (the threshold and preserve conditions are OR'd in `ensure_distribution`)
- exact key, threshold met, preserve not met, target > count → reuse
- subset key, threshold not met, preserve not met → hash repartition
- incompatible key → hash repartition in all configurations
## Describe alternatives you've considered
None.
## Additional context
- Once the matrix lands, the equivalent per-operator slt permutations in the aggregate section of `range_partitioning.slt` can also be slimmed, and future operator opt-ins only need operator-specific slt cases.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Add config-matrix tests in enforce_distribution.rs for range-satisfaction settings [datafusion]
Posted by "gabotechs (via GitHub)" <gi...@apache.org>.
gabotechs closed issue #23572: Add config-matrix tests in enforce_distribution.rs for range-satisfaction settings
URL: https://github.com/apache/datafusion/issues/23572
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Add config-matrix tests in enforce_distribution.rs for range-satisfaction settings [datafusion]
Posted by "blinding-pixels (via GitHub)" <gi...@apache.org>.
blinding-pixels commented on issue #23572:
URL: https://github.com/apache/datafusion/issues/23572#issuecomment-4977748681
take
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org