Re: [I] Allow for scaling RangePartitioning [datafusion]

["stuhood (via GitHub)" <gi...@apache.org>]

stuhood commented on issue #24712:
URL: https://github.com/apache/datafusion/issues/24712#issuecomment-5656168290

   Ok, sorry for the delay!
   
   To ensure that scaling `RangePartitioning` is exercised on `main` and improve behavior related to PR #24600, we should integrate the feature from this ticket into `EnsureRequirements` when we land it.
   
   Currently, when a child operator is already partitioned with `Partitioning::Range` that satisfies `Distribution::KeyPartitioned`, `EnsureRequirements` replaces it with `Partitioning::Hash` whenever `target_partitions > child_partitions`:
   
   https://github.com/apache/datafusion/blob/8642154323c3800948f495af9ba2ee1539399105/datafusion/physical-optimizer/src/ensure_requirements/enforce_distribution.rs#L1574-L1595
   
   Because `Distribution::KeyPartitioned::create_partitioning` [unconditionally generates `Partitioning::Hash`](https://github.com/apache/datafusion/blob/8642154323c3800948f495af9ba2ee1539399105/datafusion/physical-expr/src/partitioning.rs#L766-L777), any preexisting range partitioning is destroyed before `enforce_distribution_relationships` can align child partitions.
   
   To fix this and exercise the new scaling logic:
   1. In `EnsureRequirements`, check if the existing satisfying `Partitioning` supports scaling before falling back to `Distribution::create_partitioning`.
   2. When scaling `RangePartitioning` is possible, use the scaled range partitioning.
   3. When scaling is not possible, preserve the satisfying range partitioning rather than overriding it with `Partitioning::Hash`, avoiding unnecessary 2-sided hash shuffles.
   
   ----
   
   @goutamadwant: If you have time to do this on #24766, that would be awesome: otherwise, I may tackle it this week.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org