You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@datafusion.apache.org by "hcrosse (via GitHub)" <gi...@apache.org> on 2026/04/03 13:27:46 UTC

[PR] feat: make BatchPartitioner::partition_iter public [datafusion]

hcrosse opened a new pull request, #21341:
URL: https://github.com/apache/datafusion/pull/21341

   ## Which issue does this PR close?
   
   Closes #21311.
   
   ## Rationale for this change
   
   `BatchPartitioner::partition_iter` is already used internally as the core implementation behind the public `partition` method, and was intentionally factored out to support both sync and async consumption patterns. However, since it's private, downstream crates like Ballista can't use the iterator directly and are forced to run both CPU-bound partitioning and I/O together in a sync closure.
   
   ## What changes are included in this PR?
   
   Changed `partition_iter` visibility from private to public.
   
   ## Are these changes tested?
   
   The existing tests for `BatchPartitioner` cover `partition_iter` indirectly through the `partition` method, which delegates to it. No behavioral change was made.
   
   ## Are there any user-facing changes?
   
   `BatchPartitioner::partition_iter` is now part of the public API. This is a purely additive change with no breaking impact.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] feat: make BatchPartitioner::partition_iter public [datafusion]

Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on PR #21341:
URL: https://github.com/apache/datafusion/pull/21341#issuecomment-4183910946

   ![573567179-f08a3c78-6533-438f-8377-eb81dbcbe2a0](https://github.com/user-attachments/assets/e0e18d66-631a-4e2e-83e8-8a43c7adca61)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] feat: make BatchPartitioner::partition_iter public [datafusion]

Posted by "blaginin (via GitHub)" <gi...@apache.org>.
blaginin commented on PR #21341:
URL: https://github.com/apache/datafusion/pull/21341#issuecomment-4183903421

   it worked, it was very fast 😅
   
   ![image](https://github.com/user-attachments/assets/f08a3c78-6533-438f-8377-eb81dbcbe2a0)


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] feat: make BatchPartitioner::partition_iter public [datafusion]

Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on PR #21341:
URL: https://github.com/apache/datafusion/pull/21341#issuecomment-4183896124

   @blaginin  FYI I tried the merge queue on this PR and it seems like it just merged directly (didn't wait for the checks 🤔 )


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] feat: make BatchPartitioner::partition_iter public [datafusion]

Posted by "blaginin (via GitHub)" <gi...@apache.org>.
blaginin commented on PR #21341:
URL: https://github.com/apache/datafusion/pull/21341#issuecomment-4183930410

   Right! In this list you see two types of actions:
   - "merge group" - those are ran the merge queue. Currently, we only have dev.yml checks there
   - "push" the normal "old" CI on every push to develop. Those check involve rust.yml, dev.yml, extended.yml and trigger on every commit (that's why you see "pending" now)


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] feat: make BatchPartitioner::partition_iter public [datafusion]

Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on PR #21341:
URL: https://github.com/apache/datafusion/pull/21341#issuecomment-4183880205

   Enabled auto merge so we can test out the merge queue


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] feat: make BatchPartitioner::partition_iter public [datafusion]

Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on PR #21341:
URL: https://github.com/apache/datafusion/pull/21341#issuecomment-4183878220

   Looks good to me -- thank you @hcrosse 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] feat: make BatchPartitioner::partition_iter public [datafusion]

Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb merged PR #21341:
URL: https://github.com/apache/datafusion/pull/21341


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] feat: make BatchPartitioner::partition_iter public [datafusion]

Posted by "blaginin (via GitHub)" <gi...@apache.org>.
blaginin commented on PR #21341:
URL: https://github.com/apache/datafusion/pull/21341#issuecomment-4183936199

   ![IMG_6113](https://github.com/user-attachments/assets/a262df15-1897-4ba0-93ab-707ce47710c7)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org