You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@datafusion.apache.org by "comphead (via GitHub)" <gi...@apache.org> on 2025/01/08 22:24:08 UTC
[I] Optimize filtered SortMergeJoin to avoid producing small/empty batches [datafusion]
comphead opened a new issue, #14050:
URL: https://github.com/apache/datafusion/issues/14050
### Is your feature request related to a problem or challenge?
Related to #9846
In #9846 there is a couple of tasks to fix the correctness issues for SortMergeJoin with filter clause
As reported in https://github.com/apache/datafusion-comet/pull/1211#issuecomment-2568581478 the execution produces a lot of small/empty batches
### Describe the solution you'd like
Concat small batches and produces batches equal or near equal to `batch_size`
### Describe alternatives you've considered
_No response_
### Additional context
_No response_
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Optimize filtered SortMergeJoin to avoid producing small/empty batches [datafusion]
Posted by "comphead (via GitHub)" <gi...@apache.org>.
comphead closed issue #14050: Optimize filtered SortMergeJoin to avoid producing small/empty batches
URL: https://github.com/apache/datafusion/issues/14050
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org