You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@datafusion.apache.org by "mbutrovich (via GitHub)" <gi...@apache.org> on 2026/03/04 15:55:26 UTC
[PR] [branch-52] Backport #20482 and dependencies [datafusion]
mbutrovich opened a new pull request, #20699:
URL: https://github.com/apache/datafusion/pull/20699
## Which issue does this PR close?
Backport of #20482 to branch-52.
## Rationale for this change
Cherry-pick fix and prerequisites so that SortMergeJoin emits output incrementally instead of waiting for all input to
complete. This resolves OOM issues Comet is seeing with DataFusion 52.
## What changes are included in this PR?
Cherry-picks of the following commits from `main`:
1. #19614 — Extract sort-merge join filter logic into separate module
2. #20463 — Use zero-copy slice instead of take kernel in sort merge join
3. #20482 — Fix SortMergeJoin to not wait for all input before emitting
## Are these changes tested?
Yes, covered by existing and new tests included in #20482.
## Are there any user-facing changes?
No.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] [branch-52] Backport #20482 and dependencies [datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on PR #20699:
URL: https://github.com/apache/datafusion/pull/20699#issuecomment-3998663740
Seems there are a few test failures
https://github.com/apache/datafusion/actions/runs/22677362632/job/65737685125?pr=20699
```
/__w/datafusion/datafusion/datafusion/spark)
error[E0061]: this function takes 8 arguments but 9 arguments were supplied
--> datafusion/physical-plan/src/joins/sort_merge_join/tests.rs:3232:22
|
3232 | let oracle = HashJoinExec::try_new(
| ^^^^^^^^^^^^^^^^^^^^^
...
3241 | false,
| ----- unexpected argument #9 of type `bool`
|
note: associated function defined here
--> datafusion/physical-plan/src/joins/hash_join/exec.rs:402:12
|
402 | pub fn try_new(
| ^^^^^^^
help: remove the extra argument
|
3240 - null_equality,
3241 - false,
3240 + null_equality,
|
Compiling datafusion-ffi v52.2.0
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] [branch-52] Backport fix: SortMergeJoin don't wait for all input before emitting #20482 [datafusion]
Posted by "mbutrovich (via GitHub)" <gi...@apache.org>.
mbutrovich merged PR #20699:
URL: https://github.com/apache/datafusion/pull/20699
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] [branch-52] Backport fix: SortMergeJoin don't wait for all input before emitting #20482 [datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on PR #20699:
URL: https://github.com/apache/datafusion/pull/20699#issuecomment-4135657654
> Was this released? it doesn't seem like it
Supposedly it was released in 52.3.0:
https://github.com/apache/datafusion/blob/7cbc6b4383e6998b6a3a94ce1018a01e03c50be8/dev/changelog/52.3.0.md?plain=1#L32
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] [branch-52] Backport fix: SortMergeJoin don't wait for all input before emitting #20482 [datafusion]
Posted by "rluvaton (via GitHub)" <gi...@apache.org>.
rluvaton commented on PR #20699:
URL: https://github.com/apache/datafusion/pull/20699#issuecomment-4135671268
Oh you are right, refined-github extension misled me
<img width="1106" height="113" alt="image" src="https://github.com/user-attachments/assets/16761333-9f4f-41a8-9b40-a5e2de777a8a" />
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] [branch-52] Backport fix: SortMergeJoin don't wait for all input before emitting #20482 [datafusion]
Posted by "rluvaton (via GitHub)" <gi...@apache.org>.
rluvaton commented on PR #20699:
URL: https://github.com/apache/datafusion/pull/20699#issuecomment-4134903256
Was this released? it doesn't seem like it
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org