You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@datafusion.apache.org by "buraksenn (via GitHub)" <gi...@apache.org> on 2026/08/08 20:33:13 UTC
[PR] minor(test): strengthen bitwise sort-merge join spill coverage [datafusion]
buraksenn opened a new pull request, #24192:
URL: https://github.com/apache/datafusion/pull/24192
## Which issue does this PR close?
- Part of #13431
## Rationale for this change
Continuing the work after #23988 and #24056. `BitwiseSortMergeJoinStream` (filtered semi/anti/mark joins) buffers the inner key group and spills it under memory pressure, but existing tests only cover the successful spill path:
- when the memory pool is exhausted and disk spilling is disabled, the error path had no coverage
- the multi-spill slice retention regression test only asserted the semi join outcome; a dropped slice resurrects the anti row and flips mark to false, which row counts alone cannot catch
- no spill test exercised `NullEqualsNull`, so NULL-key group boundary detection through the spill path was untested
## What changes are included in this PR?
- new `bitwise_filtered_no_spill`: all six semi/anti/mark join types must surface a clear "Disk spilling disabled" error (not spill or panic) when the pool is exhausted with the `DiskManager` disabled
- extend `bitwise_multi_spill_inner_key_group` over `LeftAnti`/`LeftMark`, asserting the mark column value
- new `bitwise_spill_null_key_group`: under `NullEqualsNull`, a NULL-key group spanning a batch boundary must spill as one group with one spill file and match a NULL-key outer row
- extract shared setup of `bitwise_spill_with_filter` into a fixture
## Are these changes tested?
test only change and it can be tested via:
```
cargo test -p datafusion-physical-plan --lib -- sort_merge_join::tests::bitwise
```
## Are there any user-facing changes?
no test only change
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] minor(test): strengthen bitwise sort-merge join spill coverage [datafusion]
Posted by "kosiew (via GitHub)" <gi...@apache.org>.
kosiew commented on code in PR #24192:
URL: https://github.com/apache/datafusion/pull/24192#discussion_r3800963206
##########
datafusion/physical-plan/src/joins/sort_merge_join/tests.rs:
##########
@@ -5060,6 +5136,113 @@ async fn bitwise_multi_spill_inner_key_group() -> Result<()> {
// 100-byte pool: every buffered slice fails its reservation, so each
// inner batch of the key group spills separately.
+ let runtime = RuntimeEnvBuilder::new()
+ .with_memory_limit(100, 1.0)
+ .with_disk_manager_builder(
+ DiskManagerBuilder::default().with_mode(DiskManagerMode::OsTmpDirectory),
+ )
+ .build_arc()?;
+
+ for (join_type, expected_rows) in [(LeftSemi, 1), (LeftAnti, 0), (LeftMark, 1)] {
Review Comment:
Nice coverage here. It might also be worth adding a mirrored multi-spill fixture for `RightSemi`, `RightAnti`, and `RightMark`. Those variants buffer `left` while streaming `right`, so the two-batch equal-key group would need to be on `left`. Adding them directly to this fixture would not exercise multi-spill retention since its `left` input only has one row. The new no-spill test already covers the error path for these variants, so this is just a non-blocking suggestion.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] minor(test): strengthen bitwise sort-merge join spill coverage [datafusion]
Posted by "buraksenn (via GitHub)" <gi...@apache.org>.
buraksenn commented on code in PR #24192:
URL: https://github.com/apache/datafusion/pull/24192#discussion_r3801725678
##########
datafusion/physical-plan/src/joins/sort_merge_join/tests.rs:
##########
@@ -5060,6 +5136,113 @@ async fn bitwise_multi_spill_inner_key_group() -> Result<()> {
// 100-byte pool: every buffered slice fails its reservation, so each
// inner batch of the key group spills separately.
+ let runtime = RuntimeEnvBuilder::new()
+ .with_memory_limit(100, 1.0)
+ .with_disk_manager_builder(
+ DiskManagerBuilder::default().with_mode(DiskManagerMode::OsTmpDirectory),
+ )
+ .build_arc()?;
+
+ for (join_type, expected_rows) in [(LeftSemi, 1), (LeftAnti, 0), (LeftMark, 1)] {
Review Comment:
thanks for the review, I've added coverage for right joins as well
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] minor(test): strengthen bitwise sort-merge join spill coverage [datafusion]
Posted by "kosiew (via GitHub)" <gi...@apache.org>.
kosiew commented on PR #24192:
URL: https://github.com/apache/datafusion/pull/24192#issuecomment-5337140881
🚀
Thanks @buraksenn for working on this.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] minor(test): strengthen bitwise sort-merge join spill coverage [datafusion]
Posted by "kosiew (via GitHub)" <gi...@apache.org>.
kosiew merged PR #24192:
URL: https://github.com/apache/datafusion/pull/24192
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] minor(test): strengthen bitwise sort-merge join spill coverage [datafusion]
Posted by "codecov-commenter (via GitHub)" <gi...@apache.org>.
codecov-commenter commented on PR #24192:
URL: https://github.com/apache/datafusion/pull/24192#issuecomment-5228149546
## [Codecov](https://app.codecov.io/gh/apache/datafusion/pull/24192?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
:white_check_mark: All modified and coverable lines are covered by tests.
:white_check_mark: Project coverage is 81.05%. Comparing base ([`eec8b94`](https://app.codecov.io/gh/apache/datafusion/commit/eec8b947c71a916385a28a3723753f4ffc7cd8d0?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)) to head ([`4d5444b`](https://app.codecov.io/gh/apache/datafusion/commit/4d5444b169984cbccfea27af3a6db2ef6505207d?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)).
<details><summary>Additional details and impacted files</summary>
```diff
@@ Coverage Diff @@
## main #24192 +/- ##
=======================================
Coverage 81.05% 81.05%
=======================================
Files 1106 1106
Lines 382287 382287
Branches 382287 382287
=======================================
+ Hits 309851 309854 +3
+ Misses 54121 54116 -5
- Partials 18315 18317 +2
```
</details>
[:umbrella: View full report in Codecov by Harness](https://app.codecov.io/gh/apache/datafusion/pull/24192?dropdown=coverage&src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
:loudspeaker: Have feedback on the report? [Share it here](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
<details><summary> :rocket: New features to boost your workflow: </summary>
- :snowflake: [Test Analytics](https://docs.codecov.com/docs/test-analytics): Detect flaky tests, report on failures, and find test suite problems.
- :package: [JS Bundle Analysis](https://docs.codecov.com/docs/javascript-bundle-analysis): Save yourself from yourself by tracking and limiting bundle sizes in JS merges.
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org