You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@datafusion.apache.org by "mrhard9090 (via GitHub)" <gi...@apache.org> on 2026/10/03 18:40:58 UTC
[PR] fix: evaluate sort-merge join filters that reference no columns [datafusion]
mrhard9090 opened a new pull request, #26010:
URL: https://github.com/apache/datafusion/pull/26010
## Which issue does this PR close?
- Closes #26002.
## Rationale for this change
`SortMergeJoinExec` only evaluated its join filter when the filter read at least one column. A filter that reads none (a bound parameter such as `$1 > 5`, or a volatile function such as `random() > 2`) was skipped, so every key match was kept. For `FULL JOIN` that returns matched pairs the filter rejects, where hash join null-pads both sides.
## What changes are included in this PR?
- `freeze_streamed_matched` (`sort_merge_join/materializing_stream.rs`) now evaluates the filter whenever the join has one. The filter batch is built with `RecordBatchOptions::with_row_count(total_matched_rows)`, because a batch with no columns cannot infer its row count.
- Two queries in `sort_merge_join.slt`: `FULL JOIN ... AND random() > 2` (every row null-padded) and `AND random() < 2` (all key matches kept).
I did not touch `bitwise_stream.rs` (semi/anti/mark joins), which builds its filter batch in a different place.
## What is the testing strategy for this PR?
`cargo test -p datafusion-sqllogictest --test sqllogictests -- sort_merge_join`: all 4 files pass. With the `materializing_stream.rs` change reverted, `sort_merge_join.slt` fails. `cargo test -p datafusion-physical-plan --lib sort_merge_join`: 247 passed. `cargo fmt` and `cargo clippy -p datafusion-physical-plan --all-targets -- -D warnings` pass.
## Are there any user-facing changes?
`FULL JOIN` (and other join types run by sort-merge join) with a column-free `ON` condition now returns the same rows as hash join. No API change.
AI assistance: written with Claude Code; I reproduced the bug, reviewed the diff and ran the tests above.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] fix: evaluate sort-merge join filters that reference no columns [datafusion]
Posted by "mrhard9090 (via GitHub)" <gi...@apache.org>.
mrhard9090 commented on PR #26010:
URL: https://github.com/apache/datafusion/pull/26010#issuecomment-6123651170
Merged main (conflict was in `materializing_stream.rs` and `sort_merge_join.slt` around the new `filter_schema`). I also took the semi/anti/mark fix and your test into this PR, since it is the same bug: `bitwise_stream.rs` now builds the filter batch with the row count. The test fails without that change and passes with it. Thanks for the repro.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] fix: evaluate sort-merge join filters that reference no columns [datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb merged PR #26010:
URL: https://github.com/apache/datafusion/pull/26010
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] fix: evaluate sort-merge join filters that reference no columns [datafusion]
Posted by "blaginin (via GitHub)" <gi...@apache.org>.
blaginin commented on PR #26010:
URL: https://github.com/apache/datafusion/pull/26010#issuecomment-6189254559
https://github.com/apache/datafusion/actions/runs/38700092129/job/116305919332
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] fix: evaluate sort-merge join filters that reference no columns [datafusion]
Posted by "jayzhan211 (via GitHub)" <gi...@apache.org>.
jayzhan211 commented on PR #26010:
URL: https://github.com/apache/datafusion/pull/26010#issuecomment-6071772766
Same bug in the semi/anti/mark stream: with a column-free filter, `try_new` fails with `must either specify a row count or at least one column` for all six types. SQL can't reach it because the planner pushes the predicate into an input, but a `SortMergeJoinExec` built directly can. Fine to handle in a follow-up; it's a one-line fix:
```diff
- let filter_batch = RecordBatch::try_new(Arc::clone(filter.schema()), columns)?;
+ let filter_batch = RecordBatch::try_new_with_options(
+ Arc::clone(filter.schema()),
+ columns,
+ &RecordBatchOptions::new().with_row_count(Some(num_outer_rows)),
+ )?;
```
Test (fails on this branch, passes with the fix):
```rs
#[tokio::test]
async fn join_semi_anti_mark_column_free_filter() -> Result<()> {
for join_type in [
LeftSemi, LeftAnti, LeftMark, RightSemi, RightAnti, RightMark,
] {
let left = build_table(
("a1", &vec![1, 2, 3]),
("b1", &vec![4, 5, 7]),
("c1", &vec![7, 8, 9]),
);
let right = build_table(
("a2", &vec![10, 20, 30]),
("b1", &vec![4, 5, 6]),
("c2", &vec![70, 80, 90]),
);
let on = vec![(
Arc::new(Column::new_with_schema("b1", &left.schema())?) as _,
Arc::new(Column::new_with_schema("b1", &right.schema())?) as _,
)];
let filter = JoinFilter::new(
Arc::new(Literal::new(ScalarValue::Boolean(Some(false)))),
vec![],
Arc::new(Schema::empty()),
);
let (_, batches) =
join_collect_with_filter(left, right, on, filter, join_type).await?;
let rows: usize = batches.iter().map(|b| b.num_rows()).sum();
let expected = match join_type {
LeftSemi | RightSemi => 0,
_ => 3,
};
assert_eq!(rows, expected, "{join_type:?}");
}
Ok(())
}
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] fix: evaluate sort-merge join filters that reference no columns [datafusion]
Posted by "codecov-commenter (via GitHub)" <gi...@apache.org>.
codecov-commenter commented on PR #26010:
URL: https://github.com/apache/datafusion/pull/26010#issuecomment-5972508053
## [Codecov](https://app.codecov.io/gh/apache/datafusion/pull/26010?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
:x: Patch coverage is `89.36170%` with `5 lines` in your changes missing coverage. Please review.
:white_check_mark: Project coverage is 82.64%. Comparing base ([`95a1961`](https://app.codecov.io/gh/apache/datafusion/commit/95a1961cc7ec95bbb7b4015bfb0e2ed814b215c0?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)) to head ([`7c18075`](https://app.codecov.io/gh/apache/datafusion/commit/7c1807538d2058ce35ec7da505812b60f9f243c8?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)).
| [Files with missing lines](https://app.codecov.io/gh/apache/datafusion/pull/26010?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Patch % | Lines |
|---|---|---|
| [.../src/joins/sort\_merge\_join/materializing\_stream.rs](https://app.codecov.io/gh/apache/datafusion/pull/26010?src=pr&el=tree&filepath=datafusion%2Fphysical-plan%2Fsrc%2Fjoins%2Fsort_merge_join%2Fmaterializing_stream.rs&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-ZGF0YWZ1c2lvbi9waHlzaWNhbC1wbGFuL3NyYy9qb2lucy9zb3J0X21lcmdlX2pvaW4vbWF0ZXJpYWxpemluZ19zdHJlYW0ucnM=) | 89.36% | [1 Missing and 4 partials :warning: ](https://app.codecov.io/gh/apache/datafusion/pull/26010?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
<details><summary>Additional details and impacted files</summary>
```diff
@@ Coverage Diff @@
## main #26010 +/- ##
==========================================
- Coverage 82.64% 82.64% -0.01%
==========================================
Files 1147 1147
Lines 445502 445503 +1
Branches 445502 445503 +1
==========================================
- Hits 368179 368169 -10
- Misses 54969 54975 +6
- Partials 22354 22359 +5
```
</details>
[:umbrella: View full report in Codecov by Harness](https://app.codecov.io/gh/apache/datafusion/pull/26010?dropdown=coverage&src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
:loudspeaker: Have feedback on the report? [Share it here](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
<details><summary> :rocket: New features to boost your workflow: </summary>
- :snowflake: [Test Analytics](https://docs.codecov.com/docs/test-analytics): Detect flaky tests, report on failures, and find test suite problems.
- :package: [JS Bundle Analysis](https://docs.codecov.com/docs/javascript-bundle-analysis): Save yourself from yourself by tracking and limiting bundle sizes in JS merges.
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org