You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@datafusion.apache.org by "phillipleblanc (via GitHub)" <gi...@apache.org> on 2026/07/27 14:02:05 UTC
[I] Default join planning loses NOT IN null semantics in SortMergeJoinExec [datafusion-ballista]
phillipleblanc opened a new issue, #2193:
URL: https://github.com/apache/datafusion-ballista/issues/2193
## Bug
Ballista defaults `datafusion.optimizer.prefer_hash_join` to `false`. For a SQL `NOT IN` predicate, DataFusion's physical planner can therefore replace the logical null-aware anti join with a `SortMergeJoinExec` before Ballista's scheduler rules run.
`SortMergeJoinExec` does not carry the `null_aware` flag. The scheduler cannot recover the required `NOT IN` NULL semantics afterward.
## Reproduction
Use four CSV partitions for each table:
- `t1(a)`: values `0..19`
- `t2(b)`: values `0..4`, with one key replaced by `NULL`
Run Ballista with its default join preference:
```sql
select a
from t1
where a not in (select b from t2)
order by a;
```
Expected result: **0 rows**, because the subquery contains `NULL`.
Observed result: **16 rows**. Single-process DataFusion returns the correct 0 rows.
Setting `datafusion.optimizer.prefer_hash_join = true` retains a null-aware `HashJoinExec`, so the scheduler can recognize and lower it correctly. Apache #2188 addresses that hash-join and adaptive-planning path, but cannot fix a null-aware flag that was already discarded.
## Expected behavior
Physical planning should retain null-aware anti-join semantics regardless of the global hash-join preference. Possible approaches include forcing a null-aware `HashJoinExec` for this logical shape or adding equivalent null-aware support to the selected physical join.
Discovered while validating #2188 from the end-to-end reproduction in Andy Grove's review.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Default join planning loses NOT IN null semantics in SortMergeJoinExec [datafusion-ballista]
Posted by "amitvijapur (via GitHub)" <gi...@apache.org>.
amitvijapur commented on issue #2193:
URL: https://github.com/apache/datafusion-ballista/issues/2193#issuecomment-5784157551
This looks addressed on the DataFusion side by apache/datafusion#22810 (merged 2026-07-24, three days before this issue): the physical planner now never routes a null-aware anti join to `SortMergeJoinExec`, whatever `prefer_hash_join` says (`physical_planner.rs`, the `can_repartition_join` / sort-merge branch checks `!*null_aware`). That commit is in DataFusion 55.0.0, which `main` here depends on.
I have not re-run the four-partition CSV reproduction on a cluster, so this is a pointer rather than a confirmation. If the repro passes on `main` now, the issue can probably be closed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org