You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@datafusion.apache.org by "jayzhan211 (via GitHub)" <gi...@apache.org> on 2026/10/19 12:17:38 UTC
[I] Null-aware `NOT IN` hash join in `Partitioned` mode returns rows when the subquery contains NULL [datafusion]
jayzhan211 opened a new issue, #26496:
URL: https://github.com/apache/datafusion/issues/26496
### Describe the bug
An uncorrelated null-aware anti join (`build.c1 NOT IN (probe.c2)`, `JoinType::LeftAnti` with `null_aware = true`) built with `PartitionMode::Partitioned` returns rows even when the probe side contains a NULL. SQL semantics say that if the subquery produces a NULL, `x NOT IN (...)` is never true, so the result must be empty. The same join in `PartitionMode::CollectLeft` returns the correct empty result.
The cause is that in `Partitioned` mode every output partition builds its own build side, and the "probe side saw a NULL" fact is recorded on that per-partition build data (`hash_join/exec.rs` around line 1935, `hash_join/stream.rs` around line 1366). A NULL probe key is hash-routed to one partition, so it only suppresses the build rows of that partition. The others still emit their unmatched rows.
`NullAwareMode::try_new` already rejects `Partitioned` for the correlated `LeftAnti` and `LeftMark` forms, but accepts it for the uncorrelated `LeftAnti` form.
### To Reproduce
The DataFusion SQL planner never produces this plan: `JoinSelection` and the physical planner both pin null-aware joins to `CollectLeft`. It is reachable from plans built by hand or deserialized from protobuf (`HashJoinExecNode` decodes the partition mode as written).
Build side `c1` = 1..8, probe side `c2` = {1, NULL}, both sides `RepartitionExec(Hash([c], 4))`, `HashJoinExec` with `JoinType::LeftAnti`, `PartitionMode::Partitioned`, `null_aware = true`, `NullEqualsNothing`.
### Expected behavior
No rows (as `CollectLeft` returns).
### Actual behavior
`[3, 6, 8]` — the build rows hashed to partitions that did not receive the NULL probe row.
### Additional context
A fix that rejects this shape at plan construction, consistent with the existing rejection for the correlated forms, is ready; I'll link the PR.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Null-aware `NOT IN` hash join in `Partitioned` mode returns rows when the subquery contains NULL [datafusion]
Posted by "jayzhan211 (via GitHub)" <gi...@apache.org>.
jayzhan211 closed issue #26496: Null-aware `NOT IN` hash join in `Partitioned` mode returns rows when the subquery contains NULL
URL: https://github.com/apache/datafusion/issues/26496
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Null-aware `NOT IN` hash join in `Partitioned` mode returns rows when the subquery contains NULL [datafusion]
Posted by "cool9850311 (via GitHub)" <gi...@apache.org>.
cool9850311 commented on issue #26496:
URL: https://github.com/apache/datafusion/issues/26496#issuecomment-6267624093
take
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org