You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@datafusion.apache.org by "ryux1 (via GitHub)" <gi...@apache.org> on 2026/09/06 23:50:28 UTC
[PR] fix: preserve groupless aggregate subquery cardinality [datafusion]
ryux1 opened a new pull request, #25002:
URL: https://github.com/apache/datafusion/pull/25002
## Which issue does this PR close?
- Closes #24960.
## Rationale for this change
A groupless aggregate always emits one row unless an operator such as `HAVING` or `LIMIT 0` removes it. Correlated `EXISTS` and `IN` predicates currently decorrelate that aggregate into semi/anti joins, which loses the empty-input aggregate row. As a result, `EXISTS` drops outer rows without an inner match, `IN` misses comparisons against values such as `count(*) = 0`, and `NOT IN` can fail planning with a two-column null-aware anti join.
## What changes are included in this PR?
- Recognize correlated, guaranteed-single-row subqueries through projection and alias wrappers.
- Fold `EXISTS` and `NOT EXISTS` according to that cardinality.
- Rewrite `IN` and `NOT IN` to scalar-subquery comparisons so the existing scalar decorrelator preserves empty-input aggregate values.
- Keep `HAVING`, `LIMIT`, grouped aggregates, and uncorrelated subqueries on their existing paths.
- Update the affected logical-plan snapshot.
## What is the testing strategy for this PR?
A new `issue_24960.slt` regression file covers top-level `EXISTS`, `NOT EXISTS`, `IN`, and `NOT IN`, plus embedded `EXISTS`. It also verifies that `HAVING` and `LIMIT 0` are not treated as guaranteed-one-row cases.
Locally passing:
- `cargo test --test sqllogictests -- issue_24960.slt subquery.slt`
- `cargo test -p datafusion-optimizer` (776 unit tests, 26 integration tests, doc tests)
- `PATH=/home/ryu/.cargo/bin:$PATH ./dev/rust_lint.sh`
## Are there any user-facing changes?
Yes. Correlated `EXISTS` and `IN` predicates over groupless aggregates now preserve SQL aggregate cardinality and return the correct rows. There are no public API changes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] fix: preserve groupless aggregate subquery cardinality [datafusion]
Posted by "ryux1 (via GitHub)" <gi...@apache.org>.
ryux1 commented on PR #25002:
URL: https://github.com/apache/datafusion/pull/25002#issuecomment-5575591570
Understood—thanks for catching it. I missed #25008 when I checked for competing work. I’ll coordinate on the issue before starting similar work in DataFusion going forward.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] fix: preserve groupless aggregate subquery cardinality [datafusion]
Posted by "codecov-commenter (via GitHub)" <gi...@apache.org>.
codecov-commenter commented on PR #25002:
URL: https://github.com/apache/datafusion/pull/25002#issuecomment-5563288423
## [Codecov](https://app.codecov.io/gh/apache/datafusion/pull/25002?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
:x: Patch coverage is `74.35897%` with `10 lines` in your changes missing coverage. Please review.
:white_check_mark: Project coverage is 81.68%. Comparing base ([`d25ffaa`](https://app.codecov.io/gh/apache/datafusion/commit/d25ffaab493c18690028410656ed12ea6c6fb0b6?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)) to head ([`6f39c2f`](https://app.codecov.io/gh/apache/datafusion/commit/6f39c2fbcfa3972daaae275e0c2866786f4ce706?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)).
| [Files with missing lines](https://app.codecov.io/gh/apache/datafusion/pull/25002?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Patch % | Lines |
|---|---|---|
| [...on/optimizer/src/decorrelate\_predicate\_subquery.rs](https://app.codecov.io/gh/apache/datafusion/pull/25002?src=pr&el=tree&filepath=datafusion%2Foptimizer%2Fsrc%2Fdecorrelate_predicate_subquery.rs&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-ZGF0YWZ1c2lvbi9vcHRpbWl6ZXIvc3JjL2RlY29ycmVsYXRlX3ByZWRpY2F0ZV9zdWJxdWVyeS5ycw==) | 74.35% | [9 Missing and 1 partial :warning: ](https://app.codecov.io/gh/apache/datafusion/pull/25002?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
<details><summary>Additional details and impacted files</summary>
```diff
@@ Coverage Diff @@
## main #25002 +/- ##
==========================================
- Coverage 81.69% 81.68% -0.01%
==========================================
Files 1126 1126
Lines 415193 415232 +39
Branches 415193 415232 +39
==========================================
+ Hits 339182 339201 +19
- Misses 56079 56095 +16
- Partials 19932 19936 +4
```
</details>
[:umbrella: View full report in Codecov by Harness](https://app.codecov.io/gh/apache/datafusion/pull/25002?dropdown=coverage&src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
:loudspeaker: Have feedback on the report? [Share it here](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
<details><summary> :rocket: New features to boost your workflow: </summary>
- :snowflake: [Test Analytics](https://docs.codecov.com/docs/test-analytics): Detect flaky tests, report on failures, and find test suite problems.
- :package: [JS Bundle Analysis](https://docs.codecov.com/docs/javascript-bundle-analysis): Save yourself from yourself by tracking and limiting bundle sizes in JS merges.
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] fix: preserve groupless aggregate subquery cardinality [datafusion]
Posted by "kumarUjjawal (via GitHub)" <gi...@apache.org>.
kumarUjjawal closed pull request #25002: fix: preserve groupless aggregate subquery cardinality
URL: https://github.com/apache/datafusion/pull/25002
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] fix: preserve groupless aggregate subquery cardinality [datafusion]
Posted by "kumarUjjawal (via GitHub)" <gi...@apache.org>.
kumarUjjawal commented on PR #25002:
URL: https://github.com/apache/datafusion/pull/25002#issuecomment-5572080469
Hi @ryux1 I think @mohammadnaqvi04 was already planning to work on this and has a PR https://github.com/apache/datafusion/pull/25008
I will close this one. Please always coordinate with the original issue author before opening a PR to avoid duplicates.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org