You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@datafusion.apache.org by "brancz (via GitHub)" <gi...@apache.org> on 2026/03/19 12:41:26 UTC
[I] Add bounded distinct count optimization [datafusion]
brancz opened a new issue, #21051:
URL: https://github.com/apache/datafusion/issues/21051
### Is your feature request related to a problem or challenge?
Optimize queries like
```sql
SELECT count(DISTINCT col) > 1 FROM table;
```
For cases where col has high cardinality. The issue is that currently the distinct accumulates all values first and only then are they counted, so we both insert values into the distinct accumulator, leaving CPU on the table, and we pay for memory unnecessarily.
### Describe the solution you'd like
Add a bound to count distinct and only insert into the tracked values collection up to N+1.
### Describe alternatives you've considered
n/a
### Additional context
I have a prototype of this working, if the general idea sounds good, I can clean it up and submit the PR.
@alamb
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Add bounded distinct count optimization [datafusion]
Posted by "HairstonE (via GitHub)" <gi...@apache.org>.
HairstonE commented on issue #21051:
URL: https://github.com/apache/datafusion/issues/21051#issuecomment-5619011368
take
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Add bounded distinct count optimization [datafusion]
Posted by "Harsh8084 (via GitHub)" <gi...@apache.org>.
Harsh8084 commented on issue #21051:
URL: https://github.com/apache/datafusion/issues/21051#issuecomment-4105360579
Appreciate the guidance! To minimize disruption, I’m planning to:
Introduce a BoundedDistinctAccumulator that only triggers when the physical plan detects a limit requirement.
Use a simple threshold check that stops insertions once we hit N+1.
Does it make sense to you to handle this via a specialized accumulator, or would you prefer a flag on the existing one? I want to ensure the 'non-special' path stays as lean as possible.
@alamb
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Add bounded distinct count optimization [datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on issue #21051:
URL: https://github.com/apache/datafusion/issues/21051#issuecomment-4104375194
I think the general idea sounds reasonable to me
Things I would be looking for in a PR are:
1. Minimal disruption to the non special code path
2. Clear documentation about what the new code was doing and why
3. Don't add overhead for queries where this optimization applies
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Add bounded distinct count optimization [datafusion]
Posted by "brancz (via GitHub)" <gi...@apache.org>.
brancz commented on issue #21051:
URL: https://github.com/apache/datafusion/issues/21051#issuecomment-4105673982
I wrote this in the original issue:
> I have a prototype of this working, if the general idea sounds good, I can clean it up and submit the PR.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org