You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@datafusion.apache.org by "Dandandan (via GitHub)" <gi...@apache.org> on 2026/01/22 15:58:56 UTC
[I] Support grouped aggregates with known min/max statistics [datafusion]
Dandandan opened a new issue, #19938:
URL: https://github.com/apache/datafusion/issues/19938
### Is your feature request related to a problem or challenge?
Currently, grouped aggregates follow this path (simplified)
* create hashes for columns
* group by hash using a hash table / check equality
The approach is well optimized, but we can avoid a lot of work if we don't have to hash and use a hashtable.
### Describe the solution you'd like
When the column statistics includinf the range (min/max) s known for a group by column, and the range is not too large, we can store the groups in a `Vec` where each element at `i` represents the group `min + i`, using direct indexing.
This could save a lot of overhead.
This is very similar to whats implemented in https://github.com/apache/datafusion/pull/19411 for joins.
### Describe alternatives you've considered
We could also consider computing the statistics on the fly and switch dynamically to a hash table vs hash map (i.e. copy all entries to a hash table once the range exceeds the maximum).
### Additional context
_No response_
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Support grouped aggregates with known min/max statistics [datafusion]
Posted by "jizezhang (via GitHub)" <gi...@apache.org>.
jizezhang commented on issue #19938:
URL: https://github.com/apache/datafusion/issues/19938#issuecomment-3830477234
Hi @Dandandan , I am interested in this issue. If it has not yet been worked on, I would like to take a stab at it.
Based on my understanding of the code, my thought is that we could create a new implementation of trait `GroupValues` https://github.com/apache/datafusion/blob/51c04754ee9be41195d7a197bb71a66ed1b11673/datafusion/physical-plan/src/aggregates/group_values/mod.rs#L90 that is specific to integer types and separate from `GroupValuesPrimitive`https://github.com/apache/datafusion/blob/51c04754ee9be41195d7a197bb71a66ed1b11673/datafusion/physical-plan/src/aggregates/group_values/single_group_by/primitive.rs#L80, which maintains a hash table internally. Logic for the new implementation would use direct indexing following similar implementation in `ArrayMap` https://github.com/apache/datafusion/blob/51c04754ee9be41195d7a197bb71a66ed1b11673/datafusion/physical-plan/src/joins/array_map.rs#L103 for hash join. The logic for choosing specific implementations of `GroupValues` in `new_group_values` https://github.com/apache/datafusion/blob/51c04754ee9be41195d7a197bb71a66ed1b11673/datafusion/physic
al-plan/src/aggregates/group_values/mod.rs#L134 would also be updated to take column statistics, available via `AggregateExec::partition_statistics`, https://github.com/apache/datafusion/blob/51c04754ee9be41195d7a197bb71a66ed1b11673/datafusion/physical-plan/src/aggregates/mod.rs#L1403 and determine whether to apply direct indexing or not.
Please let me know whether this makes sense or not, and any suggestions/corrections. Thanks a lot!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Support grouped aggregates with known min/max statistics [datafusion]
Posted by "HairstonE (via GitHub)" <gi...@apache.org>.
HairstonE commented on issue #19938:
URL: https://github.com/apache/datafusion/issues/19938#issuecomment-5144631037
take
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org