You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@datafusion.apache.org by "Dandandan (via GitHub)" <gi...@apache.org> on 2025/05/24 09:28:52 UTC
[I] Support u32 indices in HashJoinExec [datafusion]
Dandandan opened a new issue, #16179:
URL: https://github.com/apache/datafusion/issues/16179
### Is your feature request related to a problem or challenge?
Currently we save indices to the batch always as `u64` in the `HashTable` and in the `next` `Vec`.
If we have less than `u32:MAX` (4.2B, i.e. most of the time) items, we can store them as `u32` - which should make it fit more easily in the CPU cache.
### Describe the solution you'd like
* Implement optimization to store indices as `u32` if possible.
* Run benchmarks
### Describe alternatives you've considered
_No response_
### Additional context
_No response_
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Support u32 indices in HashJoinExec [datafusion]
Posted by "jonathanc-n (via GitHub)" <gi...@apache.org>.
jonathanc-n commented on issue #16179:
URL: https://github.com/apache/datafusion/issues/16179#issuecomment-2910372703
@Dandandan For this, is the preferable solution to create a generic parameter, and when the hash join exec/ stream is created, based on the size of the build side we can assign the generic that value? or an enum could also possibly work? What do you think?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Support u32 indices in HashJoinExec [datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb closed issue #16179: Support u32 indices in HashJoinExec
URL: https://github.com/apache/datafusion/issues/16179
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Support u32 indices in HashJoinExec [datafusion]
Posted by "Dandandan (via GitHub)" <gi...@apache.org>.
Dandandan commented on issue #16179:
URL: https://github.com/apache/datafusion/issues/16179#issuecomment-2910442788
> [@Dandandan](https://github.com/Dandandan) For this, is the preferable solution to create a generic parameter, and when the hash join exec/ stream is created, based on the size of the build side we can assign the generic that value? or an enum could also possibly work? What do you think?
I think we'll likely have to:
* add a generic type to `JoinHashMap<T>`
* make a `Box<dyn JoinHashMap>` based on the `num_rows` (num_rows <= u32::MAX => u32 else u64)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Support u32 indices in HashJoinExec [datafusion]
Posted by "jonathanc-n (via GitHub)" <gi...@apache.org>.
jonathanc-n commented on issue #16179:
URL: https://github.com/apache/datafusion/issues/16179#issuecomment-2906971884
take
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org