[jira] [Created] (ARROW-11112) [Rust][DataFusion] Implement vectorized hashing

["Daniël Heres (Jira)" <ji...@apache.org>]

Daniël Heres created ARROW-11112:
------------------------------------

             Summary: [Rust][DataFusion] Implement vectorized hashing
                 Key: ARROW-11112
                 URL: https://issues.apache.org/jira/browse/ARROW-11112
             Project: Apache Arrow
          Issue Type: Improvement
          Components: Rust - DataFusion
            Reporter: Daniël Heres


Currently, the approach of the join and hash aggregates is to create a key individually from the row values. However, this is far from ideal, as it doesn't utilize the cache vectorized nature of Arrow, but instead copies data into a vec, traverses multiple arrays in the inner loop, etc.

This blog post has a summary of an approach to do this in a vectorized way.

[https://www.cockroachlabs.com/blog/vectorized-hash-joiner/]

 


TBD:
We should decide/find out whether it still makes sense to use rust `HashMap` (with () as key?) or whether to create an own? Benefit of using hashmap is that there is an API, can resize automatically, and uses SIMD, and also exposes some lower level bits we can use here.



 



--
This message was sent by Atlassian Jira
(v8.3.4#803005)