[GitHub] [arrow] michalursa commented on pull request #13501: ARROW-16695: [R][Python][C++] Extension types are not supported in joins

[GitBox <gi...@apache.org>]

michalursa commented on PR #13501:
URL: https://github.com/apache/arrow/pull/13501#issuecomment-1218423588

   > With the current changes in this PR, it fails in ` RowEncoder::Init`, which also would need to be updated to handle ExtensionTypes (delegating to the storage type).
   > 
   > What I don't fully understand here is why the "non-key field" needs to be included in creating the Encoder? The non-key fields are not used for building up the join values, so why is it important that we support that data type? (naively, I would assume that for such non-key fields, we only need to be able to "take" the resulting values needed for the result)
   
   The data on the build side of hash join is stored in a row oriented way, that is why there is a storage transformation that is not simply a Take. Storing hash table rows in a row oriented way is important for performance, especially when the entire hash table does not fit into CPU cache, because there is only an access to contiguous block of memory (potentially just one or a few cache lines next to each other) instead of multiple loads from separate addresses. 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org