You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by "Mihir Shah (Jira)" <ji...@apache.org> on 2021/01/22 05:45:00 UTC

[jira] [Commented] (HUDI-1503) Implement a Hash(Bucket)-based Index

    [ https://issues.apache.org/jira/browse/HUDI-1503?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17269828#comment-17269828 ] 

Mihir Shah commented on HUDI-1503:
----------------------------------

h4. [Shimin Yang|https://issues.apache.org/jira/secure/ViewProfile.jspa?name=dangdangdang]

Hello Mr. Yang,

I would be interested in working on this issue, I was wondering if there is some documentation about the index or the project's design so I could understand the problem better?

Thank you!

> Implement a Hash(Bucket)-based Index
> ------------------------------------
>
>                 Key: HUDI-1503
>                 URL: https://issues.apache.org/jira/browse/HUDI-1503
>             Project: Apache Hudi
>          Issue Type: Wish
>          Components: Index, Performance
>            Reporter: Shimin Yang
>            Priority: Major
>
> This ticket is to introduce a new hash based index, which can improve the performance of  write operations and speed up the queries at the same time(removing shuffle for Spark/Hive).
> The new hash-based index works with a customized hash-based partitioner, which partition records based on the hash value of index keys and a fixed bucket number. So there's no need to visit the existing files to determine which file group each record belongs.
> Meanwhile, the file group id, hash mode and bucket num can be used by the query engines to eliminate shuffle introduced by aggregation and join.
> We implemented an HoodieIndex based on hive hash function which used on production environment of ByteDance for many very-large volume dataset, and we hope this feature can be contributed to the community soon.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)