You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by "Danny Chen (Jira)" <ji...@apache.org> on 2020/12/24 11:59:00 UTC

[jira] [Commented] (HUDI-1449) Support for _hoodie_record_key as a virtual column

    [ https://issues.apache.org/jira/browse/HUDI-1449?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17254513#comment-17254513 ] 

Danny Chen commented on HUDI-1449:
----------------------------------

Makes sense, i'm wondering can we also make the `_hoodie_file_name` virtual ? Is the `_hoodie_file_name` always the file name that contains the record or it may change in the future ? Currently, the file name is a uuid string plus the instant time, so it is storage consuming.

> Support for _hoodie_record_key as a virtual column 
> ---------------------------------------------------
>
>                 Key: HUDI-1449
>                 URL: https://issues.apache.org/jira/browse/HUDI-1449
>             Project: Apache Hudi
>          Issue Type: Improvement
>          Components: Common Core
>            Reporter: Nishith Agarwal
>            Assignee: Abhishek Modi
>            Priority: Major
>
> Context:
> Currently, _hoodie_record_key is written to DFS, as a column in the Parquet
> file. In our production systems at Uber however, _hoodie_record_key
> contains data that can be found in a different column (or set of columns).
> This means that we are storing duplicated data.
> Proposal:
> In the interest of improving storage efficiency, we could add confs /
> abstract classes that can construct the _hoodie_record_key given other
> columns. That way we do not have to store duplicated data on DFS.
>  
> RFC -> https://cwiki.apache.org/confluence/display/HUDI/RFC+-+21+%3A+Allow+HoodieRecordKey+to+be+Virtual



--
This message was sent by Atlassian Jira
(v8.3.4#803005)