[jira] [Updated] (HUDI-431) Support Parquet in MOR log files

["Rajesh Mahindra (Jira)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/HUDI-431?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Rajesh Mahindra updated HUDI-431:
---------------------------------
    Sprint: Hudi-Sprint-Jan-3, Hudi-Sprint-Jan-3, Hudi-Sprint-Jan-10, Hudi-Sprint-Jan-18, Hudi-Sprint-Jan-24, Hudi-Sprint-Jan-31  (was: Hudi-Sprint-Jan-3, Hudi-Sprint-Jan-3, Hudi-Sprint-Jan-10, Hudi-Sprint-Jan-18, Hudi-Sprint-Jan-24)

> Support Parquet in MOR log files
> --------------------------------
>
>                 Key: HUDI-431
>                 URL: https://issues.apache.org/jira/browse/HUDI-431
>             Project: Apache Hudi
>          Issue Type: New Feature
>          Components: storage-management
>            Reporter: sivabalan narayanan
>            Assignee: Alexey Kudinkin
>            Priority: Blocker
>              Labels: help-requested, pull-request-available
>             Fix For: 0.11.0
>
>   Original Estimate: 4h
>  Remaining Estimate: 4h
>
> We have a basic implementation of inline filesystem, to read a file format like Parquet, embedded "inline" into another file.  
> [https://github.com/apache/hudi/blob/master/hudi-common/src/test/java/org/apache/hudi/common/fs/inline/TestInLineFileSystem.java] for sample usage.
>  This idea here is to see if we can embed parquet/hfile formats into the Hudi log files, to get columnar reads on the delta log files as well. This helps us speed up query performance, given the log is row based today. Once Inline FS is available, enable parquet logging support with HoodieLogFile. LogFile can expose a writer (essentially ParquetWriter) and users can write records as though writing to parquet files. Similarly on the read path, a reader (parquetReader) will be exposed which the user can use to read data out of it. 
> This Jira tracks work to implement such parquet inlining into the log format and have the writer and reader use it. 
>  



--
This message was sent by Atlassian Jira
(v8.20.1#820001)