[jira] [Closed] (HUDI-8031) Allow read partition path field directly from file in new filegroup reader

["Jonathan Vexler (Jira)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/HUDI-8031?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Jonathan Vexler closed HUDI-8031.
---------------------------------
    Resolution: Duplicate

> Allow read partition path field directly from file in new filegroup reader
> --------------------------------------------------------------------------
>
>                 Key: HUDI-8031
>                 URL: https://issues.apache.org/jira/browse/HUDI-8031
>             Project: Apache Hudi
>          Issue Type: Improvement
>          Components: reader-core, spark
>            Reporter: Jonathan Vexler
>            Assignee: Jonathan Vexler
>            Priority: Major
>
> Currently for spark, we append the same partition path value to the end of every record. If you use timestamp based keygen for example, your partition field can differ for every record.
> Idea for how to implement: In default source / hadoopfs factory, we figure out if the partition cols are going to all be the same values or not. If they are not, we set the partition schema as empty. 1 thing to think about is that in the file index, we need to look through the data filters and move them to partition filters if they are on the partition column.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)