[jira] [Commented] (HUDI-3232) support reload timeline Incrementally

["sivabalan narayanan (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/HUDI-3232?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17480722#comment-17480722 ] 

sivabalan narayanan commented on HUDI-3232:
-------------------------------------------

this definitely looks beneficial. 

may I know how you plan to get this done. 

Do you plan to add some filtering in below method in HoodieTableMetaClient. 
{code:java}
public List<HoodieInstant> scanHoodieInstantsFromFileSystem(Path timelinePath, Set<String> includedExtensions,
    boolean applyLayoutVersionFilters) throws IOException { {code}
 

something like,

HTMC (HoodieTableMetaClient) will maintain last loaded instant or mod time of last loaded instant file. So, every time when reload is invoked, we just filter for files modified after the last known mod time. 

 

Or you had something else in mind ? 

 

> support reload timeline Incrementally
> -------------------------------------
>
>                 Key: HUDI-3232
>                 URL: https://issues.apache.org/jira/browse/HUDI-3232
>             Project: Apache Hudi
>          Issue Type: Improvement
>          Components: Common Core, incremental-query, writer-core
>            Reporter: Yann Byron
>            Priority: Major
>
> Recently, call `HoodieTableMetaClient.reloadActiveTimeline` many times in one operation, and this will reload the timeline fully.
> Perhaps, to support to reload in Incremental mode will increase the performance.



--
This message was sent by Atlassian Jira
(v8.20.1#820001)