You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by "Danny Chen (Jira)" <ji...@apache.org> on 2023/09/06 01:07:00 UTC

[jira] [Updated] (HUDI-1623) Solid completion time on timeline

     [ https://issues.apache.org/jira/browse/HUDI-1623?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Danny Chen updated HUDI-1623:
-----------------------------
    Summary: Solid completion time on timeline  (was: Support start_commit_time & end_commit_times for serializable incremental pull)

> Solid completion time on timeline
> ---------------------------------
>
>                 Key: HUDI-1623
>                 URL: https://issues.apache.org/jira/browse/HUDI-1623
>             Project: Apache Hudi
>          Issue Type: Improvement
>          Components: Common Core
>            Reporter: Nishith Agarwal
>            Assignee: Danny Chen
>            Priority: Critical
>             Fix For: 1.0.0
>
>
> We suggest a new file naming for the *completed* metadata file:
> ${start_time}.${action}.${completion_time}
>  
> We also need a global *Time Generator* that can ensure the monotonical increasing generation of the timestamp, for example, maybe hold a mutex lock with the last generated timestamp backing up there. Say it may holds a lock {*}L1{*}. For each instant time generation, it needs guard from the lock.
>  
> Before creating the completed file, we also need a lock guard from L1.
>  
> Things need to note:
> 1. we only add completion timestamp to the completed metadata file;
> 2. we only add lock guard to the completed metadata file creation, not the whole commiting procedure;
> 3. for regular instant time generation, we also need a lock (that we should ship out by default)



--
This message was sent by Atlassian Jira
(v8.20.10#820010)