[jira] [Closed] (HUDI-2436) rollback in cloud stores w/o append, wrt collecting failed log files to be deleted/logged

["sivabalan narayanan (Jira)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/HUDI-2436?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

sivabalan narayanan closed HUDI-2436.
-------------------------------------
    Resolution: Invalid

> rollback in cloud stores w/o append, wrt collecting failed log files to be deleted/logged
> -----------------------------------------------------------------------------------------
>
>                 Key: HUDI-2436
>                 URL: https://issues.apache.org/jira/browse/HUDI-2436
>             Project: Apache Hudi
>          Issue Type: Sub-task
>          Components: Writer Core
>            Reporter: sivabalan narayanan
>            Assignee: sivabalan narayanan
>            Priority: Major
>             Fix For: 0.10.0
>
>
> I don't fully get [this|https://github.com/apache/hudi/pull/3651/files#r708820142] point. will follow up with Vinoth on the exact scenario.
>  here is my understanding: of a scenario using cloud stores that does not support append.
> If there was crash during a commit, when listing log files to be logged, the last one which got crashed may not be part of the rollback plan. but thats should be fine. anyways, its not available via listing. and so I assume even during compaction those will not be available. we will proceed on with rollback by adding another log file. and this will get replayed to metadata table.
> If you are talking about the case, where a crash happens when rollback itself is being logged and crashed just before committing to metadata table.
>  we should be ok here too. we will retry the rollback which will redo the action phase. and will add new log blocks (with same old logs that were part of failed writes, just that it may not be able to successfully delete). and this will get applied to metadata table. We just have to ensure when applying changes to metadata table, we consider all files from the plan and not just the ones that got successfully deleted.
>  
>  - with hdfs type of cloud stores, where appends are allowed, we just create new log blocks. and hence should not be an issue. 



--
This message was sent by Atlassian Jira
(v8.3.4#803005)