You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by "Udit Mehrotra (Jira)" <ji...@apache.org> on 2021/08/12 22:31:00 UTC
[jira] [Resolved] (HUDI-2286) Handle the case of failed deltacommit
on the metadata table.
[ https://issues.apache.org/jira/browse/HUDI-2286?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Udit Mehrotra resolved HUDI-2286.
---------------------------------
Resolution: Fixed
> Handle the case of failed deltacommit on the metadata table.
> ------------------------------------------------------------
>
> Key: HUDI-2286
> URL: https://issues.apache.org/jira/browse/HUDI-2286
> Project: Apache Hudi
> Issue Type: Bug
> Reporter: Prashant Wason
> Assignee: Prashant Wason
> Priority: Major
> Labels: pull-request-available
> Fix For: 0.9.0
>
>
> Assume the current timeline state is as follows:
> Dataset: C10
> Metadata Table: DC10
>
> Next ingestion run attempts a commit which succeeds. But syncing to metadata table fails. This is the new timeline state:
> Dataset: C10 C11
> Metadata Table: DC10 DC11.inflight
>
> Next ingestion run attempts to sync metadata table in preWrite(). This will do the following:
> 1. Get list of instants to sync (C11)
> 2. MetadataTable.startCommitWithTime(C11)
> 3. autoRollback DC11.inflight
> At this time the timelines will be:
> Dataset: C10. C11
> Metadata Table: DC10. Rollback12
> 4. startCommitWithTime fails here with the following exception:
> 21/07/17 15:59:43 ERROR client.AbstractHoodieWriteClient: Cannot start a new commit at time 20210717141448 as there are future commits present: [[==>20210717141448__deltacommit__REQUESTED]]
>
> Next ingestion run attempts a commit which succeeds. This is the new timeline state just after the commit has finished:
> Dataset: C10 C11 C13
> Metadata Table: DC10. Rollback12
> Metadata table sync will be called. It will pickup all instants to sync after the last timestamp on the metadata table - C13 (last timestamp on metadata = 12). Sync will succeed leading to this final timeline state:
> Dataset: C10. C11 C13
> Metadata Table: DC10. Rollback12 DC13
> So C11 never got committed to the metadata table.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)