You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by "ASF GitHub Bot (Jira)" <ji...@apache.org> on 2021/06/15 07:02:00 UTC
[jira] [Updated] (HUDI-1717) Metadata Table reader does not show
correct view of the metadata
[ https://issues.apache.org/jira/browse/HUDI-1717?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
ASF GitHub Bot updated HUDI-1717:
---------------------------------
Labels: pull-request-available sev:critical user-support-issues (was: sev:critical user-support-issues)
> Metadata Table reader does not show correct view of the metadata
> ----------------------------------------------------------------
>
> Key: HUDI-1717
> URL: https://issues.apache.org/jira/browse/HUDI-1717
> Project: Apache Hudi
> Issue Type: Bug
> Reporter: Prashant Wason
> Assignee: Prashant Wason
> Priority: Blocker
> Labels: pull-request-available, sev:critical, user-support-issues
>
> Dataset timeline: C1 C2 C3 Compaction.inflight C4 C5
> Metadata timeline: DC1 DC2 DC3. (DC=deltaCommit)
> Assume the dataset timeline has some completed commits (C1, C2 ... C5) and an async compaction operation in progress. Also assume that the metadata table is synced only till C3.
> The MetadataTableWriter will not sync any more instants to the Metadata Table since an incomplete instant is present next (Compaction.inflight).
> The same sync logic is also used by the MetadataReader to perform the in-memory merge of timeline. Hence, the reader will also not consider C4 and C5 thereby providing an incorrect and older view of the FileSlices and FileGroups.
> Any future ingestion into this table MAY insert data into older versions of the FileSlices which will end up being a data loss when queried.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)