You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by "ASF GitHub Bot (Jira)" <ji...@apache.org> on 2023/03/01 01:47:00 UTC
[jira] [Updated] (HUDI-5863) Fix HoodieMetadataFileSystemView serving stale view at the timeline server
[ https://issues.apache.org/jira/browse/HUDI-5863?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
ASF GitHub Bot updated HUDI-5863:
---------------------------------
Labels: pull-request-available (was: )
> Fix HoodieMetadataFileSystemView serving stale view at the timeline server
> --------------------------------------------------------------------------
>
> Key: HUDI-5863
> URL: https://issues.apache.org/jira/browse/HUDI-5863
> Project: Apache Hudi
> Issue Type: Bug
> Components: timeline-server, writer-core
> Reporter: Ethan Guo
> Assignee: Ethan Guo
> Priority: Blocker
> Labels: pull-request-available
> Fix For: 0.13.1
>
>
> We observe that for MOR table, occasionally (<10% for large tables with frequent updates and compactions), the deltacommit after the compaction commit may add a new log file to the old file slice, not the latest file slice, in the corresponding file group. This happens when both the metadata table and timeline server are enabled. If either is disabled, the problem does not show up.
> Deeper analysis of the code surfaces that the file system view at the timeline server may serve the stale view, causing the issue. This is because the sync of HoodieMetadataFileSystemView is not atomic when the metadata table is enabled:
>
> {code:java}
> AbstractTableFileSystemView:
> @Override
> public void sync() {
> HoodieTimeline oldTimeline = getTimeline();
> HoodieTimeline newTimeline = metaClient.reloadActiveTimeline().filterCompletedOrMajorOrMinorCompactionInstants();
> try {
> writeLock.lock();
> runSync(oldTimeline, newTimeline);
> } finally {
> writeLock.unlock();
> }
> }
> HoodieMetadataFileSystemView:
> @Override
> public void sync() {
> super.sync();
> tableMetadata.reset();
> }
> {code}
> The
> tableMetadata.reset()
> happens outside the write lock, causing the issue during race conditions (reading the inconsistent state while the table metadata is being reset).
>
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)