You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by "Surya Prasanna Yalla (Jira)" <ji...@apache.org> on 2023/06/12 03:09:00 UTC
[jira] [Updated] (HUDI-6356) Fix rollback logic on AbstractHoodieLogRecordReader when replacecommits in main table are retried and created log blocks in metadata table.
[ https://issues.apache.org/jira/browse/HUDI-6356?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Surya Prasanna Yalla updated HUDI-6356:
---------------------------------------
Description:
On metadata table deltacommmit timestamp is equivalent to main table commit's timestamp. So, if metadata sync fails it reuses the same timestamp. This change fixes the case where the log blocks are treated as valid if their corresponding rollback is already visited.
So, rollback block follow the ordering and do not effect the blocks that arrive later.
Ex:
LogBlock1 (Data log block with instant1)
LogBlock2 (Data log block with instant2)
LogBlock3 (Rollback block with instant3 and rolling back instant2)
LogBlock4 (Data log block with instant2)
After processing this change returns LogBlock1 and LogBlock4 as valid blocks.
was:
On metadata table deltacommmit timestamp is equivalent to main table commit's timestamp. So, if metadata sync fails it reuses the same timestamp. This change fixes the case where the log blocks are treated as valid if their corresponding rollback is already visited.
So, rollback block follow the ordering and do not effect the blocks that arrive later.
Ex:
LogBlock1 (Data log block with instant1)
LogBlock2 (Data log block with instant2)
LogBlock3 (Rollback block with instant3 and rolling back instant2)
LogBlock4 (Data log block with instant2)
In this change makes sure the valid blocks are LogBlock1 and LogBlock4.
> Fix rollback logic on AbstractHoodieLogRecordReader when replacecommits in main table are retried and created log blocks in metadata table.
> -------------------------------------------------------------------------------------------------------------------------------------------
>
> Key: HUDI-6356
> URL: https://issues.apache.org/jira/browse/HUDI-6356
> Project: Apache Hudi
> Issue Type: Bug
> Reporter: Surya Prasanna Yalla
> Priority: Major
>
> On metadata table deltacommmit timestamp is equivalent to main table commit's timestamp. So, if metadata sync fails it reuses the same timestamp. This change fixes the case where the log blocks are treated as valid if their corresponding rollback is already visited.
> So, rollback block follow the ordering and do not effect the blocks that arrive later.
> Ex:
> LogBlock1 (Data log block with instant1)
> LogBlock2 (Data log block with instant2)
> LogBlock3 (Rollback block with instant3 and rolling back instant2)
> LogBlock4 (Data log block with instant2)
> After processing this change returns LogBlock1 and LogBlock4 as valid blocks.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)