[jira] [Updated] (HUDI-3443) Async clustering cannot proceed due to archived deltacommit in metadata table

["Ethan Guo (Jira)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/HUDI-3443?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Ethan Guo updated HUDI-3443:
----------------------------
    Sprint: Hudi-Sprint-Feb-14

> Async clustering cannot proceed due to archived deltacommit in metadata table
> -----------------------------------------------------------------------------
>
>                 Key: HUDI-3443
>                 URL: https://issues.apache.org/jira/browse/HUDI-3443
>             Project: Apache Hudi
>          Issue Type: Bug
>            Reporter: Ethan Guo
>            Assignee: Ethan Guo
>            Priority: Blocker
>             Fix For: 0.11.0
>
>
> Environment: deltastreamer continuous mode writing COW table, async clustering and clean enabled.  Metadata table enabled.
> Repeatly hitting the following issue after the deltastreamer is running for some time:
> {code:java}
> org.apache.hudi.exception.HoodieMetadataException: The instant [20220209121825078__deltacommit__COMPLETED] required to sync rollback of 20220209121825078 has been archived
>     at org.apache.hudi.metadata.HoodieTableMetadataUtil.lambda$processRollbackMetadata$15(HoodieTableMetadataUtil.java:521)
>     at java.util.HashMap$Values.forEach(HashMap.java:981)
>     at java.util.Collections$UnmodifiableCollection.forEach(Collections.java:1082)
>     at org.apache.hudi.metadata.HoodieTableMetadataUtil.processRollbackMetadata(HoodieTableMetadataUtil.java:498)
>     at org.apache.hudi.metadata.HoodieTableMetadataUtil.convertMetadataToRollbackRecords(HoodieTableMetadataUtil.java:472)
>     at org.apache.hudi.metadata.HoodieTableMetadataUtil.convertMetadataToRecords(HoodieTableMetadataUtil.java:437)
>     at org.apache.hudi.metadata.HoodieBackedTableMetadataWriter.update(HoodieBackedTableMetadataWriter.java:726)
>     at org.apache.hudi.table.action.BaseActionExecutor.lambda$writeTableMetadata$2(BaseActionExecutor.java:77)
>     at org.apache.hudi.common.util.Option.ifPresent(Option.java:96)
>     at org.apache.hudi.table.action.BaseActionExecutor.writeTableMetadata(BaseActionExecutor.java:77)
>     at org.apache.hudi.table.action.rollback.BaseRollbackActionExecutor.finishRollback(BaseRollbackActionExecutor.java:244)
>     at org.apache.hudi.table.action.rollback.BaseRollbackActionExecutor.runRollback(BaseRollbackActionExecutor.java:122)
>     at org.apache.hudi.table.action.rollback.BaseRollbackActionExecutor.execute(BaseRollbackActionExecutor.java:144)
>     at org.apache.hudi.table.HoodieSparkCopyOnWriteTable.rollback(HoodieSparkCopyOnWriteTable.java:346)
>     at org.apache.hudi.client.BaseHoodieWriteClient.rollbackInflightClustering(BaseHoodieWriteClient.java:1211)
>     at org.apache.hudi.client.SparkRDDWriteClient.cluster(SparkRDDWriteClient.java:361)
>     at org.apache.hudi.client.HoodieSparkClusteringClient.cluster(HoodieSparkClusteringClient.java:54)
>     at org.apache.hudi.async.AsyncClusteringService.lambda$null$1(AsyncClusteringService.java:79)
>     at java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1604)
>     at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
>     at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
>     at java.lang.Thread.run(Thread.java:748) {code}



--
This message was sent by Atlassian Jira
(v8.20.1#820001)