You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by "liujinhui (Jira)" <ji...@apache.org> on 2022/04/01 03:15:00 UTC
[jira] [Updated] (HUDI-3765) Structured streaming MOR table can not asynchronous clean
[ https://issues.apache.org/jira/browse/HUDI-3765?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
liujinhui updated HUDI-3765:
----------------------------
Summary: Structured streaming MOR table can not asynchronous clean (was: structured streaming MOR table can not asynchronous clean)
> Structured streaming MOR table can not asynchronous clean
> ----------------------------------------------------------
>
> Key: HUDI-3765
> URL: https://issues.apache.org/jira/browse/HUDI-3765
> Project: Apache Hudi
> Issue Type: Bug
> Reporter: liujinhui
> Priority: Major
>
> When writing to the mor table using structured streaming, when the asynchronous clean service is enabled, the clean will only be triggered when the task is restarted. Through debugging, it is found that in the process of continuous operation, the generation of clean will not actually be triggered. ...
>
> Although there will be the log, it will not trigger clean. Pretty sure that the number of version files that triggers a clean has been reached
> {code:java}
> // Async cleaner has been spawned. Waiting for it to finish | org.apache.hudi.client.BaseHoodieWriteClient.autoCleanOnCommit(BaseHoodieWriteClient.java:541)
> 2022-03-31 19:26:06,677 | INFO | [stream execution thread for [id = ce830c81-00c6-4d83-8a60-1970b8a6a1c9, runId = d57ad29c-c63e-441d-b183-0e5eb028acc5]] | Waiting for async clean service to finish | org.apache.hudi.async.AsyncCleanerService.waitForCompletion(AsyncCleanerService.java:73)
> 2022-03-31 19:26:06,677 | INFO | [stream execution thread for [id = ce830c81-00c6-4d83-8a60-1970b8a6a1c9, runId = d57ad29c-c63e-441d-b183-0e5eb028acc5]] | Async cleaner has finished | org.apache.hudi.client.BaseHoodieWriteClient.autoCleanOnCommit(BaseHoodieWriteClient.java:543) {code}
--
This message was sent by Atlassian Jira
(v8.20.1#820001)