You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by "hudi-bot (via GitHub)" <gi...@apache.org> on 2025/11/30 08:49:06 UTC
[I] Concurrent cleaner commit same instance conflict [hudi]
hudi-bot opened a new issue, #16187:
URL: https://github.com/apache/hudi/issues/16187
Timeline
{code:java}
-rw-r--r-- 1 jon wheel 0B Aug 16 19:58 20230816195843234.commit.requested
-rw-r--r-- 1 jon wheel 0B Aug 16 19:58 20230816195845557.commit.requested
-rw-r--r-- 1 jon wheel 2.2K Aug 16 19:58 20230816195843234.inflight
-rw-r--r-- 1 jon wheel 813B Aug 16 19:58 20230816195845557.inflight
-rw-r--r-- 1 jon wheel 2.6K Aug 16 19:58 20230816195845557.commit
-rw-r--r-- 1 jon wheel 2.6K Aug 16 19:58 20230816195843234.commit
-rw-r--r-- 1 jon wheel 1.7K Aug 16 19:58 20230816195855285.clean.requested
-rw-r--r-- 1 jon wheel 1.7K Aug 16 19:58 20230816195855285.clean.inflight
-rw-r--r-- 1 jon wheel 1.8K Aug 16 19:58 20230816195855389.clean.requested
-rw-r--r-- 1 jon wheel 1.7K Aug 16 19:58 20230816195855285.clean {code}
requests:
{code:java}
avrocat hudi/output/.hoodie/20230816195855285.clean.requested
{"earliestInstantToRetain": {"HoodieActionInstant": {"timestamp": "20230816195654386", "action": "commit", "state": "COMPLETED"}}, "lastCompletedCommitTimestamp": "20230816195845557", "policy": "KEEP_LATEST_COMMITS", "filesToBeDeletedPerPartition": {"map": {}}, "version": {"int": 2}, "filePathsToBeDeletedPerPartition": {"map": {"1970/01/01": [{"filePath": {"string": "file:/tmp/hudi/output/1970/01/01/f66cf644-9e9f-477f-863c-eb62d1c6b14d-0_0-1391-2009_20230816195619275.parquet"}, "isBootstrapBaseFile": {"boolean": false}}]}}, "partitionsToBeDeleted": {"array": []}} {code}
{code:java}
avrocat hudi/output/.hoodie/20230816195855389.clean.requested {"earliestInstantToRetain": {"HoodieActionInstant": {"timestamp": "20230816195704584", "action": "commit", "state": "COMPLETED"}}, "lastCompletedCommitTimestamp": "20230816195845557", "policy": "KEEP_LATEST_COMMITS", "filesToBeDeletedPerPartition": {"map": {}}, "version": {"int": 2}, "filePathsToBeDeletedPerPartition": {"map": {"1970/01/01": [{"filePath": {"string": "file:/tmp/hudi/output/1970/01/01/f66cf644-9e9f-477f-863c-eb62d1c6b14d-0_0-1391-2009_20230816195619275.parquet"}, "isBootstrapBaseFile": {"boolean": false}}], "1970/01/20": [{"filePath": {"string": "file:/tmp/hudi/output/1970/01/20/05942caf-2d53-4345-845c-5e42abaca797-0_0-1454-2121_20230816195635690.parquet"}, "isBootstrapBaseFile": {"boolean": false}}]}}, "partitionsToBeDeleted": {"array": []}}
{code}
Console output:
notice transaction starts twice for the same instance
{code:java}
424775 [pool-75-thread-1] INFO org.apache.hudi.table.action.clean.CleanActionExecutor [] - Finishing previously unfinished cleaner instant=[==>20230816195855285__clean__INFLIGHT__20230816195855525]
424775 [pool-75-thread-1] INFO org.apache.hudi.table.action.clean.CleanActionExecutor [] - Using cleanerParallelism: 1
424779 [pool-91-thread-1] INFO org.apache.hudi.common.table.timeline.HoodieActiveTimeline [] - Loaded instants upto : Option{val=[==>20230816195855389__clean__REQUESTED__20230816195855634]}
424779 [pool-91-thread-1] INFO org.apache.hudi.client.transaction.TransactionManager [] - Transaction starting for Option{val=[==>20230816195855285__clean__INFLIGHT]} with latest completed transaction instant Optional.empty
424779 [pool-91-thread-1] INFO org.apache.hudi.client.transaction.lock.LockManager [] - LockProvider org.apache.hudi.client.transaction.lock.InProcessLockProvider
424779 [pool-91-thread-1] INFO org.apache.hudi.client.transaction.lock.InProcessLockProvider [] - Base Path file:/tmp/hudi/output, Lock Instance java.util.concurrent.locks.ReentrantReadWriteLock@78f60539[Write locks = 0, Read locks = 0], Thread pool-91-thread-1, In-process lock state ACQUIRING
424779 [pool-91-thread-1] INFO org.apache.hudi.client.transaction.lock.InProcessLockProvider [] - Base Path file:/tmp/hudi/output, Lock Instance java.util.concurrent.locks.ReentrantReadWriteLock@78f60539[Write locks = 1, Read locks = 0], Thread pool-91-thread-1, In-process lock state ACQUIRED
424779 [pool-91-thread-1] INFO org.apache.hudi.client.transaction.TransactionManager [] - Transaction started for Option{val=[==>20230816195855285__clean__INFLIGHT]} with latest completed transaction instant Optional.empty {code}
The following pr exposed the issue
[https://github.com/apache/hudi/pull/8602]
This does not cause data corruption. Writer needs to be restarted
## JIRA info
- Link: https://issues.apache.org/jira/browse/HUDI-6718
- Type: Bug
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Concurrent cleaner commit same instance conflict [hudi]
Posted by "nsivabalan (via GitHub)" <gi...@apache.org>.
nsivabalan closed issue #16187: Concurrent cleaner commit same instance conflict
URL: https://github.com/apache/hudi/issues/16187
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Concurrent cleaner commit same instance conflict [hudi]
Posted by "nsivabalan (via GitHub)" <gi...@apache.org>.
nsivabalan commented on issue #16187:
URL: https://github.com/apache/hudi/issues/16187#issuecomment-4795550761
Closing as resolved. Tracked in [HUDI-6718](https://issues.apache.org/jira/browse/HUDI-6718). The change landed in PR [#9468](https://github.com/apache/hudi/pull/9468) (commit `d7c16f56f4f9`). Please reopen if you can still reproduce on the latest release.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Concurrent cleaner commit same instance conflict [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on issue #16187:
URL: https://github.com/apache/hudi/issues/16187#issuecomment-3630127059
### Linked PR(s)
- https://github.com/apache/hudi/pull/8602
- https://github.com/apache/hudi/pull/9468
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org