You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by "hudi-bot (via GitHub)" <gi...@apache.org> on 2025/11/30 08:49:06 UTC

[I] Concurrent cleaner commit same instance conflict [hudi]

hudi-bot opened a new issue, #16187:
URL: https://github.com/apache/hudi/issues/16187

   Timeline 
   
    
   {code:java}
   -rw-r--r--   1 jon  wheel     0B Aug 16 19:58 20230816195843234.commit.requested
   -rw-r--r--   1 jon  wheel     0B Aug 16 19:58 20230816195845557.commit.requested
   -rw-r--r--   1 jon  wheel   2.2K Aug 16 19:58 20230816195843234.inflight
   -rw-r--r--   1 jon  wheel   813B Aug 16 19:58 20230816195845557.inflight
   -rw-r--r--   1 jon  wheel   2.6K Aug 16 19:58 20230816195845557.commit
   -rw-r--r--   1 jon  wheel   2.6K Aug 16 19:58 20230816195843234.commit
   -rw-r--r--   1 jon  wheel   1.7K Aug 16 19:58 20230816195855285.clean.requested
   -rw-r--r--   1 jon  wheel   1.7K Aug 16 19:58 20230816195855285.clean.inflight
   -rw-r--r--   1 jon  wheel   1.8K Aug 16 19:58 20230816195855389.clean.requested
   -rw-r--r--   1 jon  wheel   1.7K Aug 16 19:58 20230816195855285.clean {code}
   requests:
   {code:java}
   avrocat hudi/output/.hoodie/20230816195855285.clean.requested
   {"earliestInstantToRetain": {"HoodieActionInstant": {"timestamp": "20230816195654386", "action": "commit", "state": "COMPLETED"}}, "lastCompletedCommitTimestamp": "20230816195845557", "policy": "KEEP_LATEST_COMMITS", "filesToBeDeletedPerPartition": {"map": {}}, "version": {"int": 2}, "filePathsToBeDeletedPerPartition": {"map": {"1970/01/01": [{"filePath": {"string": "file:/tmp/hudi/output/1970/01/01/f66cf644-9e9f-477f-863c-eb62d1c6b14d-0_0-1391-2009_20230816195619275.parquet"}, "isBootstrapBaseFile": {"boolean": false}}]}}, "partitionsToBeDeleted": {"array": []}} {code}
   {code:java}
   avrocat hudi/output/.hoodie/20230816195855389.clean.requested {"earliestInstantToRetain": {"HoodieActionInstant": {"timestamp": "20230816195704584", "action": "commit", "state": "COMPLETED"}}, "lastCompletedCommitTimestamp": "20230816195845557", "policy": "KEEP_LATEST_COMMITS", "filesToBeDeletedPerPartition": {"map": {}}, "version": {"int": 2}, "filePathsToBeDeletedPerPartition": {"map": {"1970/01/01": [{"filePath": {"string": "file:/tmp/hudi/output/1970/01/01/f66cf644-9e9f-477f-863c-eb62d1c6b14d-0_0-1391-2009_20230816195619275.parquet"}, "isBootstrapBaseFile": {"boolean": false}}], "1970/01/20": [{"filePath": {"string": "file:/tmp/hudi/output/1970/01/20/05942caf-2d53-4345-845c-5e42abaca797-0_0-1454-2121_20230816195635690.parquet"}, "isBootstrapBaseFile": {"boolean": false}}]}}, "partitionsToBeDeleted": {"array": []}}
   {code}
   Console output:
   
   notice transaction starts twice for the same instance
   {code:java}
   424775 [pool-75-thread-1] INFO  org.apache.hudi.table.action.clean.CleanActionExecutor [] - Finishing previously unfinished cleaner instant=[==>20230816195855285__clean__INFLIGHT__20230816195855525]
   424775 [pool-75-thread-1] INFO  org.apache.hudi.table.action.clean.CleanActionExecutor [] - Using cleanerParallelism: 1
   424779 [pool-91-thread-1] INFO  org.apache.hudi.common.table.timeline.HoodieActiveTimeline [] - Loaded instants upto : Option{val=[==>20230816195855389__clean__REQUESTED__20230816195855634]}
   424779 [pool-91-thread-1] INFO  org.apache.hudi.client.transaction.TransactionManager [] - Transaction starting for Option{val=[==>20230816195855285__clean__INFLIGHT]} with latest completed transaction instant Optional.empty
   424779 [pool-91-thread-1] INFO  org.apache.hudi.client.transaction.lock.LockManager [] - LockProvider org.apache.hudi.client.transaction.lock.InProcessLockProvider
   424779 [pool-91-thread-1] INFO  org.apache.hudi.client.transaction.lock.InProcessLockProvider [] - Base Path file:/tmp/hudi/output, Lock Instance java.util.concurrent.locks.ReentrantReadWriteLock@78f60539[Write locks = 0, Read locks = 0], Thread pool-91-thread-1, In-process lock state ACQUIRING
   424779 [pool-91-thread-1] INFO  org.apache.hudi.client.transaction.lock.InProcessLockProvider [] - Base Path file:/tmp/hudi/output, Lock Instance java.util.concurrent.locks.ReentrantReadWriteLock@78f60539[Write locks = 1, Read locks = 0], Thread pool-91-thread-1, In-process lock state ACQUIRED
   424779 [pool-91-thread-1] INFO  org.apache.hudi.client.transaction.TransactionManager [] - Transaction started for Option{val=[==>20230816195855285__clean__INFLIGHT]} with latest completed transaction instant Optional.empty {code}
   The following pr exposed the issue
   
   [https://github.com/apache/hudi/pull/8602]
   
   This does not cause data corruption. Writer needs to be restarted
   
   ## JIRA info
   
   - Link: https://issues.apache.org/jira/browse/HUDI-6718
   - Type: Bug


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] Concurrent cleaner commit same instance conflict [hudi]

Posted by "nsivabalan (via GitHub)" <gi...@apache.org>.
nsivabalan closed issue #16187: Concurrent cleaner commit same instance conflict 
URL: https://github.com/apache/hudi/issues/16187


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] Concurrent cleaner commit same instance conflict [hudi]

Posted by "nsivabalan (via GitHub)" <gi...@apache.org>.
nsivabalan commented on issue #16187:
URL: https://github.com/apache/hudi/issues/16187#issuecomment-4795550761

   Closing as resolved. Tracked in [HUDI-6718](https://issues.apache.org/jira/browse/HUDI-6718). The change landed in PR [#9468](https://github.com/apache/hudi/pull/9468) (commit `d7c16f56f4f9`). Please reopen if you can still reproduce on the latest release.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] Concurrent cleaner commit same instance conflict [hudi]

Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on issue #16187:
URL: https://github.com/apache/hudi/issues/16187#issuecomment-3630127059

   ### Linked PR(s)
   
   - https://github.com/apache/hudi/pull/8602
   - https://github.com/apache/hudi/pull/9468


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org