You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by "Ethan Guo (Jira)" <ji...@apache.org> on 2024/03/28 01:18:00 UTC
[jira] [Updated] (HUDI-3031) TestHoodieDeltaStreamerWithMultiWriter time out due to async services and writer deadlock
[ https://issues.apache.org/jira/browse/HUDI-3031?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Ethan Guo updated HUDI-3031:
----------------------------
Component/s: table-service
> TestHoodieDeltaStreamerWithMultiWriter time out due to async services and writer deadlock
> -----------------------------------------------------------------------------------------
>
> Key: HUDI-3031
> URL: https://issues.apache.org/jira/browse/HUDI-3031
> Project: Apache Hudi
> Issue Type: Task
> Components: table-service, writer-core
> Reporter: Manoj Govindassamy
> Assignee: Manoj Govindassamy
> Priority: Blocker
> Fix For: 0.11.0
>
>
> Off late, TestHoodieDeltaStreamerWithMultiWriter started consistently failing for the MOR table type. The test spins off few pool threads to do table ingestion via back filling along with async compaction and clustering. After the data ingestion is completed the test endlessly waits for the the following condition to pass.
>
> {code:java}
> // Condition for parallel ingestion job
> Function<Boolean, Boolean> conditionForRegularIngestion = (r) -> {
> if (tableType.equals(HoodieTableType.MERGE_ON_READ)) {
> TestHoodieDeltaStreamer.TestHelpers.assertAtleastNDeltaCommitsAfterCommit(3, lastSuccessfulCommit, tableBasePath, fs());
> } else {
> TestHoodieDeltaStreamer.TestHelpers.assertAtleastNCompactionCommitsAfterCommit(3, lastSuccessfulCommit, tableBasePath, fs());
> }
> TestHoodieDeltaStreamer.TestHelpers.assertRecordCount(totalRecords, tableBasePath + "/*/*.parquet", sqlContext());
> TestHoodieDeltaStreamer.TestHelpers.assertDistanceCount(totalRecords, tableBasePath + "/*/*.parquet", sqlContext());
> return true;
> }; {code}
> Issue 1: The compaction thread and the writer thread are in deadlock
> {code:java}
> "async_compact_thread" #188 prio=5 os_prio=31 tid=0x00007f8c26266800 nid=0x15803 waiting for monitor entry [0x0000700009d3e000]
> java.lang.Thread.State: BLOCKED (on object monitor)
> at org.apache.hudi.client.transaction.TransactionManager.endTransaction(TransactionManager.java:70)
> - waiting to lock <0x00000006c353f528> (a org.apache.hudi.client.transaction.TransactionManager)
> at org.apache.hudi.client.SparkRDDWriteClient.completeCompaction(SparkRDDWriteClient.java:312)
> at org.apache.hudi.client.SparkRDDWriteClient.commitCompaction(SparkRDDWriteClient.java:294)
> at org.apache.hudi.client.HoodieSparkCompactor.compact(HoodieSparkCompactor.java:59)
> at org.apache.hudi.async.AsyncCompactService.lambda$null$1(AsyncCompactService.java:89)
> at org.apache.hudi.async.AsyncCompactService$$Lambda$612/2034420774.get(Unknown Source)
> at java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1604)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> "pool-22-thread-1" #143 prio=5 os_prio=31 tid=0x00007f8c0b125800 nid=0x12603 waiting on condition [0x0000700006fb7000]
> java.lang.Thread.State: TIMED_WAITING (sleeping)
> at java.lang.Thread.sleep(Native Method)
> at org.apache.hudi.client.transaction.FileSystemBasedLockProviderTestClass.tryLock(FileSystemBasedLockProviderTestClass.java:80)
> at org.apache.hudi.client.transaction.lock.LockManager.lock(LockManager.java:68)
> at org.apache.hudi.client.transaction.TransactionManager.beginTransaction(TransactionManager.java:64)
> - locked <0x00000006c353f528> (a org.apache.hudi.client.transaction.TransactionManager)
> at org.apache.hudi.client.AbstractHoodieWriteClient.commitStats(AbstractHoodieWriteClient.java:193)
> at org.apache.hudi.client.SparkRDDWriteClient.commit(SparkRDDWriteClient.java:125)
> at org.apache.hudi.utilities.deltastreamer.DeltaSync.writeToSink(DeltaSync.java:536)
> at org.apache.hudi.utilities.deltastreamer.DeltaSync.syncOnce(DeltaSync.java:308)
> {code}
> Issue 2: Even after fixing the above my replacing the hoodie.write.lock.provider with the local lock provider, the end condition of 3 DeltaCommitAfterLastCommit is not met and the test times out. This needs to be digged further.
>
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)