You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by GitBox <gi...@apache.org> on 2022/06/09 03:22:46 UTC
[GitHub] [hudi] jtchen-study opened a new issue, #5807: [SUPPORT] Multi Thread in a process is not ok ，please help
jtchen-study opened a new issue, #5807:
URL: https://github.com/apache/hudi/issues/5807
HoodieWriteConfig cfg = HoodieWriteConfig.newBuilder().withPath(tablePath)
.withSchema(tableSchema)
.forTable(tableName)
.withIndexConfig(HoodieIndexConfig.newBuilder().withIndexType(HoodieIndex.IndexType.BLOOM).build())
.withCompactionConfig(HoodieCompactionConfig.newBuilder()
.withFailedWritesCleaningPolicy(HoodieFailedWritesCleaningPolicy.LAZY)
.build())
.withWriteConcurrencyMode(WriteConcurrencyMode.OPTIMISTIC_CONCURRENCY_CONTROL)
.withLockConfig(HoodieLockConfig.newBuilder()
.withLockProvider(InProcessLockProvider.class)
.build())
.build();
my client config is above, but when i try to write to hudi with multi thread in a process , it produces the following error , is there any solution ?
org.apache.hudi.exception.HoodieUpsertException: Failed to upsert for commit time 20220609103047174
at org.apache.hudi.table.action.commit.BaseWriteHelper.write(BaseWriteHelper.java:64) ~[hudi-client-common-0.11.0.jar:0.11.0]
at org.apache.hudi.table.action.commit.JavaInsertCommitActionExecutor.execute(JavaInsertCommitActionExecutor.java:47) ~[hudi-java-client-0.11.0.jar:0.11.0]
at org.apache.hudi.table.HoodieJavaCopyOnWriteTable.insert(HoodieJavaCopyOnWriteTable.java:111) ~[hudi-java-client-0.11.0.jar:0.11.0]
at org.apache.hudi.table.HoodieJavaCopyOnWriteTable.insert(HoodieJavaCopyOnWriteTable.java:82) ~[hudi-java-client-0.11.0.jar:0.11.0]
at org.apache.hudi.client.HoodieJavaWriteClient.insert(HoodieJavaWriteClient.java:129) ~[hudi-java-client-0.11.0.jar:0.11.0]
at com.alibaba.datax.plugin.writer.hudiwriter.HudiWriterProxy.doBatchInsert(HudiWriterProxy.java:119) ~[hudiwriter-0.0.1-SNAPSHOT.jar:na]
at com.alibaba.datax.plugin.writer.hudiwriter.HudiWriter$Task.startWrite(HudiWriter.java:109) ~[hudiwriter-0.0.1-SNAPSHOT.jar:na]
at com.alibaba.datax.core.taskgroup.runner.WriterRunner.run(WriterRunner.java:114) [datax-core-0.0.1-SNAPSHOT.jar:na]
at java.lang.Thread.run(Thread.java:748) [na:1.8.0_151]
Caused by: org.apache.hudi.exception.HoodieWriteConflictException: java.util.ConcurrentModificationException: Cannot resolve conflicts for overlapping writes
at org.apache.hudi.client.transaction.SimpleConcurrentFileWritesConflictResolutionStrategy.resolveConflict(SimpleConcurrentFileWritesConflictResolutionStrategy.java:102) ~[hudi-client-common-0.11.0.jar:0.11.0]
at org.apache.hudi.client.utils.TransactionUtils.lambda$resolveWriteConflictIfAny$0(TransactionUtils.java:85) ~[hudi-client-common-0.11.0.jar:0.11.0]
at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1380) ~[na:1.8.0_151]
at java.util.stream.Streams$ConcatSpliterator.forEachRemaining(Streams.java:742) ~[na:1.8.0_151]
at java.util.stream.Streams$ConcatSpliterator.forEachRemaining(Streams.java:742) ~[na:1.8.0_151]
at java.util.stream.ReferencePipeline$Head.forEach(ReferencePipeline.java:580) ~[na:1.8.0_151]
at org.apache.hudi.client.utils.TransactionUtils.resolveWriteConflictIfAny(TransactionUtils.java:79) ~[hudi-client-common-0.11.0.jar:0.11.0]
at org.apache.hudi.table.action.commit.BaseCommitActionExecutor.autoCommit(BaseCommitActionExecutor.java:189) ~[hudi-client-common-0.11.0.jar:0.11.0]
at org.apache.hudi.table.action.commit.BaseCommitActionExecutor.commitOnAutoCommit(BaseCommitActionExecutor.java:175) ~[hudi-client-common-0.11.0.jar:0.11.0]
at org.apache.hudi.table.action.commit.BaseJavaCommitActionExecutor.updateIndexAndCommitIfNeeded(BaseJavaCommitActionExecutor.java:347) ~[hudi-java-client-0.11.0.jar:0.11.0]
at org.apache.hudi.table.action.commit.BaseJavaCommitActionExecutor.execute(BaseJavaCommitActionExecutor.java:126) ~[hudi-java-client-0.11.0.jar:0.11.0]
at org.apache.hudi.table.action.commit.BaseJavaCommitActionExecutor.execute(BaseJavaCommitActionExecutor.java:68) ~[hudi-java-client-0.11.0.jar:0.11.0]
at org.apache.hudi.table.action.commit.BaseWriteHelper.write(BaseWriteHelper.java:57) ~[hudi-client-common-0.11.0.jar:0.11.0]
... 8 common frames omitted
Caused by: java.util.ConcurrentModificationException: Cannot resolve conflicts for overlapping writes
... 21 common frames omitted
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] nsivabalan commented on issue #5807: [SUPPORT] Multi Thread in a process caused java.util.ConcurrentModificationException，please help
Posted by GitBox <gi...@apache.org>.
nsivabalan commented on issue #5807:
URL: https://github.com/apache/hudi/issues/5807#issuecomment-1152984934
We don't have multi writer support in java client yet.
We have a tracking ticket https://issues.apache.org/jira/browse/HUDI-4225
If you are interested to contribute, let us know. We can guide you through it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] nsivabalan closed issue #5807: [SUPPORT] Multi Thread in a process caused java.util.ConcurrentModificationException，please help
Posted by GitBox <gi...@apache.org>.
nsivabalan closed issue #5807: [SUPPORT] Multi Thread in a process caused java.util.ConcurrentModificationException，please help
URL: https://github.com/apache/hudi/issues/5807
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] jtchen-study commented on issue #5807: [SUPPORT] Multi Thread in a process is not ok ，please help
Posted by GitBox <gi...@apache.org>.
jtchen-study commented on issue #5807:
URL: https://github.com/apache/hudi/issues/5807#issuecomment-1150626686
It is worth mentioning that， i use the java client
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org