You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@hudi.apache.org by wei li <lw...@gmail.com> on 2020/05/25 15:00:24 UTC
[Discussion]Hudi support more complete concurrency control when write data
Now hudi just support write、compaction concurrency control. But some
scenario need write concurrency control.Such as two spark job with
different data source ,need to write to the same hudi table.
I have two Proposal：
1. first step :support write concurrency control on different partition
but now when two client write data to different partition, will meet these
error
a、Rolling back commits failed
b、instants version already exist
[2020-05-25 21:20:34,732] INFO Checking for file exists
?/tmp/HudiDLATestPartition/.hoodie/20200525212031.clean.inflight
(org.apache.hudi.common.table.timeline.HoodieActiveTimeline)
Exception in thread "main" org.apache.hudi.exception.HoodieIOException:
Failed to create file /tmp/HudiDLATestPartition/.hoodie/20200525212031.clean
at
org.apache.hudi.common.table.timeline.HoodieActiveTimeline.createImmutableFileInPath(HoodieActiveTimeline.java:437)
at
org.apache.hudi.common.table.timeline.HoodieActiveTimeline.transitionState(HoodieActiveTimeline.java:327)
at
org.apache.hudi.common.table.timeline.HoodieActiveTimeline.transitionCleanInflightToComplete(HoodieActiveTimeline.java:290)
at
org.apache.hudi.client.HoodieCleanClient.runClean(HoodieCleanClient.java:183)
at
org.apache.hudi.client.HoodieCleanClient.runClean(HoodieCleanClient.java:142)
at
org.apache.hudi.client.HoodieCleanClient.lambda$clean$0(HoodieCleanClient.java:88)
at
java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1382)
c、two client's archiving conflict
d、the read client meets "Unable to infer schema for Parquet. It must be
specified manually.;"
2. second step: support insert、upsert、compaction concurrency control on
different isolation level such as Serializable、WriteSerializable.
hudi can design a mechanism to check the confict in
AbstractHoodieWriteClient.commit()
I created a issue https://issues.apache.org/jira/browse/HUDI-944
Best Regards,
Wei Li.
Re: [Discussion]Hudi support more complete concurrency control when
write data
Posted by Vinoth Chandar <vi...@apache.org>.
Hi Wei Li,
Left some detailed comments on the JIRA
https://issues.apache.org/jira/browse/HUDI-944?focusedCommentId=17119668&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-17119668
This is a great discussion.. thanks for getting this kicked off
Thanks
Vinoth
On Mon, May 25, 2020 at 8:10 AM wei li <lw...@gmail.com> wrote:
> Now hudi just support write、compaction concurrency control. But some
> scenario need write concurrency control.Such as two spark job with
> different data source ,need to write to the same hudi table.
>
> I have two Proposal：
>
> 1. first step :support write concurrency control on different partition
> but now when two client write data to different partition, will meet these
> error
>
> a、Rolling back commits failed
>
> b、instants version already exist
> [2020-05-25 21:20:34,732] INFO Checking for file exists
> ?/tmp/HudiDLATestPartition/.hoodie/20200525212031.clean.inflight
> (org.apache.hudi.common.table.timeline.HoodieActiveTimeline)
> Exception in thread "main" org.apache.hudi.exception.HoodieIOException:
> Failed to create file
> /tmp/HudiDLATestPartition/.hoodie/20200525212031.clean
> at
>
> org.apache.hudi.common.table.timeline.HoodieActiveTimeline.createImmutableFileInPath(HoodieActiveTimeline.java:437)
> at
>
> org.apache.hudi.common.table.timeline.HoodieActiveTimeline.transitionState(HoodieActiveTimeline.java:327)
> at
>
> org.apache.hudi.common.table.timeline.HoodieActiveTimeline.transitionCleanInflightToComplete(HoodieActiveTimeline.java:290)
> at
>
> org.apache.hudi.client.HoodieCleanClient.runClean(HoodieCleanClient.java:183)
> at
>
> org.apache.hudi.client.HoodieCleanClient.runClean(HoodieCleanClient.java:142)
> at
>
> org.apache.hudi.client.HoodieCleanClient.lambda$clean$0(HoodieCleanClient.java:88)
> at
>
> java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1382)
>
> c、two client's archiving conflict
>
> d、the read client meets "Unable to infer schema for Parquet. It must be
> specified manually.;"
>
> 2. second step: support insert、upsert、compaction concurrency control on
> different isolation level such as Serializable、WriteSerializable.
>
> hudi can design a mechanism to check the confict in
> AbstractHoodieWriteClient.commit()
>
> I created a issue https://issues.apache.org/jira/browse/HUDI-944
>
> Best Regards,
> Wei Li.
>