[jira] [Commented] (HUDI-2559) Ensure unique timestamps are generated for commit times with concurrent writers

["Raymond Xu (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/HUDI-2559?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17513803#comment-17513803 ] 

Raymond Xu commented on HUDI-2559:
----------------------------------

We need to eliminate the issue with locks or giving identifier to writers for multi-writer scenario. the last patch was a mitigation. More work to continue here.

> Ensure unique timestamps are generated for commit times with concurrent writers
> -------------------------------------------------------------------------------
>
>                 Key: HUDI-2559
>                 URL: https://issues.apache.org/jira/browse/HUDI-2559
>             Project: Apache Hudi
>          Issue Type: Improvement
>          Components: multi-writer, writer-core
>            Reporter: sivabalan narayanan
>            Assignee: sivabalan narayanan
>            Priority: Major
>              Labels: pull-request-available, release-blocker
>             Fix For: 0.11.0
>
>
> Ensure unique timestamps are generated for commit times with concurrent writers.
> this is the piece of code in HoodieActiveTimeline which creates a new commit time.
> {code:java}
> public static String createNewInstantTime(long milliseconds) {
>   return lastInstantTime.updateAndGet((oldVal) -> {
>     String newCommitTime;
>     do {
>       newCommitTime = HoodieActiveTimeline.COMMIT_FORMATTER.format(new Date(System.currentTimeMillis() + milliseconds));
>     } while (HoodieTimeline.compareTimestamps(newCommitTime, LESSER_THAN_OR_EQUALS, oldVal));
>     return newCommitTime;
>   });
> }
> {code}
> There are chances that a deltastreamer and a concurrent spark ds writer gets same timestamp and one of them fails. 
> Related issues and github jiras: 
> [https://github.com/apache/hudi/issues/3782]
> https://issues.apache.org/jira/browse/HUDI-2549
>  
>  
>  



--
This message was sent by Atlassian Jira
(v8.20.1#820001)