[jira] [Created] (HUDI-2559) Ensure unique timestamps are generated for commit times with concurrent writers

["sivabalan narayanan (Jira)" <ji...@apache.org>]

sivabalan narayanan created HUDI-2559:
-----------------------------------------

             Summary: Ensure unique timestamps are generated for commit times with concurrent writers
                 Key: HUDI-2559
                 URL: https://issues.apache.org/jira/browse/HUDI-2559
             Project: Apache Hudi
          Issue Type: Improvement
            Reporter: sivabalan narayanan


Ensure unique timestamps are generated for commit times with concurrent writers.

this is the piece of code in HoodieActiveTimeline which creates a new commit time.
{code:java}
public static String createNewInstantTime(long milliseconds) {
  return lastInstantTime.updateAndGet((oldVal) -> {
    String newCommitTime;
    do {
      newCommitTime = HoodieActiveTimeline.COMMIT_FORMATTER.format(new Date(System.currentTimeMillis() + milliseconds));
    } while (HoodieTimeline.compareTimestamps(newCommitTime, LESSER_THAN_OR_EQUALS, oldVal));
    return newCommitTime;
  });
}
{code}
There are chances that a deltastreamer and a concurrent spark ds writer gets same timestamp and one of them fails. 

Related issues and github jiras: 

[https://github.com/apache/hudi/issues/3782]

https://issues.apache.org/jira/browse/HUDI-2549

 

 

 



--
This message was sent by Atlassian Jira
(v8.3.4#803005)