You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by "nsivabalan (via GitHub)" <gi...@apache.org> on 2024/05/27 23:50:19 UTC

[PR] [HUDI-7507] Adding timestamp ordering validation before creating requested timeli… [hudi]

nsivabalan opened a new pull request, #11344:
URL: https://github.com/apache/hudi/pull/11344

   ### Change Logs
   
   When multiple writers trigger table services, there is a chance that one of them could create requested in a different ordering compared to the actual timestamp. Linkedin jira has more details of the scenario w/ an illustration. This patch, ensure that before creating a requested entry in the timeline, there is no other instant greater than the current instant time. 
   
   ### Impact
   
   No unintended gaps wrt multi-writers and timestamp ordering intricacies. 
   
   ### Risk level (write none, low medium or high below)
   
   low
   
   ### Documentation Update
   
   _Describe any necessary documentation update if there is any new feature, config, or user-facing change. If not, put "none"._
   
   - _The config description must be updated if new configs are added or the default value of the configs are changed_
   - _Any new feature or user-facing change requires updating the Hudi website. Please create a Jira ticket, attach the
     ticket number here and follow the [instruction](https://hudi.apache.org/contribute/developer-setup#website) to make
     changes to the website._
   
   ### Contributor's checklist
   
   - [ ] Read through [contributor's guide](https://hudi.apache.org/contribute/how-to-contribute)
   - [ ] Change Logs and Impact were stated clearly
   - [ ] Adequate tests were added if applicable
   - [ ] CI passed
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [HUDI-7507] Adding timestamp ordering validation before creating requested timeli… [hudi]

Posted by "nsivabalan (via GitHub)" <gi...@apache.org>.
nsivabalan commented on PR #11344:
URL: https://github.com/apache/hudi/pull/11344#issuecomment-2211324444

   here is the patch for 0.x branch https://github.com/apache/hudi/pull/11580 
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [HUDI-7507] Adding timestamp ordering validation before creating requested instant [hudi]

Posted by "nsivabalan (via GitHub)" <gi...@apache.org>.
nsivabalan commented on code in PR #11344:
URL: https://github.com/apache/hudi/pull/11344#discussion_r1667085731


##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/client/BaseHoodieWriteClient.java:
##########
@@ -924,11 +925,11 @@ private void startCommit(String instantTime, String actionType, HoodieTableMetaC
         "Found pending restore in active timeline. Please complete the restore fully before proceeding. As of now, "
             + "table could be in an inconsistent state. Pending restores: " + Arrays.toString(inflightRestoreTimeline.getInstantsAsStream()
             .map(instant -> instant.getTimestamp()).collect(Collectors.toList()).toArray()));
+    TimestampUtils.validateForLatestTimestamp(metaClient, instantTime);

Review Comment:
   hey @kbuci : for 0.x branch, we are not taking a lock for regular writes, but table service schedule is all under the block. but we are reloading the timeline while we do this validation which achieves the same purpose. 
   
   in 1.x, commit time generation is w/n a lock. so, we should be good. 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [HUDI-7507] Adding timestamp ordering validation before creating requested timeli… [hudi]

Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #11344:
URL: https://github.com/apache/hudi/pull/11344#issuecomment-2136102973

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "0397e80a3f71a4c9180a08cdd03ad16d7f313661",
       "status" : "PENDING",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24098",
       "triggerID" : "0397e80a3f71a4c9180a08cdd03ad16d7f313661",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 0397e80a3f71a4c9180a08cdd03ad16d7f313661 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24098) 
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [HUDI-7507] Adding timestamp ordering validation before creating requested timeli… [hudi]

Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #11344:
URL: https://github.com/apache/hudi/pull/11344#issuecomment-2136272563

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "0397e80a3f71a4c9180a08cdd03ad16d7f313661",
       "status" : "FAILURE",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24098",
       "triggerID" : "0397e80a3f71a4c9180a08cdd03ad16d7f313661",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 0397e80a3f71a4c9180a08cdd03ad16d7f313661 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24098) 
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [HUDI-7507] Adding timestamp ordering validation before creating requested timeli… [hudi]

Posted by "nsivabalan (via GitHub)" <gi...@apache.org>.
nsivabalan commented on PR #11344:
URL: https://github.com/apache/hudi/pull/11344#issuecomment-2211301073

   hey @danny0405 : yes. I will raise a patch against 0.x branch. we may not need it for 1.x. Or we can debate if its required for 1.x. but for 0.x branch. we definitely need it. 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [HUDI-7507] Adding timestamp ordering validation before creating requested instant [hudi]

Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #11344:
URL: https://github.com/apache/hudi/pull/11344#issuecomment-2211384069

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "0397e80a3f71a4c9180a08cdd03ad16d7f313661",
       "status" : "FAILURE",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24098",
       "triggerID" : "0397e80a3f71a4c9180a08cdd03ad16d7f313661",
       "triggerType" : "PUSH"
     }, {
       "hash" : "e9c58f48cb1d142f18f362632af28dcf651b51a4",
       "status" : "PENDING",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24735",
       "triggerID" : "e9c58f48cb1d142f18f362632af28dcf651b51a4",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 0397e80a3f71a4c9180a08cdd03ad16d7f313661 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24098) 
   * e9c58f48cb1d142f18f362632af28dcf651b51a4 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24735) 
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [HUDI-7507] Adding timestamp ordering validation before creating requested timeli… [hudi]

Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #11344:
URL: https://github.com/apache/hudi/pull/11344#issuecomment-2136084282

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "0397e80a3f71a4c9180a08cdd03ad16d7f313661",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "0397e80a3f71a4c9180a08cdd03ad16d7f313661",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 0397e80a3f71a4c9180a08cdd03ad16d7f313661 UNKNOWN
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [HUDI-7507] Adding timestamp ordering validation before creating requested instant [hudi]

Posted by "yihua (via GitHub)" <gi...@apache.org>.
yihua closed pull request #11344: [HUDI-7507] Adding timestamp ordering validation before creating requested instant
URL: https://github.com/apache/hudi/pull/11344


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [HUDI-7507] Adding timestamp ordering validation before creating requested instant [hudi]

Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #11344:
URL: https://github.com/apache/hudi/pull/11344#issuecomment-2211421877

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "0397e80a3f71a4c9180a08cdd03ad16d7f313661",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24098",
       "triggerID" : "0397e80a3f71a4c9180a08cdd03ad16d7f313661",
       "triggerType" : "PUSH"
     }, {
       "hash" : "e9c58f48cb1d142f18f362632af28dcf651b51a4",
       "status" : "FAILURE",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24735",
       "triggerID" : "e9c58f48cb1d142f18f362632af28dcf651b51a4",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * e9c58f48cb1d142f18f362632af28dcf651b51a4 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24735) 
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [HUDI-7507] Adding timestamp ordering validation before creating requested instant [hudi]

Posted by "yihua (via GitHub)" <gi...@apache.org>.
yihua commented on PR #11344:
URL: https://github.com/apache/hudi/pull/11344#issuecomment-2359992992

   Closing this as Hudi 1.0 / master does not necessarily need the start time ordering.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [HUDI-7507] Adding timestamp ordering validation before creating requested instant [hudi]

Posted by "nsivabalan (via GitHub)" <gi...@apache.org>.
nsivabalan commented on code in PR #11344:
URL: https://github.com/apache/hudi/pull/11344#discussion_r1667085731


##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/client/BaseHoodieWriteClient.java:
##########
@@ -924,11 +925,11 @@ private void startCommit(String instantTime, String actionType, HoodieTableMetaC
         "Found pending restore in active timeline. Please complete the restore fully before proceeding. As of now, "
             + "table could be in an inconsistent state. Pending restores: " + Arrays.toString(inflightRestoreTimeline.getInstantsAsStream()
             .map(instant -> instant.getTimestamp()).collect(Collectors.toList()).toArray()));
+    TimestampUtils.validateForLatestTimestamp(metaClient, instantTime);

Review Comment:
   hey @kbuci : for 0.x branch, we are not taking a lock. but we are reloading the timeline while we do this validation which achieves the same purpose. in 1.x, commit time generation is w/n a lock. so, we should be good. 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [HUDI-7507] Adding timestamp ordering validation before creating requested instant [hudi]

Posted by "nsivabalan (via GitHub)" <gi...@apache.org>.
nsivabalan commented on code in PR #11344:
URL: https://github.com/apache/hudi/pull/11344#discussion_r1667092643


##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/table/HoodieTable.java:
##########
@@ -908,6 +909,14 @@ public void validateInsertSchema() throws HoodieInsertException {
     }
   }
 
+  /**
+   * Validates that the instantTime is latest in the write timeline.
+   * @param instantTime instant time of interest.
+   */
+  public void validateForLatestTimestamp(String instantTime) {

Review Comment:
   In a way, this ensure monotonically increasing commit times in 0.x 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [HUDI-7507] Adding timestamp ordering validation before creating requested instant [hudi]

Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #11344:
URL: https://github.com/apache/hudi/pull/11344#issuecomment-2211377788

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "0397e80a3f71a4c9180a08cdd03ad16d7f313661",
       "status" : "FAILURE",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24098",
       "triggerID" : "0397e80a3f71a4c9180a08cdd03ad16d7f313661",
       "triggerType" : "PUSH"
     }, {
       "hash" : "e9c58f48cb1d142f18f362632af28dcf651b51a4",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "e9c58f48cb1d142f18f362632af28dcf651b51a4",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 0397e80a3f71a4c9180a08cdd03ad16d7f313661 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=24098) 
   * e9c58f48cb1d142f18f362632af28dcf651b51a4 UNKNOWN
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org