You are viewing a plain text version of this content. The canonical link for it is here.

Posted to commits@hudi.apache.org by "CTTY (via GitHub)" <gi...@apache.org> on 2024/09/14 21:01:04 UTC

[PR] [HUDI-8141] WIP:Always use completion time to find begin Instant for incremental source [hudi]

CTTY opened a new pull request, #11947:
URL: https://github.com/apache/hudi/pull/11947

   ### Change Logs
   
   Existing Hudi incremental query would scan the entire timeline and handle hollow commits(commits that started but have not completed) along the way. 
   Hudi 1.x supports Instant completion time, providing a more elegant and intuitive completion timeline for incremental queries to leverage. 
   
   This PR is a part of effort of making incremental query use completion timeline by default
   
   ### Impact
   
   `hoodie.read.timeline.holes.resolution.policy` would no longer has effect on `HoodieIncrSource`, `GcsEventsHoodieIncrSource` and `S3EventsHoodieIncrSource`. These sources would always use completion time to find the start instant for incremental query.
   
   ### Risk level (write none, low medium or high below)
   
   low
   
   ### Documentation Update
   
   `hoodie.read.timeline.holes.resolution.policy` would no longer has effect on `HoodieIncrSource`, `GcsEventsHoodieIncrSource` and `S3EventsHoodieIncrSource`.
   
   - _The config description must be updated if new configs are added or the default value of the configs are changed_
   - _Any new feature or user-facing change requires updating the Hudi website. Please create a Jira ticket, attach the
     ticket number here and follow the [instruction](https://hudi.apache.org/contribute/developer-setup#website) to make
     changes to the website._
   
   ### Contributor's checklist
   
   - [ ] Read through [contributor's guide](https://hudi.apache.org/contribute/how-to-contribute)
   - [ ] Change Logs and Impact were stated clearly
   - [ ] Adequate tests were added if applicable
   - [ ] CI passed
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] [HUDI-8141] WIP:Always use completion time to find begin Instant for incremental source [hudi]

Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.

hudi-bot commented on PR #11947:
URL: https://github.com/apache/hudi/pull/11947#issuecomment-2403631518

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "090f256a315cbbf0c6a54d93cc2739dd693360e6",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=484",
       "triggerID" : "090f256a315cbbf0c6a54d93cc2739dd693360e6",
       "triggerType" : "PUSH"
     }, {
       "hash" : "0df5649013c1cf9e782fcbb67b8a2e7174f94512",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=985",
       "triggerID" : "0df5649013c1cf9e782fcbb67b8a2e7174f94512",
       "triggerType" : "PUSH"
     }, {
       "hash" : "d52e0128f409918644255abbe4376dc4a780bfc8",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=986",
       "triggerID" : "d52e0128f409918644255abbe4376dc4a780bfc8",
       "triggerType" : "PUSH"
     }, {
       "hash" : "9e668f65c6d4f73c76573236f8fa507dfb80edac",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "9e668f65c6d4f73c76573236f8fa507dfb80edac",
       "triggerType" : "PUSH"
     }, {
       "hash" : "f145131d524bf190986e5de600a326e32a27beca",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=987",
       "triggerID" : "f145131d524bf190986e5de600a326e32a27beca",
       "triggerType" : "PUSH"
     }, {
       "hash" : "0b008f807b42aa6fb2686922bb88d63e84a1f299",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=993",
       "triggerID" : "0b008f807b42aa6fb2686922bb88d63e84a1f299",
       "triggerType" : "PUSH"
     }, {
       "hash" : "65c01ad9f2b33d36d4890355feef91844438e732",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=995",
       "triggerID" : "65c01ad9f2b33d36d4890355feef91844438e732",
       "triggerType" : "PUSH"
     }, {
       "hash" : "955493fdc93842e8c99529cec35e39cecdbe9638",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=996",
       "triggerID" : "955493fdc93842e8c99529cec35e39cecdbe9638",
       "triggerType" : "PUSH"
     }, {
       "hash" : "533f00e5546da0026bbe51e1c030ddd1c19fdd08",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "533f00e5546da0026bbe51e1c030ddd1c19fdd08",
       "triggerType" : "PUSH"
     }, {
       "hash" : "05777be67e37f3a6a8eaa59e54d9e17e5fc6ba9b",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1016",
       "triggerID" : "05777be67e37f3a6a8eaa59e54d9e17e5fc6ba9b",
       "triggerType" : "PUSH"
     }, {
       "hash" : "a2ee90c549332a7f17a7d9b2799849b77c08a8e6",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "a2ee90c549332a7f17a7d9b2799849b77c08a8e6",
       "triggerType" : "PUSH"
     }, {
       "hash" : "05ce568b0b6ba14da9ca40d26b39ba478837d747",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1017",
       "triggerID" : "05ce568b0b6ba14da9ca40d26b39ba478837d747",
       "triggerType" : "PUSH"
     }, {
       "hash" : "840e3b33d39b1f86367bb6b0e65eadeb33de7765",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1025",
       "triggerID" : "840e3b33d39b1f86367bb6b0e65eadeb33de7765",
       "triggerType" : "PUSH"
     }, {
       "hash" : "8d2e2dd83f51d03e890d424a51d3e0e84e904a43",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1031",
       "triggerID" : "8d2e2dd83f51d03e890d424a51d3e0e84e904a43",
       "triggerType" : "PUSH"
     }, {
       "hash" : "688381914087ff11acf2f8c9372dbf39afa0bd40",
       "status" : "SUCCESS",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1035",
       "triggerID" : "688381914087ff11acf2f8c9372dbf39afa0bd40",
       "triggerType" : "PUSH"
     }, {
       "hash" : "fc2c138901e82ed5f3fbf363c2a0ce8140ade050",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "fc2c138901e82ed5f3fbf363c2a0ce8140ade050",
       "triggerType" : "PUSH"
     }, {
       "hash" : "6ed9e1cb1e5a191752d6d39c7a9d5d41840f0cdf",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "6ed9e1cb1e5a191752d6d39c7a9d5d41840f0cdf",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 9e668f65c6d4f73c76573236f8fa507dfb80edac UNKNOWN
   * 533f00e5546da0026bbe51e1c030ddd1c19fdd08 UNKNOWN
   * a2ee90c549332a7f17a7d9b2799849b77c08a8e6 UNKNOWN
   * 688381914087ff11acf2f8c9372dbf39afa0bd40 Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1035) 
   * fc2c138901e82ed5f3fbf363c2a0ce8140ade050 UNKNOWN
   * 6ed9e1cb1e5a191752d6d39c7a9d5d41840f0cdf UNKNOWN
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] [HUDI-8141] Incremental Query with Completion Time [hudi]

Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.

hudi-bot commented on PR #11947:
URL: https://github.com/apache/hudi/pull/11947#issuecomment-2418517187

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "090f256a315cbbf0c6a54d93cc2739dd693360e6",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=484",
       "triggerID" : "090f256a315cbbf0c6a54d93cc2739dd693360e6",
       "triggerType" : "PUSH"
     }, {
       "hash" : "0df5649013c1cf9e782fcbb67b8a2e7174f94512",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=985",
       "triggerID" : "0df5649013c1cf9e782fcbb67b8a2e7174f94512",
       "triggerType" : "PUSH"
     }, {
       "hash" : "d52e0128f409918644255abbe4376dc4a780bfc8",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=986",
       "triggerID" : "d52e0128f409918644255abbe4376dc4a780bfc8",
       "triggerType" : "PUSH"
     }, {
       "hash" : "9e668f65c6d4f73c76573236f8fa507dfb80edac",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "9e668f65c6d4f73c76573236f8fa507dfb80edac",
       "triggerType" : "PUSH"
     }, {
       "hash" : "f145131d524bf190986e5de600a326e32a27beca",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=987",
       "triggerID" : "f145131d524bf190986e5de600a326e32a27beca",
       "triggerType" : "PUSH"
     }, {
       "hash" : "0b008f807b42aa6fb2686922bb88d63e84a1f299",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=993",
       "triggerID" : "0b008f807b42aa6fb2686922bb88d63e84a1f299",
       "triggerType" : "PUSH"
     }, {
       "hash" : "65c01ad9f2b33d36d4890355feef91844438e732",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=995",
       "triggerID" : "65c01ad9f2b33d36d4890355feef91844438e732",
       "triggerType" : "PUSH"
     }, {
       "hash" : "955493fdc93842e8c99529cec35e39cecdbe9638",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=996",
       "triggerID" : "955493fdc93842e8c99529cec35e39cecdbe9638",
       "triggerType" : "PUSH"
     }, {
       "hash" : "533f00e5546da0026bbe51e1c030ddd1c19fdd08",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "533f00e5546da0026bbe51e1c030ddd1c19fdd08",
       "triggerType" : "PUSH"
     }, {
       "hash" : "05777be67e37f3a6a8eaa59e54d9e17e5fc6ba9b",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1016",
       "triggerID" : "05777be67e37f3a6a8eaa59e54d9e17e5fc6ba9b",
       "triggerType" : "PUSH"
     }, {
       "hash" : "a2ee90c549332a7f17a7d9b2799849b77c08a8e6",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "a2ee90c549332a7f17a7d9b2799849b77c08a8e6",
       "triggerType" : "PUSH"
     }, {
       "hash" : "05ce568b0b6ba14da9ca40d26b39ba478837d747",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1017",
       "triggerID" : "05ce568b0b6ba14da9ca40d26b39ba478837d747",
       "triggerType" : "PUSH"
     }, {
       "hash" : "840e3b33d39b1f86367bb6b0e65eadeb33de7765",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1025",
       "triggerID" : "840e3b33d39b1f86367bb6b0e65eadeb33de7765",
       "triggerType" : "PUSH"
     }, {
       "hash" : "8d2e2dd83f51d03e890d424a51d3e0e84e904a43",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1031",
       "triggerID" : "8d2e2dd83f51d03e890d424a51d3e0e84e904a43",
       "triggerType" : "PUSH"
     }, {
       "hash" : "688381914087ff11acf2f8c9372dbf39afa0bd40",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1035",
       "triggerID" : "688381914087ff11acf2f8c9372dbf39afa0bd40",
       "triggerType" : "PUSH"
     }, {
       "hash" : "fc2c138901e82ed5f3fbf363c2a0ce8140ade050",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "fc2c138901e82ed5f3fbf363c2a0ce8140ade050",
       "triggerType" : "PUSH"
     }, {
       "hash" : "6ed9e1cb1e5a191752d6d39c7a9d5d41840f0cdf",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1039",
       "triggerID" : "6ed9e1cb1e5a191752d6d39c7a9d5d41840f0cdf",
       "triggerType" : "PUSH"
     }, {
       "hash" : "a510a7de1458bb996df5bbb341e8d010e4c91381",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "a510a7de1458bb996df5bbb341e8d010e4c91381",
       "triggerType" : "PUSH"
     }, {
       "hash" : "49677397cd117228742205944606f7a789090376",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1042",
       "triggerID" : "49677397cd117228742205944606f7a789090376",
       "triggerType" : "PUSH"
     }, {
       "hash" : "1630db0a37422aed942ff280560a9d6329df5015",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1044",
       "triggerID" : "1630db0a37422aed942ff280560a9d6329df5015",
       "triggerType" : "PUSH"
     }, {
       "hash" : "f11c76301d4ab524db0af57aa831be3a898ea0fb",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1048",
       "triggerID" : "f11c76301d4ab524db0af57aa831be3a898ea0fb",
       "triggerType" : "PUSH"
     }, {
       "hash" : "d1a0dc4c2017853a7ba0c3302f886043a63f29da",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1050",
       "triggerID" : "d1a0dc4c2017853a7ba0c3302f886043a63f29da",
       "triggerType" : "PUSH"
     }, {
       "hash" : "6a4c15ec39ded1f90b113b272554217698a98e1c",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1087",
       "triggerID" : "6a4c15ec39ded1f90b113b272554217698a98e1c",
       "triggerType" : "PUSH"
     }, {
       "hash" : "288e730d645a5e58e50a87ef5f44ba11a8fb0533",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1092",
       "triggerID" : "288e730d645a5e58e50a87ef5f44ba11a8fb0533",
       "triggerType" : "PUSH"
     }, {
       "hash" : "70a5c21321a0da3a0be8d85e206c24befa7b89f1",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1095",
       "triggerID" : "70a5c21321a0da3a0be8d85e206c24befa7b89f1",
       "triggerType" : "PUSH"
     }, {
       "hash" : "f072a986114949c5e426ce64a5f5568fc718e25d",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "f072a986114949c5e426ce64a5f5568fc718e25d",
       "triggerType" : "PUSH"
     }, {
       "hash" : "45b27475b1e8806ec8c566d6f151f4bc79c8549d",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1096",
       "triggerID" : "45b27475b1e8806ec8c566d6f151f4bc79c8549d",
       "triggerType" : "PUSH"
     }, {
       "hash" : "3c62ef22f6cde28110e5eb2fca8760a008d77723",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "3c62ef22f6cde28110e5eb2fca8760a008d77723",
       "triggerType" : "PUSH"
     }, {
       "hash" : "dc785fa04e542b2ccc45494e0292f5153c7ecea8",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1097",
       "triggerID" : "dc785fa04e542b2ccc45494e0292f5153c7ecea8",
       "triggerType" : "PUSH"
     }, {
       "hash" : "585ddb8049b9b9ccb057027e265b08a1b586df8c",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "585ddb8049b9b9ccb057027e265b08a1b586df8c",
       "triggerType" : "PUSH"
     }, {
       "hash" : "b62ff8675beed2c4487c788b9ca1a0ad4271b913",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "b62ff8675beed2c4487c788b9ca1a0ad4271b913",
       "triggerType" : "PUSH"
     }, {
       "hash" : "2d46048a7402907f348cc96da2839ba1b3e4932a",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1099",
       "triggerID" : "2d46048a7402907f348cc96da2839ba1b3e4932a",
       "triggerType" : "PUSH"
     }, {
       "hash" : "004fb6ebf23f796a4b77650e0530f0f1145e8b91",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1101",
       "triggerID" : "004fb6ebf23f796a4b77650e0530f0f1145e8b91",
       "triggerType" : "PUSH"
     }, {
       "hash" : "af0c33333b5e9ae6d59f20a6db1d7c016ea015d3",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1165",
       "triggerID" : "af0c33333b5e9ae6d59f20a6db1d7c016ea015d3",
       "triggerType" : "PUSH"
     }, {
       "hash" : "da8111118d71306500229619d3b7996fcd2526e8",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "da8111118d71306500229619d3b7996fcd2526e8",
       "triggerType" : "PUSH"
     }, {
       "hash" : "83257e2b9bd3c426cf90e11003a5f244a750d7fa",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1167",
       "triggerID" : "83257e2b9bd3c426cf90e11003a5f244a750d7fa",
       "triggerType" : "PUSH"
     }, {
       "hash" : "e0de408f1b8153fdaaea4c87b27a7b69213b75b5",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1177",
       "triggerID" : "e0de408f1b8153fdaaea4c87b27a7b69213b75b5",
       "triggerType" : "PUSH"
     }, {
       "hash" : "3f7951f7ad7931f5dc0b977d7c33bc5429623535",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1180",
       "triggerID" : "3f7951f7ad7931f5dc0b977d7c33bc5429623535",
       "triggerType" : "PUSH"
     }, {
       "hash" : "9cd6f06a753b63630e8245d51338d2bf9ab86b84",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "9cd6f06a753b63630e8245d51338d2bf9ab86b84",
       "triggerType" : "PUSH"
     }, {
       "hash" : "7380b902731a92e0645c02397e14591a05798620",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "7380b902731a92e0645c02397e14591a05798620",
       "triggerType" : "PUSH"
     }, {
       "hash" : "0bd5cca01a541845081d5430d48b021d4bdf0969",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "0bd5cca01a541845081d5430d48b021d4bdf0969",
       "triggerType" : "PUSH"
     }, {
       "hash" : "8f4e175ef554f015e34e0968b656b5de714ab430",
       "status" : "SUCCESS",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1182",
       "triggerID" : "8f4e175ef554f015e34e0968b656b5de714ab430",
       "triggerType" : "PUSH"
     }, {
       "hash" : "2023f3c882d0c2f5b96e28d2f21812dc0c64aaee",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "2023f3c882d0c2f5b96e28d2f21812dc0c64aaee",
       "triggerType" : "PUSH"
     }, {
       "hash" : "8394acf060d5c3953dbd67cf930b9b1d3b92d8b5",
       "status" : "PENDING",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1198",
       "triggerID" : "8394acf060d5c3953dbd67cf930b9b1d3b92d8b5",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 9e668f65c6d4f73c76573236f8fa507dfb80edac UNKNOWN
   * 533f00e5546da0026bbe51e1c030ddd1c19fdd08 UNKNOWN
   * a2ee90c549332a7f17a7d9b2799849b77c08a8e6 UNKNOWN
   * fc2c138901e82ed5f3fbf363c2a0ce8140ade050 UNKNOWN
   * a510a7de1458bb996df5bbb341e8d010e4c91381 UNKNOWN
   * f072a986114949c5e426ce64a5f5568fc718e25d UNKNOWN
   * 3c62ef22f6cde28110e5eb2fca8760a008d77723 UNKNOWN
   * 585ddb8049b9b9ccb057027e265b08a1b586df8c UNKNOWN
   * b62ff8675beed2c4487c788b9ca1a0ad4271b913 UNKNOWN
   * da8111118d71306500229619d3b7996fcd2526e8 UNKNOWN
   * 9cd6f06a753b63630e8245d51338d2bf9ab86b84 UNKNOWN
   * 7380b902731a92e0645c02397e14591a05798620 UNKNOWN
   * 0bd5cca01a541845081d5430d48b021d4bdf0969 UNKNOWN
   * 8f4e175ef554f015e34e0968b656b5de714ab430 Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1182) 
   * 2023f3c882d0c2f5b96e28d2f21812dc0c64aaee UNKNOWN
   * 8394acf060d5c3953dbd67cf930b9b1d3b92d8b5 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1198) 
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] [HUDI-8141] Incremental Query with Completion Time [hudi]

Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.

hudi-bot commented on PR #11947:
URL: https://github.com/apache/hudi/pull/11947#issuecomment-2418749973

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "090f256a315cbbf0c6a54d93cc2739dd693360e6",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=484",
       "triggerID" : "090f256a315cbbf0c6a54d93cc2739dd693360e6",
       "triggerType" : "PUSH"
     }, {
       "hash" : "0df5649013c1cf9e782fcbb67b8a2e7174f94512",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=985",
       "triggerID" : "0df5649013c1cf9e782fcbb67b8a2e7174f94512",
       "triggerType" : "PUSH"
     }, {
       "hash" : "d52e0128f409918644255abbe4376dc4a780bfc8",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=986",
       "triggerID" : "d52e0128f409918644255abbe4376dc4a780bfc8",
       "triggerType" : "PUSH"
     }, {
       "hash" : "9e668f65c6d4f73c76573236f8fa507dfb80edac",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "9e668f65c6d4f73c76573236f8fa507dfb80edac",
       "triggerType" : "PUSH"
     }, {
       "hash" : "f145131d524bf190986e5de600a326e32a27beca",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=987",
       "triggerID" : "f145131d524bf190986e5de600a326e32a27beca",
       "triggerType" : "PUSH"
     }, {
       "hash" : "0b008f807b42aa6fb2686922bb88d63e84a1f299",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=993",
       "triggerID" : "0b008f807b42aa6fb2686922bb88d63e84a1f299",
       "triggerType" : "PUSH"
     }, {
       "hash" : "65c01ad9f2b33d36d4890355feef91844438e732",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=995",
       "triggerID" : "65c01ad9f2b33d36d4890355feef91844438e732",
       "triggerType" : "PUSH"
     }, {
       "hash" : "955493fdc93842e8c99529cec35e39cecdbe9638",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=996",
       "triggerID" : "955493fdc93842e8c99529cec35e39cecdbe9638",
       "triggerType" : "PUSH"
     }, {
       "hash" : "533f00e5546da0026bbe51e1c030ddd1c19fdd08",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "533f00e5546da0026bbe51e1c030ddd1c19fdd08",
       "triggerType" : "PUSH"
     }, {
       "hash" : "05777be67e37f3a6a8eaa59e54d9e17e5fc6ba9b",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1016",
       "triggerID" : "05777be67e37f3a6a8eaa59e54d9e17e5fc6ba9b",
       "triggerType" : "PUSH"
     }, {
       "hash" : "a2ee90c549332a7f17a7d9b2799849b77c08a8e6",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "a2ee90c549332a7f17a7d9b2799849b77c08a8e6",
       "triggerType" : "PUSH"
     }, {
       "hash" : "05ce568b0b6ba14da9ca40d26b39ba478837d747",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1017",
       "triggerID" : "05ce568b0b6ba14da9ca40d26b39ba478837d747",
       "triggerType" : "PUSH"
     }, {
       "hash" : "840e3b33d39b1f86367bb6b0e65eadeb33de7765",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1025",
       "triggerID" : "840e3b33d39b1f86367bb6b0e65eadeb33de7765",
       "triggerType" : "PUSH"
     }, {
       "hash" : "8d2e2dd83f51d03e890d424a51d3e0e84e904a43",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1031",
       "triggerID" : "8d2e2dd83f51d03e890d424a51d3e0e84e904a43",
       "triggerType" : "PUSH"
     }, {
       "hash" : "688381914087ff11acf2f8c9372dbf39afa0bd40",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1035",
       "triggerID" : "688381914087ff11acf2f8c9372dbf39afa0bd40",
       "triggerType" : "PUSH"
     }, {
       "hash" : "fc2c138901e82ed5f3fbf363c2a0ce8140ade050",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "fc2c138901e82ed5f3fbf363c2a0ce8140ade050",
       "triggerType" : "PUSH"
     }, {
       "hash" : "6ed9e1cb1e5a191752d6d39c7a9d5d41840f0cdf",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1039",
       "triggerID" : "6ed9e1cb1e5a191752d6d39c7a9d5d41840f0cdf",
       "triggerType" : "PUSH"
     }, {
       "hash" : "a510a7de1458bb996df5bbb341e8d010e4c91381",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "a510a7de1458bb996df5bbb341e8d010e4c91381",
       "triggerType" : "PUSH"
     }, {
       "hash" : "49677397cd117228742205944606f7a789090376",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1042",
       "triggerID" : "49677397cd117228742205944606f7a789090376",
       "triggerType" : "PUSH"
     }, {
       "hash" : "1630db0a37422aed942ff280560a9d6329df5015",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1044",
       "triggerID" : "1630db0a37422aed942ff280560a9d6329df5015",
       "triggerType" : "PUSH"
     }, {
       "hash" : "f11c76301d4ab524db0af57aa831be3a898ea0fb",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1048",
       "triggerID" : "f11c76301d4ab524db0af57aa831be3a898ea0fb",
       "triggerType" : "PUSH"
     }, {
       "hash" : "d1a0dc4c2017853a7ba0c3302f886043a63f29da",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1050",
       "triggerID" : "d1a0dc4c2017853a7ba0c3302f886043a63f29da",
       "triggerType" : "PUSH"
     }, {
       "hash" : "6a4c15ec39ded1f90b113b272554217698a98e1c",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1087",
       "triggerID" : "6a4c15ec39ded1f90b113b272554217698a98e1c",
       "triggerType" : "PUSH"
     }, {
       "hash" : "288e730d645a5e58e50a87ef5f44ba11a8fb0533",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1092",
       "triggerID" : "288e730d645a5e58e50a87ef5f44ba11a8fb0533",
       "triggerType" : "PUSH"
     }, {
       "hash" : "70a5c21321a0da3a0be8d85e206c24befa7b89f1",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1095",
       "triggerID" : "70a5c21321a0da3a0be8d85e206c24befa7b89f1",
       "triggerType" : "PUSH"
     }, {
       "hash" : "f072a986114949c5e426ce64a5f5568fc718e25d",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "f072a986114949c5e426ce64a5f5568fc718e25d",
       "triggerType" : "PUSH"
     }, {
       "hash" : "45b27475b1e8806ec8c566d6f151f4bc79c8549d",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1096",
       "triggerID" : "45b27475b1e8806ec8c566d6f151f4bc79c8549d",
       "triggerType" : "PUSH"
     }, {
       "hash" : "3c62ef22f6cde28110e5eb2fca8760a008d77723",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "3c62ef22f6cde28110e5eb2fca8760a008d77723",
       "triggerType" : "PUSH"
     }, {
       "hash" : "dc785fa04e542b2ccc45494e0292f5153c7ecea8",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1097",
       "triggerID" : "dc785fa04e542b2ccc45494e0292f5153c7ecea8",
       "triggerType" : "PUSH"
     }, {
       "hash" : "585ddb8049b9b9ccb057027e265b08a1b586df8c",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "585ddb8049b9b9ccb057027e265b08a1b586df8c",
       "triggerType" : "PUSH"
     }, {
       "hash" : "b62ff8675beed2c4487c788b9ca1a0ad4271b913",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "b62ff8675beed2c4487c788b9ca1a0ad4271b913",
       "triggerType" : "PUSH"
     }, {
       "hash" : "2d46048a7402907f348cc96da2839ba1b3e4932a",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1099",
       "triggerID" : "2d46048a7402907f348cc96da2839ba1b3e4932a",
       "triggerType" : "PUSH"
     }, {
       "hash" : "004fb6ebf23f796a4b77650e0530f0f1145e8b91",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1101",
       "triggerID" : "004fb6ebf23f796a4b77650e0530f0f1145e8b91",
       "triggerType" : "PUSH"
     }, {
       "hash" : "af0c33333b5e9ae6d59f20a6db1d7c016ea015d3",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1165",
       "triggerID" : "af0c33333b5e9ae6d59f20a6db1d7c016ea015d3",
       "triggerType" : "PUSH"
     }, {
       "hash" : "da8111118d71306500229619d3b7996fcd2526e8",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "da8111118d71306500229619d3b7996fcd2526e8",
       "triggerType" : "PUSH"
     }, {
       "hash" : "83257e2b9bd3c426cf90e11003a5f244a750d7fa",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1167",
       "triggerID" : "83257e2b9bd3c426cf90e11003a5f244a750d7fa",
       "triggerType" : "PUSH"
     }, {
       "hash" : "e0de408f1b8153fdaaea4c87b27a7b69213b75b5",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1177",
       "triggerID" : "e0de408f1b8153fdaaea4c87b27a7b69213b75b5",
       "triggerType" : "PUSH"
     }, {
       "hash" : "3f7951f7ad7931f5dc0b977d7c33bc5429623535",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1180",
       "triggerID" : "3f7951f7ad7931f5dc0b977d7c33bc5429623535",
       "triggerType" : "PUSH"
     }, {
       "hash" : "9cd6f06a753b63630e8245d51338d2bf9ab86b84",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "9cd6f06a753b63630e8245d51338d2bf9ab86b84",
       "triggerType" : "PUSH"
     }, {
       "hash" : "7380b902731a92e0645c02397e14591a05798620",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "7380b902731a92e0645c02397e14591a05798620",
       "triggerType" : "PUSH"
     }, {
       "hash" : "0bd5cca01a541845081d5430d48b021d4bdf0969",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "0bd5cca01a541845081d5430d48b021d4bdf0969",
       "triggerType" : "PUSH"
     }, {
       "hash" : "8f4e175ef554f015e34e0968b656b5de714ab430",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1182",
       "triggerID" : "8f4e175ef554f015e34e0968b656b5de714ab430",
       "triggerType" : "PUSH"
     }, {
       "hash" : "2023f3c882d0c2f5b96e28d2f21812dc0c64aaee",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "2023f3c882d0c2f5b96e28d2f21812dc0c64aaee",
       "triggerType" : "PUSH"
     }, {
       "hash" : "8394acf060d5c3953dbd67cf930b9b1d3b92d8b5",
       "status" : "SUCCESS",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1198",
       "triggerID" : "8394acf060d5c3953dbd67cf930b9b1d3b92d8b5",
       "triggerType" : "PUSH"
     }, {
       "hash" : "7ec42c105971bff463e359019405a7f4e7947d86",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "7ec42c105971bff463e359019405a7f4e7947d86",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 9e668f65c6d4f73c76573236f8fa507dfb80edac UNKNOWN
   * 533f00e5546da0026bbe51e1c030ddd1c19fdd08 UNKNOWN
   * a2ee90c549332a7f17a7d9b2799849b77c08a8e6 UNKNOWN
   * fc2c138901e82ed5f3fbf363c2a0ce8140ade050 UNKNOWN
   * a510a7de1458bb996df5bbb341e8d010e4c91381 UNKNOWN
   * f072a986114949c5e426ce64a5f5568fc718e25d UNKNOWN
   * 3c62ef22f6cde28110e5eb2fca8760a008d77723 UNKNOWN
   * 585ddb8049b9b9ccb057027e265b08a1b586df8c UNKNOWN
   * b62ff8675beed2c4487c788b9ca1a0ad4271b913 UNKNOWN
   * da8111118d71306500229619d3b7996fcd2526e8 UNKNOWN
   * 9cd6f06a753b63630e8245d51338d2bf9ab86b84 UNKNOWN
   * 7380b902731a92e0645c02397e14591a05798620 UNKNOWN
   * 0bd5cca01a541845081d5430d48b021d4bdf0969 UNKNOWN
   * 2023f3c882d0c2f5b96e28d2f21812dc0c64aaee UNKNOWN
   * 8394acf060d5c3953dbd67cf930b9b1d3b92d8b5 Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1198) 
   * 7ec42c105971bff463e359019405a7f4e7947d86 UNKNOWN
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] [HUDI-8141] Incremental Query with Completion Time [hudi]

Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.

hudi-bot commented on PR #11947:
URL: https://github.com/apache/hudi/pull/11947#issuecomment-2408835668

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "090f256a315cbbf0c6a54d93cc2739dd693360e6",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=484",
       "triggerID" : "090f256a315cbbf0c6a54d93cc2739dd693360e6",
       "triggerType" : "PUSH"
     }, {
       "hash" : "0df5649013c1cf9e782fcbb67b8a2e7174f94512",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=985",
       "triggerID" : "0df5649013c1cf9e782fcbb67b8a2e7174f94512",
       "triggerType" : "PUSH"
     }, {
       "hash" : "d52e0128f409918644255abbe4376dc4a780bfc8",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=986",
       "triggerID" : "d52e0128f409918644255abbe4376dc4a780bfc8",
       "triggerType" : "PUSH"
     }, {
       "hash" : "9e668f65c6d4f73c76573236f8fa507dfb80edac",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "9e668f65c6d4f73c76573236f8fa507dfb80edac",
       "triggerType" : "PUSH"
     }, {
       "hash" : "f145131d524bf190986e5de600a326e32a27beca",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=987",
       "triggerID" : "f145131d524bf190986e5de600a326e32a27beca",
       "triggerType" : "PUSH"
     }, {
       "hash" : "0b008f807b42aa6fb2686922bb88d63e84a1f299",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=993",
       "triggerID" : "0b008f807b42aa6fb2686922bb88d63e84a1f299",
       "triggerType" : "PUSH"
     }, {
       "hash" : "65c01ad9f2b33d36d4890355feef91844438e732",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=995",
       "triggerID" : "65c01ad9f2b33d36d4890355feef91844438e732",
       "triggerType" : "PUSH"
     }, {
       "hash" : "955493fdc93842e8c99529cec35e39cecdbe9638",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=996",
       "triggerID" : "955493fdc93842e8c99529cec35e39cecdbe9638",
       "triggerType" : "PUSH"
     }, {
       "hash" : "533f00e5546da0026bbe51e1c030ddd1c19fdd08",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "533f00e5546da0026bbe51e1c030ddd1c19fdd08",
       "triggerType" : "PUSH"
     }, {
       "hash" : "05777be67e37f3a6a8eaa59e54d9e17e5fc6ba9b",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1016",
       "triggerID" : "05777be67e37f3a6a8eaa59e54d9e17e5fc6ba9b",
       "triggerType" : "PUSH"
     }, {
       "hash" : "a2ee90c549332a7f17a7d9b2799849b77c08a8e6",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "a2ee90c549332a7f17a7d9b2799849b77c08a8e6",
       "triggerType" : "PUSH"
     }, {
       "hash" : "05ce568b0b6ba14da9ca40d26b39ba478837d747",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1017",
       "triggerID" : "05ce568b0b6ba14da9ca40d26b39ba478837d747",
       "triggerType" : "PUSH"
     }, {
       "hash" : "840e3b33d39b1f86367bb6b0e65eadeb33de7765",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1025",
       "triggerID" : "840e3b33d39b1f86367bb6b0e65eadeb33de7765",
       "triggerType" : "PUSH"
     }, {
       "hash" : "8d2e2dd83f51d03e890d424a51d3e0e84e904a43",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1031",
       "triggerID" : "8d2e2dd83f51d03e890d424a51d3e0e84e904a43",
       "triggerType" : "PUSH"
     }, {
       "hash" : "688381914087ff11acf2f8c9372dbf39afa0bd40",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1035",
       "triggerID" : "688381914087ff11acf2f8c9372dbf39afa0bd40",
       "triggerType" : "PUSH"
     }, {
       "hash" : "fc2c138901e82ed5f3fbf363c2a0ce8140ade050",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "fc2c138901e82ed5f3fbf363c2a0ce8140ade050",
       "triggerType" : "PUSH"
     }, {
       "hash" : "6ed9e1cb1e5a191752d6d39c7a9d5d41840f0cdf",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1039",
       "triggerID" : "6ed9e1cb1e5a191752d6d39c7a9d5d41840f0cdf",
       "triggerType" : "PUSH"
     }, {
       "hash" : "a510a7de1458bb996df5bbb341e8d010e4c91381",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "a510a7de1458bb996df5bbb341e8d010e4c91381",
       "triggerType" : "PUSH"
     }, {
       "hash" : "49677397cd117228742205944606f7a789090376",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1042",
       "triggerID" : "49677397cd117228742205944606f7a789090376",
       "triggerType" : "PUSH"
     }, {
       "hash" : "1630db0a37422aed942ff280560a9d6329df5015",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1044",
       "triggerID" : "1630db0a37422aed942ff280560a9d6329df5015",
       "triggerType" : "PUSH"
     }, {
       "hash" : "f11c76301d4ab524db0af57aa831be3a898ea0fb",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1048",
       "triggerID" : "f11c76301d4ab524db0af57aa831be3a898ea0fb",
       "triggerType" : "PUSH"
     }, {
       "hash" : "d1a0dc4c2017853a7ba0c3302f886043a63f29da",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1050",
       "triggerID" : "d1a0dc4c2017853a7ba0c3302f886043a63f29da",
       "triggerType" : "PUSH"
     }, {
       "hash" : "6a4c15ec39ded1f90b113b272554217698a98e1c",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1087",
       "triggerID" : "6a4c15ec39ded1f90b113b272554217698a98e1c",
       "triggerType" : "PUSH"
     }, {
       "hash" : "288e730d645a5e58e50a87ef5f44ba11a8fb0533",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1092",
       "triggerID" : "288e730d645a5e58e50a87ef5f44ba11a8fb0533",
       "triggerType" : "PUSH"
     }, {
       "hash" : "70a5c21321a0da3a0be8d85e206c24befa7b89f1",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1095",
       "triggerID" : "70a5c21321a0da3a0be8d85e206c24befa7b89f1",
       "triggerType" : "PUSH"
     }, {
       "hash" : "f072a986114949c5e426ce64a5f5568fc718e25d",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "f072a986114949c5e426ce64a5f5568fc718e25d",
       "triggerType" : "PUSH"
     }, {
       "hash" : "45b27475b1e8806ec8c566d6f151f4bc79c8549d",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1096",
       "triggerID" : "45b27475b1e8806ec8c566d6f151f4bc79c8549d",
       "triggerType" : "PUSH"
     }, {
       "hash" : "3c62ef22f6cde28110e5eb2fca8760a008d77723",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "3c62ef22f6cde28110e5eb2fca8760a008d77723",
       "triggerType" : "PUSH"
     }, {
       "hash" : "dc785fa04e542b2ccc45494e0292f5153c7ecea8",
       "status" : "FAILURE",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1097",
       "triggerID" : "dc785fa04e542b2ccc45494e0292f5153c7ecea8",
       "triggerType" : "PUSH"
     }, {
       "hash" : "585ddb8049b9b9ccb057027e265b08a1b586df8c",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "585ddb8049b9b9ccb057027e265b08a1b586df8c",
       "triggerType" : "PUSH"
     }, {
       "hash" : "b62ff8675beed2c4487c788b9ca1a0ad4271b913",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "b62ff8675beed2c4487c788b9ca1a0ad4271b913",
       "triggerType" : "PUSH"
     }, {
       "hash" : "2d46048a7402907f348cc96da2839ba1b3e4932a",
       "status" : "PENDING",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1099",
       "triggerID" : "2d46048a7402907f348cc96da2839ba1b3e4932a",
       "triggerType" : "PUSH"
     }, {
       "hash" : "004fb6ebf23f796a4b77650e0530f0f1145e8b91",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "004fb6ebf23f796a4b77650e0530f0f1145e8b91",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 9e668f65c6d4f73c76573236f8fa507dfb80edac UNKNOWN
   * 533f00e5546da0026bbe51e1c030ddd1c19fdd08 UNKNOWN
   * a2ee90c549332a7f17a7d9b2799849b77c08a8e6 UNKNOWN
   * fc2c138901e82ed5f3fbf363c2a0ce8140ade050 UNKNOWN
   * a510a7de1458bb996df5bbb341e8d010e4c91381 UNKNOWN
   * f072a986114949c5e426ce64a5f5568fc718e25d UNKNOWN
   * 3c62ef22f6cde28110e5eb2fca8760a008d77723 UNKNOWN
   * dc785fa04e542b2ccc45494e0292f5153c7ecea8 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1097) 
   * 585ddb8049b9b9ccb057027e265b08a1b586df8c UNKNOWN
   * b62ff8675beed2c4487c788b9ca1a0ad4271b913 UNKNOWN
   * 2d46048a7402907f348cc96da2839ba1b3e4932a Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1099) 
   * 004fb6ebf23f796a4b77650e0530f0f1145e8b91 UNKNOWN
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] [HUDI-8141] WIP:Always use completion time to find begin Instant for incremental source [hudi]

Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.

hudi-bot commented on PR #11947:
URL: https://github.com/apache/hudi/pull/11947#issuecomment-2401419924

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "090f256a315cbbf0c6a54d93cc2739dd693360e6",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=484",
       "triggerID" : "090f256a315cbbf0c6a54d93cc2739dd693360e6",
       "triggerType" : "PUSH"
     }, {
       "hash" : "0df5649013c1cf9e782fcbb67b8a2e7174f94512",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=985",
       "triggerID" : "0df5649013c1cf9e782fcbb67b8a2e7174f94512",
       "triggerType" : "PUSH"
     }, {
       "hash" : "d52e0128f409918644255abbe4376dc4a780bfc8",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=986",
       "triggerID" : "d52e0128f409918644255abbe4376dc4a780bfc8",
       "triggerType" : "PUSH"
     }, {
       "hash" : "9e668f65c6d4f73c76573236f8fa507dfb80edac",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "9e668f65c6d4f73c76573236f8fa507dfb80edac",
       "triggerType" : "PUSH"
     }, {
       "hash" : "f145131d524bf190986e5de600a326e32a27beca",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=987",
       "triggerID" : "f145131d524bf190986e5de600a326e32a27beca",
       "triggerType" : "PUSH"
     }, {
       "hash" : "0b008f807b42aa6fb2686922bb88d63e84a1f299",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=993",
       "triggerID" : "0b008f807b42aa6fb2686922bb88d63e84a1f299",
       "triggerType" : "PUSH"
     }, {
       "hash" : "65c01ad9f2b33d36d4890355feef91844438e732",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=995",
       "triggerID" : "65c01ad9f2b33d36d4890355feef91844438e732",
       "triggerType" : "PUSH"
     }, {
       "hash" : "955493fdc93842e8c99529cec35e39cecdbe9638",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=996",
       "triggerID" : "955493fdc93842e8c99529cec35e39cecdbe9638",
       "triggerType" : "PUSH"
     }, {
       "hash" : "533f00e5546da0026bbe51e1c030ddd1c19fdd08",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "533f00e5546da0026bbe51e1c030ddd1c19fdd08",
       "triggerType" : "PUSH"
     }, {
       "hash" : "05777be67e37f3a6a8eaa59e54d9e17e5fc6ba9b",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1016",
       "triggerID" : "05777be67e37f3a6a8eaa59e54d9e17e5fc6ba9b",
       "triggerType" : "PUSH"
     }, {
       "hash" : "a2ee90c549332a7f17a7d9b2799849b77c08a8e6",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "a2ee90c549332a7f17a7d9b2799849b77c08a8e6",
       "triggerType" : "PUSH"
     }, {
       "hash" : "05ce568b0b6ba14da9ca40d26b39ba478837d747",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1017",
       "triggerID" : "05ce568b0b6ba14da9ca40d26b39ba478837d747",
       "triggerType" : "PUSH"
     }, {
       "hash" : "840e3b33d39b1f86367bb6b0e65eadeb33de7765",
       "status" : "FAILURE",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1025",
       "triggerID" : "840e3b33d39b1f86367bb6b0e65eadeb33de7765",
       "triggerType" : "PUSH"
     }, {
       "hash" : "8d2e2dd83f51d03e890d424a51d3e0e84e904a43",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "8d2e2dd83f51d03e890d424a51d3e0e84e904a43",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 9e668f65c6d4f73c76573236f8fa507dfb80edac UNKNOWN
   * 533f00e5546da0026bbe51e1c030ddd1c19fdd08 UNKNOWN
   * a2ee90c549332a7f17a7d9b2799849b77c08a8e6 UNKNOWN
   * 840e3b33d39b1f86367bb6b0e65eadeb33de7765 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1025) 
   * 8d2e2dd83f51d03e890d424a51d3e0e84e904a43 UNKNOWN
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] [HUDI-8141] Incremental Query with Completion Time [hudi]

Posted by "yihua (via GitHub)" <gi...@apache.org>.

yihua commented on code in PR #11947:
URL: https://github.com/apache/hudi/pull/11947#discussion_r1805752041


##########
hudi-utilities/src/main/java/org/apache/hudi/utilities/sources/HoodieIncrSource.java:
##########
@@ -212,35 +209,58 @@ public Pair<Option<Dataset<Row>>, String> fetchNextBatch(Option<String> lastCkpt
           .collect(Collectors.toMap(Pair::getLeft, Pair::getRight));
       reader = reader.options(optionsMap);
     }
+
+    boolean shouldFullScan =
+        missingCheckpointStrategy == MissingCheckpointStrategy.READ_UPTO_LATEST_COMMIT
+            && queryContext.getActiveTimeline()
+            .isBeforeTimelineStartsByCompletionTime(analyzer.getBeginCompletionTime().get());
     Dataset<Row> source;
-    // Do Incr pull. Set end instant if available
-    if (queryInfo.isIncremental()) {
-      source = reader
-          .options(readOpts)
-          .option(QUERY_TYPE().key(), QUERY_TYPE_INCREMENTAL_OPT_VAL())
-          .option(BEGIN_INSTANTTIME().key(), queryInfo.getStartInstant())
-          .option(END_INSTANTTIME().key(), queryInfo.getEndInstant())
-          .option(INCREMENTAL_FALLBACK_TO_FULL_TABLE_SCAN().key(),
-              props.getString(INCREMENTAL_FALLBACK_TO_FULL_TABLE_SCAN().key(),
-                  INCREMENTAL_FALLBACK_TO_FULL_TABLE_SCAN().defaultValue()))
-          .option(INCREMENTAL_READ_HANDLE_HOLLOW_COMMIT().key(), handlingMode.name())
-          .load(srcPath);
-    } else {
-      // if checkpoint is missing from source table, and if strategy is set to READ_UPTO_LATEST_COMMIT, we have to issue snapshot query
+    if (instantRange.isEmpty() || shouldFullScan) {
+      // snapshot query
       Dataset<Row> snapshot = reader
           .options(readOpts)
           .option(DataSourceReadOptions.QUERY_TYPE().key(), DataSourceReadOptions.QUERY_TYPE_SNAPSHOT_OPT_VAL())
           .load(srcPath);
+
+      Option<String> predicate = Option.empty();
+      List<String> instantTimeList = queryContext.getInstantTimeList();
       if (snapshotLoadQuerySplitter.isPresent()) {
-        queryInfo = snapshotLoadQuerySplitter.get().getNextCheckpoint(snapshot, queryInfo, sourceProfileSupplier);
+        Option<SnapshotLoadQuerySplitter.CheckpointWithPredicates> newCheckpointAndPredicate =
+            snapshotLoadQuerySplitter.get().getNextCheckpoint(snapshot, queryContext, sourceProfileSupplier);
+        if (newCheckpointAndPredicate.isPresent()) {
+          endCompletionTime = newCheckpointAndPredicate.get().endInstant;
+          predicate = Option.of(newCheckpointAndPredicate.get().predicateFilter);
+          instantTimeList = queryContext.getInstants().stream()
+              .filter(instant -> HoodieTimeline.compareTimestamps(
+                  instant.getCompletionTime(), HoodieTimeline.LESSER_THAN_OR_EQUALS, newCheckpointAndPredicate.get().endInstant))
+              .map(HoodieInstant::getTimestamp)
+              .collect(Collectors.toList());
+        } else {
+          endCompletionTime = queryContext.getEndInstant().orElse(queryContext.getLastInstant());

Review Comment:
   Good catch.  Fixed.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] [HUDI-8141] Incremental Query with Completion Time [hudi]

Posted by "CTTY (via GitHub)" <gi...@apache.org>.

CTTY commented on code in PR #11947:
URL: https://github.com/apache/hudi/pull/11947#discussion_r1797748275


##########
hudi-utilities/src/main/java/org/apache/hudi/utilities/sources/HoodieIncrSource.java:
##########
@@ -212,35 +257,43 @@ public Pair<Option<Dataset<Row>>, String> fetchNextBatch(Option<String> lastCkpt
           .collect(Collectors.toMap(Pair::getLeft, Pair::getRight));
       reader = reader.options(optionsMap);
     }
+    boolean shouldFullScan =
+        queryContext.getActiveTimeline().isBeforeTimelineStartsByCompletionTime(startTime)
+        && missingCheckpointStrategy == MissingCheckpointStrategy.READ_UPTO_LATEST_COMMIT;
     Dataset<Row> source;
-    // Do Incr pull. Set end instant if available
-    if (queryInfo.isIncremental()) {
-      source = reader
-          .options(readOpts)
-          .option(QUERY_TYPE().key(), QUERY_TYPE_INCREMENTAL_OPT_VAL())
-          .option(BEGIN_INSTANTTIME().key(), queryInfo.getStartInstant())
-          .option(END_INSTANTTIME().key(), queryInfo.getEndInstant())
-          .option(INCREMENTAL_FALLBACK_TO_FULL_TABLE_SCAN().key(),
-              props.getString(INCREMENTAL_FALLBACK_TO_FULL_TABLE_SCAN().key(),
-                  INCREMENTAL_FALLBACK_TO_FULL_TABLE_SCAN().defaultValue()))
-          .option(INCREMENTAL_READ_HANDLE_HOLLOW_COMMIT().key(), handlingMode.name())
-          .load(srcPath);
-    } else {
-      // if checkpoint is missing from source table, and if strategy is set to READ_UPTO_LATEST_COMMIT, we have to issue snapshot query
+    if (instantRange.isEmpty() || shouldFullScan) {
+      // snapshot query
       Dataset<Row> snapshot = reader
           .options(readOpts)
           .option(DataSourceReadOptions.QUERY_TYPE().key(), DataSourceReadOptions.QUERY_TYPE_SNAPSHOT_OPT_VAL())
           .load(srcPath);
       if (snapshotLoadQuerySplitter.isPresent()) {
-        queryInfo = snapshotLoadQuerySplitter.get().getNextCheckpoint(snapshot, queryInfo, sourceProfileSupplier);
+        queryContext = snapshotLoadQuerySplitter.get().getNextCheckpoint(snapshot, queryContext, sourceProfileSupplier);
+        // update endTime/next checkpoint
+        endTime = queryContext.getEndInstant().orElse(queryContext.getLastInstant());
       }
+      Set<String> validInstants = new HashSet<>(queryContext.getInstants());
+      snapshot = queryContext.getPredicateFilter().map(snapshot::filter).orElse(snapshot);

Review Comment:
   It's set in `IncrSourceHelper#withUpdatedCheckpoint`. This is hidden because this feature is  only available for `SnapshotLoadQuerySplitter`. 
   
   In the existing code, predicateFilter is set for `QueryInfo` in the same way here: https://github.com/apache/hudi/commit/7a242fec8d9d9f1fc1619a2b49d3ad931c10e35f#diff-8f93bd18cf9a608544e74b53ba4c379e0cd0ff4a763c28576ba2f5b651fc377bR87



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] [HUDI-8141] Incremental Query with Completion Time [hudi]

Posted by "CTTY (via GitHub)" <gi...@apache.org>.

CTTY commented on code in PR #11947:
URL: https://github.com/apache/hudi/pull/11947#discussion_r1797780363


##########
hudi-common/src/main/java/org/apache/hudi/common/table/timeline/HoodieDefaultTimeline.java:
##########
@@ -532,6 +532,13 @@ public boolean isBeforeTimelineStarts(String instant) {
         && compareTimestamps(instant, LESSER_THAN, firstNonSavepointCommit.get().getTimestamp());
   }
 
+  @Override
+  public boolean isBeforeTimelineStartsByCompletionTime(String completionTime) {
+    Option<HoodieInstant> firstNonSavepointCommit = getFirstNonSavepointCommit();

Review Comment:
   Good catch! Will fix this. I think adding a new variable `firstNonSavepointCommitByCompletionTime` in timeline would make sense so we can cache this result. And we can reuse the existing logic to get `firstNonSavepointCommit` but use `completionTime` instead when comparing instants



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] [HUDI-8141] Incremental Query with Completion Time [hudi]

Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.

hudi-bot commented on PR #11947:
URL: https://github.com/apache/hudi/pull/11947#issuecomment-2408794128

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "090f256a315cbbf0c6a54d93cc2739dd693360e6",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=484",
       "triggerID" : "090f256a315cbbf0c6a54d93cc2739dd693360e6",
       "triggerType" : "PUSH"
     }, {
       "hash" : "0df5649013c1cf9e782fcbb67b8a2e7174f94512",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=985",
       "triggerID" : "0df5649013c1cf9e782fcbb67b8a2e7174f94512",
       "triggerType" : "PUSH"
     }, {
       "hash" : "d52e0128f409918644255abbe4376dc4a780bfc8",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=986",
       "triggerID" : "d52e0128f409918644255abbe4376dc4a780bfc8",
       "triggerType" : "PUSH"
     }, {
       "hash" : "9e668f65c6d4f73c76573236f8fa507dfb80edac",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "9e668f65c6d4f73c76573236f8fa507dfb80edac",
       "triggerType" : "PUSH"
     }, {
       "hash" : "f145131d524bf190986e5de600a326e32a27beca",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=987",
       "triggerID" : "f145131d524bf190986e5de600a326e32a27beca",
       "triggerType" : "PUSH"
     }, {
       "hash" : "0b008f807b42aa6fb2686922bb88d63e84a1f299",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=993",
       "triggerID" : "0b008f807b42aa6fb2686922bb88d63e84a1f299",
       "triggerType" : "PUSH"
     }, {
       "hash" : "65c01ad9f2b33d36d4890355feef91844438e732",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=995",
       "triggerID" : "65c01ad9f2b33d36d4890355feef91844438e732",
       "triggerType" : "PUSH"
     }, {
       "hash" : "955493fdc93842e8c99529cec35e39cecdbe9638",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=996",
       "triggerID" : "955493fdc93842e8c99529cec35e39cecdbe9638",
       "triggerType" : "PUSH"
     }, {
       "hash" : "533f00e5546da0026bbe51e1c030ddd1c19fdd08",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "533f00e5546da0026bbe51e1c030ddd1c19fdd08",
       "triggerType" : "PUSH"
     }, {
       "hash" : "05777be67e37f3a6a8eaa59e54d9e17e5fc6ba9b",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1016",
       "triggerID" : "05777be67e37f3a6a8eaa59e54d9e17e5fc6ba9b",
       "triggerType" : "PUSH"
     }, {
       "hash" : "a2ee90c549332a7f17a7d9b2799849b77c08a8e6",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "a2ee90c549332a7f17a7d9b2799849b77c08a8e6",
       "triggerType" : "PUSH"
     }, {
       "hash" : "05ce568b0b6ba14da9ca40d26b39ba478837d747",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1017",
       "triggerID" : "05ce568b0b6ba14da9ca40d26b39ba478837d747",
       "triggerType" : "PUSH"
     }, {
       "hash" : "840e3b33d39b1f86367bb6b0e65eadeb33de7765",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1025",
       "triggerID" : "840e3b33d39b1f86367bb6b0e65eadeb33de7765",
       "triggerType" : "PUSH"
     }, {
       "hash" : "8d2e2dd83f51d03e890d424a51d3e0e84e904a43",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1031",
       "triggerID" : "8d2e2dd83f51d03e890d424a51d3e0e84e904a43",
       "triggerType" : "PUSH"
     }, {
       "hash" : "688381914087ff11acf2f8c9372dbf39afa0bd40",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1035",
       "triggerID" : "688381914087ff11acf2f8c9372dbf39afa0bd40",
       "triggerType" : "PUSH"
     }, {
       "hash" : "fc2c138901e82ed5f3fbf363c2a0ce8140ade050",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "fc2c138901e82ed5f3fbf363c2a0ce8140ade050",
       "triggerType" : "PUSH"
     }, {
       "hash" : "6ed9e1cb1e5a191752d6d39c7a9d5d41840f0cdf",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1039",
       "triggerID" : "6ed9e1cb1e5a191752d6d39c7a9d5d41840f0cdf",
       "triggerType" : "PUSH"
     }, {
       "hash" : "a510a7de1458bb996df5bbb341e8d010e4c91381",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "a510a7de1458bb996df5bbb341e8d010e4c91381",
       "triggerType" : "PUSH"
     }, {
       "hash" : "49677397cd117228742205944606f7a789090376",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1042",
       "triggerID" : "49677397cd117228742205944606f7a789090376",
       "triggerType" : "PUSH"
     }, {
       "hash" : "1630db0a37422aed942ff280560a9d6329df5015",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1044",
       "triggerID" : "1630db0a37422aed942ff280560a9d6329df5015",
       "triggerType" : "PUSH"
     }, {
       "hash" : "f11c76301d4ab524db0af57aa831be3a898ea0fb",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1048",
       "triggerID" : "f11c76301d4ab524db0af57aa831be3a898ea0fb",
       "triggerType" : "PUSH"
     }, {
       "hash" : "d1a0dc4c2017853a7ba0c3302f886043a63f29da",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1050",
       "triggerID" : "d1a0dc4c2017853a7ba0c3302f886043a63f29da",
       "triggerType" : "PUSH"
     }, {
       "hash" : "6a4c15ec39ded1f90b113b272554217698a98e1c",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1087",
       "triggerID" : "6a4c15ec39ded1f90b113b272554217698a98e1c",
       "triggerType" : "PUSH"
     }, {
       "hash" : "288e730d645a5e58e50a87ef5f44ba11a8fb0533",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1092",
       "triggerID" : "288e730d645a5e58e50a87ef5f44ba11a8fb0533",
       "triggerType" : "PUSH"
     }, {
       "hash" : "70a5c21321a0da3a0be8d85e206c24befa7b89f1",
       "status" : "CANCELED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1095",
       "triggerID" : "70a5c21321a0da3a0be8d85e206c24befa7b89f1",
       "triggerType" : "PUSH"
     }, {
       "hash" : "f072a986114949c5e426ce64a5f5568fc718e25d",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "f072a986114949c5e426ce64a5f5568fc718e25d",
       "triggerType" : "PUSH"
     }, {
       "hash" : "45b27475b1e8806ec8c566d6f151f4bc79c8549d",
       "status" : "PENDING",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1096",
       "triggerID" : "45b27475b1e8806ec8c566d6f151f4bc79c8549d",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 9e668f65c6d4f73c76573236f8fa507dfb80edac UNKNOWN
   * 533f00e5546da0026bbe51e1c030ddd1c19fdd08 UNKNOWN
   * a2ee90c549332a7f17a7d9b2799849b77c08a8e6 UNKNOWN
   * fc2c138901e82ed5f3fbf363c2a0ce8140ade050 UNKNOWN
   * a510a7de1458bb996df5bbb341e8d010e4c91381 UNKNOWN
   * 70a5c21321a0da3a0be8d85e206c24befa7b89f1 Azure: [CANCELED](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1095) 
   * f072a986114949c5e426ce64a5f5568fc718e25d UNKNOWN
   * 45b27475b1e8806ec8c566d6f151f4bc79c8549d Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1096) 
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] [HUDI-8141] Incremental Query with Completion Time [hudi]

Posted by "yihua (via GitHub)" <gi...@apache.org>.

yihua commented on code in PR #11947:
URL: https://github.com/apache/hudi/pull/11947#discussion_r1797785933


##########
hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/hudi/MergeOnReadIncrementalRelation.scala:
##########
@@ -94,9 +94,10 @@ case class MergeOnReadIncrementalRelation(override val sqlContext: SQLContext,
       tableState = tableState,
       mergeType = mergeType,
       fileSplits = fileSplits,
-      includeStartTime = includeStartTime,
+      includeStartTime = false,
       startTimestamp = startTs,
-      endTimestamp = endTs)
+      endTimestamp = endTs,
+      includedTimestamps = includedCommits.map(_.getTimestamp).toSet)

Review Comment:
   Similar here, clearly define what timestamps are passed in here.



##########
hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/hudi/MergeOnReadIncrementalRelation.scala:
##########
@@ -185,13 +186,11 @@ trait HoodieIncrementalRelationTrait extends HoodieBaseRelation {
   protected def startTimestamp: String = optParams(DataSourceReadOptions.BEGIN_INSTANTTIME.key)
 
   protected def endTimestamp: String = optParams.getOrElse(
-    DataSourceReadOptions.END_INSTANTTIME.key,
-    if (hollowCommitHandling == USE_TRANSITION_TIME) super.timeline.lastInstant().get.getCompletionTime
-    else super.timeline.lastInstant().get.getTimestamp)
+    DataSourceReadOptions.END_INSTANTTIME.key, super.timeline.lastInstant().get.getCompletionTime)

Review Comment:
   `lastInstant()` might not be the max completion time on the timeline.



##########
hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/hudi/MergeOnReadIncrementalRelation.scala:
##########
@@ -185,13 +186,11 @@ trait HoodieIncrementalRelationTrait extends HoodieBaseRelation {
   protected def startTimestamp: String = optParams(DataSourceReadOptions.BEGIN_INSTANTTIME.key)
 
   protected def endTimestamp: String = optParams.getOrElse(
-    DataSourceReadOptions.END_INSTANTTIME.key,
-    if (hollowCommitHandling == USE_TRANSITION_TIME) super.timeline.lastInstant().get.getCompletionTime
-    else super.timeline.lastInstant().get.getTimestamp)
+    DataSourceReadOptions.END_INSTANTTIME.key, super.timeline.lastInstant().get.getCompletionTime)
 
-  protected def startInstantArchived: Boolean = super.timeline.isBeforeTimelineStarts(startTimestamp)
+  protected def startInstantArchived: Boolean = super.timeline.isBeforeTimelineStartsByCompletionTime(startTimestamp)
 
-  protected def endInstantArchived: Boolean = super.timeline.isBeforeTimelineStarts(endTimestamp)
+  protected def endInstantArchived: Boolean = super.timeline.isBeforeTimelineStartsByCompletionTime(endTimestamp)

Review Comment:
   Same here.  These are no longer needed if query context splits the instants into active and archived.



##########
hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/hudi/MergeOnReadIncrementalRelation.scala:
##########
@@ -94,9 +94,10 @@ case class MergeOnReadIncrementalRelation(override val sqlContext: SQLContext,
       tableState = tableState,
       mergeType = mergeType,
       fileSplits = fileSplits,
-      includeStartTime = includeStartTime,
+      includeStartTime = false,

Review Comment:
   If `includeStartTime` is not used, should we get rid of it?



##########
hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/hudi/MergeOnReadIncrementalRelation.scala:
##########
@@ -205,51 +204,46 @@ trait HoodieIncrementalRelationTrait extends HoodieBaseRelation {
       || affectedFilesInCommits.asScala.exists(fileStatus => !metaClient.getStorage.exists(fileStatus.getPath)))
   }
 
-  protected lazy val includedCommits: immutable.Seq[HoodieInstant] = {
-    if (!startInstantArchived || !endInstantArchived) {
-      // If endTimestamp commit is not archived, will filter instants
-      // before endTimestamp.
-      if (hollowCommitHandling == USE_TRANSITION_TIME) {
-        super.timeline.findInstantsInRangeByCompletionTime(startTimestamp, endTimestamp).getInstants.asScala.toList
-      } else {
-        super.timeline.findInstantsInRange(startTimestamp, endTimestamp).getInstants.asScala.toList
-      }
-    } else {
-      super.timeline.getInstants.asScala.toList
-    }
-  }
+  protected lazy val queryContext: IncrementalQueryAnalyzer.QueryContext =
+    IncrementalQueryAnalyzer.builder()
+      .metaClient(metaClient)
+      .startTime(optParams(DataSourceReadOptions.BEGIN_INSTANTTIME.key))
+      .endTime(optParams.getOrElse(DataSourceReadOptions.END_INSTANTTIME.key, null))
+      .rangeType(InstantRange.RangeType.OPEN_CLOSED)
+      .limit(optParams.getOrElse(
+        DataSourceReadOptions.INCREMENTAL_LIMIT.key,
+        DataSourceReadOptions.INCREMENTAL_LIMIT.defaultValue).toInt)
+      .build()
+      .analyze()
+
+  protected lazy val includedCommits: immutable.Seq[HoodieInstant] = List.concat(
+    queryContext.getArchivedInstants.asScala,
+    queryContext.getActiveInstants.asScala)
 
   protected lazy val commitsMetadata = includedCommits.map(getCommitMetadata(_, super.timeline)).asJava
 
   protected lazy val affectedFilesInCommits: java.util.List[StoragePathInfo] = {
     listAffectedFilesForCommits(conf, metaClient.getBasePath, commitsMetadata)
   }
 
-  protected lazy val (includeStartTime, startTs) = if (startInstantArchived) {
-    (false, startTimestamp)

Review Comment:
   Do you know why `startInstantArchived` determines `includeStartTime`?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] [HUDI-8141] Incremental Query with Completion Time [hudi]

Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.

hudi-bot commented on PR #11947:
URL: https://github.com/apache/hudi/pull/11947#issuecomment-2408776420

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "090f256a315cbbf0c6a54d93cc2739dd693360e6",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=484",
       "triggerID" : "090f256a315cbbf0c6a54d93cc2739dd693360e6",
       "triggerType" : "PUSH"
     }, {
       "hash" : "0df5649013c1cf9e782fcbb67b8a2e7174f94512",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=985",
       "triggerID" : "0df5649013c1cf9e782fcbb67b8a2e7174f94512",
       "triggerType" : "PUSH"
     }, {
       "hash" : "d52e0128f409918644255abbe4376dc4a780bfc8",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=986",
       "triggerID" : "d52e0128f409918644255abbe4376dc4a780bfc8",
       "triggerType" : "PUSH"
     }, {
       "hash" : "9e668f65c6d4f73c76573236f8fa507dfb80edac",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "9e668f65c6d4f73c76573236f8fa507dfb80edac",
       "triggerType" : "PUSH"
     }, {
       "hash" : "f145131d524bf190986e5de600a326e32a27beca",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=987",
       "triggerID" : "f145131d524bf190986e5de600a326e32a27beca",
       "triggerType" : "PUSH"
     }, {
       "hash" : "0b008f807b42aa6fb2686922bb88d63e84a1f299",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=993",
       "triggerID" : "0b008f807b42aa6fb2686922bb88d63e84a1f299",
       "triggerType" : "PUSH"
     }, {
       "hash" : "65c01ad9f2b33d36d4890355feef91844438e732",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=995",
       "triggerID" : "65c01ad9f2b33d36d4890355feef91844438e732",
       "triggerType" : "PUSH"
     }, {
       "hash" : "955493fdc93842e8c99529cec35e39cecdbe9638",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=996",
       "triggerID" : "955493fdc93842e8c99529cec35e39cecdbe9638",
       "triggerType" : "PUSH"
     }, {
       "hash" : "533f00e5546da0026bbe51e1c030ddd1c19fdd08",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "533f00e5546da0026bbe51e1c030ddd1c19fdd08",
       "triggerType" : "PUSH"
     }, {
       "hash" : "05777be67e37f3a6a8eaa59e54d9e17e5fc6ba9b",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1016",
       "triggerID" : "05777be67e37f3a6a8eaa59e54d9e17e5fc6ba9b",
       "triggerType" : "PUSH"
     }, {
       "hash" : "a2ee90c549332a7f17a7d9b2799849b77c08a8e6",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "a2ee90c549332a7f17a7d9b2799849b77c08a8e6",
       "triggerType" : "PUSH"
     }, {
       "hash" : "05ce568b0b6ba14da9ca40d26b39ba478837d747",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1017",
       "triggerID" : "05ce568b0b6ba14da9ca40d26b39ba478837d747",
       "triggerType" : "PUSH"
     }, {
       "hash" : "840e3b33d39b1f86367bb6b0e65eadeb33de7765",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1025",
       "triggerID" : "840e3b33d39b1f86367bb6b0e65eadeb33de7765",
       "triggerType" : "PUSH"
     }, {
       "hash" : "8d2e2dd83f51d03e890d424a51d3e0e84e904a43",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1031",
       "triggerID" : "8d2e2dd83f51d03e890d424a51d3e0e84e904a43",
       "triggerType" : "PUSH"
     }, {
       "hash" : "688381914087ff11acf2f8c9372dbf39afa0bd40",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1035",
       "triggerID" : "688381914087ff11acf2f8c9372dbf39afa0bd40",
       "triggerType" : "PUSH"
     }, {
       "hash" : "fc2c138901e82ed5f3fbf363c2a0ce8140ade050",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "fc2c138901e82ed5f3fbf363c2a0ce8140ade050",
       "triggerType" : "PUSH"
     }, {
       "hash" : "6ed9e1cb1e5a191752d6d39c7a9d5d41840f0cdf",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1039",
       "triggerID" : "6ed9e1cb1e5a191752d6d39c7a9d5d41840f0cdf",
       "triggerType" : "PUSH"
     }, {
       "hash" : "a510a7de1458bb996df5bbb341e8d010e4c91381",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "a510a7de1458bb996df5bbb341e8d010e4c91381",
       "triggerType" : "PUSH"
     }, {
       "hash" : "49677397cd117228742205944606f7a789090376",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1042",
       "triggerID" : "49677397cd117228742205944606f7a789090376",
       "triggerType" : "PUSH"
     }, {
       "hash" : "1630db0a37422aed942ff280560a9d6329df5015",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1044",
       "triggerID" : "1630db0a37422aed942ff280560a9d6329df5015",
       "triggerType" : "PUSH"
     }, {
       "hash" : "f11c76301d4ab524db0af57aa831be3a898ea0fb",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1048",
       "triggerID" : "f11c76301d4ab524db0af57aa831be3a898ea0fb",
       "triggerType" : "PUSH"
     }, {
       "hash" : "d1a0dc4c2017853a7ba0c3302f886043a63f29da",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1050",
       "triggerID" : "d1a0dc4c2017853a7ba0c3302f886043a63f29da",
       "triggerType" : "PUSH"
     }, {
       "hash" : "6a4c15ec39ded1f90b113b272554217698a98e1c",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1087",
       "triggerID" : "6a4c15ec39ded1f90b113b272554217698a98e1c",
       "triggerType" : "PUSH"
     }, {
       "hash" : "288e730d645a5e58e50a87ef5f44ba11a8fb0533",
       "status" : "FAILURE",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1092",
       "triggerID" : "288e730d645a5e58e50a87ef5f44ba11a8fb0533",
       "triggerType" : "PUSH"
     }, {
       "hash" : "70a5c21321a0da3a0be8d85e206c24befa7b89f1",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "70a5c21321a0da3a0be8d85e206c24befa7b89f1",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 9e668f65c6d4f73c76573236f8fa507dfb80edac UNKNOWN
   * 533f00e5546da0026bbe51e1c030ddd1c19fdd08 UNKNOWN
   * a2ee90c549332a7f17a7d9b2799849b77c08a8e6 UNKNOWN
   * fc2c138901e82ed5f3fbf363c2a0ce8140ade050 UNKNOWN
   * a510a7de1458bb996df5bbb341e8d010e4c91381 UNKNOWN
   * 288e730d645a5e58e50a87ef5f44ba11a8fb0533 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1092) 
   * 70a5c21321a0da3a0be8d85e206c24befa7b89f1 UNKNOWN
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] [HUDI-8141] Incremental Query with Completion Time [hudi]

Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.

hudi-bot commented on PR #11947:
URL: https://github.com/apache/hudi/pull/11947#issuecomment-2408850473

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "090f256a315cbbf0c6a54d93cc2739dd693360e6",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=484",
       "triggerID" : "090f256a315cbbf0c6a54d93cc2739dd693360e6",
       "triggerType" : "PUSH"
     }, {
       "hash" : "0df5649013c1cf9e782fcbb67b8a2e7174f94512",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=985",
       "triggerID" : "0df5649013c1cf9e782fcbb67b8a2e7174f94512",
       "triggerType" : "PUSH"
     }, {
       "hash" : "d52e0128f409918644255abbe4376dc4a780bfc8",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=986",
       "triggerID" : "d52e0128f409918644255abbe4376dc4a780bfc8",
       "triggerType" : "PUSH"
     }, {
       "hash" : "9e668f65c6d4f73c76573236f8fa507dfb80edac",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "9e668f65c6d4f73c76573236f8fa507dfb80edac",
       "triggerType" : "PUSH"
     }, {
       "hash" : "f145131d524bf190986e5de600a326e32a27beca",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=987",
       "triggerID" : "f145131d524bf190986e5de600a326e32a27beca",
       "triggerType" : "PUSH"
     }, {
       "hash" : "0b008f807b42aa6fb2686922bb88d63e84a1f299",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=993",
       "triggerID" : "0b008f807b42aa6fb2686922bb88d63e84a1f299",
       "triggerType" : "PUSH"
     }, {
       "hash" : "65c01ad9f2b33d36d4890355feef91844438e732",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=995",
       "triggerID" : "65c01ad9f2b33d36d4890355feef91844438e732",
       "triggerType" : "PUSH"
     }, {
       "hash" : "955493fdc93842e8c99529cec35e39cecdbe9638",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=996",
       "triggerID" : "955493fdc93842e8c99529cec35e39cecdbe9638",
       "triggerType" : "PUSH"
     }, {
       "hash" : "533f00e5546da0026bbe51e1c030ddd1c19fdd08",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "533f00e5546da0026bbe51e1c030ddd1c19fdd08",
       "triggerType" : "PUSH"
     }, {
       "hash" : "05777be67e37f3a6a8eaa59e54d9e17e5fc6ba9b",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1016",
       "triggerID" : "05777be67e37f3a6a8eaa59e54d9e17e5fc6ba9b",
       "triggerType" : "PUSH"
     }, {
       "hash" : "a2ee90c549332a7f17a7d9b2799849b77c08a8e6",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "a2ee90c549332a7f17a7d9b2799849b77c08a8e6",
       "triggerType" : "PUSH"
     }, {
       "hash" : "05ce568b0b6ba14da9ca40d26b39ba478837d747",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1017",
       "triggerID" : "05ce568b0b6ba14da9ca40d26b39ba478837d747",
       "triggerType" : "PUSH"
     }, {
       "hash" : "840e3b33d39b1f86367bb6b0e65eadeb33de7765",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1025",
       "triggerID" : "840e3b33d39b1f86367bb6b0e65eadeb33de7765",
       "triggerType" : "PUSH"
     }, {
       "hash" : "8d2e2dd83f51d03e890d424a51d3e0e84e904a43",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1031",
       "triggerID" : "8d2e2dd83f51d03e890d424a51d3e0e84e904a43",
       "triggerType" : "PUSH"
     }, {
       "hash" : "688381914087ff11acf2f8c9372dbf39afa0bd40",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1035",
       "triggerID" : "688381914087ff11acf2f8c9372dbf39afa0bd40",
       "triggerType" : "PUSH"
     }, {
       "hash" : "fc2c138901e82ed5f3fbf363c2a0ce8140ade050",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "fc2c138901e82ed5f3fbf363c2a0ce8140ade050",
       "triggerType" : "PUSH"
     }, {
       "hash" : "6ed9e1cb1e5a191752d6d39c7a9d5d41840f0cdf",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1039",
       "triggerID" : "6ed9e1cb1e5a191752d6d39c7a9d5d41840f0cdf",
       "triggerType" : "PUSH"
     }, {
       "hash" : "a510a7de1458bb996df5bbb341e8d010e4c91381",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "a510a7de1458bb996df5bbb341e8d010e4c91381",
       "triggerType" : "PUSH"
     }, {
       "hash" : "49677397cd117228742205944606f7a789090376",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1042",
       "triggerID" : "49677397cd117228742205944606f7a789090376",
       "triggerType" : "PUSH"
     }, {
       "hash" : "1630db0a37422aed942ff280560a9d6329df5015",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1044",
       "triggerID" : "1630db0a37422aed942ff280560a9d6329df5015",
       "triggerType" : "PUSH"
     }, {
       "hash" : "f11c76301d4ab524db0af57aa831be3a898ea0fb",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1048",
       "triggerID" : "f11c76301d4ab524db0af57aa831be3a898ea0fb",
       "triggerType" : "PUSH"
     }, {
       "hash" : "d1a0dc4c2017853a7ba0c3302f886043a63f29da",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1050",
       "triggerID" : "d1a0dc4c2017853a7ba0c3302f886043a63f29da",
       "triggerType" : "PUSH"
     }, {
       "hash" : "6a4c15ec39ded1f90b113b272554217698a98e1c",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1087",
       "triggerID" : "6a4c15ec39ded1f90b113b272554217698a98e1c",
       "triggerType" : "PUSH"
     }, {
       "hash" : "288e730d645a5e58e50a87ef5f44ba11a8fb0533",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1092",
       "triggerID" : "288e730d645a5e58e50a87ef5f44ba11a8fb0533",
       "triggerType" : "PUSH"
     }, {
       "hash" : "70a5c21321a0da3a0be8d85e206c24befa7b89f1",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1095",
       "triggerID" : "70a5c21321a0da3a0be8d85e206c24befa7b89f1",
       "triggerType" : "PUSH"
     }, {
       "hash" : "f072a986114949c5e426ce64a5f5568fc718e25d",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "f072a986114949c5e426ce64a5f5568fc718e25d",
       "triggerType" : "PUSH"
     }, {
       "hash" : "45b27475b1e8806ec8c566d6f151f4bc79c8549d",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1096",
       "triggerID" : "45b27475b1e8806ec8c566d6f151f4bc79c8549d",
       "triggerType" : "PUSH"
     }, {
       "hash" : "3c62ef22f6cde28110e5eb2fca8760a008d77723",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "3c62ef22f6cde28110e5eb2fca8760a008d77723",
       "triggerType" : "PUSH"
     }, {
       "hash" : "dc785fa04e542b2ccc45494e0292f5153c7ecea8",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1097",
       "triggerID" : "dc785fa04e542b2ccc45494e0292f5153c7ecea8",
       "triggerType" : "PUSH"
     }, {
       "hash" : "585ddb8049b9b9ccb057027e265b08a1b586df8c",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "585ddb8049b9b9ccb057027e265b08a1b586df8c",
       "triggerType" : "PUSH"
     }, {
       "hash" : "b62ff8675beed2c4487c788b9ca1a0ad4271b913",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "b62ff8675beed2c4487c788b9ca1a0ad4271b913",
       "triggerType" : "PUSH"
     }, {
       "hash" : "2d46048a7402907f348cc96da2839ba1b3e4932a",
       "status" : "CANCELED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1099",
       "triggerID" : "2d46048a7402907f348cc96da2839ba1b3e4932a",
       "triggerType" : "PUSH"
     }, {
       "hash" : "004fb6ebf23f796a4b77650e0530f0f1145e8b91",
       "status" : "PENDING",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1101",
       "triggerID" : "004fb6ebf23f796a4b77650e0530f0f1145e8b91",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 9e668f65c6d4f73c76573236f8fa507dfb80edac UNKNOWN
   * 533f00e5546da0026bbe51e1c030ddd1c19fdd08 UNKNOWN
   * a2ee90c549332a7f17a7d9b2799849b77c08a8e6 UNKNOWN
   * fc2c138901e82ed5f3fbf363c2a0ce8140ade050 UNKNOWN
   * a510a7de1458bb996df5bbb341e8d010e4c91381 UNKNOWN
   * f072a986114949c5e426ce64a5f5568fc718e25d UNKNOWN
   * 3c62ef22f6cde28110e5eb2fca8760a008d77723 UNKNOWN
   * 585ddb8049b9b9ccb057027e265b08a1b586df8c UNKNOWN
   * b62ff8675beed2c4487c788b9ca1a0ad4271b913 UNKNOWN
   * 2d46048a7402907f348cc96da2839ba1b3e4932a Azure: [CANCELED](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1099) 
   * 004fb6ebf23f796a4b77650e0530f0f1145e8b91 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1101) 
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] [HUDI-8141] Incremental Query with Completion Time [hudi]

Posted by "CTTY (via GitHub)" <gi...@apache.org>.

CTTY commented on code in PR #11947:
URL: https://github.com/apache/hudi/pull/11947#discussion_r1798106048


##########
hudi-common/src/main/java/org/apache/hudi/common/table/timeline/TimelineUtils.java:
##########
@@ -308,6 +309,38 @@ public static HoodieTimeline getCommitsTimelineAfter(
     return timelineSinceLastSync;
   }
 
+  public static HoodieTimeline getCommitsTimeLineAfterByCompletionTimeRange(

Review Comment:
   Fixed in this PR, and filed JIRA to track this issue for Hudi 0.x: https://issues.apache.org/jira/browse/HUDI-8357



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] [HUDI-8141] Incremental Query with Completion Time [hudi]

Posted by "danny0405 (via GitHub)" <gi...@apache.org>.

danny0405 commented on code in PR #11947:
URL: https://github.com/apache/hudi/pull/11947#discussion_r1798653208


##########
hudi-common/src/main/java/org/apache/hudi/common/table/timeline/CompletionTimeQueryView.java:
##########
@@ -189,46 +193,78 @@ public Option<String> getCompletionTime(String beginTime) {
    * <p>By default, assumes there is at most 1 day time of duration for an instant to accelerate the queries.
    *
    * @param timeline The timeline.
-   * @param rangeStart   The query range start completion time.
-   * @param rangeEnd     The query range end completion time.
+   * @param rangeStartCompletionTime   The query range start completion time.
+   * @param rangeEndCompletionTime     The query range end completion time.
    * @param rangeType    The range type.
    *
    * @return The sorted instant time list.
    */
   public List<String> getStartTimes(
       HoodieTimeline timeline,
-      Option<String> rangeStart,
-      Option<String> rangeEnd,
+      Option<String> rangeStartCompletionTime,
+      Option<String> rangeEndCompletionTime,
       InstantRange.RangeType rangeType) {
     // assumes any instant/transaction lasts at most 1 day to optimize the query efficiency.
-    return getStartTimes(timeline, rangeStart, rangeEnd, rangeType, s -> HoodieInstantTimeGenerator.instantTimeMinusMillis(s, MILLI_SECONDS_IN_ONE_DAY));
+    return getStartTimes(
+        timeline,
+        rangeStartCompletionTime,
+        rangeEndCompletionTime,
+        rangeType,
+        GET_INSTANT_ONE_DAY_BEFORE);
   }
 
   /**
    * Queries the instant start time with given completion time range.
    *
-   * @param rangeStart              The query range start completion time.
-   * @param rangeEnd                The query range end completion time.
+   * @param rangeStartCompletionTime              The query range start completion time.
+   * @param rangeEndCompletionTime                The query range end completion time.
    * @param earliestInstantTimeFunc The function to generate the earliest start time boundary
    *                                with the minimum completion time.
    *
    * @return The sorted instant time list.
    */
   @VisibleForTesting
   public List<String> getStartTimes(
-      String rangeStart,
-      String rangeEnd,
+      String rangeStartCompletionTime,
+      String rangeEndCompletionTime,

Review Comment:
   ditto



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] [HUDI-8141] Incremental Query with Completion Time [hudi]

Posted by "yihua (via GitHub)" <gi...@apache.org>.

yihua commented on code in PR #11947:
URL: https://github.com/apache/hudi/pull/11947#discussion_r1802141309


##########
hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/spark/sql/hudi/streaming/HoodieStreamSource.scala:
##########
@@ -71,19 +69,6 @@ class HoodieStreamSource(
     parameters.get(DataSourceReadOptions.QUERY_TYPE.key).contains(DataSourceReadOptions.QUERY_TYPE_INCREMENTAL_OPT_VAL) &&
     parameters.get(DataSourceReadOptions.INCREMENTAL_FORMAT.key).contains(DataSourceReadOptions.INCREMENTAL_FORMAT_CDC_VAL)
 
-  /**
-   * When hollow commits are found while doing streaming read , unlike batch incremental query,
-   * we do not use [[HollowCommitHandling.FAIL]] by default, instead we use [[HollowCommitHandling.BLOCK]]
-   * to block processing data from going beyond the hollow commits to avoid unintentional skip.
-   *
-   * Users can set [[DataSourceReadOptions.INCREMENTAL_READ_HANDLE_HOLLOW_COMMIT]] to
-   * [[HollowCommitHandling.USE_TRANSITION_TIME]] to avoid the blocking behavior.
-   */
-  private val hollowCommitHandling: HollowCommitHandling =

Review Comment:
   We still need to handle hollow commits if the source table is version 6 or below.  So keeping this for now and let HUDI-8359 fix it properly.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] [HUDI-8141] Incremental Query with Completion Time [hudi]

Posted by "yihua (via GitHub)" <gi...@apache.org>.

yihua commented on code in PR #11947:
URL: https://github.com/apache/hudi/pull/11947#discussion_r1802062995


##########
hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/hudi/IncrementalRelation.scala:
##########
@@ -90,24 +78,24 @@ class IncrementalRelation(val sqlContext: SQLContext,
     throw new HoodieException("Incremental queries are not supported when meta fields are disabled")
   }
 
+  private val queryContext: IncrementalQueryAnalyzer.QueryContext =
+    IncrementalQueryAnalyzer.builder()
+      .metaClient(metaClient)
+      .startTime(optParams(DataSourceReadOptions.BEGIN_INSTANTTIME.key))
+      .endTime(optParams.getOrElse(DataSourceReadOptions.END_INSTANTTIME.key, null))
+      .rangeType(InstantRange.RangeType.OPEN_CLOSED)
+      .build()
+      .analyze()
+
+  private val commitsToReturn = List.concat(
+    queryContext.getArchivedInstants.asScala,
+    queryContext.getActiveInstants.asScala)
+  // TODO: investigate failure: TestHoodieIncrSource#testHoodieIncrSourceInflightCommitBeforeCompletedCommit
+  //    .sortBy(_.getCompletionTime)  // why would sorting by completion time fail this test?

Review Comment:
   This needs investigation.



##########
hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/hudi/MergeOnReadIncrementalRelation.scala:
##########
@@ -205,51 +204,46 @@ trait HoodieIncrementalRelationTrait extends HoodieBaseRelation {
       || affectedFilesInCommits.asScala.exists(fileStatus => !metaClient.getStorage.exists(fileStatus.getPath)))
   }
 
-  protected lazy val includedCommits: immutable.Seq[HoodieInstant] = {
-    if (!startInstantArchived || !endInstantArchived) {
-      // If endTimestamp commit is not archived, will filter instants
-      // before endTimestamp.
-      if (hollowCommitHandling == USE_TRANSITION_TIME) {
-        super.timeline.findInstantsInRangeByCompletionTime(startTimestamp, endTimestamp).getInstants.asScala.toList
-      } else {
-        super.timeline.findInstantsInRange(startTimestamp, endTimestamp).getInstants.asScala.toList
-      }
-    } else {
-      super.timeline.getInstants.asScala.toList
-    }
-  }
+  protected lazy val queryContext: IncrementalQueryAnalyzer.QueryContext =
+    IncrementalQueryAnalyzer.builder()
+      .metaClient(metaClient)
+      .startTime(optParams(DataSourceReadOptions.BEGIN_INSTANTTIME.key))
+      .endTime(optParams.getOrElse(DataSourceReadOptions.END_INSTANTTIME.key, null))
+      .rangeType(InstantRange.RangeType.OPEN_CLOSED)
+      .limit(optParams.getOrElse(
+        DataSourceReadOptions.INCREMENTAL_LIMIT.key,
+        DataSourceReadOptions.INCREMENTAL_LIMIT.defaultValue).toInt)
+      .build()
+      .analyze()
+
+  protected lazy val includedCommits: immutable.Seq[HoodieInstant] = List.concat(
+    queryContext.getArchivedInstants.asScala,
+    queryContext.getActiveInstants.asScala)
 
   protected lazy val commitsMetadata = includedCommits.map(getCommitMetadata(_, super.timeline)).asJava
 
   protected lazy val affectedFilesInCommits: java.util.List[StoragePathInfo] = {
     listAffectedFilesForCommits(conf, metaClient.getBasePath, commitsMetadata)
   }
 
-  protected lazy val (includeStartTime, startTs) = if (startInstantArchived) {
-    (false, startTimestamp)

Review Comment:
   Got it.  I think this part of code was hard to understand so it's good that we clean it up and make the logic unified without unnecessary flags.



##########
hudi-common/src/test/java/org/apache/hudi/common/testutils/HoodieTestDataGenerator.java:
##########
@@ -681,6 +681,10 @@ public List<HoodieRecord> generateInserts(String instantTime, Integer n) {
     return generateInserts(instantTime, n, false);
   }
 
+  public List<HoodieRecord> generateInserts(Long instantTime, Integer n) {

Review Comment:
   Nit: to remove this.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] [HUDI-8141] Incremental Query with Completion Time [hudi]

Posted by "yihua (via GitHub)" <gi...@apache.org>.

yihua commented on code in PR #11947:
URL: https://github.com/apache/hudi/pull/11947#discussion_r1802125468


##########
hudi-common/src/main/java/org/apache/hudi/common/table/timeline/HoodieDefaultTimeline.java:
##########
@@ -629,16 +649,23 @@ private Set<String> getOrCreatePendingClusteringInstantSet() {
   /**
    * Returns the first non savepoint commit on the timeline.
    */
-  private static Option<HoodieInstant> findFirstNonSavepointCommit(List<HoodieInstant> instants) {
+  private static Option<HoodieInstant> findFirstNonSavepointCommit(
+      List<HoodieInstant> instants,
+      boolean byCompletionTime) {
     Set<String> savepointTimestamps = instants.stream()
         .filter(entry -> entry.getAction().equals(HoodieTimeline.SAVEPOINT_ACTION))
-        .map(HoodieInstant::getTimestamp)
+        .map(byCompletionTime
+            ? HoodieInstant::getCompletionTime
+            : HoodieInstant::getTimestamp)
         .collect(Collectors.toSet());
     if (!savepointTimestamps.isEmpty()) {
       // There are chances that there could be holes in the timeline due to archival and savepoint interplay.
       // So, the first non-savepoint commit is considered as beginning of the active timeline.
       return Option.fromJavaOptional(instants.stream()
-          .filter(entry -> !savepointTimestamps.contains(entry.getTimestamp()))
+          .filter(entry -> !savepointTimestamps.contains(

Review Comment:
   Fixed.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] [HUDI-8141] Incremental Query with Completion Time [hudi]

Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.

hudi-bot commented on PR #11947:
URL: https://github.com/apache/hudi/pull/11947#issuecomment-2421522724

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "090f256a315cbbf0c6a54d93cc2739dd693360e6",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=484",
       "triggerID" : "090f256a315cbbf0c6a54d93cc2739dd693360e6",
       "triggerType" : "PUSH"
     }, {
       "hash" : "0df5649013c1cf9e782fcbb67b8a2e7174f94512",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=985",
       "triggerID" : "0df5649013c1cf9e782fcbb67b8a2e7174f94512",
       "triggerType" : "PUSH"
     }, {
       "hash" : "d52e0128f409918644255abbe4376dc4a780bfc8",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=986",
       "triggerID" : "d52e0128f409918644255abbe4376dc4a780bfc8",
       "triggerType" : "PUSH"
     }, {
       "hash" : "9e668f65c6d4f73c76573236f8fa507dfb80edac",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "9e668f65c6d4f73c76573236f8fa507dfb80edac",
       "triggerType" : "PUSH"
     }, {
       "hash" : "f145131d524bf190986e5de600a326e32a27beca",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=987",
       "triggerID" : "f145131d524bf190986e5de600a326e32a27beca",
       "triggerType" : "PUSH"
     }, {
       "hash" : "0b008f807b42aa6fb2686922bb88d63e84a1f299",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=993",
       "triggerID" : "0b008f807b42aa6fb2686922bb88d63e84a1f299",
       "triggerType" : "PUSH"
     }, {
       "hash" : "65c01ad9f2b33d36d4890355feef91844438e732",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=995",
       "triggerID" : "65c01ad9f2b33d36d4890355feef91844438e732",
       "triggerType" : "PUSH"
     }, {
       "hash" : "955493fdc93842e8c99529cec35e39cecdbe9638",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=996",
       "triggerID" : "955493fdc93842e8c99529cec35e39cecdbe9638",
       "triggerType" : "PUSH"
     }, {
       "hash" : "533f00e5546da0026bbe51e1c030ddd1c19fdd08",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "533f00e5546da0026bbe51e1c030ddd1c19fdd08",
       "triggerType" : "PUSH"
     }, {
       "hash" : "05777be67e37f3a6a8eaa59e54d9e17e5fc6ba9b",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1016",
       "triggerID" : "05777be67e37f3a6a8eaa59e54d9e17e5fc6ba9b",
       "triggerType" : "PUSH"
     }, {
       "hash" : "a2ee90c549332a7f17a7d9b2799849b77c08a8e6",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "a2ee90c549332a7f17a7d9b2799849b77c08a8e6",
       "triggerType" : "PUSH"
     }, {
       "hash" : "05ce568b0b6ba14da9ca40d26b39ba478837d747",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1017",
       "triggerID" : "05ce568b0b6ba14da9ca40d26b39ba478837d747",
       "triggerType" : "PUSH"
     }, {
       "hash" : "840e3b33d39b1f86367bb6b0e65eadeb33de7765",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1025",
       "triggerID" : "840e3b33d39b1f86367bb6b0e65eadeb33de7765",
       "triggerType" : "PUSH"
     }, {
       "hash" : "8d2e2dd83f51d03e890d424a51d3e0e84e904a43",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1031",
       "triggerID" : "8d2e2dd83f51d03e890d424a51d3e0e84e904a43",
       "triggerType" : "PUSH"
     }, {
       "hash" : "688381914087ff11acf2f8c9372dbf39afa0bd40",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1035",
       "triggerID" : "688381914087ff11acf2f8c9372dbf39afa0bd40",
       "triggerType" : "PUSH"
     }, {
       "hash" : "fc2c138901e82ed5f3fbf363c2a0ce8140ade050",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "fc2c138901e82ed5f3fbf363c2a0ce8140ade050",
       "triggerType" : "PUSH"
     }, {
       "hash" : "6ed9e1cb1e5a191752d6d39c7a9d5d41840f0cdf",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1039",
       "triggerID" : "6ed9e1cb1e5a191752d6d39c7a9d5d41840f0cdf",
       "triggerType" : "PUSH"
     }, {
       "hash" : "a510a7de1458bb996df5bbb341e8d010e4c91381",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "a510a7de1458bb996df5bbb341e8d010e4c91381",
       "triggerType" : "PUSH"
     }, {
       "hash" : "49677397cd117228742205944606f7a789090376",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1042",
       "triggerID" : "49677397cd117228742205944606f7a789090376",
       "triggerType" : "PUSH"
     }, {
       "hash" : "1630db0a37422aed942ff280560a9d6329df5015",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1044",
       "triggerID" : "1630db0a37422aed942ff280560a9d6329df5015",
       "triggerType" : "PUSH"
     }, {
       "hash" : "f11c76301d4ab524db0af57aa831be3a898ea0fb",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1048",
       "triggerID" : "f11c76301d4ab524db0af57aa831be3a898ea0fb",
       "triggerType" : "PUSH"
     }, {
       "hash" : "d1a0dc4c2017853a7ba0c3302f886043a63f29da",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1050",
       "triggerID" : "d1a0dc4c2017853a7ba0c3302f886043a63f29da",
       "triggerType" : "PUSH"
     }, {
       "hash" : "6a4c15ec39ded1f90b113b272554217698a98e1c",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1087",
       "triggerID" : "6a4c15ec39ded1f90b113b272554217698a98e1c",
       "triggerType" : "PUSH"
     }, {
       "hash" : "288e730d645a5e58e50a87ef5f44ba11a8fb0533",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1092",
       "triggerID" : "288e730d645a5e58e50a87ef5f44ba11a8fb0533",
       "triggerType" : "PUSH"
     }, {
       "hash" : "70a5c21321a0da3a0be8d85e206c24befa7b89f1",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1095",
       "triggerID" : "70a5c21321a0da3a0be8d85e206c24befa7b89f1",
       "triggerType" : "PUSH"
     }, {
       "hash" : "f072a986114949c5e426ce64a5f5568fc718e25d",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "f072a986114949c5e426ce64a5f5568fc718e25d",
       "triggerType" : "PUSH"
     }, {
       "hash" : "45b27475b1e8806ec8c566d6f151f4bc79c8549d",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1096",
       "triggerID" : "45b27475b1e8806ec8c566d6f151f4bc79c8549d",
       "triggerType" : "PUSH"
     }, {
       "hash" : "3c62ef22f6cde28110e5eb2fca8760a008d77723",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "3c62ef22f6cde28110e5eb2fca8760a008d77723",
       "triggerType" : "PUSH"
     }, {
       "hash" : "dc785fa04e542b2ccc45494e0292f5153c7ecea8",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1097",
       "triggerID" : "dc785fa04e542b2ccc45494e0292f5153c7ecea8",
       "triggerType" : "PUSH"
     }, {
       "hash" : "585ddb8049b9b9ccb057027e265b08a1b586df8c",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "585ddb8049b9b9ccb057027e265b08a1b586df8c",
       "triggerType" : "PUSH"
     }, {
       "hash" : "b62ff8675beed2c4487c788b9ca1a0ad4271b913",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "b62ff8675beed2c4487c788b9ca1a0ad4271b913",
       "triggerType" : "PUSH"
     }, {
       "hash" : "2d46048a7402907f348cc96da2839ba1b3e4932a",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1099",
       "triggerID" : "2d46048a7402907f348cc96da2839ba1b3e4932a",
       "triggerType" : "PUSH"
     }, {
       "hash" : "004fb6ebf23f796a4b77650e0530f0f1145e8b91",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1101",
       "triggerID" : "004fb6ebf23f796a4b77650e0530f0f1145e8b91",
       "triggerType" : "PUSH"
     }, {
       "hash" : "af0c33333b5e9ae6d59f20a6db1d7c016ea015d3",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1165",
       "triggerID" : "af0c33333b5e9ae6d59f20a6db1d7c016ea015d3",
       "triggerType" : "PUSH"
     }, {
       "hash" : "da8111118d71306500229619d3b7996fcd2526e8",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "da8111118d71306500229619d3b7996fcd2526e8",
       "triggerType" : "PUSH"
     }, {
       "hash" : "83257e2b9bd3c426cf90e11003a5f244a750d7fa",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1167",
       "triggerID" : "83257e2b9bd3c426cf90e11003a5f244a750d7fa",
       "triggerType" : "PUSH"
     }, {
       "hash" : "e0de408f1b8153fdaaea4c87b27a7b69213b75b5",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1177",
       "triggerID" : "e0de408f1b8153fdaaea4c87b27a7b69213b75b5",
       "triggerType" : "PUSH"
     }, {
       "hash" : "3f7951f7ad7931f5dc0b977d7c33bc5429623535",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1180",
       "triggerID" : "3f7951f7ad7931f5dc0b977d7c33bc5429623535",
       "triggerType" : "PUSH"
     }, {
       "hash" : "9cd6f06a753b63630e8245d51338d2bf9ab86b84",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "9cd6f06a753b63630e8245d51338d2bf9ab86b84",
       "triggerType" : "PUSH"
     }, {
       "hash" : "7380b902731a92e0645c02397e14591a05798620",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "7380b902731a92e0645c02397e14591a05798620",
       "triggerType" : "PUSH"
     }, {
       "hash" : "0bd5cca01a541845081d5430d48b021d4bdf0969",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "0bd5cca01a541845081d5430d48b021d4bdf0969",
       "triggerType" : "PUSH"
     }, {
       "hash" : "8f4e175ef554f015e34e0968b656b5de714ab430",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1182",
       "triggerID" : "8f4e175ef554f015e34e0968b656b5de714ab430",
       "triggerType" : "PUSH"
     }, {
       "hash" : "2023f3c882d0c2f5b96e28d2f21812dc0c64aaee",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "2023f3c882d0c2f5b96e28d2f21812dc0c64aaee",
       "triggerType" : "PUSH"
     }, {
       "hash" : "8394acf060d5c3953dbd67cf930b9b1d3b92d8b5",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1198",
       "triggerID" : "8394acf060d5c3953dbd67cf930b9b1d3b92d8b5",
       "triggerType" : "PUSH"
     }, {
       "hash" : "7ec42c105971bff463e359019405a7f4e7947d86",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "7ec42c105971bff463e359019405a7f4e7947d86",
       "triggerType" : "PUSH"
     }, {
       "hash" : "4b930786a7b2de09ac1653cd884ee76a7d1b2ca5",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1202",
       "triggerID" : "4b930786a7b2de09ac1653cd884ee76a7d1b2ca5",
       "triggerType" : "PUSH"
     }, {
       "hash" : "0e9a648c4e03b5a74d6a0085831805041a9a26a2",
       "status" : "SUCCESS",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1216",
       "triggerID" : "0e9a648c4e03b5a74d6a0085831805041a9a26a2",
       "triggerType" : "PUSH"
     }, {
       "hash" : "326a8729a5d1dd88bc474a5c273aec6ca219f298",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "326a8729a5d1dd88bc474a5c273aec6ca219f298",
       "triggerType" : "PUSH"
     }, {
       "hash" : "cad2f761fc50fba28e2fb3595e4b1270e39ac48a",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "cad2f761fc50fba28e2fb3595e4b1270e39ac48a",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 9e668f65c6d4f73c76573236f8fa507dfb80edac UNKNOWN
   * 533f00e5546da0026bbe51e1c030ddd1c19fdd08 UNKNOWN
   * a2ee90c549332a7f17a7d9b2799849b77c08a8e6 UNKNOWN
   * fc2c138901e82ed5f3fbf363c2a0ce8140ade050 UNKNOWN
   * a510a7de1458bb996df5bbb341e8d010e4c91381 UNKNOWN
   * f072a986114949c5e426ce64a5f5568fc718e25d UNKNOWN
   * 3c62ef22f6cde28110e5eb2fca8760a008d77723 UNKNOWN
   * 585ddb8049b9b9ccb057027e265b08a1b586df8c UNKNOWN
   * b62ff8675beed2c4487c788b9ca1a0ad4271b913 UNKNOWN
   * da8111118d71306500229619d3b7996fcd2526e8 UNKNOWN
   * 9cd6f06a753b63630e8245d51338d2bf9ab86b84 UNKNOWN
   * 7380b902731a92e0645c02397e14591a05798620 UNKNOWN
   * 0bd5cca01a541845081d5430d48b021d4bdf0969 UNKNOWN
   * 2023f3c882d0c2f5b96e28d2f21812dc0c64aaee UNKNOWN
   * 7ec42c105971bff463e359019405a7f4e7947d86 UNKNOWN
   * 0e9a648c4e03b5a74d6a0085831805041a9a26a2 Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1216) 
   * 326a8729a5d1dd88bc474a5c273aec6ca219f298 UNKNOWN
   * cad2f761fc50fba28e2fb3595e4b1270e39ac48a UNKNOWN
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] [HUDI-8141] Incremental Query with Completion Time [hudi]

Posted by "danny0405 (via GitHub)" <gi...@apache.org>.

danny0405 commented on code in PR #11947:
URL: https://github.com/apache/hudi/pull/11947#discussion_r1805984220


##########
hudi-common/src/main/java/org/apache/hudi/common/table/read/IncrementalQueryAnalyzer.java:
##########
@@ -413,9 +425,14 @@ public Option<String> getEndInstant() {
      * Returns the latest instant time which should be included physically in reading.
      */
     public String getLastInstant() {
+      ValidationUtils.checkArgument(!this.instants.isEmpty());

Review Comment:
   Use `checkState` instead.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] [HUDI-8141] Incremental Query with Completion Time [hudi]

Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.

hudi-bot commented on PR #11947:
URL: https://github.com/apache/hudi/pull/11947#issuecomment-2423378605

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "090f256a315cbbf0c6a54d93cc2739dd693360e6",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=484",
       "triggerID" : "090f256a315cbbf0c6a54d93cc2739dd693360e6",
       "triggerType" : "PUSH"
     }, {
       "hash" : "0df5649013c1cf9e782fcbb67b8a2e7174f94512",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=985",
       "triggerID" : "0df5649013c1cf9e782fcbb67b8a2e7174f94512",
       "triggerType" : "PUSH"
     }, {
       "hash" : "d52e0128f409918644255abbe4376dc4a780bfc8",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=986",
       "triggerID" : "d52e0128f409918644255abbe4376dc4a780bfc8",
       "triggerType" : "PUSH"
     }, {
       "hash" : "9e668f65c6d4f73c76573236f8fa507dfb80edac",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "9e668f65c6d4f73c76573236f8fa507dfb80edac",
       "triggerType" : "PUSH"
     }, {
       "hash" : "f145131d524bf190986e5de600a326e32a27beca",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=987",
       "triggerID" : "f145131d524bf190986e5de600a326e32a27beca",
       "triggerType" : "PUSH"
     }, {
       "hash" : "0b008f807b42aa6fb2686922bb88d63e84a1f299",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=993",
       "triggerID" : "0b008f807b42aa6fb2686922bb88d63e84a1f299",
       "triggerType" : "PUSH"
     }, {
       "hash" : "65c01ad9f2b33d36d4890355feef91844438e732",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=995",
       "triggerID" : "65c01ad9f2b33d36d4890355feef91844438e732",
       "triggerType" : "PUSH"
     }, {
       "hash" : "955493fdc93842e8c99529cec35e39cecdbe9638",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=996",
       "triggerID" : "955493fdc93842e8c99529cec35e39cecdbe9638",
       "triggerType" : "PUSH"
     }, {
       "hash" : "533f00e5546da0026bbe51e1c030ddd1c19fdd08",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "533f00e5546da0026bbe51e1c030ddd1c19fdd08",
       "triggerType" : "PUSH"
     }, {
       "hash" : "05777be67e37f3a6a8eaa59e54d9e17e5fc6ba9b",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1016",
       "triggerID" : "05777be67e37f3a6a8eaa59e54d9e17e5fc6ba9b",
       "triggerType" : "PUSH"
     }, {
       "hash" : "a2ee90c549332a7f17a7d9b2799849b77c08a8e6",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "a2ee90c549332a7f17a7d9b2799849b77c08a8e6",
       "triggerType" : "PUSH"
     }, {
       "hash" : "05ce568b0b6ba14da9ca40d26b39ba478837d747",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1017",
       "triggerID" : "05ce568b0b6ba14da9ca40d26b39ba478837d747",
       "triggerType" : "PUSH"
     }, {
       "hash" : "840e3b33d39b1f86367bb6b0e65eadeb33de7765",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1025",
       "triggerID" : "840e3b33d39b1f86367bb6b0e65eadeb33de7765",
       "triggerType" : "PUSH"
     }, {
       "hash" : "8d2e2dd83f51d03e890d424a51d3e0e84e904a43",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1031",
       "triggerID" : "8d2e2dd83f51d03e890d424a51d3e0e84e904a43",
       "triggerType" : "PUSH"
     }, {
       "hash" : "688381914087ff11acf2f8c9372dbf39afa0bd40",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1035",
       "triggerID" : "688381914087ff11acf2f8c9372dbf39afa0bd40",
       "triggerType" : "PUSH"
     }, {
       "hash" : "fc2c138901e82ed5f3fbf363c2a0ce8140ade050",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "fc2c138901e82ed5f3fbf363c2a0ce8140ade050",
       "triggerType" : "PUSH"
     }, {
       "hash" : "6ed9e1cb1e5a191752d6d39c7a9d5d41840f0cdf",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1039",
       "triggerID" : "6ed9e1cb1e5a191752d6d39c7a9d5d41840f0cdf",
       "triggerType" : "PUSH"
     }, {
       "hash" : "a510a7de1458bb996df5bbb341e8d010e4c91381",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "a510a7de1458bb996df5bbb341e8d010e4c91381",
       "triggerType" : "PUSH"
     }, {
       "hash" : "49677397cd117228742205944606f7a789090376",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1042",
       "triggerID" : "49677397cd117228742205944606f7a789090376",
       "triggerType" : "PUSH"
     }, {
       "hash" : "1630db0a37422aed942ff280560a9d6329df5015",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1044",
       "triggerID" : "1630db0a37422aed942ff280560a9d6329df5015",
       "triggerType" : "PUSH"
     }, {
       "hash" : "f11c76301d4ab524db0af57aa831be3a898ea0fb",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1048",
       "triggerID" : "f11c76301d4ab524db0af57aa831be3a898ea0fb",
       "triggerType" : "PUSH"
     }, {
       "hash" : "d1a0dc4c2017853a7ba0c3302f886043a63f29da",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1050",
       "triggerID" : "d1a0dc4c2017853a7ba0c3302f886043a63f29da",
       "triggerType" : "PUSH"
     }, {
       "hash" : "6a4c15ec39ded1f90b113b272554217698a98e1c",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1087",
       "triggerID" : "6a4c15ec39ded1f90b113b272554217698a98e1c",
       "triggerType" : "PUSH"
     }, {
       "hash" : "288e730d645a5e58e50a87ef5f44ba11a8fb0533",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1092",
       "triggerID" : "288e730d645a5e58e50a87ef5f44ba11a8fb0533",
       "triggerType" : "PUSH"
     }, {
       "hash" : "70a5c21321a0da3a0be8d85e206c24befa7b89f1",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1095",
       "triggerID" : "70a5c21321a0da3a0be8d85e206c24befa7b89f1",
       "triggerType" : "PUSH"
     }, {
       "hash" : "f072a986114949c5e426ce64a5f5568fc718e25d",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "f072a986114949c5e426ce64a5f5568fc718e25d",
       "triggerType" : "PUSH"
     }, {
       "hash" : "45b27475b1e8806ec8c566d6f151f4bc79c8549d",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1096",
       "triggerID" : "45b27475b1e8806ec8c566d6f151f4bc79c8549d",
       "triggerType" : "PUSH"
     }, {
       "hash" : "3c62ef22f6cde28110e5eb2fca8760a008d77723",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "3c62ef22f6cde28110e5eb2fca8760a008d77723",
       "triggerType" : "PUSH"
     }, {
       "hash" : "dc785fa04e542b2ccc45494e0292f5153c7ecea8",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1097",
       "triggerID" : "dc785fa04e542b2ccc45494e0292f5153c7ecea8",
       "triggerType" : "PUSH"
     }, {
       "hash" : "585ddb8049b9b9ccb057027e265b08a1b586df8c",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "585ddb8049b9b9ccb057027e265b08a1b586df8c",
       "triggerType" : "PUSH"
     }, {
       "hash" : "b62ff8675beed2c4487c788b9ca1a0ad4271b913",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "b62ff8675beed2c4487c788b9ca1a0ad4271b913",
       "triggerType" : "PUSH"
     }, {
       "hash" : "2d46048a7402907f348cc96da2839ba1b3e4932a",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1099",
       "triggerID" : "2d46048a7402907f348cc96da2839ba1b3e4932a",
       "triggerType" : "PUSH"
     }, {
       "hash" : "004fb6ebf23f796a4b77650e0530f0f1145e8b91",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1101",
       "triggerID" : "004fb6ebf23f796a4b77650e0530f0f1145e8b91",
       "triggerType" : "PUSH"
     }, {
       "hash" : "af0c33333b5e9ae6d59f20a6db1d7c016ea015d3",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1165",
       "triggerID" : "af0c33333b5e9ae6d59f20a6db1d7c016ea015d3",
       "triggerType" : "PUSH"
     }, {
       "hash" : "da8111118d71306500229619d3b7996fcd2526e8",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "da8111118d71306500229619d3b7996fcd2526e8",
       "triggerType" : "PUSH"
     }, {
       "hash" : "83257e2b9bd3c426cf90e11003a5f244a750d7fa",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1167",
       "triggerID" : "83257e2b9bd3c426cf90e11003a5f244a750d7fa",
       "triggerType" : "PUSH"
     }, {
       "hash" : "e0de408f1b8153fdaaea4c87b27a7b69213b75b5",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1177",
       "triggerID" : "e0de408f1b8153fdaaea4c87b27a7b69213b75b5",
       "triggerType" : "PUSH"
     }, {
       "hash" : "3f7951f7ad7931f5dc0b977d7c33bc5429623535",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1180",
       "triggerID" : "3f7951f7ad7931f5dc0b977d7c33bc5429623535",
       "triggerType" : "PUSH"
     }, {
       "hash" : "9cd6f06a753b63630e8245d51338d2bf9ab86b84",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "9cd6f06a753b63630e8245d51338d2bf9ab86b84",
       "triggerType" : "PUSH"
     }, {
       "hash" : "7380b902731a92e0645c02397e14591a05798620",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "7380b902731a92e0645c02397e14591a05798620",
       "triggerType" : "PUSH"
     }, {
       "hash" : "0bd5cca01a541845081d5430d48b021d4bdf0969",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "0bd5cca01a541845081d5430d48b021d4bdf0969",
       "triggerType" : "PUSH"
     }, {
       "hash" : "8f4e175ef554f015e34e0968b656b5de714ab430",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1182",
       "triggerID" : "8f4e175ef554f015e34e0968b656b5de714ab430",
       "triggerType" : "PUSH"
     }, {
       "hash" : "2023f3c882d0c2f5b96e28d2f21812dc0c64aaee",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "2023f3c882d0c2f5b96e28d2f21812dc0c64aaee",
       "triggerType" : "PUSH"
     }, {
       "hash" : "8394acf060d5c3953dbd67cf930b9b1d3b92d8b5",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1198",
       "triggerID" : "8394acf060d5c3953dbd67cf930b9b1d3b92d8b5",
       "triggerType" : "PUSH"
     }, {
       "hash" : "7ec42c105971bff463e359019405a7f4e7947d86",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "7ec42c105971bff463e359019405a7f4e7947d86",
       "triggerType" : "PUSH"
     }, {
       "hash" : "4b930786a7b2de09ac1653cd884ee76a7d1b2ca5",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1202",
       "triggerID" : "4b930786a7b2de09ac1653cd884ee76a7d1b2ca5",
       "triggerType" : "PUSH"
     }, {
       "hash" : "0e9a648c4e03b5a74d6a0085831805041a9a26a2",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1216",
       "triggerID" : "0e9a648c4e03b5a74d6a0085831805041a9a26a2",
       "triggerType" : "PUSH"
     }, {
       "hash" : "326a8729a5d1dd88bc474a5c273aec6ca219f298",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "326a8729a5d1dd88bc474a5c273aec6ca219f298",
       "triggerType" : "PUSH"
     }, {
       "hash" : "cad2f761fc50fba28e2fb3595e4b1270e39ac48a",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1221",
       "triggerID" : "cad2f761fc50fba28e2fb3595e4b1270e39ac48a",
       "triggerType" : "PUSH"
     }, {
       "hash" : "d485a50aec34bc5707377afe96d696772832558e",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "d485a50aec34bc5707377afe96d696772832558e",
       "triggerType" : "PUSH"
     }, {
       "hash" : "38a98106f19fe4e51632a2c4289ec0f01805ba5e",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1224",
       "triggerID" : "38a98106f19fe4e51632a2c4289ec0f01805ba5e",
       "triggerType" : "PUSH"
     }, {
       "hash" : "7a6116cfbf4eba54e77194664225ed1c372d71be",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1250",
       "triggerID" : "7a6116cfbf4eba54e77194664225ed1c372d71be",
       "triggerType" : "PUSH"
     }, {
       "hash" : "df721d8b5f610e156430f46b1a704310c70a4725",
       "status" : "SUCCESS",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1254",
       "triggerID" : "df721d8b5f610e156430f46b1a704310c70a4725",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 9e668f65c6d4f73c76573236f8fa507dfb80edac UNKNOWN
   * 533f00e5546da0026bbe51e1c030ddd1c19fdd08 UNKNOWN
   * a2ee90c549332a7f17a7d9b2799849b77c08a8e6 UNKNOWN
   * fc2c138901e82ed5f3fbf363c2a0ce8140ade050 UNKNOWN
   * a510a7de1458bb996df5bbb341e8d010e4c91381 UNKNOWN
   * f072a986114949c5e426ce64a5f5568fc718e25d UNKNOWN
   * 3c62ef22f6cde28110e5eb2fca8760a008d77723 UNKNOWN
   * 585ddb8049b9b9ccb057027e265b08a1b586df8c UNKNOWN
   * b62ff8675beed2c4487c788b9ca1a0ad4271b913 UNKNOWN
   * da8111118d71306500229619d3b7996fcd2526e8 UNKNOWN
   * 9cd6f06a753b63630e8245d51338d2bf9ab86b84 UNKNOWN
   * 7380b902731a92e0645c02397e14591a05798620 UNKNOWN
   * 0bd5cca01a541845081d5430d48b021d4bdf0969 UNKNOWN
   * 2023f3c882d0c2f5b96e28d2f21812dc0c64aaee UNKNOWN
   * 7ec42c105971bff463e359019405a7f4e7947d86 UNKNOWN
   * 326a8729a5d1dd88bc474a5c273aec6ca219f298 UNKNOWN
   * d485a50aec34bc5707377afe96d696772832558e UNKNOWN
   * df721d8b5f610e156430f46b1a704310c70a4725 Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1254) 
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] [HUDI-8141] Incremental Query with Completion Time [hudi]

Posted by "danny0405 (via GitHub)" <gi...@apache.org>.

danny0405 commented on code in PR #11947:
URL: https://github.com/apache/hudi/pull/11947#discussion_r1798656460


##########
hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/spark/sql/hudi/streaming/HoodieStreamSource.scala:
##########
@@ -71,19 +69,6 @@ class HoodieStreamSource(
     parameters.get(DataSourceReadOptions.QUERY_TYPE.key).contains(DataSourceReadOptions.QUERY_TYPE_INCREMENTAL_OPT_VAL) &&
     parameters.get(DataSourceReadOptions.INCREMENTAL_FORMAT.key).contains(DataSourceReadOptions.INCREMENTAL_FORMAT_CDC_VAL)
 
-  /**
-   * When hollow commits are found while doing streaming read , unlike batch incremental query,
-   * we do not use [[HollowCommitHandling.FAIL]] by default, instead we use [[HollowCommitHandling.BLOCK]]
-   * to block processing data from going beyond the hollow commits to avoid unintentional skip.
-   *
-   * Users can set [[DataSourceReadOptions.INCREMENTAL_READ_HANDLE_HOLLOW_COMMIT]] to
-   * [[HollowCommitHandling.USE_TRANSITION_TIME]] to avoid the blocking behavior.
-   */
-  private val hollowCommitHandling: HollowCommitHandling =

Review Comment:
   Should we remove the class `HollowCommitHandling` because it is not used now.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] [HUDI-8141] WIP:Always use completion time to find begin Instant for incremental source [hudi]

Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.

hudi-bot commented on PR #11947:
URL: https://github.com/apache/hudi/pull/11947#issuecomment-2403654674

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "090f256a315cbbf0c6a54d93cc2739dd693360e6",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=484",
       "triggerID" : "090f256a315cbbf0c6a54d93cc2739dd693360e6",
       "triggerType" : "PUSH"
     }, {
       "hash" : "0df5649013c1cf9e782fcbb67b8a2e7174f94512",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=985",
       "triggerID" : "0df5649013c1cf9e782fcbb67b8a2e7174f94512",
       "triggerType" : "PUSH"
     }, {
       "hash" : "d52e0128f409918644255abbe4376dc4a780bfc8",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=986",
       "triggerID" : "d52e0128f409918644255abbe4376dc4a780bfc8",
       "triggerType" : "PUSH"
     }, {
       "hash" : "9e668f65c6d4f73c76573236f8fa507dfb80edac",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "9e668f65c6d4f73c76573236f8fa507dfb80edac",
       "triggerType" : "PUSH"
     }, {
       "hash" : "f145131d524bf190986e5de600a326e32a27beca",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=987",
       "triggerID" : "f145131d524bf190986e5de600a326e32a27beca",
       "triggerType" : "PUSH"
     }, {
       "hash" : "0b008f807b42aa6fb2686922bb88d63e84a1f299",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=993",
       "triggerID" : "0b008f807b42aa6fb2686922bb88d63e84a1f299",
       "triggerType" : "PUSH"
     }, {
       "hash" : "65c01ad9f2b33d36d4890355feef91844438e732",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=995",
       "triggerID" : "65c01ad9f2b33d36d4890355feef91844438e732",
       "triggerType" : "PUSH"
     }, {
       "hash" : "955493fdc93842e8c99529cec35e39cecdbe9638",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=996",
       "triggerID" : "955493fdc93842e8c99529cec35e39cecdbe9638",
       "triggerType" : "PUSH"
     }, {
       "hash" : "533f00e5546da0026bbe51e1c030ddd1c19fdd08",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "533f00e5546da0026bbe51e1c030ddd1c19fdd08",
       "triggerType" : "PUSH"
     }, {
       "hash" : "05777be67e37f3a6a8eaa59e54d9e17e5fc6ba9b",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1016",
       "triggerID" : "05777be67e37f3a6a8eaa59e54d9e17e5fc6ba9b",
       "triggerType" : "PUSH"
     }, {
       "hash" : "a2ee90c549332a7f17a7d9b2799849b77c08a8e6",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "a2ee90c549332a7f17a7d9b2799849b77c08a8e6",
       "triggerType" : "PUSH"
     }, {
       "hash" : "05ce568b0b6ba14da9ca40d26b39ba478837d747",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1017",
       "triggerID" : "05ce568b0b6ba14da9ca40d26b39ba478837d747",
       "triggerType" : "PUSH"
     }, {
       "hash" : "840e3b33d39b1f86367bb6b0e65eadeb33de7765",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1025",
       "triggerID" : "840e3b33d39b1f86367bb6b0e65eadeb33de7765",
       "triggerType" : "PUSH"
     }, {
       "hash" : "8d2e2dd83f51d03e890d424a51d3e0e84e904a43",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1031",
       "triggerID" : "8d2e2dd83f51d03e890d424a51d3e0e84e904a43",
       "triggerType" : "PUSH"
     }, {
       "hash" : "688381914087ff11acf2f8c9372dbf39afa0bd40",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1035",
       "triggerID" : "688381914087ff11acf2f8c9372dbf39afa0bd40",
       "triggerType" : "PUSH"
     }, {
       "hash" : "fc2c138901e82ed5f3fbf363c2a0ce8140ade050",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "fc2c138901e82ed5f3fbf363c2a0ce8140ade050",
       "triggerType" : "PUSH"
     }, {
       "hash" : "6ed9e1cb1e5a191752d6d39c7a9d5d41840f0cdf",
       "status" : "CANCELED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1039",
       "triggerID" : "6ed9e1cb1e5a191752d6d39c7a9d5d41840f0cdf",
       "triggerType" : "PUSH"
     }, {
       "hash" : "a510a7de1458bb996df5bbb341e8d010e4c91381",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "a510a7de1458bb996df5bbb341e8d010e4c91381",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 9e668f65c6d4f73c76573236f8fa507dfb80edac UNKNOWN
   * 533f00e5546da0026bbe51e1c030ddd1c19fdd08 UNKNOWN
   * a2ee90c549332a7f17a7d9b2799849b77c08a8e6 UNKNOWN
   * fc2c138901e82ed5f3fbf363c2a0ce8140ade050 UNKNOWN
   * 6ed9e1cb1e5a191752d6d39c7a9d5d41840f0cdf Azure: [CANCELED](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1039) 
   * a510a7de1458bb996df5bbb341e8d010e4c91381 UNKNOWN
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] [HUDI-8141] Incremental Query with Completion Time [hudi]

Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.

hudi-bot commented on PR #11947:
URL: https://github.com/apache/hudi/pull/11947#issuecomment-2418180240

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "090f256a315cbbf0c6a54d93cc2739dd693360e6",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=484",
       "triggerID" : "090f256a315cbbf0c6a54d93cc2739dd693360e6",
       "triggerType" : "PUSH"
     }, {
       "hash" : "0df5649013c1cf9e782fcbb67b8a2e7174f94512",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=985",
       "triggerID" : "0df5649013c1cf9e782fcbb67b8a2e7174f94512",
       "triggerType" : "PUSH"
     }, {
       "hash" : "d52e0128f409918644255abbe4376dc4a780bfc8",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=986",
       "triggerID" : "d52e0128f409918644255abbe4376dc4a780bfc8",
       "triggerType" : "PUSH"
     }, {
       "hash" : "9e668f65c6d4f73c76573236f8fa507dfb80edac",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "9e668f65c6d4f73c76573236f8fa507dfb80edac",
       "triggerType" : "PUSH"
     }, {
       "hash" : "f145131d524bf190986e5de600a326e32a27beca",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=987",
       "triggerID" : "f145131d524bf190986e5de600a326e32a27beca",
       "triggerType" : "PUSH"
     }, {
       "hash" : "0b008f807b42aa6fb2686922bb88d63e84a1f299",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=993",
       "triggerID" : "0b008f807b42aa6fb2686922bb88d63e84a1f299",
       "triggerType" : "PUSH"
     }, {
       "hash" : "65c01ad9f2b33d36d4890355feef91844438e732",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=995",
       "triggerID" : "65c01ad9f2b33d36d4890355feef91844438e732",
       "triggerType" : "PUSH"
     }, {
       "hash" : "955493fdc93842e8c99529cec35e39cecdbe9638",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=996",
       "triggerID" : "955493fdc93842e8c99529cec35e39cecdbe9638",
       "triggerType" : "PUSH"
     }, {
       "hash" : "533f00e5546da0026bbe51e1c030ddd1c19fdd08",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "533f00e5546da0026bbe51e1c030ddd1c19fdd08",
       "triggerType" : "PUSH"
     }, {
       "hash" : "05777be67e37f3a6a8eaa59e54d9e17e5fc6ba9b",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1016",
       "triggerID" : "05777be67e37f3a6a8eaa59e54d9e17e5fc6ba9b",
       "triggerType" : "PUSH"
     }, {
       "hash" : "a2ee90c549332a7f17a7d9b2799849b77c08a8e6",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "a2ee90c549332a7f17a7d9b2799849b77c08a8e6",
       "triggerType" : "PUSH"
     }, {
       "hash" : "05ce568b0b6ba14da9ca40d26b39ba478837d747",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1017",
       "triggerID" : "05ce568b0b6ba14da9ca40d26b39ba478837d747",
       "triggerType" : "PUSH"
     }, {
       "hash" : "840e3b33d39b1f86367bb6b0e65eadeb33de7765",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1025",
       "triggerID" : "840e3b33d39b1f86367bb6b0e65eadeb33de7765",
       "triggerType" : "PUSH"
     }, {
       "hash" : "8d2e2dd83f51d03e890d424a51d3e0e84e904a43",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1031",
       "triggerID" : "8d2e2dd83f51d03e890d424a51d3e0e84e904a43",
       "triggerType" : "PUSH"
     }, {
       "hash" : "688381914087ff11acf2f8c9372dbf39afa0bd40",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1035",
       "triggerID" : "688381914087ff11acf2f8c9372dbf39afa0bd40",
       "triggerType" : "PUSH"
     }, {
       "hash" : "fc2c138901e82ed5f3fbf363c2a0ce8140ade050",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "fc2c138901e82ed5f3fbf363c2a0ce8140ade050",
       "triggerType" : "PUSH"
     }, {
       "hash" : "6ed9e1cb1e5a191752d6d39c7a9d5d41840f0cdf",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1039",
       "triggerID" : "6ed9e1cb1e5a191752d6d39c7a9d5d41840f0cdf",
       "triggerType" : "PUSH"
     }, {
       "hash" : "a510a7de1458bb996df5bbb341e8d010e4c91381",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "a510a7de1458bb996df5bbb341e8d010e4c91381",
       "triggerType" : "PUSH"
     }, {
       "hash" : "49677397cd117228742205944606f7a789090376",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1042",
       "triggerID" : "49677397cd117228742205944606f7a789090376",
       "triggerType" : "PUSH"
     }, {
       "hash" : "1630db0a37422aed942ff280560a9d6329df5015",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1044",
       "triggerID" : "1630db0a37422aed942ff280560a9d6329df5015",
       "triggerType" : "PUSH"
     }, {
       "hash" : "f11c76301d4ab524db0af57aa831be3a898ea0fb",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1048",
       "triggerID" : "f11c76301d4ab524db0af57aa831be3a898ea0fb",
       "triggerType" : "PUSH"
     }, {
       "hash" : "d1a0dc4c2017853a7ba0c3302f886043a63f29da",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1050",
       "triggerID" : "d1a0dc4c2017853a7ba0c3302f886043a63f29da",
       "triggerType" : "PUSH"
     }, {
       "hash" : "6a4c15ec39ded1f90b113b272554217698a98e1c",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1087",
       "triggerID" : "6a4c15ec39ded1f90b113b272554217698a98e1c",
       "triggerType" : "PUSH"
     }, {
       "hash" : "288e730d645a5e58e50a87ef5f44ba11a8fb0533",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1092",
       "triggerID" : "288e730d645a5e58e50a87ef5f44ba11a8fb0533",
       "triggerType" : "PUSH"
     }, {
       "hash" : "70a5c21321a0da3a0be8d85e206c24befa7b89f1",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1095",
       "triggerID" : "70a5c21321a0da3a0be8d85e206c24befa7b89f1",
       "triggerType" : "PUSH"
     }, {
       "hash" : "f072a986114949c5e426ce64a5f5568fc718e25d",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "f072a986114949c5e426ce64a5f5568fc718e25d",
       "triggerType" : "PUSH"
     }, {
       "hash" : "45b27475b1e8806ec8c566d6f151f4bc79c8549d",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1096",
       "triggerID" : "45b27475b1e8806ec8c566d6f151f4bc79c8549d",
       "triggerType" : "PUSH"
     }, {
       "hash" : "3c62ef22f6cde28110e5eb2fca8760a008d77723",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "3c62ef22f6cde28110e5eb2fca8760a008d77723",
       "triggerType" : "PUSH"
     }, {
       "hash" : "dc785fa04e542b2ccc45494e0292f5153c7ecea8",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1097",
       "triggerID" : "dc785fa04e542b2ccc45494e0292f5153c7ecea8",
       "triggerType" : "PUSH"
     }, {
       "hash" : "585ddb8049b9b9ccb057027e265b08a1b586df8c",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "585ddb8049b9b9ccb057027e265b08a1b586df8c",
       "triggerType" : "PUSH"
     }, {
       "hash" : "b62ff8675beed2c4487c788b9ca1a0ad4271b913",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "b62ff8675beed2c4487c788b9ca1a0ad4271b913",
       "triggerType" : "PUSH"
     }, {
       "hash" : "2d46048a7402907f348cc96da2839ba1b3e4932a",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1099",
       "triggerID" : "2d46048a7402907f348cc96da2839ba1b3e4932a",
       "triggerType" : "PUSH"
     }, {
       "hash" : "004fb6ebf23f796a4b77650e0530f0f1145e8b91",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1101",
       "triggerID" : "004fb6ebf23f796a4b77650e0530f0f1145e8b91",
       "triggerType" : "PUSH"
     }, {
       "hash" : "af0c33333b5e9ae6d59f20a6db1d7c016ea015d3",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1165",
       "triggerID" : "af0c33333b5e9ae6d59f20a6db1d7c016ea015d3",
       "triggerType" : "PUSH"
     }, {
       "hash" : "da8111118d71306500229619d3b7996fcd2526e8",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "da8111118d71306500229619d3b7996fcd2526e8",
       "triggerType" : "PUSH"
     }, {
       "hash" : "83257e2b9bd3c426cf90e11003a5f244a750d7fa",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1167",
       "triggerID" : "83257e2b9bd3c426cf90e11003a5f244a750d7fa",
       "triggerType" : "PUSH"
     }, {
       "hash" : "e0de408f1b8153fdaaea4c87b27a7b69213b75b5",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1177",
       "triggerID" : "e0de408f1b8153fdaaea4c87b27a7b69213b75b5",
       "triggerType" : "PUSH"
     }, {
       "hash" : "3f7951f7ad7931f5dc0b977d7c33bc5429623535",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1180",
       "triggerID" : "3f7951f7ad7931f5dc0b977d7c33bc5429623535",
       "triggerType" : "PUSH"
     }, {
       "hash" : "9cd6f06a753b63630e8245d51338d2bf9ab86b84",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "9cd6f06a753b63630e8245d51338d2bf9ab86b84",
       "triggerType" : "PUSH"
     }, {
       "hash" : "7380b902731a92e0645c02397e14591a05798620",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "7380b902731a92e0645c02397e14591a05798620",
       "triggerType" : "PUSH"
     }, {
       "hash" : "0bd5cca01a541845081d5430d48b021d4bdf0969",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "0bd5cca01a541845081d5430d48b021d4bdf0969",
       "triggerType" : "PUSH"
     }, {
       "hash" : "8f4e175ef554f015e34e0968b656b5de714ab430",
       "status" : "SUCCESS",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1182",
       "triggerID" : "8f4e175ef554f015e34e0968b656b5de714ab430",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 9e668f65c6d4f73c76573236f8fa507dfb80edac UNKNOWN
   * 533f00e5546da0026bbe51e1c030ddd1c19fdd08 UNKNOWN
   * a2ee90c549332a7f17a7d9b2799849b77c08a8e6 UNKNOWN
   * fc2c138901e82ed5f3fbf363c2a0ce8140ade050 UNKNOWN
   * a510a7de1458bb996df5bbb341e8d010e4c91381 UNKNOWN
   * f072a986114949c5e426ce64a5f5568fc718e25d UNKNOWN
   * 3c62ef22f6cde28110e5eb2fca8760a008d77723 UNKNOWN
   * 585ddb8049b9b9ccb057027e265b08a1b586df8c UNKNOWN
   * b62ff8675beed2c4487c788b9ca1a0ad4271b913 UNKNOWN
   * da8111118d71306500229619d3b7996fcd2526e8 UNKNOWN
   * 9cd6f06a753b63630e8245d51338d2bf9ab86b84 UNKNOWN
   * 7380b902731a92e0645c02397e14591a05798620 UNKNOWN
   * 0bd5cca01a541845081d5430d48b021d4bdf0969 UNKNOWN
   * 8f4e175ef554f015e34e0968b656b5de714ab430 Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1182) 
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] [HUDI-8141] Incremental Query with Completion Time [hudi]

Posted by "yihua (via GitHub)" <gi...@apache.org>.

yihua commented on code in PR #11947:
URL: https://github.com/apache/hudi/pull/11947#discussion_r1802528193


##########
hudi-common/src/main/java/org/apache/hudi/common/table/timeline/CompletionTimeQueryView.java:
##########
@@ -189,46 +193,78 @@ public Option<String> getCompletionTime(String beginTime) {
    * <p>By default, assumes there is at most 1 day time of duration for an instant to accelerate the queries.
    *
    * @param timeline The timeline.
-   * @param rangeStart   The query range start completion time.
-   * @param rangeEnd     The query range end completion time.
+   * @param rangeStartCompletionTime   The query range start completion time.
+   * @param rangeEndCompletionTime     The query range end completion time.
    * @param rangeType    The range type.
    *
    * @return The sorted instant time list.
    */
   public List<String> getStartTimes(
       HoodieTimeline timeline,
-      Option<String> rangeStart,
-      Option<String> rangeEnd,
+      Option<String> rangeStartCompletionTime,

Review Comment:
   I've made the variable names consistent in `IncrementalQueryAnalyzer` and  `CompletionTimeQueryView` to avoid confusion.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] [HUDI-8141] Incremental Query with Completion Time [hudi]

Posted by "yihua (via GitHub)" <gi...@apache.org>.

yihua commented on code in PR #11947:
URL: https://github.com/apache/hudi/pull/11947#discussion_r1802578981


##########
hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/hudi/MergeOnReadIncrementalRelation.scala:
##########
@@ -201,55 +199,42 @@ trait HoodieIncrementalRelationTrait extends HoodieBaseRelation {
     val fallbackToFullTableScan = optParams.getOrElse(DataSourceReadOptions.INCREMENTAL_FALLBACK_TO_FULL_TABLE_SCAN.key,
       DataSourceReadOptions.INCREMENTAL_FALLBACK_TO_FULL_TABLE_SCAN.defaultValue).toBoolean
 
-    fallbackToFullTableScan && (startInstantArchived || endInstantArchived
+    fallbackToFullTableScan && (startInstantArchived
       || affectedFilesInCommits.asScala.exists(fileStatus => !metaClient.getStorage.exists(fileStatus.getPath)))
   }
 
-  protected lazy val includedCommits: immutable.Seq[HoodieInstant] = {
-    if (!startInstantArchived || !endInstantArchived) {
-      // If endTimestamp commit is not archived, will filter instants
-      // before endTimestamp.
-      if (hollowCommitHandling == USE_TRANSITION_TIME) {
-        super.timeline.findInstantsInRangeByCompletionTime(startTimestamp, endTimestamp).getInstants.asScala.toList
-      } else {
-        super.timeline.findInstantsInRange(startTimestamp, endTimestamp).getInstants.asScala.toList
-      }
-    } else {
-      super.timeline.getInstants.asScala.toList
-    }
-  }
+  protected lazy val queryContext: IncrementalQueryAnalyzer.QueryContext =
+    IncrementalQueryAnalyzer.builder()
+      .metaClient(metaClient)
+      .startTime(optParams(DataSourceReadOptions.BEGIN_INSTANTTIME.key))
+      .endTime(optParams.getOrElse(DataSourceReadOptions.END_INSTANTTIME.key, null))
+      .rangeType(InstantRange.RangeType.OPEN_CLOSED)
+      .build()
+      .analyze()
+
+  protected lazy val includedCommits: immutable.Seq[HoodieInstant] = List.concat(
+    queryContext.getArchivedInstants.asScala,

Review Comment:
   Added a new API to abstract this.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] [HUDI-8141] Incremental Query with Completion Time [hudi]

Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.

hudi-bot commented on PR #11947:
URL: https://github.com/apache/hudi/pull/11947#issuecomment-2416324615

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "090f256a315cbbf0c6a54d93cc2739dd693360e6",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=484",
       "triggerID" : "090f256a315cbbf0c6a54d93cc2739dd693360e6",
       "triggerType" : "PUSH"
     }, {
       "hash" : "0df5649013c1cf9e782fcbb67b8a2e7174f94512",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=985",
       "triggerID" : "0df5649013c1cf9e782fcbb67b8a2e7174f94512",
       "triggerType" : "PUSH"
     }, {
       "hash" : "d52e0128f409918644255abbe4376dc4a780bfc8",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=986",
       "triggerID" : "d52e0128f409918644255abbe4376dc4a780bfc8",
       "triggerType" : "PUSH"
     }, {
       "hash" : "9e668f65c6d4f73c76573236f8fa507dfb80edac",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "9e668f65c6d4f73c76573236f8fa507dfb80edac",
       "triggerType" : "PUSH"
     }, {
       "hash" : "f145131d524bf190986e5de600a326e32a27beca",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=987",
       "triggerID" : "f145131d524bf190986e5de600a326e32a27beca",
       "triggerType" : "PUSH"
     }, {
       "hash" : "0b008f807b42aa6fb2686922bb88d63e84a1f299",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=993",
       "triggerID" : "0b008f807b42aa6fb2686922bb88d63e84a1f299",
       "triggerType" : "PUSH"
     }, {
       "hash" : "65c01ad9f2b33d36d4890355feef91844438e732",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=995",
       "triggerID" : "65c01ad9f2b33d36d4890355feef91844438e732",
       "triggerType" : "PUSH"
     }, {
       "hash" : "955493fdc93842e8c99529cec35e39cecdbe9638",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=996",
       "triggerID" : "955493fdc93842e8c99529cec35e39cecdbe9638",
       "triggerType" : "PUSH"
     }, {
       "hash" : "533f00e5546da0026bbe51e1c030ddd1c19fdd08",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "533f00e5546da0026bbe51e1c030ddd1c19fdd08",
       "triggerType" : "PUSH"
     }, {
       "hash" : "05777be67e37f3a6a8eaa59e54d9e17e5fc6ba9b",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1016",
       "triggerID" : "05777be67e37f3a6a8eaa59e54d9e17e5fc6ba9b",
       "triggerType" : "PUSH"
     }, {
       "hash" : "a2ee90c549332a7f17a7d9b2799849b77c08a8e6",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "a2ee90c549332a7f17a7d9b2799849b77c08a8e6",
       "triggerType" : "PUSH"
     }, {
       "hash" : "05ce568b0b6ba14da9ca40d26b39ba478837d747",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1017",
       "triggerID" : "05ce568b0b6ba14da9ca40d26b39ba478837d747",
       "triggerType" : "PUSH"
     }, {
       "hash" : "840e3b33d39b1f86367bb6b0e65eadeb33de7765",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1025",
       "triggerID" : "840e3b33d39b1f86367bb6b0e65eadeb33de7765",
       "triggerType" : "PUSH"
     }, {
       "hash" : "8d2e2dd83f51d03e890d424a51d3e0e84e904a43",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1031",
       "triggerID" : "8d2e2dd83f51d03e890d424a51d3e0e84e904a43",
       "triggerType" : "PUSH"
     }, {
       "hash" : "688381914087ff11acf2f8c9372dbf39afa0bd40",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1035",
       "triggerID" : "688381914087ff11acf2f8c9372dbf39afa0bd40",
       "triggerType" : "PUSH"
     }, {
       "hash" : "fc2c138901e82ed5f3fbf363c2a0ce8140ade050",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "fc2c138901e82ed5f3fbf363c2a0ce8140ade050",
       "triggerType" : "PUSH"
     }, {
       "hash" : "6ed9e1cb1e5a191752d6d39c7a9d5d41840f0cdf",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1039",
       "triggerID" : "6ed9e1cb1e5a191752d6d39c7a9d5d41840f0cdf",
       "triggerType" : "PUSH"
     }, {
       "hash" : "a510a7de1458bb996df5bbb341e8d010e4c91381",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "a510a7de1458bb996df5bbb341e8d010e4c91381",
       "triggerType" : "PUSH"
     }, {
       "hash" : "49677397cd117228742205944606f7a789090376",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1042",
       "triggerID" : "49677397cd117228742205944606f7a789090376",
       "triggerType" : "PUSH"
     }, {
       "hash" : "1630db0a37422aed942ff280560a9d6329df5015",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1044",
       "triggerID" : "1630db0a37422aed942ff280560a9d6329df5015",
       "triggerType" : "PUSH"
     }, {
       "hash" : "f11c76301d4ab524db0af57aa831be3a898ea0fb",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1048",
       "triggerID" : "f11c76301d4ab524db0af57aa831be3a898ea0fb",
       "triggerType" : "PUSH"
     }, {
       "hash" : "d1a0dc4c2017853a7ba0c3302f886043a63f29da",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1050",
       "triggerID" : "d1a0dc4c2017853a7ba0c3302f886043a63f29da",
       "triggerType" : "PUSH"
     }, {
       "hash" : "6a4c15ec39ded1f90b113b272554217698a98e1c",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1087",
       "triggerID" : "6a4c15ec39ded1f90b113b272554217698a98e1c",
       "triggerType" : "PUSH"
     }, {
       "hash" : "288e730d645a5e58e50a87ef5f44ba11a8fb0533",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1092",
       "triggerID" : "288e730d645a5e58e50a87ef5f44ba11a8fb0533",
       "triggerType" : "PUSH"
     }, {
       "hash" : "70a5c21321a0da3a0be8d85e206c24befa7b89f1",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1095",
       "triggerID" : "70a5c21321a0da3a0be8d85e206c24befa7b89f1",
       "triggerType" : "PUSH"
     }, {
       "hash" : "f072a986114949c5e426ce64a5f5568fc718e25d",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "f072a986114949c5e426ce64a5f5568fc718e25d",
       "triggerType" : "PUSH"
     }, {
       "hash" : "45b27475b1e8806ec8c566d6f151f4bc79c8549d",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1096",
       "triggerID" : "45b27475b1e8806ec8c566d6f151f4bc79c8549d",
       "triggerType" : "PUSH"
     }, {
       "hash" : "3c62ef22f6cde28110e5eb2fca8760a008d77723",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "3c62ef22f6cde28110e5eb2fca8760a008d77723",
       "triggerType" : "PUSH"
     }, {
       "hash" : "dc785fa04e542b2ccc45494e0292f5153c7ecea8",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1097",
       "triggerID" : "dc785fa04e542b2ccc45494e0292f5153c7ecea8",
       "triggerType" : "PUSH"
     }, {
       "hash" : "585ddb8049b9b9ccb057027e265b08a1b586df8c",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "585ddb8049b9b9ccb057027e265b08a1b586df8c",
       "triggerType" : "PUSH"
     }, {
       "hash" : "b62ff8675beed2c4487c788b9ca1a0ad4271b913",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "b62ff8675beed2c4487c788b9ca1a0ad4271b913",
       "triggerType" : "PUSH"
     }, {
       "hash" : "2d46048a7402907f348cc96da2839ba1b3e4932a",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1099",
       "triggerID" : "2d46048a7402907f348cc96da2839ba1b3e4932a",
       "triggerType" : "PUSH"
     }, {
       "hash" : "004fb6ebf23f796a4b77650e0530f0f1145e8b91",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1101",
       "triggerID" : "004fb6ebf23f796a4b77650e0530f0f1145e8b91",
       "triggerType" : "PUSH"
     }, {
       "hash" : "af0c33333b5e9ae6d59f20a6db1d7c016ea015d3",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1165",
       "triggerID" : "af0c33333b5e9ae6d59f20a6db1d7c016ea015d3",
       "triggerType" : "PUSH"
     }, {
       "hash" : "da8111118d71306500229619d3b7996fcd2526e8",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "da8111118d71306500229619d3b7996fcd2526e8",
       "triggerType" : "PUSH"
     }, {
       "hash" : "83257e2b9bd3c426cf90e11003a5f244a750d7fa",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1167",
       "triggerID" : "83257e2b9bd3c426cf90e11003a5f244a750d7fa",
       "triggerType" : "PUSH"
     }, {
       "hash" : "e0de408f1b8153fdaaea4c87b27a7b69213b75b5",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1177",
       "triggerID" : "e0de408f1b8153fdaaea4c87b27a7b69213b75b5",
       "triggerType" : "PUSH"
     }, {
       "hash" : "3f7951f7ad7931f5dc0b977d7c33bc5429623535",
       "status" : "FAILURE",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1180",
       "triggerID" : "3f7951f7ad7931f5dc0b977d7c33bc5429623535",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 9e668f65c6d4f73c76573236f8fa507dfb80edac UNKNOWN
   * 533f00e5546da0026bbe51e1c030ddd1c19fdd08 UNKNOWN
   * a2ee90c549332a7f17a7d9b2799849b77c08a8e6 UNKNOWN
   * fc2c138901e82ed5f3fbf363c2a0ce8140ade050 UNKNOWN
   * a510a7de1458bb996df5bbb341e8d010e4c91381 UNKNOWN
   * f072a986114949c5e426ce64a5f5568fc718e25d UNKNOWN
   * 3c62ef22f6cde28110e5eb2fca8760a008d77723 UNKNOWN
   * 585ddb8049b9b9ccb057027e265b08a1b586df8c UNKNOWN
   * b62ff8675beed2c4487c788b9ca1a0ad4271b913 UNKNOWN
   * da8111118d71306500229619d3b7996fcd2526e8 UNKNOWN
   * 3f7951f7ad7931f5dc0b977d7c33bc5429623535 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1180) 
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] [HUDI-8141] Incremental Query with Completion Time [hudi]

Posted by "yihua (via GitHub)" <gi...@apache.org>.

yihua commented on code in PR #11947:
URL: https://github.com/apache/hudi/pull/11947#discussion_r1802099864


##########
hudi-common/src/main/java/org/apache/hudi/common/table/read/IncrementalQueryAnalyzer.java:
##########
@@ -376,14 +382,16 @@ private QueryContext(
         List<HoodieInstant> archivedInstants,
         List<HoodieInstant> activeInstants,
         HoodieTimeline activeTimeline,
-        @Nullable HoodieTimeline archivedTimeline) {
+        @Nullable HoodieTimeline archivedTimeline,
+        @Nullable String predicateFilter) {
       this.startInstant = Option.ofNullable(startInstant);
       this.endInstant = Option.ofNullable(endInstant);
       this.archivedInstants = archivedInstants;
       this.activeInstants = activeInstants;
       this.activeTimeline = activeTimeline;
       this.archivedTimeline = archivedTimeline;
       this.instants = instants;
+      this.predicateFilter = Option.ofNullable(predicateFilter);

Review Comment:
   - The `SnapshotLoadQuerySplitter` may change the end instant time and add the predicate in the case of full scan (see the following code in `HoodieIncrSource`).  To pass the predicate around, the `QueryContext` instance is passed to `SnapshotLoadQuerySplitter` for further processing before returning a new `QueryContext` instance that can contain the predicate.
   
   ```
   if (instantRange.isEmpty() || shouldFullScan) {
         // snapshot query
         Dataset<Row> snapshot = reader
             .options(readOpts)
             .option(DataSourceReadOptions.QUERY_TYPE().key(), DataSourceReadOptions.QUERY_TYPE_SNAPSHOT_OPT_VAL())
             .load(srcPath);
         if (snapshotLoadQuerySplitter.isPresent()) {
           queryContext = snapshotLoadQuerySplitter.get().getNextCheckpoint(snapshot, queryContext, sourceProfileSupplier);
           // update endTime/next checkpoint
           completionTimeToStopAt = queryContext.getEndInstant().orElse(queryContext.getLastInstant());
         }
   
         snapshot = queryContext.getPredicateFilter().map(snapshot::filter).orElse(snapshot);
         source = snapshot
             // add filtering so that only interested records are returned.
             // completion time comparison uses ( , ], but when comparing start time we need to use [, ]
             .filter(String.format("%s >= '%s'", HoodieRecord.COMMIT_TIME_METADATA_FIELD,
                 queryContext.getStartInstant().get()))
             .filter(String.format("%s <= '%s'", HoodieRecord.COMMIT_TIME_METADATA_FIELD,
                 queryContext.getEndInstant().orElse(queryContext.getLastInstant())))
             .filter(String.format("%s IN ('%s')", HoodieRecord.COMMIT_TIME_METADATA_FIELD,
                 String.join("','", queryContext.getInstants())));
       }
   ```



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] [HUDI-8141] Incremental Query with Completion Time [hudi]

Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.

hudi-bot commented on PR #11947:
URL: https://github.com/apache/hudi/pull/11947#issuecomment-2415347501

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "090f256a315cbbf0c6a54d93cc2739dd693360e6",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=484",
       "triggerID" : "090f256a315cbbf0c6a54d93cc2739dd693360e6",
       "triggerType" : "PUSH"
     }, {
       "hash" : "0df5649013c1cf9e782fcbb67b8a2e7174f94512",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=985",
       "triggerID" : "0df5649013c1cf9e782fcbb67b8a2e7174f94512",
       "triggerType" : "PUSH"
     }, {
       "hash" : "d52e0128f409918644255abbe4376dc4a780bfc8",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=986",
       "triggerID" : "d52e0128f409918644255abbe4376dc4a780bfc8",
       "triggerType" : "PUSH"
     }, {
       "hash" : "9e668f65c6d4f73c76573236f8fa507dfb80edac",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "9e668f65c6d4f73c76573236f8fa507dfb80edac",
       "triggerType" : "PUSH"
     }, {
       "hash" : "f145131d524bf190986e5de600a326e32a27beca",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=987",
       "triggerID" : "f145131d524bf190986e5de600a326e32a27beca",
       "triggerType" : "PUSH"
     }, {
       "hash" : "0b008f807b42aa6fb2686922bb88d63e84a1f299",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=993",
       "triggerID" : "0b008f807b42aa6fb2686922bb88d63e84a1f299",
       "triggerType" : "PUSH"
     }, {
       "hash" : "65c01ad9f2b33d36d4890355feef91844438e732",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=995",
       "triggerID" : "65c01ad9f2b33d36d4890355feef91844438e732",
       "triggerType" : "PUSH"
     }, {
       "hash" : "955493fdc93842e8c99529cec35e39cecdbe9638",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=996",
       "triggerID" : "955493fdc93842e8c99529cec35e39cecdbe9638",
       "triggerType" : "PUSH"
     }, {
       "hash" : "533f00e5546da0026bbe51e1c030ddd1c19fdd08",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "533f00e5546da0026bbe51e1c030ddd1c19fdd08",
       "triggerType" : "PUSH"
     }, {
       "hash" : "05777be67e37f3a6a8eaa59e54d9e17e5fc6ba9b",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1016",
       "triggerID" : "05777be67e37f3a6a8eaa59e54d9e17e5fc6ba9b",
       "triggerType" : "PUSH"
     }, {
       "hash" : "a2ee90c549332a7f17a7d9b2799849b77c08a8e6",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "a2ee90c549332a7f17a7d9b2799849b77c08a8e6",
       "triggerType" : "PUSH"
     }, {
       "hash" : "05ce568b0b6ba14da9ca40d26b39ba478837d747",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1017",
       "triggerID" : "05ce568b0b6ba14da9ca40d26b39ba478837d747",
       "triggerType" : "PUSH"
     }, {
       "hash" : "840e3b33d39b1f86367bb6b0e65eadeb33de7765",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1025",
       "triggerID" : "840e3b33d39b1f86367bb6b0e65eadeb33de7765",
       "triggerType" : "PUSH"
     }, {
       "hash" : "8d2e2dd83f51d03e890d424a51d3e0e84e904a43",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1031",
       "triggerID" : "8d2e2dd83f51d03e890d424a51d3e0e84e904a43",
       "triggerType" : "PUSH"
     }, {
       "hash" : "688381914087ff11acf2f8c9372dbf39afa0bd40",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1035",
       "triggerID" : "688381914087ff11acf2f8c9372dbf39afa0bd40",
       "triggerType" : "PUSH"
     }, {
       "hash" : "fc2c138901e82ed5f3fbf363c2a0ce8140ade050",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "fc2c138901e82ed5f3fbf363c2a0ce8140ade050",
       "triggerType" : "PUSH"
     }, {
       "hash" : "6ed9e1cb1e5a191752d6d39c7a9d5d41840f0cdf",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1039",
       "triggerID" : "6ed9e1cb1e5a191752d6d39c7a9d5d41840f0cdf",
       "triggerType" : "PUSH"
     }, {
       "hash" : "a510a7de1458bb996df5bbb341e8d010e4c91381",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "a510a7de1458bb996df5bbb341e8d010e4c91381",
       "triggerType" : "PUSH"
     }, {
       "hash" : "49677397cd117228742205944606f7a789090376",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1042",
       "triggerID" : "49677397cd117228742205944606f7a789090376",
       "triggerType" : "PUSH"
     }, {
       "hash" : "1630db0a37422aed942ff280560a9d6329df5015",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1044",
       "triggerID" : "1630db0a37422aed942ff280560a9d6329df5015",
       "triggerType" : "PUSH"
     }, {
       "hash" : "f11c76301d4ab524db0af57aa831be3a898ea0fb",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1048",
       "triggerID" : "f11c76301d4ab524db0af57aa831be3a898ea0fb",
       "triggerType" : "PUSH"
     }, {
       "hash" : "d1a0dc4c2017853a7ba0c3302f886043a63f29da",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1050",
       "triggerID" : "d1a0dc4c2017853a7ba0c3302f886043a63f29da",
       "triggerType" : "PUSH"
     }, {
       "hash" : "6a4c15ec39ded1f90b113b272554217698a98e1c",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1087",
       "triggerID" : "6a4c15ec39ded1f90b113b272554217698a98e1c",
       "triggerType" : "PUSH"
     }, {
       "hash" : "288e730d645a5e58e50a87ef5f44ba11a8fb0533",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1092",
       "triggerID" : "288e730d645a5e58e50a87ef5f44ba11a8fb0533",
       "triggerType" : "PUSH"
     }, {
       "hash" : "70a5c21321a0da3a0be8d85e206c24befa7b89f1",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1095",
       "triggerID" : "70a5c21321a0da3a0be8d85e206c24befa7b89f1",
       "triggerType" : "PUSH"
     }, {
       "hash" : "f072a986114949c5e426ce64a5f5568fc718e25d",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "f072a986114949c5e426ce64a5f5568fc718e25d",
       "triggerType" : "PUSH"
     }, {
       "hash" : "45b27475b1e8806ec8c566d6f151f4bc79c8549d",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1096",
       "triggerID" : "45b27475b1e8806ec8c566d6f151f4bc79c8549d",
       "triggerType" : "PUSH"
     }, {
       "hash" : "3c62ef22f6cde28110e5eb2fca8760a008d77723",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "3c62ef22f6cde28110e5eb2fca8760a008d77723",
       "triggerType" : "PUSH"
     }, {
       "hash" : "dc785fa04e542b2ccc45494e0292f5153c7ecea8",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1097",
       "triggerID" : "dc785fa04e542b2ccc45494e0292f5153c7ecea8",
       "triggerType" : "PUSH"
     }, {
       "hash" : "585ddb8049b9b9ccb057027e265b08a1b586df8c",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "585ddb8049b9b9ccb057027e265b08a1b586df8c",
       "triggerType" : "PUSH"
     }, {
       "hash" : "b62ff8675beed2c4487c788b9ca1a0ad4271b913",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "b62ff8675beed2c4487c788b9ca1a0ad4271b913",
       "triggerType" : "PUSH"
     }, {
       "hash" : "2d46048a7402907f348cc96da2839ba1b3e4932a",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1099",
       "triggerID" : "2d46048a7402907f348cc96da2839ba1b3e4932a",
       "triggerType" : "PUSH"
     }, {
       "hash" : "004fb6ebf23f796a4b77650e0530f0f1145e8b91",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1101",
       "triggerID" : "004fb6ebf23f796a4b77650e0530f0f1145e8b91",
       "triggerType" : "PUSH"
     }, {
       "hash" : "af0c33333b5e9ae6d59f20a6db1d7c016ea015d3",
       "status" : "CANCELED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1165",
       "triggerID" : "af0c33333b5e9ae6d59f20a6db1d7c016ea015d3",
       "triggerType" : "PUSH"
     }, {
       "hash" : "da8111118d71306500229619d3b7996fcd2526e8",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "da8111118d71306500229619d3b7996fcd2526e8",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 9e668f65c6d4f73c76573236f8fa507dfb80edac UNKNOWN
   * 533f00e5546da0026bbe51e1c030ddd1c19fdd08 UNKNOWN
   * a2ee90c549332a7f17a7d9b2799849b77c08a8e6 UNKNOWN
   * fc2c138901e82ed5f3fbf363c2a0ce8140ade050 UNKNOWN
   * a510a7de1458bb996df5bbb341e8d010e4c91381 UNKNOWN
   * f072a986114949c5e426ce64a5f5568fc718e25d UNKNOWN
   * 3c62ef22f6cde28110e5eb2fca8760a008d77723 UNKNOWN
   * 585ddb8049b9b9ccb057027e265b08a1b586df8c UNKNOWN
   * b62ff8675beed2c4487c788b9ca1a0ad4271b913 UNKNOWN
   * af0c33333b5e9ae6d59f20a6db1d7c016ea015d3 Azure: [CANCELED](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1165) 
   * da8111118d71306500229619d3b7996fcd2526e8 UNKNOWN
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] [HUDI-8141] Incremental Query with Completion Time [hudi]

Posted by "yihua (via GitHub)" <gi...@apache.org>.

yihua commented on code in PR #11947:
URL: https://github.com/apache/hudi/pull/11947#discussion_r1802563846


##########
hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/spark/sql/hudi/streaming/HoodieStreamSource.scala:
##########
@@ -71,19 +69,6 @@ class HoodieStreamSource(
     parameters.get(DataSourceReadOptions.QUERY_TYPE.key).contains(DataSourceReadOptions.QUERY_TYPE_INCREMENTAL_OPT_VAL) &&
     parameters.get(DataSourceReadOptions.INCREMENTAL_FORMAT.key).contains(DataSourceReadOptions.INCREMENTAL_FORMAT_CDC_VAL)
 
-  /**
-   * When hollow commits are found while doing streaming read , unlike batch incremental query,
-   * we do not use [[HollowCommitHandling.FAIL]] by default, instead we use [[HollowCommitHandling.BLOCK]]
-   * to block processing data from going beyond the hollow commits to avoid unintentional skip.
-   *
-   * Users can set [[DataSourceReadOptions.INCREMENTAL_READ_HANDLE_HOLLOW_COMMIT]] to
-   * [[HollowCommitHandling.USE_TRANSITION_TIME]] to avoid the blocking behavior.
-   */
-  private val hollowCommitHandling: HollowCommitHandling =

Review Comment:
   We discussed this issue and agreed that it's simpler to deprecate the hollow commit handling in Hudi 1.0 as the completion time-based incremental query can properly support serializability; user should upgrade the table to Hudi 1.0 for the correct behavior.
   
   Regarding this PR, there are many places that need clean-up on the hollow commit handling.  I'll defer this to a separate PR.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] [HUDI-8141] Incremental Query with Completion Time [hudi]

Posted by "CTTY (via GitHub)" <gi...@apache.org>.

CTTY commented on code in PR #11947:
URL: https://github.com/apache/hudi/pull/11947#discussion_r1797786560


##########
hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/hudi/IncrementalRelation.scala:
##########
@@ -90,24 +78,25 @@ class IncrementalRelation(val sqlContext: SQLContext,
     throw new HoodieException("Incremental queries are not supported when meta fields are disabled")
   }
 
+  private val queryContext: IncrementalQueryAnalyzer.QueryContext =
+    IncrementalQueryAnalyzer.builder()
+      .metaClient(metaClient)
+      .startTime(optParams(DataSourceReadOptions.BEGIN_INSTANTTIME.key))
+      .endTime(optParams.getOrElse(DataSourceReadOptions.END_INSTANTTIME.key, null))
+      .rangeType(InstantRange.RangeType.OPEN_CLOSED)
+      .limit(optParams.getOrElse(
+        DataSourceReadOptions.INCREMENTAL_LIMIT.key,
+        DataSourceReadOptions.INCREMENTAL_LIMIT.defaultValue).toInt)

Review Comment:
   Yes, this is only used with Hudi streamer, I've moved this config back to `HoodieIncrSourceConfig`



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] [HUDI-8141] Incremental Query with Completion Time [hudi]

Posted by "CTTY (via GitHub)" <gi...@apache.org>.

CTTY commented on code in PR #11947:
URL: https://github.com/apache/hudi/pull/11947#discussion_r1797786059


##########
hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/hudi/HoodieMergeOnReadRDD.scala:
##########
@@ -82,7 +82,8 @@ class HoodieMergeOnReadRDD(@transient sc: SparkContext,
                            @transient fileSplits: Seq[HoodieMergeOnReadFileSplit],
                            includeStartTime: Boolean = false,
                            startTimestamp: String = null,
-                           endTimestamp: String = null)
+                           endTimestamp: String = null,
+                           includedTimestamps: Set[String] = null)

Review Comment:
   Yes, these are still using instant times. Because they are used to filter records based on `HoodieRecord.COMMIT_TIME_METADATA_FIELD` in each record.
   
   I'll fix the scaladoc.
   
   Also, I think we can remove `startTimestamp` and `endTimestamp` since they are not used anymore, will file a jira to track this 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] [HUDI-8141] Incremental Query with Completion Time [hudi]

Posted by "yihua (via GitHub)" <gi...@apache.org>.

yihua commented on code in PR #11947:
URL: https://github.com/apache/hudi/pull/11947#discussion_r1797787989


##########
hudi-spark-datasource/hudi-spark/src/test/scala/org/apache/hudi/functional/TestCOWDataSourceStorage.scala:
##########
@@ -185,7 +188,7 @@ class TestCOWDataSourceStorage extends SparkClientFunctionalTestHarness {
     val hoodieIncViewDF1 = spark.read.format("org.apache.hudi")
       .option(DataSourceReadOptions.QUERY_TYPE.key, DataSourceReadOptions.QUERY_TYPE_INCREMENTAL_OPT_VAL)
       .option(DataSourceReadOptions.BEGIN_INSTANTTIME.key, "000")
-      .option(DataSourceReadOptions.END_INSTANTTIME.key, firstCommit)
+      .option(DataSourceReadOptions.END_INSTANTTIME.key, completionTime1)

Review Comment:
   Should this be from `completionTime1` to `completionTime2`?



##########
hudi-spark-datasource/hudi-spark/src/test/scala/org/apache/hudi/functional/TestIncrementalReadWithFullTableScan.scala:
##########
@@ -70,9 +70,11 @@ class TestIncrementalReadWithFullTableScan extends HoodieSparkClientTestBase {
       HoodieWriteConfig.TBL_NAME.key -> "hoodie_test",
       HoodieMetadataConfig.COMPACT_NUM_DELTA_COMMITS.key -> "1"
     )
+
+

Review Comment:
   Remove redundant empty line.



##########
hudi-spark-datasource/hudi-spark/src/test/scala/org/apache/hudi/functional/TestIncrementalReadWithFullTableScan.scala:
##########
@@ -70,9 +70,11 @@ class TestIncrementalReadWithFullTableScan extends HoodieSparkClientTestBase {
       HoodieWriteConfig.TBL_NAME.key -> "hoodie_test",
       HoodieMetadataConfig.COMPACT_NUM_DELTA_COMMITS.key -> "1"
     )
+
+
     // Create 10 commits
     for (i <- 1 to 10) {
-      val records = recordsToStrings(dataGen.generateInserts("%05d".format(i), perBatchSize)).asScala.toList
+      val records = recordsToStrings(dataGen.generateInserts(System.currentTimeMillis(), perBatchSize)).asScala.toList

Review Comment:
   Have you added any new test cases that cover the scenarios where the completion time ordering is different than instant time ordering and make sure the incremental queries return the correct results, for both incremental and fallback full scan?



##########
hudi-spark-datasource/hudi-spark/src/test/scala/org/apache/hudi/functional/TestCOWDataSourceStorage.scala:
##########
@@ -232,7 +235,7 @@ class TestCOWDataSourceStorage extends SparkClientFunctionalTestHarness {
     val timeTravelDF = spark.read.format("org.apache.hudi")
       .option(DataSourceReadOptions.QUERY_TYPE.key, DataSourceReadOptions.QUERY_TYPE_INCREMENTAL_OPT_VAL)
       .option(DataSourceReadOptions.BEGIN_INSTANTTIME.key, "000")
-      .option(DataSourceReadOptions.END_INSTANTTIME.key, firstCommit)
+      .option(DataSourceReadOptions.END_INSTANTTIME.key, completionTime1)
       .load(basePath)

Review Comment:
   Should this be from `completionTime1` to `completionTime2`?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] [HUDI-8141] Incremental Query with Completion Time [hudi]

Posted by "CTTY (via GitHub)" <gi...@apache.org>.

CTTY commented on code in PR #11947:
URL: https://github.com/apache/hudi/pull/11947#discussion_r1797905952


##########
hudi-spark-datasource/hudi-spark/src/test/scala/org/apache/hudi/functional/TestCOWDataSourceStorage.scala:
##########
@@ -232,7 +235,7 @@ class TestCOWDataSourceStorage extends SparkClientFunctionalTestHarness {
     val timeTravelDF = spark.read.format("org.apache.hudi")
       .option(DataSourceReadOptions.QUERY_TYPE.key, DataSourceReadOptions.QUERY_TYPE_INCREMENTAL_OPT_VAL)
       .option(DataSourceReadOptions.BEGIN_INSTANTTIME.key, "000")
-      .option(DataSourceReadOptions.END_INSTANTTIME.key, firstCommit)
+      .option(DataSourceReadOptions.END_INSTANTTIME.key, completionTime1)
       .load(basePath)

Review Comment:
   Same as above, this range is supposed to be `(completionTime1, completionTime1]`. And any timestamp that's before completionTime1 would suffice as the start time



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] [HUDI-8141] Incremental Query with Completion Time [hudi]

Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.

hudi-bot commented on PR #11947:
URL: https://github.com/apache/hudi/pull/11947#issuecomment-2408796587

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "090f256a315cbbf0c6a54d93cc2739dd693360e6",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=484",
       "triggerID" : "090f256a315cbbf0c6a54d93cc2739dd693360e6",
       "triggerType" : "PUSH"
     }, {
       "hash" : "0df5649013c1cf9e782fcbb67b8a2e7174f94512",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=985",
       "triggerID" : "0df5649013c1cf9e782fcbb67b8a2e7174f94512",
       "triggerType" : "PUSH"
     }, {
       "hash" : "d52e0128f409918644255abbe4376dc4a780bfc8",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=986",
       "triggerID" : "d52e0128f409918644255abbe4376dc4a780bfc8",
       "triggerType" : "PUSH"
     }, {
       "hash" : "9e668f65c6d4f73c76573236f8fa507dfb80edac",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "9e668f65c6d4f73c76573236f8fa507dfb80edac",
       "triggerType" : "PUSH"
     }, {
       "hash" : "f145131d524bf190986e5de600a326e32a27beca",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=987",
       "triggerID" : "f145131d524bf190986e5de600a326e32a27beca",
       "triggerType" : "PUSH"
     }, {
       "hash" : "0b008f807b42aa6fb2686922bb88d63e84a1f299",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=993",
       "triggerID" : "0b008f807b42aa6fb2686922bb88d63e84a1f299",
       "triggerType" : "PUSH"
     }, {
       "hash" : "65c01ad9f2b33d36d4890355feef91844438e732",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=995",
       "triggerID" : "65c01ad9f2b33d36d4890355feef91844438e732",
       "triggerType" : "PUSH"
     }, {
       "hash" : "955493fdc93842e8c99529cec35e39cecdbe9638",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=996",
       "triggerID" : "955493fdc93842e8c99529cec35e39cecdbe9638",
       "triggerType" : "PUSH"
     }, {
       "hash" : "533f00e5546da0026bbe51e1c030ddd1c19fdd08",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "533f00e5546da0026bbe51e1c030ddd1c19fdd08",
       "triggerType" : "PUSH"
     }, {
       "hash" : "05777be67e37f3a6a8eaa59e54d9e17e5fc6ba9b",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1016",
       "triggerID" : "05777be67e37f3a6a8eaa59e54d9e17e5fc6ba9b",
       "triggerType" : "PUSH"
     }, {
       "hash" : "a2ee90c549332a7f17a7d9b2799849b77c08a8e6",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "a2ee90c549332a7f17a7d9b2799849b77c08a8e6",
       "triggerType" : "PUSH"
     }, {
       "hash" : "05ce568b0b6ba14da9ca40d26b39ba478837d747",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1017",
       "triggerID" : "05ce568b0b6ba14da9ca40d26b39ba478837d747",
       "triggerType" : "PUSH"
     }, {
       "hash" : "840e3b33d39b1f86367bb6b0e65eadeb33de7765",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1025",
       "triggerID" : "840e3b33d39b1f86367bb6b0e65eadeb33de7765",
       "triggerType" : "PUSH"
     }, {
       "hash" : "8d2e2dd83f51d03e890d424a51d3e0e84e904a43",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1031",
       "triggerID" : "8d2e2dd83f51d03e890d424a51d3e0e84e904a43",
       "triggerType" : "PUSH"
     }, {
       "hash" : "688381914087ff11acf2f8c9372dbf39afa0bd40",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1035",
       "triggerID" : "688381914087ff11acf2f8c9372dbf39afa0bd40",
       "triggerType" : "PUSH"
     }, {
       "hash" : "fc2c138901e82ed5f3fbf363c2a0ce8140ade050",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "fc2c138901e82ed5f3fbf363c2a0ce8140ade050",
       "triggerType" : "PUSH"
     }, {
       "hash" : "6ed9e1cb1e5a191752d6d39c7a9d5d41840f0cdf",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1039",
       "triggerID" : "6ed9e1cb1e5a191752d6d39c7a9d5d41840f0cdf",
       "triggerType" : "PUSH"
     }, {
       "hash" : "a510a7de1458bb996df5bbb341e8d010e4c91381",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "a510a7de1458bb996df5bbb341e8d010e4c91381",
       "triggerType" : "PUSH"
     }, {
       "hash" : "49677397cd117228742205944606f7a789090376",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1042",
       "triggerID" : "49677397cd117228742205944606f7a789090376",
       "triggerType" : "PUSH"
     }, {
       "hash" : "1630db0a37422aed942ff280560a9d6329df5015",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1044",
       "triggerID" : "1630db0a37422aed942ff280560a9d6329df5015",
       "triggerType" : "PUSH"
     }, {
       "hash" : "f11c76301d4ab524db0af57aa831be3a898ea0fb",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1048",
       "triggerID" : "f11c76301d4ab524db0af57aa831be3a898ea0fb",
       "triggerType" : "PUSH"
     }, {
       "hash" : "d1a0dc4c2017853a7ba0c3302f886043a63f29da",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1050",
       "triggerID" : "d1a0dc4c2017853a7ba0c3302f886043a63f29da",
       "triggerType" : "PUSH"
     }, {
       "hash" : "6a4c15ec39ded1f90b113b272554217698a98e1c",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1087",
       "triggerID" : "6a4c15ec39ded1f90b113b272554217698a98e1c",
       "triggerType" : "PUSH"
     }, {
       "hash" : "288e730d645a5e58e50a87ef5f44ba11a8fb0533",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1092",
       "triggerID" : "288e730d645a5e58e50a87ef5f44ba11a8fb0533",
       "triggerType" : "PUSH"
     }, {
       "hash" : "70a5c21321a0da3a0be8d85e206c24befa7b89f1",
       "status" : "CANCELED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1095",
       "triggerID" : "70a5c21321a0da3a0be8d85e206c24befa7b89f1",
       "triggerType" : "PUSH"
     }, {
       "hash" : "f072a986114949c5e426ce64a5f5568fc718e25d",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "f072a986114949c5e426ce64a5f5568fc718e25d",
       "triggerType" : "PUSH"
     }, {
       "hash" : "45b27475b1e8806ec8c566d6f151f4bc79c8549d",
       "status" : "PENDING",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1096",
       "triggerID" : "45b27475b1e8806ec8c566d6f151f4bc79c8549d",
       "triggerType" : "PUSH"
     }, {
       "hash" : "3c62ef22f6cde28110e5eb2fca8760a008d77723",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "3c62ef22f6cde28110e5eb2fca8760a008d77723",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 9e668f65c6d4f73c76573236f8fa507dfb80edac UNKNOWN
   * 533f00e5546da0026bbe51e1c030ddd1c19fdd08 UNKNOWN
   * a2ee90c549332a7f17a7d9b2799849b77c08a8e6 UNKNOWN
   * fc2c138901e82ed5f3fbf363c2a0ce8140ade050 UNKNOWN
   * a510a7de1458bb996df5bbb341e8d010e4c91381 UNKNOWN
   * 70a5c21321a0da3a0be8d85e206c24befa7b89f1 Azure: [CANCELED](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1095) 
   * f072a986114949c5e426ce64a5f5568fc718e25d UNKNOWN
   * 45b27475b1e8806ec8c566d6f151f4bc79c8549d Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1096) 
   * 3c62ef22f6cde28110e5eb2fca8760a008d77723 UNKNOWN
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] [HUDI-8141] Incremental Query with Completion Time [hudi]

Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.

hudi-bot commented on PR #11947:
URL: https://github.com/apache/hudi/pull/11947#issuecomment-2408826246

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "090f256a315cbbf0c6a54d93cc2739dd693360e6",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=484",
       "triggerID" : "090f256a315cbbf0c6a54d93cc2739dd693360e6",
       "triggerType" : "PUSH"
     }, {
       "hash" : "0df5649013c1cf9e782fcbb67b8a2e7174f94512",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=985",
       "triggerID" : "0df5649013c1cf9e782fcbb67b8a2e7174f94512",
       "triggerType" : "PUSH"
     }, {
       "hash" : "d52e0128f409918644255abbe4376dc4a780bfc8",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=986",
       "triggerID" : "d52e0128f409918644255abbe4376dc4a780bfc8",
       "triggerType" : "PUSH"
     }, {
       "hash" : "9e668f65c6d4f73c76573236f8fa507dfb80edac",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "9e668f65c6d4f73c76573236f8fa507dfb80edac",
       "triggerType" : "PUSH"
     }, {
       "hash" : "f145131d524bf190986e5de600a326e32a27beca",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=987",
       "triggerID" : "f145131d524bf190986e5de600a326e32a27beca",
       "triggerType" : "PUSH"
     }, {
       "hash" : "0b008f807b42aa6fb2686922bb88d63e84a1f299",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=993",
       "triggerID" : "0b008f807b42aa6fb2686922bb88d63e84a1f299",
       "triggerType" : "PUSH"
     }, {
       "hash" : "65c01ad9f2b33d36d4890355feef91844438e732",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=995",
       "triggerID" : "65c01ad9f2b33d36d4890355feef91844438e732",
       "triggerType" : "PUSH"
     }, {
       "hash" : "955493fdc93842e8c99529cec35e39cecdbe9638",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=996",
       "triggerID" : "955493fdc93842e8c99529cec35e39cecdbe9638",
       "triggerType" : "PUSH"
     }, {
       "hash" : "533f00e5546da0026bbe51e1c030ddd1c19fdd08",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "533f00e5546da0026bbe51e1c030ddd1c19fdd08",
       "triggerType" : "PUSH"
     }, {
       "hash" : "05777be67e37f3a6a8eaa59e54d9e17e5fc6ba9b",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1016",
       "triggerID" : "05777be67e37f3a6a8eaa59e54d9e17e5fc6ba9b",
       "triggerType" : "PUSH"
     }, {
       "hash" : "a2ee90c549332a7f17a7d9b2799849b77c08a8e6",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "a2ee90c549332a7f17a7d9b2799849b77c08a8e6",
       "triggerType" : "PUSH"
     }, {
       "hash" : "05ce568b0b6ba14da9ca40d26b39ba478837d747",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1017",
       "triggerID" : "05ce568b0b6ba14da9ca40d26b39ba478837d747",
       "triggerType" : "PUSH"
     }, {
       "hash" : "840e3b33d39b1f86367bb6b0e65eadeb33de7765",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1025",
       "triggerID" : "840e3b33d39b1f86367bb6b0e65eadeb33de7765",
       "triggerType" : "PUSH"
     }, {
       "hash" : "8d2e2dd83f51d03e890d424a51d3e0e84e904a43",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1031",
       "triggerID" : "8d2e2dd83f51d03e890d424a51d3e0e84e904a43",
       "triggerType" : "PUSH"
     }, {
       "hash" : "688381914087ff11acf2f8c9372dbf39afa0bd40",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1035",
       "triggerID" : "688381914087ff11acf2f8c9372dbf39afa0bd40",
       "triggerType" : "PUSH"
     }, {
       "hash" : "fc2c138901e82ed5f3fbf363c2a0ce8140ade050",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "fc2c138901e82ed5f3fbf363c2a0ce8140ade050",
       "triggerType" : "PUSH"
     }, {
       "hash" : "6ed9e1cb1e5a191752d6d39c7a9d5d41840f0cdf",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1039",
       "triggerID" : "6ed9e1cb1e5a191752d6d39c7a9d5d41840f0cdf",
       "triggerType" : "PUSH"
     }, {
       "hash" : "a510a7de1458bb996df5bbb341e8d010e4c91381",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "a510a7de1458bb996df5bbb341e8d010e4c91381",
       "triggerType" : "PUSH"
     }, {
       "hash" : "49677397cd117228742205944606f7a789090376",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1042",
       "triggerID" : "49677397cd117228742205944606f7a789090376",
       "triggerType" : "PUSH"
     }, {
       "hash" : "1630db0a37422aed942ff280560a9d6329df5015",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1044",
       "triggerID" : "1630db0a37422aed942ff280560a9d6329df5015",
       "triggerType" : "PUSH"
     }, {
       "hash" : "f11c76301d4ab524db0af57aa831be3a898ea0fb",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1048",
       "triggerID" : "f11c76301d4ab524db0af57aa831be3a898ea0fb",
       "triggerType" : "PUSH"
     }, {
       "hash" : "d1a0dc4c2017853a7ba0c3302f886043a63f29da",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1050",
       "triggerID" : "d1a0dc4c2017853a7ba0c3302f886043a63f29da",
       "triggerType" : "PUSH"
     }, {
       "hash" : "6a4c15ec39ded1f90b113b272554217698a98e1c",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1087",
       "triggerID" : "6a4c15ec39ded1f90b113b272554217698a98e1c",
       "triggerType" : "PUSH"
     }, {
       "hash" : "288e730d645a5e58e50a87ef5f44ba11a8fb0533",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1092",
       "triggerID" : "288e730d645a5e58e50a87ef5f44ba11a8fb0533",
       "triggerType" : "PUSH"
     }, {
       "hash" : "70a5c21321a0da3a0be8d85e206c24befa7b89f1",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1095",
       "triggerID" : "70a5c21321a0da3a0be8d85e206c24befa7b89f1",
       "triggerType" : "PUSH"
     }, {
       "hash" : "f072a986114949c5e426ce64a5f5568fc718e25d",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "f072a986114949c5e426ce64a5f5568fc718e25d",
       "triggerType" : "PUSH"
     }, {
       "hash" : "45b27475b1e8806ec8c566d6f151f4bc79c8549d",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1096",
       "triggerID" : "45b27475b1e8806ec8c566d6f151f4bc79c8549d",
       "triggerType" : "PUSH"
     }, {
       "hash" : "3c62ef22f6cde28110e5eb2fca8760a008d77723",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "3c62ef22f6cde28110e5eb2fca8760a008d77723",
       "triggerType" : "PUSH"
     }, {
       "hash" : "dc785fa04e542b2ccc45494e0292f5153c7ecea8",
       "status" : "FAILURE",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1097",
       "triggerID" : "dc785fa04e542b2ccc45494e0292f5153c7ecea8",
       "triggerType" : "PUSH"
     }, {
       "hash" : "585ddb8049b9b9ccb057027e265b08a1b586df8c",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "585ddb8049b9b9ccb057027e265b08a1b586df8c",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 9e668f65c6d4f73c76573236f8fa507dfb80edac UNKNOWN
   * 533f00e5546da0026bbe51e1c030ddd1c19fdd08 UNKNOWN
   * a2ee90c549332a7f17a7d9b2799849b77c08a8e6 UNKNOWN
   * fc2c138901e82ed5f3fbf363c2a0ce8140ade050 UNKNOWN
   * a510a7de1458bb996df5bbb341e8d010e4c91381 UNKNOWN
   * f072a986114949c5e426ce64a5f5568fc718e25d UNKNOWN
   * 3c62ef22f6cde28110e5eb2fca8760a008d77723 UNKNOWN
   * dc785fa04e542b2ccc45494e0292f5153c7ecea8 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1097) 
   * 585ddb8049b9b9ccb057027e265b08a1b586df8c UNKNOWN
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] [HUDI-8141] Incremental Query with Completion Time [hudi]

Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.

hudi-bot commented on PR #11947:
URL: https://github.com/apache/hudi/pull/11947#issuecomment-2421156648

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "090f256a315cbbf0c6a54d93cc2739dd693360e6",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=484",
       "triggerID" : "090f256a315cbbf0c6a54d93cc2739dd693360e6",
       "triggerType" : "PUSH"
     }, {
       "hash" : "0df5649013c1cf9e782fcbb67b8a2e7174f94512",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=985",
       "triggerID" : "0df5649013c1cf9e782fcbb67b8a2e7174f94512",
       "triggerType" : "PUSH"
     }, {
       "hash" : "d52e0128f409918644255abbe4376dc4a780bfc8",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=986",
       "triggerID" : "d52e0128f409918644255abbe4376dc4a780bfc8",
       "triggerType" : "PUSH"
     }, {
       "hash" : "9e668f65c6d4f73c76573236f8fa507dfb80edac",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "9e668f65c6d4f73c76573236f8fa507dfb80edac",
       "triggerType" : "PUSH"
     }, {
       "hash" : "f145131d524bf190986e5de600a326e32a27beca",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=987",
       "triggerID" : "f145131d524bf190986e5de600a326e32a27beca",
       "triggerType" : "PUSH"
     }, {
       "hash" : "0b008f807b42aa6fb2686922bb88d63e84a1f299",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=993",
       "triggerID" : "0b008f807b42aa6fb2686922bb88d63e84a1f299",
       "triggerType" : "PUSH"
     }, {
       "hash" : "65c01ad9f2b33d36d4890355feef91844438e732",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=995",
       "triggerID" : "65c01ad9f2b33d36d4890355feef91844438e732",
       "triggerType" : "PUSH"
     }, {
       "hash" : "955493fdc93842e8c99529cec35e39cecdbe9638",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=996",
       "triggerID" : "955493fdc93842e8c99529cec35e39cecdbe9638",
       "triggerType" : "PUSH"
     }, {
       "hash" : "533f00e5546da0026bbe51e1c030ddd1c19fdd08",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "533f00e5546da0026bbe51e1c030ddd1c19fdd08",
       "triggerType" : "PUSH"
     }, {
       "hash" : "05777be67e37f3a6a8eaa59e54d9e17e5fc6ba9b",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1016",
       "triggerID" : "05777be67e37f3a6a8eaa59e54d9e17e5fc6ba9b",
       "triggerType" : "PUSH"
     }, {
       "hash" : "a2ee90c549332a7f17a7d9b2799849b77c08a8e6",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "a2ee90c549332a7f17a7d9b2799849b77c08a8e6",
       "triggerType" : "PUSH"
     }, {
       "hash" : "05ce568b0b6ba14da9ca40d26b39ba478837d747",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1017",
       "triggerID" : "05ce568b0b6ba14da9ca40d26b39ba478837d747",
       "triggerType" : "PUSH"
     }, {
       "hash" : "840e3b33d39b1f86367bb6b0e65eadeb33de7765",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1025",
       "triggerID" : "840e3b33d39b1f86367bb6b0e65eadeb33de7765",
       "triggerType" : "PUSH"
     }, {
       "hash" : "8d2e2dd83f51d03e890d424a51d3e0e84e904a43",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1031",
       "triggerID" : "8d2e2dd83f51d03e890d424a51d3e0e84e904a43",
       "triggerType" : "PUSH"
     }, {
       "hash" : "688381914087ff11acf2f8c9372dbf39afa0bd40",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1035",
       "triggerID" : "688381914087ff11acf2f8c9372dbf39afa0bd40",
       "triggerType" : "PUSH"
     }, {
       "hash" : "fc2c138901e82ed5f3fbf363c2a0ce8140ade050",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "fc2c138901e82ed5f3fbf363c2a0ce8140ade050",
       "triggerType" : "PUSH"
     }, {
       "hash" : "6ed9e1cb1e5a191752d6d39c7a9d5d41840f0cdf",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1039",
       "triggerID" : "6ed9e1cb1e5a191752d6d39c7a9d5d41840f0cdf",
       "triggerType" : "PUSH"
     }, {
       "hash" : "a510a7de1458bb996df5bbb341e8d010e4c91381",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "a510a7de1458bb996df5bbb341e8d010e4c91381",
       "triggerType" : "PUSH"
     }, {
       "hash" : "49677397cd117228742205944606f7a789090376",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1042",
       "triggerID" : "49677397cd117228742205944606f7a789090376",
       "triggerType" : "PUSH"
     }, {
       "hash" : "1630db0a37422aed942ff280560a9d6329df5015",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1044",
       "triggerID" : "1630db0a37422aed942ff280560a9d6329df5015",
       "triggerType" : "PUSH"
     }, {
       "hash" : "f11c76301d4ab524db0af57aa831be3a898ea0fb",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1048",
       "triggerID" : "f11c76301d4ab524db0af57aa831be3a898ea0fb",
       "triggerType" : "PUSH"
     }, {
       "hash" : "d1a0dc4c2017853a7ba0c3302f886043a63f29da",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1050",
       "triggerID" : "d1a0dc4c2017853a7ba0c3302f886043a63f29da",
       "triggerType" : "PUSH"
     }, {
       "hash" : "6a4c15ec39ded1f90b113b272554217698a98e1c",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1087",
       "triggerID" : "6a4c15ec39ded1f90b113b272554217698a98e1c",
       "triggerType" : "PUSH"
     }, {
       "hash" : "288e730d645a5e58e50a87ef5f44ba11a8fb0533",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1092",
       "triggerID" : "288e730d645a5e58e50a87ef5f44ba11a8fb0533",
       "triggerType" : "PUSH"
     }, {
       "hash" : "70a5c21321a0da3a0be8d85e206c24befa7b89f1",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1095",
       "triggerID" : "70a5c21321a0da3a0be8d85e206c24befa7b89f1",
       "triggerType" : "PUSH"
     }, {
       "hash" : "f072a986114949c5e426ce64a5f5568fc718e25d",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "f072a986114949c5e426ce64a5f5568fc718e25d",
       "triggerType" : "PUSH"
     }, {
       "hash" : "45b27475b1e8806ec8c566d6f151f4bc79c8549d",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1096",
       "triggerID" : "45b27475b1e8806ec8c566d6f151f4bc79c8549d",
       "triggerType" : "PUSH"
     }, {
       "hash" : "3c62ef22f6cde28110e5eb2fca8760a008d77723",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "3c62ef22f6cde28110e5eb2fca8760a008d77723",
       "triggerType" : "PUSH"
     }, {
       "hash" : "dc785fa04e542b2ccc45494e0292f5153c7ecea8",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1097",
       "triggerID" : "dc785fa04e542b2ccc45494e0292f5153c7ecea8",
       "triggerType" : "PUSH"
     }, {
       "hash" : "585ddb8049b9b9ccb057027e265b08a1b586df8c",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "585ddb8049b9b9ccb057027e265b08a1b586df8c",
       "triggerType" : "PUSH"
     }, {
       "hash" : "b62ff8675beed2c4487c788b9ca1a0ad4271b913",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "b62ff8675beed2c4487c788b9ca1a0ad4271b913",
       "triggerType" : "PUSH"
     }, {
       "hash" : "2d46048a7402907f348cc96da2839ba1b3e4932a",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1099",
       "triggerID" : "2d46048a7402907f348cc96da2839ba1b3e4932a",
       "triggerType" : "PUSH"
     }, {
       "hash" : "004fb6ebf23f796a4b77650e0530f0f1145e8b91",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1101",
       "triggerID" : "004fb6ebf23f796a4b77650e0530f0f1145e8b91",
       "triggerType" : "PUSH"
     }, {
       "hash" : "af0c33333b5e9ae6d59f20a6db1d7c016ea015d3",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1165",
       "triggerID" : "af0c33333b5e9ae6d59f20a6db1d7c016ea015d3",
       "triggerType" : "PUSH"
     }, {
       "hash" : "da8111118d71306500229619d3b7996fcd2526e8",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "da8111118d71306500229619d3b7996fcd2526e8",
       "triggerType" : "PUSH"
     }, {
       "hash" : "83257e2b9bd3c426cf90e11003a5f244a750d7fa",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1167",
       "triggerID" : "83257e2b9bd3c426cf90e11003a5f244a750d7fa",
       "triggerType" : "PUSH"
     }, {
       "hash" : "e0de408f1b8153fdaaea4c87b27a7b69213b75b5",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1177",
       "triggerID" : "e0de408f1b8153fdaaea4c87b27a7b69213b75b5",
       "triggerType" : "PUSH"
     }, {
       "hash" : "3f7951f7ad7931f5dc0b977d7c33bc5429623535",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1180",
       "triggerID" : "3f7951f7ad7931f5dc0b977d7c33bc5429623535",
       "triggerType" : "PUSH"
     }, {
       "hash" : "9cd6f06a753b63630e8245d51338d2bf9ab86b84",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "9cd6f06a753b63630e8245d51338d2bf9ab86b84",
       "triggerType" : "PUSH"
     }, {
       "hash" : "7380b902731a92e0645c02397e14591a05798620",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "7380b902731a92e0645c02397e14591a05798620",
       "triggerType" : "PUSH"
     }, {
       "hash" : "0bd5cca01a541845081d5430d48b021d4bdf0969",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "0bd5cca01a541845081d5430d48b021d4bdf0969",
       "triggerType" : "PUSH"
     }, {
       "hash" : "8f4e175ef554f015e34e0968b656b5de714ab430",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1182",
       "triggerID" : "8f4e175ef554f015e34e0968b656b5de714ab430",
       "triggerType" : "PUSH"
     }, {
       "hash" : "2023f3c882d0c2f5b96e28d2f21812dc0c64aaee",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "2023f3c882d0c2f5b96e28d2f21812dc0c64aaee",
       "triggerType" : "PUSH"
     }, {
       "hash" : "8394acf060d5c3953dbd67cf930b9b1d3b92d8b5",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1198",
       "triggerID" : "8394acf060d5c3953dbd67cf930b9b1d3b92d8b5",
       "triggerType" : "PUSH"
     }, {
       "hash" : "7ec42c105971bff463e359019405a7f4e7947d86",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "7ec42c105971bff463e359019405a7f4e7947d86",
       "triggerType" : "PUSH"
     }, {
       "hash" : "4b930786a7b2de09ac1653cd884ee76a7d1b2ca5",
       "status" : "SUCCESS",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1202",
       "triggerID" : "4b930786a7b2de09ac1653cd884ee76a7d1b2ca5",
       "triggerType" : "PUSH"
     }, {
       "hash" : "0e9a648c4e03b5a74d6a0085831805041a9a26a2",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "0e9a648c4e03b5a74d6a0085831805041a9a26a2",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 9e668f65c6d4f73c76573236f8fa507dfb80edac UNKNOWN
   * 533f00e5546da0026bbe51e1c030ddd1c19fdd08 UNKNOWN
   * a2ee90c549332a7f17a7d9b2799849b77c08a8e6 UNKNOWN
   * fc2c138901e82ed5f3fbf363c2a0ce8140ade050 UNKNOWN
   * a510a7de1458bb996df5bbb341e8d010e4c91381 UNKNOWN
   * f072a986114949c5e426ce64a5f5568fc718e25d UNKNOWN
   * 3c62ef22f6cde28110e5eb2fca8760a008d77723 UNKNOWN
   * 585ddb8049b9b9ccb057027e265b08a1b586df8c UNKNOWN
   * b62ff8675beed2c4487c788b9ca1a0ad4271b913 UNKNOWN
   * da8111118d71306500229619d3b7996fcd2526e8 UNKNOWN
   * 9cd6f06a753b63630e8245d51338d2bf9ab86b84 UNKNOWN
   * 7380b902731a92e0645c02397e14591a05798620 UNKNOWN
   * 0bd5cca01a541845081d5430d48b021d4bdf0969 UNKNOWN
   * 2023f3c882d0c2f5b96e28d2f21812dc0c64aaee UNKNOWN
   * 7ec42c105971bff463e359019405a7f4e7947d86 UNKNOWN
   * 4b930786a7b2de09ac1653cd884ee76a7d1b2ca5 Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1202) 
   * 0e9a648c4e03b5a74d6a0085831805041a9a26a2 UNKNOWN
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] [HUDI-8141] Incremental Query with Completion Time [hudi]

Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.

hudi-bot commented on PR #11947:
URL: https://github.com/apache/hudi/pull/11947#issuecomment-2421369797

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "090f256a315cbbf0c6a54d93cc2739dd693360e6",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=484",
       "triggerID" : "090f256a315cbbf0c6a54d93cc2739dd693360e6",
       "triggerType" : "PUSH"
     }, {
       "hash" : "0df5649013c1cf9e782fcbb67b8a2e7174f94512",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=985",
       "triggerID" : "0df5649013c1cf9e782fcbb67b8a2e7174f94512",
       "triggerType" : "PUSH"
     }, {
       "hash" : "d52e0128f409918644255abbe4376dc4a780bfc8",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=986",
       "triggerID" : "d52e0128f409918644255abbe4376dc4a780bfc8",
       "triggerType" : "PUSH"
     }, {
       "hash" : "9e668f65c6d4f73c76573236f8fa507dfb80edac",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "9e668f65c6d4f73c76573236f8fa507dfb80edac",
       "triggerType" : "PUSH"
     }, {
       "hash" : "f145131d524bf190986e5de600a326e32a27beca",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=987",
       "triggerID" : "f145131d524bf190986e5de600a326e32a27beca",
       "triggerType" : "PUSH"
     }, {
       "hash" : "0b008f807b42aa6fb2686922bb88d63e84a1f299",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=993",
       "triggerID" : "0b008f807b42aa6fb2686922bb88d63e84a1f299",
       "triggerType" : "PUSH"
     }, {
       "hash" : "65c01ad9f2b33d36d4890355feef91844438e732",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=995",
       "triggerID" : "65c01ad9f2b33d36d4890355feef91844438e732",
       "triggerType" : "PUSH"
     }, {
       "hash" : "955493fdc93842e8c99529cec35e39cecdbe9638",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=996",
       "triggerID" : "955493fdc93842e8c99529cec35e39cecdbe9638",
       "triggerType" : "PUSH"
     }, {
       "hash" : "533f00e5546da0026bbe51e1c030ddd1c19fdd08",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "533f00e5546da0026bbe51e1c030ddd1c19fdd08",
       "triggerType" : "PUSH"
     }, {
       "hash" : "05777be67e37f3a6a8eaa59e54d9e17e5fc6ba9b",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1016",
       "triggerID" : "05777be67e37f3a6a8eaa59e54d9e17e5fc6ba9b",
       "triggerType" : "PUSH"
     }, {
       "hash" : "a2ee90c549332a7f17a7d9b2799849b77c08a8e6",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "a2ee90c549332a7f17a7d9b2799849b77c08a8e6",
       "triggerType" : "PUSH"
     }, {
       "hash" : "05ce568b0b6ba14da9ca40d26b39ba478837d747",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1017",
       "triggerID" : "05ce568b0b6ba14da9ca40d26b39ba478837d747",
       "triggerType" : "PUSH"
     }, {
       "hash" : "840e3b33d39b1f86367bb6b0e65eadeb33de7765",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1025",
       "triggerID" : "840e3b33d39b1f86367bb6b0e65eadeb33de7765",
       "triggerType" : "PUSH"
     }, {
       "hash" : "8d2e2dd83f51d03e890d424a51d3e0e84e904a43",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1031",
       "triggerID" : "8d2e2dd83f51d03e890d424a51d3e0e84e904a43",
       "triggerType" : "PUSH"
     }, {
       "hash" : "688381914087ff11acf2f8c9372dbf39afa0bd40",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1035",
       "triggerID" : "688381914087ff11acf2f8c9372dbf39afa0bd40",
       "triggerType" : "PUSH"
     }, {
       "hash" : "fc2c138901e82ed5f3fbf363c2a0ce8140ade050",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "fc2c138901e82ed5f3fbf363c2a0ce8140ade050",
       "triggerType" : "PUSH"
     }, {
       "hash" : "6ed9e1cb1e5a191752d6d39c7a9d5d41840f0cdf",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1039",
       "triggerID" : "6ed9e1cb1e5a191752d6d39c7a9d5d41840f0cdf",
       "triggerType" : "PUSH"
     }, {
       "hash" : "a510a7de1458bb996df5bbb341e8d010e4c91381",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "a510a7de1458bb996df5bbb341e8d010e4c91381",
       "triggerType" : "PUSH"
     }, {
       "hash" : "49677397cd117228742205944606f7a789090376",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1042",
       "triggerID" : "49677397cd117228742205944606f7a789090376",
       "triggerType" : "PUSH"
     }, {
       "hash" : "1630db0a37422aed942ff280560a9d6329df5015",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1044",
       "triggerID" : "1630db0a37422aed942ff280560a9d6329df5015",
       "triggerType" : "PUSH"
     }, {
       "hash" : "f11c76301d4ab524db0af57aa831be3a898ea0fb",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1048",
       "triggerID" : "f11c76301d4ab524db0af57aa831be3a898ea0fb",
       "triggerType" : "PUSH"
     }, {
       "hash" : "d1a0dc4c2017853a7ba0c3302f886043a63f29da",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1050",
       "triggerID" : "d1a0dc4c2017853a7ba0c3302f886043a63f29da",
       "triggerType" : "PUSH"
     }, {
       "hash" : "6a4c15ec39ded1f90b113b272554217698a98e1c",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1087",
       "triggerID" : "6a4c15ec39ded1f90b113b272554217698a98e1c",
       "triggerType" : "PUSH"
     }, {
       "hash" : "288e730d645a5e58e50a87ef5f44ba11a8fb0533",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1092",
       "triggerID" : "288e730d645a5e58e50a87ef5f44ba11a8fb0533",
       "triggerType" : "PUSH"
     }, {
       "hash" : "70a5c21321a0da3a0be8d85e206c24befa7b89f1",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1095",
       "triggerID" : "70a5c21321a0da3a0be8d85e206c24befa7b89f1",
       "triggerType" : "PUSH"
     }, {
       "hash" : "f072a986114949c5e426ce64a5f5568fc718e25d",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "f072a986114949c5e426ce64a5f5568fc718e25d",
       "triggerType" : "PUSH"
     }, {
       "hash" : "45b27475b1e8806ec8c566d6f151f4bc79c8549d",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1096",
       "triggerID" : "45b27475b1e8806ec8c566d6f151f4bc79c8549d",
       "triggerType" : "PUSH"
     }, {
       "hash" : "3c62ef22f6cde28110e5eb2fca8760a008d77723",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "3c62ef22f6cde28110e5eb2fca8760a008d77723",
       "triggerType" : "PUSH"
     }, {
       "hash" : "dc785fa04e542b2ccc45494e0292f5153c7ecea8",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1097",
       "triggerID" : "dc785fa04e542b2ccc45494e0292f5153c7ecea8",
       "triggerType" : "PUSH"
     }, {
       "hash" : "585ddb8049b9b9ccb057027e265b08a1b586df8c",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "585ddb8049b9b9ccb057027e265b08a1b586df8c",
       "triggerType" : "PUSH"
     }, {
       "hash" : "b62ff8675beed2c4487c788b9ca1a0ad4271b913",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "b62ff8675beed2c4487c788b9ca1a0ad4271b913",
       "triggerType" : "PUSH"
     }, {
       "hash" : "2d46048a7402907f348cc96da2839ba1b3e4932a",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1099",
       "triggerID" : "2d46048a7402907f348cc96da2839ba1b3e4932a",
       "triggerType" : "PUSH"
     }, {
       "hash" : "004fb6ebf23f796a4b77650e0530f0f1145e8b91",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1101",
       "triggerID" : "004fb6ebf23f796a4b77650e0530f0f1145e8b91",
       "triggerType" : "PUSH"
     }, {
       "hash" : "af0c33333b5e9ae6d59f20a6db1d7c016ea015d3",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1165",
       "triggerID" : "af0c33333b5e9ae6d59f20a6db1d7c016ea015d3",
       "triggerType" : "PUSH"
     }, {
       "hash" : "da8111118d71306500229619d3b7996fcd2526e8",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "da8111118d71306500229619d3b7996fcd2526e8",
       "triggerType" : "PUSH"
     }, {
       "hash" : "83257e2b9bd3c426cf90e11003a5f244a750d7fa",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1167",
       "triggerID" : "83257e2b9bd3c426cf90e11003a5f244a750d7fa",
       "triggerType" : "PUSH"
     }, {
       "hash" : "e0de408f1b8153fdaaea4c87b27a7b69213b75b5",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1177",
       "triggerID" : "e0de408f1b8153fdaaea4c87b27a7b69213b75b5",
       "triggerType" : "PUSH"
     }, {
       "hash" : "3f7951f7ad7931f5dc0b977d7c33bc5429623535",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1180",
       "triggerID" : "3f7951f7ad7931f5dc0b977d7c33bc5429623535",
       "triggerType" : "PUSH"
     }, {
       "hash" : "9cd6f06a753b63630e8245d51338d2bf9ab86b84",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "9cd6f06a753b63630e8245d51338d2bf9ab86b84",
       "triggerType" : "PUSH"
     }, {
       "hash" : "7380b902731a92e0645c02397e14591a05798620",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "7380b902731a92e0645c02397e14591a05798620",
       "triggerType" : "PUSH"
     }, {
       "hash" : "0bd5cca01a541845081d5430d48b021d4bdf0969",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "0bd5cca01a541845081d5430d48b021d4bdf0969",
       "triggerType" : "PUSH"
     }, {
       "hash" : "8f4e175ef554f015e34e0968b656b5de714ab430",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1182",
       "triggerID" : "8f4e175ef554f015e34e0968b656b5de714ab430",
       "triggerType" : "PUSH"
     }, {
       "hash" : "2023f3c882d0c2f5b96e28d2f21812dc0c64aaee",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "2023f3c882d0c2f5b96e28d2f21812dc0c64aaee",
       "triggerType" : "PUSH"
     }, {
       "hash" : "8394acf060d5c3953dbd67cf930b9b1d3b92d8b5",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1198",
       "triggerID" : "8394acf060d5c3953dbd67cf930b9b1d3b92d8b5",
       "triggerType" : "PUSH"
     }, {
       "hash" : "7ec42c105971bff463e359019405a7f4e7947d86",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "7ec42c105971bff463e359019405a7f4e7947d86",
       "triggerType" : "PUSH"
     }, {
       "hash" : "4b930786a7b2de09ac1653cd884ee76a7d1b2ca5",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1202",
       "triggerID" : "4b930786a7b2de09ac1653cd884ee76a7d1b2ca5",
       "triggerType" : "PUSH"
     }, {
       "hash" : "0e9a648c4e03b5a74d6a0085831805041a9a26a2",
       "status" : "SUCCESS",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1216",
       "triggerID" : "0e9a648c4e03b5a74d6a0085831805041a9a26a2",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 9e668f65c6d4f73c76573236f8fa507dfb80edac UNKNOWN
   * 533f00e5546da0026bbe51e1c030ddd1c19fdd08 UNKNOWN
   * a2ee90c549332a7f17a7d9b2799849b77c08a8e6 UNKNOWN
   * fc2c138901e82ed5f3fbf363c2a0ce8140ade050 UNKNOWN
   * a510a7de1458bb996df5bbb341e8d010e4c91381 UNKNOWN
   * f072a986114949c5e426ce64a5f5568fc718e25d UNKNOWN
   * 3c62ef22f6cde28110e5eb2fca8760a008d77723 UNKNOWN
   * 585ddb8049b9b9ccb057027e265b08a1b586df8c UNKNOWN
   * b62ff8675beed2c4487c788b9ca1a0ad4271b913 UNKNOWN
   * da8111118d71306500229619d3b7996fcd2526e8 UNKNOWN
   * 9cd6f06a753b63630e8245d51338d2bf9ab86b84 UNKNOWN
   * 7380b902731a92e0645c02397e14591a05798620 UNKNOWN
   * 0bd5cca01a541845081d5430d48b021d4bdf0969 UNKNOWN
   * 2023f3c882d0c2f5b96e28d2f21812dc0c64aaee UNKNOWN
   * 7ec42c105971bff463e359019405a7f4e7947d86 UNKNOWN
   * 0e9a648c4e03b5a74d6a0085831805041a9a26a2 Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1216) 
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] [HUDI-8141] Incremental Query with Completion Time [hudi]

Posted by "yihua (via GitHub)" <gi...@apache.org>.

yihua commented on code in PR #11947:
URL: https://github.com/apache/hudi/pull/11947#discussion_r1806020254


##########
hudi-utilities/src/main/java/org/apache/hudi/utilities/sources/HoodieIncrSource.java:
##########
@@ -175,33 +177,27 @@ public Pair<Option<Dataset<Row>>, String> fetchNextBatch(Option<String> lastCkpt
     IncrSourceHelper.MissingCheckpointStrategy missingCheckpointStrategy =
         (containsConfigProperty(props, HoodieIncrSourceConfig.MISSING_CHECKPOINT_STRATEGY))
             ? IncrSourceHelper.MissingCheckpointStrategy.valueOf(
-            getStringWithAltKeys(props, HoodieIncrSourceConfig.MISSING_CHECKPOINT_STRATEGY))
+                getStringWithAltKeys(props, HoodieIncrSourceConfig.MISSING_CHECKPOINT_STRATEGY))
             : null;
     if (readLatestOnMissingCkpt) {
       missingCheckpointStrategy = IncrSourceHelper.MissingCheckpointStrategy.READ_LATEST;
     }
 
-    // Use begin Instant if set and non-empty
-    Option<String> beginInstant =
-        lastCkptStr.isPresent() ? lastCkptStr.get().isEmpty() ? Option.empty() : lastCkptStr : Option.empty();
+    IncrementalQueryAnalyzer analyzer = IncrSourceHelper.getIncrementalQueryAnalyzer(
+        sparkContext, srcPath, lastCkptStr, missingCheckpointStrategy,
+        getIntWithAltKeys(props, HoodieIncrSourceConfig.NUM_INSTANTS_PER_FETCH),
+        getLatestSourceProfile());
+    QueryContext queryContext = analyzer.analyze();
+    Option<InstantRange> instantRange = queryContext.getInstantRange();
 
-    // If source profile exists, use the numInstants from source profile.
-    final int numInstantsFromConfig = getIntWithAltKeys(props, HoodieIncrSourceConfig.NUM_INSTANTS_PER_FETCH);
-    int numInstantsPerFetch = getLatestSourceProfile().map(sourceProfile -> {
-      int numInstantsFromSourceProfile = sourceProfile.getSourceSpecificContext();
-      LOG.info("Overriding numInstantsPerFetch from source profile numInstantsFromSourceProfile {} , numInstantsFromConfig {}", numInstantsFromSourceProfile, numInstantsFromConfig);
-      return numInstantsFromSourceProfile;
-    }).orElse(numInstantsFromConfig);
-
-    HollowCommitHandling handlingMode = getHollowCommitHandleMode(props);
-    QueryInfo queryInfo = generateQueryInfo(sparkContext, srcPath,
-        numInstantsPerFetch, beginInstant, missingCheckpointStrategy, handlingMode,
-        HoodieRecord.COMMIT_TIME_METADATA_FIELD, HoodieRecord.RECORD_KEY_METADATA_FIELD,
-        null, false, Option.empty());
-
-    if (queryInfo.areStartAndEndInstantsEqual()) {
+    String endCompletionTime;
+    // analyzer.getBeginCompletionTime() is empty only when reading the latest instant
+    // in the first batch
+    if (queryContext.isEmpty()
+        || (endCompletionTime = queryContext.getMaxCompletionTime())
+        .equals(analyzer.getStartCompletionTime().orElseGet(() -> null))) {
       LOG.info("Already caught up. No new data to process");
-      return Pair.of(Option.empty(), queryInfo.getEndInstant());
+      return Pair.of(Option.empty(), analyzer.getStartCompletionTime().get());

Review Comment:
   `analyzer.getStartCompletionTime().get()` and `queryContext.getMaxCompletionTime()` are the same in this case, or queryContext is empty.  I changed it to `lastCkptStr.orElse(null)` for readability.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] [HUDI-8141] Incremental Query with Completion Time [hudi]

Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.

hudi-bot commented on PR #11947:
URL: https://github.com/apache/hudi/pull/11947#issuecomment-2421698240

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "090f256a315cbbf0c6a54d93cc2739dd693360e6",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=484",
       "triggerID" : "090f256a315cbbf0c6a54d93cc2739dd693360e6",
       "triggerType" : "PUSH"
     }, {
       "hash" : "0df5649013c1cf9e782fcbb67b8a2e7174f94512",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=985",
       "triggerID" : "0df5649013c1cf9e782fcbb67b8a2e7174f94512",
       "triggerType" : "PUSH"
     }, {
       "hash" : "d52e0128f409918644255abbe4376dc4a780bfc8",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=986",
       "triggerID" : "d52e0128f409918644255abbe4376dc4a780bfc8",
       "triggerType" : "PUSH"
     }, {
       "hash" : "9e668f65c6d4f73c76573236f8fa507dfb80edac",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "9e668f65c6d4f73c76573236f8fa507dfb80edac",
       "triggerType" : "PUSH"
     }, {
       "hash" : "f145131d524bf190986e5de600a326e32a27beca",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=987",
       "triggerID" : "f145131d524bf190986e5de600a326e32a27beca",
       "triggerType" : "PUSH"
     }, {
       "hash" : "0b008f807b42aa6fb2686922bb88d63e84a1f299",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=993",
       "triggerID" : "0b008f807b42aa6fb2686922bb88d63e84a1f299",
       "triggerType" : "PUSH"
     }, {
       "hash" : "65c01ad9f2b33d36d4890355feef91844438e732",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=995",
       "triggerID" : "65c01ad9f2b33d36d4890355feef91844438e732",
       "triggerType" : "PUSH"
     }, {
       "hash" : "955493fdc93842e8c99529cec35e39cecdbe9638",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=996",
       "triggerID" : "955493fdc93842e8c99529cec35e39cecdbe9638",
       "triggerType" : "PUSH"
     }, {
       "hash" : "533f00e5546da0026bbe51e1c030ddd1c19fdd08",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "533f00e5546da0026bbe51e1c030ddd1c19fdd08",
       "triggerType" : "PUSH"
     }, {
       "hash" : "05777be67e37f3a6a8eaa59e54d9e17e5fc6ba9b",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1016",
       "triggerID" : "05777be67e37f3a6a8eaa59e54d9e17e5fc6ba9b",
       "triggerType" : "PUSH"
     }, {
       "hash" : "a2ee90c549332a7f17a7d9b2799849b77c08a8e6",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "a2ee90c549332a7f17a7d9b2799849b77c08a8e6",
       "triggerType" : "PUSH"
     }, {
       "hash" : "05ce568b0b6ba14da9ca40d26b39ba478837d747",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1017",
       "triggerID" : "05ce568b0b6ba14da9ca40d26b39ba478837d747",
       "triggerType" : "PUSH"
     }, {
       "hash" : "840e3b33d39b1f86367bb6b0e65eadeb33de7765",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1025",
       "triggerID" : "840e3b33d39b1f86367bb6b0e65eadeb33de7765",
       "triggerType" : "PUSH"
     }, {
       "hash" : "8d2e2dd83f51d03e890d424a51d3e0e84e904a43",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1031",
       "triggerID" : "8d2e2dd83f51d03e890d424a51d3e0e84e904a43",
       "triggerType" : "PUSH"
     }, {
       "hash" : "688381914087ff11acf2f8c9372dbf39afa0bd40",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1035",
       "triggerID" : "688381914087ff11acf2f8c9372dbf39afa0bd40",
       "triggerType" : "PUSH"
     }, {
       "hash" : "fc2c138901e82ed5f3fbf363c2a0ce8140ade050",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "fc2c138901e82ed5f3fbf363c2a0ce8140ade050",
       "triggerType" : "PUSH"
     }, {
       "hash" : "6ed9e1cb1e5a191752d6d39c7a9d5d41840f0cdf",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1039",
       "triggerID" : "6ed9e1cb1e5a191752d6d39c7a9d5d41840f0cdf",
       "triggerType" : "PUSH"
     }, {
       "hash" : "a510a7de1458bb996df5bbb341e8d010e4c91381",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "a510a7de1458bb996df5bbb341e8d010e4c91381",
       "triggerType" : "PUSH"
     }, {
       "hash" : "49677397cd117228742205944606f7a789090376",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1042",
       "triggerID" : "49677397cd117228742205944606f7a789090376",
       "triggerType" : "PUSH"
     }, {
       "hash" : "1630db0a37422aed942ff280560a9d6329df5015",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1044",
       "triggerID" : "1630db0a37422aed942ff280560a9d6329df5015",
       "triggerType" : "PUSH"
     }, {
       "hash" : "f11c76301d4ab524db0af57aa831be3a898ea0fb",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1048",
       "triggerID" : "f11c76301d4ab524db0af57aa831be3a898ea0fb",
       "triggerType" : "PUSH"
     }, {
       "hash" : "d1a0dc4c2017853a7ba0c3302f886043a63f29da",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1050",
       "triggerID" : "d1a0dc4c2017853a7ba0c3302f886043a63f29da",
       "triggerType" : "PUSH"
     }, {
       "hash" : "6a4c15ec39ded1f90b113b272554217698a98e1c",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1087",
       "triggerID" : "6a4c15ec39ded1f90b113b272554217698a98e1c",
       "triggerType" : "PUSH"
     }, {
       "hash" : "288e730d645a5e58e50a87ef5f44ba11a8fb0533",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1092",
       "triggerID" : "288e730d645a5e58e50a87ef5f44ba11a8fb0533",
       "triggerType" : "PUSH"
     }, {
       "hash" : "70a5c21321a0da3a0be8d85e206c24befa7b89f1",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1095",
       "triggerID" : "70a5c21321a0da3a0be8d85e206c24befa7b89f1",
       "triggerType" : "PUSH"
     }, {
       "hash" : "f072a986114949c5e426ce64a5f5568fc718e25d",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "f072a986114949c5e426ce64a5f5568fc718e25d",
       "triggerType" : "PUSH"
     }, {
       "hash" : "45b27475b1e8806ec8c566d6f151f4bc79c8549d",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1096",
       "triggerID" : "45b27475b1e8806ec8c566d6f151f4bc79c8549d",
       "triggerType" : "PUSH"
     }, {
       "hash" : "3c62ef22f6cde28110e5eb2fca8760a008d77723",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "3c62ef22f6cde28110e5eb2fca8760a008d77723",
       "triggerType" : "PUSH"
     }, {
       "hash" : "dc785fa04e542b2ccc45494e0292f5153c7ecea8",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1097",
       "triggerID" : "dc785fa04e542b2ccc45494e0292f5153c7ecea8",
       "triggerType" : "PUSH"
     }, {
       "hash" : "585ddb8049b9b9ccb057027e265b08a1b586df8c",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "585ddb8049b9b9ccb057027e265b08a1b586df8c",
       "triggerType" : "PUSH"
     }, {
       "hash" : "b62ff8675beed2c4487c788b9ca1a0ad4271b913",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "b62ff8675beed2c4487c788b9ca1a0ad4271b913",
       "triggerType" : "PUSH"
     }, {
       "hash" : "2d46048a7402907f348cc96da2839ba1b3e4932a",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1099",
       "triggerID" : "2d46048a7402907f348cc96da2839ba1b3e4932a",
       "triggerType" : "PUSH"
     }, {
       "hash" : "004fb6ebf23f796a4b77650e0530f0f1145e8b91",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1101",
       "triggerID" : "004fb6ebf23f796a4b77650e0530f0f1145e8b91",
       "triggerType" : "PUSH"
     }, {
       "hash" : "af0c33333b5e9ae6d59f20a6db1d7c016ea015d3",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1165",
       "triggerID" : "af0c33333b5e9ae6d59f20a6db1d7c016ea015d3",
       "triggerType" : "PUSH"
     }, {
       "hash" : "da8111118d71306500229619d3b7996fcd2526e8",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "da8111118d71306500229619d3b7996fcd2526e8",
       "triggerType" : "PUSH"
     }, {
       "hash" : "83257e2b9bd3c426cf90e11003a5f244a750d7fa",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1167",
       "triggerID" : "83257e2b9bd3c426cf90e11003a5f244a750d7fa",
       "triggerType" : "PUSH"
     }, {
       "hash" : "e0de408f1b8153fdaaea4c87b27a7b69213b75b5",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1177",
       "triggerID" : "e0de408f1b8153fdaaea4c87b27a7b69213b75b5",
       "triggerType" : "PUSH"
     }, {
       "hash" : "3f7951f7ad7931f5dc0b977d7c33bc5429623535",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1180",
       "triggerID" : "3f7951f7ad7931f5dc0b977d7c33bc5429623535",
       "triggerType" : "PUSH"
     }, {
       "hash" : "9cd6f06a753b63630e8245d51338d2bf9ab86b84",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "9cd6f06a753b63630e8245d51338d2bf9ab86b84",
       "triggerType" : "PUSH"
     }, {
       "hash" : "7380b902731a92e0645c02397e14591a05798620",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "7380b902731a92e0645c02397e14591a05798620",
       "triggerType" : "PUSH"
     }, {
       "hash" : "0bd5cca01a541845081d5430d48b021d4bdf0969",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "0bd5cca01a541845081d5430d48b021d4bdf0969",
       "triggerType" : "PUSH"
     }, {
       "hash" : "8f4e175ef554f015e34e0968b656b5de714ab430",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1182",
       "triggerID" : "8f4e175ef554f015e34e0968b656b5de714ab430",
       "triggerType" : "PUSH"
     }, {
       "hash" : "2023f3c882d0c2f5b96e28d2f21812dc0c64aaee",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "2023f3c882d0c2f5b96e28d2f21812dc0c64aaee",
       "triggerType" : "PUSH"
     }, {
       "hash" : "8394acf060d5c3953dbd67cf930b9b1d3b92d8b5",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1198",
       "triggerID" : "8394acf060d5c3953dbd67cf930b9b1d3b92d8b5",
       "triggerType" : "PUSH"
     }, {
       "hash" : "7ec42c105971bff463e359019405a7f4e7947d86",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "7ec42c105971bff463e359019405a7f4e7947d86",
       "triggerType" : "PUSH"
     }, {
       "hash" : "4b930786a7b2de09ac1653cd884ee76a7d1b2ca5",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1202",
       "triggerID" : "4b930786a7b2de09ac1653cd884ee76a7d1b2ca5",
       "triggerType" : "PUSH"
     }, {
       "hash" : "0e9a648c4e03b5a74d6a0085831805041a9a26a2",
       "status" : "SUCCESS",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1216",
       "triggerID" : "0e9a648c4e03b5a74d6a0085831805041a9a26a2",
       "triggerType" : "PUSH"
     }, {
       "hash" : "326a8729a5d1dd88bc474a5c273aec6ca219f298",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "326a8729a5d1dd88bc474a5c273aec6ca219f298",
       "triggerType" : "PUSH"
     }, {
       "hash" : "cad2f761fc50fba28e2fb3595e4b1270e39ac48a",
       "status" : "PENDING",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1221",
       "triggerID" : "cad2f761fc50fba28e2fb3595e4b1270e39ac48a",
       "triggerType" : "PUSH"
     }, {
       "hash" : "d485a50aec34bc5707377afe96d696772832558e",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "d485a50aec34bc5707377afe96d696772832558e",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 9e668f65c6d4f73c76573236f8fa507dfb80edac UNKNOWN
   * 533f00e5546da0026bbe51e1c030ddd1c19fdd08 UNKNOWN
   * a2ee90c549332a7f17a7d9b2799849b77c08a8e6 UNKNOWN
   * fc2c138901e82ed5f3fbf363c2a0ce8140ade050 UNKNOWN
   * a510a7de1458bb996df5bbb341e8d010e4c91381 UNKNOWN
   * f072a986114949c5e426ce64a5f5568fc718e25d UNKNOWN
   * 3c62ef22f6cde28110e5eb2fca8760a008d77723 UNKNOWN
   * 585ddb8049b9b9ccb057027e265b08a1b586df8c UNKNOWN
   * b62ff8675beed2c4487c788b9ca1a0ad4271b913 UNKNOWN
   * da8111118d71306500229619d3b7996fcd2526e8 UNKNOWN
   * 9cd6f06a753b63630e8245d51338d2bf9ab86b84 UNKNOWN
   * 7380b902731a92e0645c02397e14591a05798620 UNKNOWN
   * 0bd5cca01a541845081d5430d48b021d4bdf0969 UNKNOWN
   * 2023f3c882d0c2f5b96e28d2f21812dc0c64aaee UNKNOWN
   * 7ec42c105971bff463e359019405a7f4e7947d86 UNKNOWN
   * 0e9a648c4e03b5a74d6a0085831805041a9a26a2 Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1216) 
   * 326a8729a5d1dd88bc474a5c273aec6ca219f298 UNKNOWN
   * cad2f761fc50fba28e2fb3595e4b1270e39ac48a Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1221) 
   * d485a50aec34bc5707377afe96d696772832558e UNKNOWN
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] [HUDI-8141] WIP:Always use completion time to find begin Instant for incremental source [hudi]

Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.

hudi-bot commented on PR #11947:
URL: https://github.com/apache/hudi/pull/11947#issuecomment-2403648051

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "090f256a315cbbf0c6a54d93cc2739dd693360e6",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=484",
       "triggerID" : "090f256a315cbbf0c6a54d93cc2739dd693360e6",
       "triggerType" : "PUSH"
     }, {
       "hash" : "0df5649013c1cf9e782fcbb67b8a2e7174f94512",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=985",
       "triggerID" : "0df5649013c1cf9e782fcbb67b8a2e7174f94512",
       "triggerType" : "PUSH"
     }, {
       "hash" : "d52e0128f409918644255abbe4376dc4a780bfc8",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=986",
       "triggerID" : "d52e0128f409918644255abbe4376dc4a780bfc8",
       "triggerType" : "PUSH"
     }, {
       "hash" : "9e668f65c6d4f73c76573236f8fa507dfb80edac",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "9e668f65c6d4f73c76573236f8fa507dfb80edac",
       "triggerType" : "PUSH"
     }, {
       "hash" : "f145131d524bf190986e5de600a326e32a27beca",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=987",
       "triggerID" : "f145131d524bf190986e5de600a326e32a27beca",
       "triggerType" : "PUSH"
     }, {
       "hash" : "0b008f807b42aa6fb2686922bb88d63e84a1f299",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=993",
       "triggerID" : "0b008f807b42aa6fb2686922bb88d63e84a1f299",
       "triggerType" : "PUSH"
     }, {
       "hash" : "65c01ad9f2b33d36d4890355feef91844438e732",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=995",
       "triggerID" : "65c01ad9f2b33d36d4890355feef91844438e732",
       "triggerType" : "PUSH"
     }, {
       "hash" : "955493fdc93842e8c99529cec35e39cecdbe9638",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=996",
       "triggerID" : "955493fdc93842e8c99529cec35e39cecdbe9638",
       "triggerType" : "PUSH"
     }, {
       "hash" : "533f00e5546da0026bbe51e1c030ddd1c19fdd08",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "533f00e5546da0026bbe51e1c030ddd1c19fdd08",
       "triggerType" : "PUSH"
     }, {
       "hash" : "05777be67e37f3a6a8eaa59e54d9e17e5fc6ba9b",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1016",
       "triggerID" : "05777be67e37f3a6a8eaa59e54d9e17e5fc6ba9b",
       "triggerType" : "PUSH"
     }, {
       "hash" : "a2ee90c549332a7f17a7d9b2799849b77c08a8e6",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "a2ee90c549332a7f17a7d9b2799849b77c08a8e6",
       "triggerType" : "PUSH"
     }, {
       "hash" : "05ce568b0b6ba14da9ca40d26b39ba478837d747",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1017",
       "triggerID" : "05ce568b0b6ba14da9ca40d26b39ba478837d747",
       "triggerType" : "PUSH"
     }, {
       "hash" : "840e3b33d39b1f86367bb6b0e65eadeb33de7765",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1025",
       "triggerID" : "840e3b33d39b1f86367bb6b0e65eadeb33de7765",
       "triggerType" : "PUSH"
     }, {
       "hash" : "8d2e2dd83f51d03e890d424a51d3e0e84e904a43",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1031",
       "triggerID" : "8d2e2dd83f51d03e890d424a51d3e0e84e904a43",
       "triggerType" : "PUSH"
     }, {
       "hash" : "688381914087ff11acf2f8c9372dbf39afa0bd40",
       "status" : "SUCCESS",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1035",
       "triggerID" : "688381914087ff11acf2f8c9372dbf39afa0bd40",
       "triggerType" : "PUSH"
     }, {
       "hash" : "fc2c138901e82ed5f3fbf363c2a0ce8140ade050",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "fc2c138901e82ed5f3fbf363c2a0ce8140ade050",
       "triggerType" : "PUSH"
     }, {
       "hash" : "6ed9e1cb1e5a191752d6d39c7a9d5d41840f0cdf",
       "status" : "PENDING",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1039",
       "triggerID" : "6ed9e1cb1e5a191752d6d39c7a9d5d41840f0cdf",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 9e668f65c6d4f73c76573236f8fa507dfb80edac UNKNOWN
   * 533f00e5546da0026bbe51e1c030ddd1c19fdd08 UNKNOWN
   * a2ee90c549332a7f17a7d9b2799849b77c08a8e6 UNKNOWN
   * 688381914087ff11acf2f8c9372dbf39afa0bd40 Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1035) 
   * fc2c138901e82ed5f3fbf363c2a0ce8140ade050 UNKNOWN
   * 6ed9e1cb1e5a191752d6d39c7a9d5d41840f0cdf Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1039) 
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] [HUDI-8141] Incremental Query with Completion Time [hudi]

Posted by "yihua (via GitHub)" <gi...@apache.org>.

yihua commented on code in PR #11947:
URL: https://github.com/apache/hudi/pull/11947#discussion_r1797776979


##########
hudi-common/src/main/java/org/apache/hudi/common/table/timeline/TimelineUtils.java:
##########
@@ -308,6 +309,38 @@ public static HoodieTimeline getCommitsTimelineAfter(
     return timelineSinceLastSync;
   }
 
+  public static HoodieTimeline getCommitsTimeLineAfterByCompletionTimeRange(

Review Comment:
   If the completion time to resume from for a incremental query corresponds to an instant in the archived timeline, we should fallback to the snapshot query instead of pulling the archived timeline, as reading archived timeline can be expensive.



##########
hudi-common/src/main/java/org/apache/hudi/common/table/timeline/HoodieDefaultTimeline.java:
##########
@@ -532,6 +532,13 @@ public boolean isBeforeTimelineStarts(String instant) {
         && compareTimestamps(instant, LESSER_THAN, firstNonSavepointCommit.get().getTimestamp());
   }
 
+  @Override
+  public boolean isBeforeTimelineStartsByCompletionTime(String completionTime) {
+    Option<HoodieInstant> firstNonSavepointCommit = getFirstNonSavepointCommit();

Review Comment:
   `getInstantsOrderedByCompletionTime()` would be useful.



##########
hudi-common/src/main/java/org/apache/hudi/common/table/timeline/HoodieInstantTimeGenerator.java:
##########
@@ -140,6 +140,10 @@ public static String fixInstantTimeCompatibility(String instantTime) {
     }
   }
 
+  public static String getStrictlyLowerTimestamp(String instantTime) {
+    return instantTimeMinusMillis(instantTime, 1);
+  }

Review Comment:
   If this method is not widely used, IMO we should directly use `instantTimeMinusMillis(instantTime, 1)` for clarity instead of introducing a new utility method.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] [HUDI-8141] Incremental Query with Completion Time [hudi]

Posted by "CTTY (via GitHub)" <gi...@apache.org>.

CTTY commented on code in PR #11947:
URL: https://github.com/apache/hudi/pull/11947#discussion_r1797786835


##########
hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/hudi/MergeOnReadIncrementalRelation.scala:
##########
@@ -94,9 +94,10 @@ case class MergeOnReadIncrementalRelation(override val sqlContext: SQLContext,
       tableState = tableState,
       mergeType = mergeType,
       fileSplits = fileSplits,
-      includeStartTime = includeStartTime,
+      includeStartTime = false,

Review Comment:
   Yes, we should get rid of it. Same applies to `startTimestamp`, and `endTimestamp`. I'll file a jira to track this



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] [HUDI-8141] Incremental Query with Completion Time [hudi]

Posted by "yihua (via GitHub)" <gi...@apache.org>.

yihua commented on code in PR #11947:
URL: https://github.com/apache/hudi/pull/11947#discussion_r1797905109


##########
hudi-utilities/src/main/java/org/apache/hudi/utilities/sources/GcsEventsHoodieIncrSource.java:
##########
@@ -170,7 +170,8 @@ public Pair<Option<Dataset<Row>>, String> fetchNextBatch(Option<String> lastChec
     QueryInfo queryInfo = generateQueryInfo(
         sparkContext, srcPath, numInstantsPerFetch,
         Option.of(cloudObjectIncrCheckpoint.getCommit()),
-        missingCheckpointStrategy, handlingMode, HoodieRecord.COMMIT_TIME_METADATA_FIELD,
+        missingCheckpointStrategy, handlingMode,
+        HoodieRecord.COMMIT_TIME_METADATA_FIELD,

Review Comment:
   Revert the format change only.



##########
hudi-spark-datasource/hudi-spark/src/test/scala/org/apache/spark/sql/hudi/procedure/TestCopyToTableProcedure.scala:
##########
@@ -188,10 +191,11 @@ class TestCopyToTableProcedure extends HoodieSparkProcedureTestBase {
       spark.sql(s"insert into $tableName select 2, 'a2', 20, 1500")
 
       // mark beginTime
-      val beginTime = spark.sql(s"select max(_hoodie_commit_time) from $tableName").collectAsList().get(0).get(0)
+      val fs = HadoopFSUtils.getFs(tablePath, spark.sessionState.newHadoopConf())
+      val beginCompletionTime = HoodieDataSourceHelpers.latestCommitCompletionTime(fs, tablePath)
       spark.sql(s"insert into $tableName select 3, 'a3', 30, 2000")
       spark.sql(s"insert into $tableName select 4, 'a4', 40, 2500")
-      val endTime = spark.sql(s"select max(_hoodie_commit_time) from $tableName").collectAsList().get(0).get(0)
+      val endCompletionTime = HoodieDataSourceHelpers.latestCommitCompletionTime(fs, tablePath)

Review Comment:
   Why is `endCompletionTime` the same as `beginCompletionTime` in this test?



##########
hudi-spark-datasource/hudi-spark/src/test/scala/org/apache/spark/sql/hudi/dml/TestMergeIntoTable.scala:
##########
@@ -1002,13 +1003,14 @@ class TestMergeIntoTable extends HoodieSparkSqlTestBase with ScalaAssertionSuppo
         val hudiIncDF1 = spark.read.format("org.apache.hudi")
           .option(DataSourceReadOptions.QUERY_TYPE.key, DataSourceReadOptions.QUERY_TYPE_INCREMENTAL_OPT_VAL)
           .option(DataSourceReadOptions.BEGIN_INSTANTTIME.key, "000")
-          .option(DataSourceReadOptions.END_INSTANTTIME.key, firstCommitTime)
+          .option(DataSourceReadOptions.END_INSTANTTIME.key, firstCompletionTime)

Review Comment:
   Check `BEGIN_INSTANTTIME` to see if it needs update too.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] [HUDI-8141] Incremental Query with Completion Time [hudi]

Posted by "yihua (via GitHub)" <gi...@apache.org>.

yihua commented on code in PR #11947:
URL: https://github.com/apache/hudi/pull/11947#discussion_r1797872110


##########
hudi-spark-datasource/hudi-spark/src/test/scala/org/apache/hudi/functional/TestStructuredStreaming.scala:
##########
@@ -175,7 +175,7 @@ class TestStructuredStreaming extends HoodieSparkClientTestBase {
       val hoodieIncViewDF1 = spark.read.format("org.apache.hudi")
         .option(DataSourceReadOptions.QUERY_TYPE.key, DataSourceReadOptions.QUERY_TYPE_INCREMENTAL_OPT_VAL)
         .option(DataSourceReadOptions.BEGIN_INSTANTTIME.key, "000")
-        .option(DataSourceReadOptions.END_INSTANTTIME.key, firstCommit)
+        .option(DataSourceReadOptions.END_INSTANTTIME.key, commitCompletionTime1)

Review Comment:
   Same here on changing the begin instant time.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] [HUDI-8141] Incremental Query with Completion Time [hudi]

Posted by "yihua (via GitHub)" <gi...@apache.org>.

yihua commented on code in PR #11947:
URL: https://github.com/apache/hudi/pull/11947#discussion_r1802144497


##########
hudi-utilities/src/main/java/org/apache/hudi/utilities/sources/helpers/IncrSourceHelper.java:
##########
@@ -174,6 +170,64 @@ public static QueryInfo generateQueryInfo(JavaSparkContext jssc, String srcBaseP
     }
   }
 
+  public static IncrementalQueryAnalyzer getIncrementalQueryAnalyzer(
+      JavaSparkContext jssc,
+      String srcPath,
+      Option<String> lastCkptStr,
+      MissingCheckpointStrategy missingCheckpointStrategy,
+      int numInstantsFromConfig,
+      Option<SourceProfile<Integer>> latestSourceProfile) {
+    HoodieTableMetaClient metaClient = HoodieTableMetaClient.builder()
+        .setConf(HadoopFSUtils.getStorageConfWithCopy(jssc.hadoopConfiguration()))
+        .setBasePath(srcPath)
+        .setLoadActiveTimelineOnLoad(true)
+        .build();
+
+    String startTime;
+    if (lastCkptStr.isPresent() && !lastCkptStr.get().isEmpty()) {
+      startTime = lastCkptStr.get();
+    } else if (missingCheckpointStrategy != null) {
+      switch (missingCheckpointStrategy) {
+        case READ_UPTO_LATEST_COMMIT:
+          startTime = DEFAULT_BEGIN_TIMESTAMP;
+          // disrespect numInstantsFromConfig when reading up to latest
+          numInstantsFromConfig = -1;
+          break;
+        case READ_LATEST:
+          Option<HoodieInstant> lastInstant = metaClient
+              .getCommitsAndCompactionTimeline()
+              .filterCompletedInstants()
+              .lastInstant();
+          startTime = lastInstant
+              .map(hoodieInstant -> instantTimeMinusMillis(hoodieInstant.getCompletionTime(), 1))

Review Comment:
   This is actually consistent with the logic in `IncrSourceHelper#generateQueryInfo` where we treat begin checkpoint as exclusive.  The reason of doing this is to make sure the subsequent can be applied.  Another way is to special case this scenario by not going through the incremental query analyzer and direct construct the `QueryContext` as the instant to read is already known, i.e., the latest commit by completion time on the timeline.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] [HUDI-8141] Incremental Query with Completion Time [hudi]

Posted by "CTTY (via GitHub)" <gi...@apache.org>.

CTTY commented on code in PR #11947:
URL: https://github.com/apache/hudi/pull/11947#discussion_r1797784983


##########
hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/hudi/DataSourceOptions.scala:
##########
@@ -125,6 +125,15 @@ object DataSourceReadOptions {
       + HollowCommitHandling.USE_TRANSITION_TIME + "`, will use instant's "
       + "`stateTransitionTime` to perform comparison.")
 
+  /* TODO: this is a replacement of HoodieIncrSourceConfig.NUM_INSTANTS_PER_FETCH
+      to make incremental instant limit available in spark incremental relations
+      Need to link HoodieIncrSourceConfig.NUM_INSTANTS_PER_FETCH to this correctly and add doc*/
+  val INCREMENTAL_LIMIT: ConfigProperty[String] = ConfigProperty
+    .key("hoodie.streamer.source.hoodieincr.num_instants")

Review Comment:
   I think this makes sense, will remove this config and its usages



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] [HUDI-8141] Incremental Query with Completion Time [hudi]

Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.

hudi-bot commented on PR #11947:
URL: https://github.com/apache/hudi/pull/11947#issuecomment-2408655945

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "090f256a315cbbf0c6a54d93cc2739dd693360e6",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=484",
       "triggerID" : "090f256a315cbbf0c6a54d93cc2739dd693360e6",
       "triggerType" : "PUSH"
     }, {
       "hash" : "0df5649013c1cf9e782fcbb67b8a2e7174f94512",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=985",
       "triggerID" : "0df5649013c1cf9e782fcbb67b8a2e7174f94512",
       "triggerType" : "PUSH"
     }, {
       "hash" : "d52e0128f409918644255abbe4376dc4a780bfc8",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=986",
       "triggerID" : "d52e0128f409918644255abbe4376dc4a780bfc8",
       "triggerType" : "PUSH"
     }, {
       "hash" : "9e668f65c6d4f73c76573236f8fa507dfb80edac",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "9e668f65c6d4f73c76573236f8fa507dfb80edac",
       "triggerType" : "PUSH"
     }, {
       "hash" : "f145131d524bf190986e5de600a326e32a27beca",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=987",
       "triggerID" : "f145131d524bf190986e5de600a326e32a27beca",
       "triggerType" : "PUSH"
     }, {
       "hash" : "0b008f807b42aa6fb2686922bb88d63e84a1f299",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=993",
       "triggerID" : "0b008f807b42aa6fb2686922bb88d63e84a1f299",
       "triggerType" : "PUSH"
     }, {
       "hash" : "65c01ad9f2b33d36d4890355feef91844438e732",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=995",
       "triggerID" : "65c01ad9f2b33d36d4890355feef91844438e732",
       "triggerType" : "PUSH"
     }, {
       "hash" : "955493fdc93842e8c99529cec35e39cecdbe9638",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=996",
       "triggerID" : "955493fdc93842e8c99529cec35e39cecdbe9638",
       "triggerType" : "PUSH"
     }, {
       "hash" : "533f00e5546da0026bbe51e1c030ddd1c19fdd08",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "533f00e5546da0026bbe51e1c030ddd1c19fdd08",
       "triggerType" : "PUSH"
     }, {
       "hash" : "05777be67e37f3a6a8eaa59e54d9e17e5fc6ba9b",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1016",
       "triggerID" : "05777be67e37f3a6a8eaa59e54d9e17e5fc6ba9b",
       "triggerType" : "PUSH"
     }, {
       "hash" : "a2ee90c549332a7f17a7d9b2799849b77c08a8e6",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "a2ee90c549332a7f17a7d9b2799849b77c08a8e6",
       "triggerType" : "PUSH"
     }, {
       "hash" : "05ce568b0b6ba14da9ca40d26b39ba478837d747",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1017",
       "triggerID" : "05ce568b0b6ba14da9ca40d26b39ba478837d747",
       "triggerType" : "PUSH"
     }, {
       "hash" : "840e3b33d39b1f86367bb6b0e65eadeb33de7765",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1025",
       "triggerID" : "840e3b33d39b1f86367bb6b0e65eadeb33de7765",
       "triggerType" : "PUSH"
     }, {
       "hash" : "8d2e2dd83f51d03e890d424a51d3e0e84e904a43",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1031",
       "triggerID" : "8d2e2dd83f51d03e890d424a51d3e0e84e904a43",
       "triggerType" : "PUSH"
     }, {
       "hash" : "688381914087ff11acf2f8c9372dbf39afa0bd40",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1035",
       "triggerID" : "688381914087ff11acf2f8c9372dbf39afa0bd40",
       "triggerType" : "PUSH"
     }, {
       "hash" : "fc2c138901e82ed5f3fbf363c2a0ce8140ade050",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "fc2c138901e82ed5f3fbf363c2a0ce8140ade050",
       "triggerType" : "PUSH"
     }, {
       "hash" : "6ed9e1cb1e5a191752d6d39c7a9d5d41840f0cdf",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1039",
       "triggerID" : "6ed9e1cb1e5a191752d6d39c7a9d5d41840f0cdf",
       "triggerType" : "PUSH"
     }, {
       "hash" : "a510a7de1458bb996df5bbb341e8d010e4c91381",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "a510a7de1458bb996df5bbb341e8d010e4c91381",
       "triggerType" : "PUSH"
     }, {
       "hash" : "49677397cd117228742205944606f7a789090376",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1042",
       "triggerID" : "49677397cd117228742205944606f7a789090376",
       "triggerType" : "PUSH"
     }, {
       "hash" : "1630db0a37422aed942ff280560a9d6329df5015",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1044",
       "triggerID" : "1630db0a37422aed942ff280560a9d6329df5015",
       "triggerType" : "PUSH"
     }, {
       "hash" : "f11c76301d4ab524db0af57aa831be3a898ea0fb",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1048",
       "triggerID" : "f11c76301d4ab524db0af57aa831be3a898ea0fb",
       "triggerType" : "PUSH"
     }, {
       "hash" : "d1a0dc4c2017853a7ba0c3302f886043a63f29da",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1050",
       "triggerID" : "d1a0dc4c2017853a7ba0c3302f886043a63f29da",
       "triggerType" : "PUSH"
     }, {
       "hash" : "6a4c15ec39ded1f90b113b272554217698a98e1c",
       "status" : "SUCCESS",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1087",
       "triggerID" : "6a4c15ec39ded1f90b113b272554217698a98e1c",
       "triggerType" : "PUSH"
     }, {
       "hash" : "288e730d645a5e58e50a87ef5f44ba11a8fb0533",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "288e730d645a5e58e50a87ef5f44ba11a8fb0533",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 9e668f65c6d4f73c76573236f8fa507dfb80edac UNKNOWN
   * 533f00e5546da0026bbe51e1c030ddd1c19fdd08 UNKNOWN
   * a2ee90c549332a7f17a7d9b2799849b77c08a8e6 UNKNOWN
   * fc2c138901e82ed5f3fbf363c2a0ce8140ade050 UNKNOWN
   * a510a7de1458bb996df5bbb341e8d010e4c91381 UNKNOWN
   * 6a4c15ec39ded1f90b113b272554217698a98e1c Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1087) 
   * 288e730d645a5e58e50a87ef5f44ba11a8fb0533 UNKNOWN
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] [HUDI-8141] Incremental Query with Completion Time [hudi]

Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.

hudi-bot commented on PR #11947:
URL: https://github.com/apache/hudi/pull/11947#issuecomment-2408779416

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "090f256a315cbbf0c6a54d93cc2739dd693360e6",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=484",
       "triggerID" : "090f256a315cbbf0c6a54d93cc2739dd693360e6",
       "triggerType" : "PUSH"
     }, {
       "hash" : "0df5649013c1cf9e782fcbb67b8a2e7174f94512",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=985",
       "triggerID" : "0df5649013c1cf9e782fcbb67b8a2e7174f94512",
       "triggerType" : "PUSH"
     }, {
       "hash" : "d52e0128f409918644255abbe4376dc4a780bfc8",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=986",
       "triggerID" : "d52e0128f409918644255abbe4376dc4a780bfc8",
       "triggerType" : "PUSH"
     }, {
       "hash" : "9e668f65c6d4f73c76573236f8fa507dfb80edac",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "9e668f65c6d4f73c76573236f8fa507dfb80edac",
       "triggerType" : "PUSH"
     }, {
       "hash" : "f145131d524bf190986e5de600a326e32a27beca",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=987",
       "triggerID" : "f145131d524bf190986e5de600a326e32a27beca",
       "triggerType" : "PUSH"
     }, {
       "hash" : "0b008f807b42aa6fb2686922bb88d63e84a1f299",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=993",
       "triggerID" : "0b008f807b42aa6fb2686922bb88d63e84a1f299",
       "triggerType" : "PUSH"
     }, {
       "hash" : "65c01ad9f2b33d36d4890355feef91844438e732",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=995",
       "triggerID" : "65c01ad9f2b33d36d4890355feef91844438e732",
       "triggerType" : "PUSH"
     }, {
       "hash" : "955493fdc93842e8c99529cec35e39cecdbe9638",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=996",
       "triggerID" : "955493fdc93842e8c99529cec35e39cecdbe9638",
       "triggerType" : "PUSH"
     }, {
       "hash" : "533f00e5546da0026bbe51e1c030ddd1c19fdd08",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "533f00e5546da0026bbe51e1c030ddd1c19fdd08",
       "triggerType" : "PUSH"
     }, {
       "hash" : "05777be67e37f3a6a8eaa59e54d9e17e5fc6ba9b",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1016",
       "triggerID" : "05777be67e37f3a6a8eaa59e54d9e17e5fc6ba9b",
       "triggerType" : "PUSH"
     }, {
       "hash" : "a2ee90c549332a7f17a7d9b2799849b77c08a8e6",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "a2ee90c549332a7f17a7d9b2799849b77c08a8e6",
       "triggerType" : "PUSH"
     }, {
       "hash" : "05ce568b0b6ba14da9ca40d26b39ba478837d747",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1017",
       "triggerID" : "05ce568b0b6ba14da9ca40d26b39ba478837d747",
       "triggerType" : "PUSH"
     }, {
       "hash" : "840e3b33d39b1f86367bb6b0e65eadeb33de7765",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1025",
       "triggerID" : "840e3b33d39b1f86367bb6b0e65eadeb33de7765",
       "triggerType" : "PUSH"
     }, {
       "hash" : "8d2e2dd83f51d03e890d424a51d3e0e84e904a43",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1031",
       "triggerID" : "8d2e2dd83f51d03e890d424a51d3e0e84e904a43",
       "triggerType" : "PUSH"
     }, {
       "hash" : "688381914087ff11acf2f8c9372dbf39afa0bd40",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1035",
       "triggerID" : "688381914087ff11acf2f8c9372dbf39afa0bd40",
       "triggerType" : "PUSH"
     }, {
       "hash" : "fc2c138901e82ed5f3fbf363c2a0ce8140ade050",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "fc2c138901e82ed5f3fbf363c2a0ce8140ade050",
       "triggerType" : "PUSH"
     }, {
       "hash" : "6ed9e1cb1e5a191752d6d39c7a9d5d41840f0cdf",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1039",
       "triggerID" : "6ed9e1cb1e5a191752d6d39c7a9d5d41840f0cdf",
       "triggerType" : "PUSH"
     }, {
       "hash" : "a510a7de1458bb996df5bbb341e8d010e4c91381",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "a510a7de1458bb996df5bbb341e8d010e4c91381",
       "triggerType" : "PUSH"
     }, {
       "hash" : "49677397cd117228742205944606f7a789090376",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1042",
       "triggerID" : "49677397cd117228742205944606f7a789090376",
       "triggerType" : "PUSH"
     }, {
       "hash" : "1630db0a37422aed942ff280560a9d6329df5015",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1044",
       "triggerID" : "1630db0a37422aed942ff280560a9d6329df5015",
       "triggerType" : "PUSH"
     }, {
       "hash" : "f11c76301d4ab524db0af57aa831be3a898ea0fb",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1048",
       "triggerID" : "f11c76301d4ab524db0af57aa831be3a898ea0fb",
       "triggerType" : "PUSH"
     }, {
       "hash" : "d1a0dc4c2017853a7ba0c3302f886043a63f29da",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1050",
       "triggerID" : "d1a0dc4c2017853a7ba0c3302f886043a63f29da",
       "triggerType" : "PUSH"
     }, {
       "hash" : "6a4c15ec39ded1f90b113b272554217698a98e1c",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1087",
       "triggerID" : "6a4c15ec39ded1f90b113b272554217698a98e1c",
       "triggerType" : "PUSH"
     }, {
       "hash" : "288e730d645a5e58e50a87ef5f44ba11a8fb0533",
       "status" : "FAILURE",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1092",
       "triggerID" : "288e730d645a5e58e50a87ef5f44ba11a8fb0533",
       "triggerType" : "PUSH"
     }, {
       "hash" : "70a5c21321a0da3a0be8d85e206c24befa7b89f1",
       "status" : "PENDING",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1095",
       "triggerID" : "70a5c21321a0da3a0be8d85e206c24befa7b89f1",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 9e668f65c6d4f73c76573236f8fa507dfb80edac UNKNOWN
   * 533f00e5546da0026bbe51e1c030ddd1c19fdd08 UNKNOWN
   * a2ee90c549332a7f17a7d9b2799849b77c08a8e6 UNKNOWN
   * fc2c138901e82ed5f3fbf363c2a0ce8140ade050 UNKNOWN
   * a510a7de1458bb996df5bbb341e8d010e4c91381 UNKNOWN
   * 288e730d645a5e58e50a87ef5f44ba11a8fb0533 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1092) 
   * 70a5c21321a0da3a0be8d85e206c24befa7b89f1 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1095) 
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] [HUDI-8141] Incremental Query with Completion Time [hudi]

Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.

hudi-bot commented on PR #11947:
URL: https://github.com/apache/hudi/pull/11947#issuecomment-2408876462

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "090f256a315cbbf0c6a54d93cc2739dd693360e6",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=484",
       "triggerID" : "090f256a315cbbf0c6a54d93cc2739dd693360e6",
       "triggerType" : "PUSH"
     }, {
       "hash" : "0df5649013c1cf9e782fcbb67b8a2e7174f94512",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=985",
       "triggerID" : "0df5649013c1cf9e782fcbb67b8a2e7174f94512",
       "triggerType" : "PUSH"
     }, {
       "hash" : "d52e0128f409918644255abbe4376dc4a780bfc8",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=986",
       "triggerID" : "d52e0128f409918644255abbe4376dc4a780bfc8",
       "triggerType" : "PUSH"
     }, {
       "hash" : "9e668f65c6d4f73c76573236f8fa507dfb80edac",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "9e668f65c6d4f73c76573236f8fa507dfb80edac",
       "triggerType" : "PUSH"
     }, {
       "hash" : "f145131d524bf190986e5de600a326e32a27beca",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=987",
       "triggerID" : "f145131d524bf190986e5de600a326e32a27beca",
       "triggerType" : "PUSH"
     }, {
       "hash" : "0b008f807b42aa6fb2686922bb88d63e84a1f299",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=993",
       "triggerID" : "0b008f807b42aa6fb2686922bb88d63e84a1f299",
       "triggerType" : "PUSH"
     }, {
       "hash" : "65c01ad9f2b33d36d4890355feef91844438e732",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=995",
       "triggerID" : "65c01ad9f2b33d36d4890355feef91844438e732",
       "triggerType" : "PUSH"
     }, {
       "hash" : "955493fdc93842e8c99529cec35e39cecdbe9638",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=996",
       "triggerID" : "955493fdc93842e8c99529cec35e39cecdbe9638",
       "triggerType" : "PUSH"
     }, {
       "hash" : "533f00e5546da0026bbe51e1c030ddd1c19fdd08",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "533f00e5546da0026bbe51e1c030ddd1c19fdd08",
       "triggerType" : "PUSH"
     }, {
       "hash" : "05777be67e37f3a6a8eaa59e54d9e17e5fc6ba9b",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1016",
       "triggerID" : "05777be67e37f3a6a8eaa59e54d9e17e5fc6ba9b",
       "triggerType" : "PUSH"
     }, {
       "hash" : "a2ee90c549332a7f17a7d9b2799849b77c08a8e6",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "a2ee90c549332a7f17a7d9b2799849b77c08a8e6",
       "triggerType" : "PUSH"
     }, {
       "hash" : "05ce568b0b6ba14da9ca40d26b39ba478837d747",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1017",
       "triggerID" : "05ce568b0b6ba14da9ca40d26b39ba478837d747",
       "triggerType" : "PUSH"
     }, {
       "hash" : "840e3b33d39b1f86367bb6b0e65eadeb33de7765",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1025",
       "triggerID" : "840e3b33d39b1f86367bb6b0e65eadeb33de7765",
       "triggerType" : "PUSH"
     }, {
       "hash" : "8d2e2dd83f51d03e890d424a51d3e0e84e904a43",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1031",
       "triggerID" : "8d2e2dd83f51d03e890d424a51d3e0e84e904a43",
       "triggerType" : "PUSH"
     }, {
       "hash" : "688381914087ff11acf2f8c9372dbf39afa0bd40",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1035",
       "triggerID" : "688381914087ff11acf2f8c9372dbf39afa0bd40",
       "triggerType" : "PUSH"
     }, {
       "hash" : "fc2c138901e82ed5f3fbf363c2a0ce8140ade050",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "fc2c138901e82ed5f3fbf363c2a0ce8140ade050",
       "triggerType" : "PUSH"
     }, {
       "hash" : "6ed9e1cb1e5a191752d6d39c7a9d5d41840f0cdf",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1039",
       "triggerID" : "6ed9e1cb1e5a191752d6d39c7a9d5d41840f0cdf",
       "triggerType" : "PUSH"
     }, {
       "hash" : "a510a7de1458bb996df5bbb341e8d010e4c91381",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "a510a7de1458bb996df5bbb341e8d010e4c91381",
       "triggerType" : "PUSH"
     }, {
       "hash" : "49677397cd117228742205944606f7a789090376",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1042",
       "triggerID" : "49677397cd117228742205944606f7a789090376",
       "triggerType" : "PUSH"
     }, {
       "hash" : "1630db0a37422aed942ff280560a9d6329df5015",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1044",
       "triggerID" : "1630db0a37422aed942ff280560a9d6329df5015",
       "triggerType" : "PUSH"
     }, {
       "hash" : "f11c76301d4ab524db0af57aa831be3a898ea0fb",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1048",
       "triggerID" : "f11c76301d4ab524db0af57aa831be3a898ea0fb",
       "triggerType" : "PUSH"
     }, {
       "hash" : "d1a0dc4c2017853a7ba0c3302f886043a63f29da",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1050",
       "triggerID" : "d1a0dc4c2017853a7ba0c3302f886043a63f29da",
       "triggerType" : "PUSH"
     }, {
       "hash" : "6a4c15ec39ded1f90b113b272554217698a98e1c",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1087",
       "triggerID" : "6a4c15ec39ded1f90b113b272554217698a98e1c",
       "triggerType" : "PUSH"
     }, {
       "hash" : "288e730d645a5e58e50a87ef5f44ba11a8fb0533",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1092",
       "triggerID" : "288e730d645a5e58e50a87ef5f44ba11a8fb0533",
       "triggerType" : "PUSH"
     }, {
       "hash" : "70a5c21321a0da3a0be8d85e206c24befa7b89f1",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1095",
       "triggerID" : "70a5c21321a0da3a0be8d85e206c24befa7b89f1",
       "triggerType" : "PUSH"
     }, {
       "hash" : "f072a986114949c5e426ce64a5f5568fc718e25d",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "f072a986114949c5e426ce64a5f5568fc718e25d",
       "triggerType" : "PUSH"
     }, {
       "hash" : "45b27475b1e8806ec8c566d6f151f4bc79c8549d",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1096",
       "triggerID" : "45b27475b1e8806ec8c566d6f151f4bc79c8549d",
       "triggerType" : "PUSH"
     }, {
       "hash" : "3c62ef22f6cde28110e5eb2fca8760a008d77723",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "3c62ef22f6cde28110e5eb2fca8760a008d77723",
       "triggerType" : "PUSH"
     }, {
       "hash" : "dc785fa04e542b2ccc45494e0292f5153c7ecea8",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1097",
       "triggerID" : "dc785fa04e542b2ccc45494e0292f5153c7ecea8",
       "triggerType" : "PUSH"
     }, {
       "hash" : "585ddb8049b9b9ccb057027e265b08a1b586df8c",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "585ddb8049b9b9ccb057027e265b08a1b586df8c",
       "triggerType" : "PUSH"
     }, {
       "hash" : "b62ff8675beed2c4487c788b9ca1a0ad4271b913",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "b62ff8675beed2c4487c788b9ca1a0ad4271b913",
       "triggerType" : "PUSH"
     }, {
       "hash" : "2d46048a7402907f348cc96da2839ba1b3e4932a",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1099",
       "triggerID" : "2d46048a7402907f348cc96da2839ba1b3e4932a",
       "triggerType" : "PUSH"
     }, {
       "hash" : "004fb6ebf23f796a4b77650e0530f0f1145e8b91",
       "status" : "SUCCESS",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1101",
       "triggerID" : "004fb6ebf23f796a4b77650e0530f0f1145e8b91",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 9e668f65c6d4f73c76573236f8fa507dfb80edac UNKNOWN
   * 533f00e5546da0026bbe51e1c030ddd1c19fdd08 UNKNOWN
   * a2ee90c549332a7f17a7d9b2799849b77c08a8e6 UNKNOWN
   * fc2c138901e82ed5f3fbf363c2a0ce8140ade050 UNKNOWN
   * a510a7de1458bb996df5bbb341e8d010e4c91381 UNKNOWN
   * f072a986114949c5e426ce64a5f5568fc718e25d UNKNOWN
   * 3c62ef22f6cde28110e5eb2fca8760a008d77723 UNKNOWN
   * 585ddb8049b9b9ccb057027e265b08a1b586df8c UNKNOWN
   * b62ff8675beed2c4487c788b9ca1a0ad4271b913 UNKNOWN
   * 004fb6ebf23f796a4b77650e0530f0f1145e8b91 Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1101) 
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] [HUDI-8141] Incremental Query with Completion Time [hudi]

Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.

hudi-bot commented on PR #11947:
URL: https://github.com/apache/hudi/pull/11947#issuecomment-2408786311

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "090f256a315cbbf0c6a54d93cc2739dd693360e6",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=484",
       "triggerID" : "090f256a315cbbf0c6a54d93cc2739dd693360e6",
       "triggerType" : "PUSH"
     }, {
       "hash" : "0df5649013c1cf9e782fcbb67b8a2e7174f94512",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=985",
       "triggerID" : "0df5649013c1cf9e782fcbb67b8a2e7174f94512",
       "triggerType" : "PUSH"
     }, {
       "hash" : "d52e0128f409918644255abbe4376dc4a780bfc8",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=986",
       "triggerID" : "d52e0128f409918644255abbe4376dc4a780bfc8",
       "triggerType" : "PUSH"
     }, {
       "hash" : "9e668f65c6d4f73c76573236f8fa507dfb80edac",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "9e668f65c6d4f73c76573236f8fa507dfb80edac",
       "triggerType" : "PUSH"
     }, {
       "hash" : "f145131d524bf190986e5de600a326e32a27beca",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=987",
       "triggerID" : "f145131d524bf190986e5de600a326e32a27beca",
       "triggerType" : "PUSH"
     }, {
       "hash" : "0b008f807b42aa6fb2686922bb88d63e84a1f299",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=993",
       "triggerID" : "0b008f807b42aa6fb2686922bb88d63e84a1f299",
       "triggerType" : "PUSH"
     }, {
       "hash" : "65c01ad9f2b33d36d4890355feef91844438e732",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=995",
       "triggerID" : "65c01ad9f2b33d36d4890355feef91844438e732",
       "triggerType" : "PUSH"
     }, {
       "hash" : "955493fdc93842e8c99529cec35e39cecdbe9638",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=996",
       "triggerID" : "955493fdc93842e8c99529cec35e39cecdbe9638",
       "triggerType" : "PUSH"
     }, {
       "hash" : "533f00e5546da0026bbe51e1c030ddd1c19fdd08",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "533f00e5546da0026bbe51e1c030ddd1c19fdd08",
       "triggerType" : "PUSH"
     }, {
       "hash" : "05777be67e37f3a6a8eaa59e54d9e17e5fc6ba9b",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1016",
       "triggerID" : "05777be67e37f3a6a8eaa59e54d9e17e5fc6ba9b",
       "triggerType" : "PUSH"
     }, {
       "hash" : "a2ee90c549332a7f17a7d9b2799849b77c08a8e6",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "a2ee90c549332a7f17a7d9b2799849b77c08a8e6",
       "triggerType" : "PUSH"
     }, {
       "hash" : "05ce568b0b6ba14da9ca40d26b39ba478837d747",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1017",
       "triggerID" : "05ce568b0b6ba14da9ca40d26b39ba478837d747",
       "triggerType" : "PUSH"
     }, {
       "hash" : "840e3b33d39b1f86367bb6b0e65eadeb33de7765",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1025",
       "triggerID" : "840e3b33d39b1f86367bb6b0e65eadeb33de7765",
       "triggerType" : "PUSH"
     }, {
       "hash" : "8d2e2dd83f51d03e890d424a51d3e0e84e904a43",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1031",
       "triggerID" : "8d2e2dd83f51d03e890d424a51d3e0e84e904a43",
       "triggerType" : "PUSH"
     }, {
       "hash" : "688381914087ff11acf2f8c9372dbf39afa0bd40",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1035",
       "triggerID" : "688381914087ff11acf2f8c9372dbf39afa0bd40",
       "triggerType" : "PUSH"
     }, {
       "hash" : "fc2c138901e82ed5f3fbf363c2a0ce8140ade050",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "fc2c138901e82ed5f3fbf363c2a0ce8140ade050",
       "triggerType" : "PUSH"
     }, {
       "hash" : "6ed9e1cb1e5a191752d6d39c7a9d5d41840f0cdf",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1039",
       "triggerID" : "6ed9e1cb1e5a191752d6d39c7a9d5d41840f0cdf",
       "triggerType" : "PUSH"
     }, {
       "hash" : "a510a7de1458bb996df5bbb341e8d010e4c91381",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "a510a7de1458bb996df5bbb341e8d010e4c91381",
       "triggerType" : "PUSH"
     }, {
       "hash" : "49677397cd117228742205944606f7a789090376",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1042",
       "triggerID" : "49677397cd117228742205944606f7a789090376",
       "triggerType" : "PUSH"
     }, {
       "hash" : "1630db0a37422aed942ff280560a9d6329df5015",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1044",
       "triggerID" : "1630db0a37422aed942ff280560a9d6329df5015",
       "triggerType" : "PUSH"
     }, {
       "hash" : "f11c76301d4ab524db0af57aa831be3a898ea0fb",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1048",
       "triggerID" : "f11c76301d4ab524db0af57aa831be3a898ea0fb",
       "triggerType" : "PUSH"
     }, {
       "hash" : "d1a0dc4c2017853a7ba0c3302f886043a63f29da",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1050",
       "triggerID" : "d1a0dc4c2017853a7ba0c3302f886043a63f29da",
       "triggerType" : "PUSH"
     }, {
       "hash" : "6a4c15ec39ded1f90b113b272554217698a98e1c",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1087",
       "triggerID" : "6a4c15ec39ded1f90b113b272554217698a98e1c",
       "triggerType" : "PUSH"
     }, {
       "hash" : "288e730d645a5e58e50a87ef5f44ba11a8fb0533",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1092",
       "triggerID" : "288e730d645a5e58e50a87ef5f44ba11a8fb0533",
       "triggerType" : "PUSH"
     }, {
       "hash" : "70a5c21321a0da3a0be8d85e206c24befa7b89f1",
       "status" : "CANCELED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1095",
       "triggerID" : "70a5c21321a0da3a0be8d85e206c24befa7b89f1",
       "triggerType" : "PUSH"
     }, {
       "hash" : "f072a986114949c5e426ce64a5f5568fc718e25d",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "f072a986114949c5e426ce64a5f5568fc718e25d",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 9e668f65c6d4f73c76573236f8fa507dfb80edac UNKNOWN
   * 533f00e5546da0026bbe51e1c030ddd1c19fdd08 UNKNOWN
   * a2ee90c549332a7f17a7d9b2799849b77c08a8e6 UNKNOWN
   * fc2c138901e82ed5f3fbf363c2a0ce8140ade050 UNKNOWN
   * a510a7de1458bb996df5bbb341e8d010e4c91381 UNKNOWN
   * 70a5c21321a0da3a0be8d85e206c24befa7b89f1 Azure: [CANCELED](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1095) 
   * f072a986114949c5e426ce64a5f5568fc718e25d UNKNOWN
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] [HUDI-8141] WIP:Always use completion time to find begin Instant for incremental source [hudi]

Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.

hudi-bot commented on PR #11947:
URL: https://github.com/apache/hudi/pull/11947#issuecomment-2400582188

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "090f256a315cbbf0c6a54d93cc2739dd693360e6",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=484",
       "triggerID" : "090f256a315cbbf0c6a54d93cc2739dd693360e6",
       "triggerType" : "PUSH"
     }, {
       "hash" : "0df5649013c1cf9e782fcbb67b8a2e7174f94512",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=985",
       "triggerID" : "0df5649013c1cf9e782fcbb67b8a2e7174f94512",
       "triggerType" : "PUSH"
     }, {
       "hash" : "d52e0128f409918644255abbe4376dc4a780bfc8",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=986",
       "triggerID" : "d52e0128f409918644255abbe4376dc4a780bfc8",
       "triggerType" : "PUSH"
     }, {
       "hash" : "9e668f65c6d4f73c76573236f8fa507dfb80edac",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "9e668f65c6d4f73c76573236f8fa507dfb80edac",
       "triggerType" : "PUSH"
     }, {
       "hash" : "f145131d524bf190986e5de600a326e32a27beca",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=987",
       "triggerID" : "f145131d524bf190986e5de600a326e32a27beca",
       "triggerType" : "PUSH"
     }, {
       "hash" : "0b008f807b42aa6fb2686922bb88d63e84a1f299",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=993",
       "triggerID" : "0b008f807b42aa6fb2686922bb88d63e84a1f299",
       "triggerType" : "PUSH"
     }, {
       "hash" : "65c01ad9f2b33d36d4890355feef91844438e732",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=995",
       "triggerID" : "65c01ad9f2b33d36d4890355feef91844438e732",
       "triggerType" : "PUSH"
     }, {
       "hash" : "955493fdc93842e8c99529cec35e39cecdbe9638",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=996",
       "triggerID" : "955493fdc93842e8c99529cec35e39cecdbe9638",
       "triggerType" : "PUSH"
     }, {
       "hash" : "533f00e5546da0026bbe51e1c030ddd1c19fdd08",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "533f00e5546da0026bbe51e1c030ddd1c19fdd08",
       "triggerType" : "PUSH"
     }, {
       "hash" : "05777be67e37f3a6a8eaa59e54d9e17e5fc6ba9b",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1016",
       "triggerID" : "05777be67e37f3a6a8eaa59e54d9e17e5fc6ba9b",
       "triggerType" : "PUSH"
     }, {
       "hash" : "a2ee90c549332a7f17a7d9b2799849b77c08a8e6",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "a2ee90c549332a7f17a7d9b2799849b77c08a8e6",
       "triggerType" : "PUSH"
     }, {
       "hash" : "05ce568b0b6ba14da9ca40d26b39ba478837d747",
       "status" : "FAILURE",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1017",
       "triggerID" : "05ce568b0b6ba14da9ca40d26b39ba478837d747",
       "triggerType" : "PUSH"
     }, {
       "hash" : "840e3b33d39b1f86367bb6b0e65eadeb33de7765",
       "status" : "PENDING",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1025",
       "triggerID" : "840e3b33d39b1f86367bb6b0e65eadeb33de7765",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 9e668f65c6d4f73c76573236f8fa507dfb80edac UNKNOWN
   * 533f00e5546da0026bbe51e1c030ddd1c19fdd08 UNKNOWN
   * a2ee90c549332a7f17a7d9b2799849b77c08a8e6 UNKNOWN
   * 05ce568b0b6ba14da9ca40d26b39ba478837d747 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1017) 
   * 840e3b33d39b1f86367bb6b0e65eadeb33de7765 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1025) 
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] [HUDI-8141] WIP:Always use completion time to find begin Instant for incremental source [hudi]

Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.

hudi-bot commented on PR #11947:
URL: https://github.com/apache/hudi/pull/11947#issuecomment-2402920520

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "090f256a315cbbf0c6a54d93cc2739dd693360e6",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=484",
       "triggerID" : "090f256a315cbbf0c6a54d93cc2739dd693360e6",
       "triggerType" : "PUSH"
     }, {
       "hash" : "0df5649013c1cf9e782fcbb67b8a2e7174f94512",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=985",
       "triggerID" : "0df5649013c1cf9e782fcbb67b8a2e7174f94512",
       "triggerType" : "PUSH"
     }, {
       "hash" : "d52e0128f409918644255abbe4376dc4a780bfc8",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=986",
       "triggerID" : "d52e0128f409918644255abbe4376dc4a780bfc8",
       "triggerType" : "PUSH"
     }, {
       "hash" : "9e668f65c6d4f73c76573236f8fa507dfb80edac",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "9e668f65c6d4f73c76573236f8fa507dfb80edac",
       "triggerType" : "PUSH"
     }, {
       "hash" : "f145131d524bf190986e5de600a326e32a27beca",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=987",
       "triggerID" : "f145131d524bf190986e5de600a326e32a27beca",
       "triggerType" : "PUSH"
     }, {
       "hash" : "0b008f807b42aa6fb2686922bb88d63e84a1f299",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=993",
       "triggerID" : "0b008f807b42aa6fb2686922bb88d63e84a1f299",
       "triggerType" : "PUSH"
     }, {
       "hash" : "65c01ad9f2b33d36d4890355feef91844438e732",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=995",
       "triggerID" : "65c01ad9f2b33d36d4890355feef91844438e732",
       "triggerType" : "PUSH"
     }, {
       "hash" : "955493fdc93842e8c99529cec35e39cecdbe9638",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=996",
       "triggerID" : "955493fdc93842e8c99529cec35e39cecdbe9638",
       "triggerType" : "PUSH"
     }, {
       "hash" : "533f00e5546da0026bbe51e1c030ddd1c19fdd08",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "533f00e5546da0026bbe51e1c030ddd1c19fdd08",
       "triggerType" : "PUSH"
     }, {
       "hash" : "05777be67e37f3a6a8eaa59e54d9e17e5fc6ba9b",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1016",
       "triggerID" : "05777be67e37f3a6a8eaa59e54d9e17e5fc6ba9b",
       "triggerType" : "PUSH"
     }, {
       "hash" : "a2ee90c549332a7f17a7d9b2799849b77c08a8e6",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "a2ee90c549332a7f17a7d9b2799849b77c08a8e6",
       "triggerType" : "PUSH"
     }, {
       "hash" : "05ce568b0b6ba14da9ca40d26b39ba478837d747",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1017",
       "triggerID" : "05ce568b0b6ba14da9ca40d26b39ba478837d747",
       "triggerType" : "PUSH"
     }, {
       "hash" : "840e3b33d39b1f86367bb6b0e65eadeb33de7765",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1025",
       "triggerID" : "840e3b33d39b1f86367bb6b0e65eadeb33de7765",
       "triggerType" : "PUSH"
     }, {
       "hash" : "8d2e2dd83f51d03e890d424a51d3e0e84e904a43",
       "status" : "FAILURE",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1031",
       "triggerID" : "8d2e2dd83f51d03e890d424a51d3e0e84e904a43",
       "triggerType" : "PUSH"
     }, {
       "hash" : "688381914087ff11acf2f8c9372dbf39afa0bd40",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "688381914087ff11acf2f8c9372dbf39afa0bd40",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 9e668f65c6d4f73c76573236f8fa507dfb80edac UNKNOWN
   * 533f00e5546da0026bbe51e1c030ddd1c19fdd08 UNKNOWN
   * a2ee90c549332a7f17a7d9b2799849b77c08a8e6 UNKNOWN
   * 8d2e2dd83f51d03e890d424a51d3e0e84e904a43 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1031) 
   * 688381914087ff11acf2f8c9372dbf39afa0bd40 UNKNOWN
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] [HUDI-8141] WIP:Always use completion time to find begin Instant for incremental source [hudi]

Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.

hudi-bot commented on PR #11947:
URL: https://github.com/apache/hudi/pull/11947#issuecomment-2403653432

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "090f256a315cbbf0c6a54d93cc2739dd693360e6",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=484",
       "triggerID" : "090f256a315cbbf0c6a54d93cc2739dd693360e6",
       "triggerType" : "PUSH"
     }, {
       "hash" : "0df5649013c1cf9e782fcbb67b8a2e7174f94512",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=985",
       "triggerID" : "0df5649013c1cf9e782fcbb67b8a2e7174f94512",
       "triggerType" : "PUSH"
     }, {
       "hash" : "d52e0128f409918644255abbe4376dc4a780bfc8",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=986",
       "triggerID" : "d52e0128f409918644255abbe4376dc4a780bfc8",
       "triggerType" : "PUSH"
     }, {
       "hash" : "9e668f65c6d4f73c76573236f8fa507dfb80edac",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "9e668f65c6d4f73c76573236f8fa507dfb80edac",
       "triggerType" : "PUSH"
     }, {
       "hash" : "f145131d524bf190986e5de600a326e32a27beca",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=987",
       "triggerID" : "f145131d524bf190986e5de600a326e32a27beca",
       "triggerType" : "PUSH"
     }, {
       "hash" : "0b008f807b42aa6fb2686922bb88d63e84a1f299",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=993",
       "triggerID" : "0b008f807b42aa6fb2686922bb88d63e84a1f299",
       "triggerType" : "PUSH"
     }, {
       "hash" : "65c01ad9f2b33d36d4890355feef91844438e732",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=995",
       "triggerID" : "65c01ad9f2b33d36d4890355feef91844438e732",
       "triggerType" : "PUSH"
     }, {
       "hash" : "955493fdc93842e8c99529cec35e39cecdbe9638",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=996",
       "triggerID" : "955493fdc93842e8c99529cec35e39cecdbe9638",
       "triggerType" : "PUSH"
     }, {
       "hash" : "533f00e5546da0026bbe51e1c030ddd1c19fdd08",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "533f00e5546da0026bbe51e1c030ddd1c19fdd08",
       "triggerType" : "PUSH"
     }, {
       "hash" : "05777be67e37f3a6a8eaa59e54d9e17e5fc6ba9b",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1016",
       "triggerID" : "05777be67e37f3a6a8eaa59e54d9e17e5fc6ba9b",
       "triggerType" : "PUSH"
     }, {
       "hash" : "a2ee90c549332a7f17a7d9b2799849b77c08a8e6",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "a2ee90c549332a7f17a7d9b2799849b77c08a8e6",
       "triggerType" : "PUSH"
     }, {
       "hash" : "05ce568b0b6ba14da9ca40d26b39ba478837d747",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1017",
       "triggerID" : "05ce568b0b6ba14da9ca40d26b39ba478837d747",
       "triggerType" : "PUSH"
     }, {
       "hash" : "840e3b33d39b1f86367bb6b0e65eadeb33de7765",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1025",
       "triggerID" : "840e3b33d39b1f86367bb6b0e65eadeb33de7765",
       "triggerType" : "PUSH"
     }, {
       "hash" : "8d2e2dd83f51d03e890d424a51d3e0e84e904a43",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1031",
       "triggerID" : "8d2e2dd83f51d03e890d424a51d3e0e84e904a43",
       "triggerType" : "PUSH"
     }, {
       "hash" : "688381914087ff11acf2f8c9372dbf39afa0bd40",
       "status" : "SUCCESS",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1035",
       "triggerID" : "688381914087ff11acf2f8c9372dbf39afa0bd40",
       "triggerType" : "PUSH"
     }, {
       "hash" : "fc2c138901e82ed5f3fbf363c2a0ce8140ade050",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "fc2c138901e82ed5f3fbf363c2a0ce8140ade050",
       "triggerType" : "PUSH"
     }, {
       "hash" : "6ed9e1cb1e5a191752d6d39c7a9d5d41840f0cdf",
       "status" : "PENDING",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1039",
       "triggerID" : "6ed9e1cb1e5a191752d6d39c7a9d5d41840f0cdf",
       "triggerType" : "PUSH"
     }, {
       "hash" : "a510a7de1458bb996df5bbb341e8d010e4c91381",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "a510a7de1458bb996df5bbb341e8d010e4c91381",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 9e668f65c6d4f73c76573236f8fa507dfb80edac UNKNOWN
   * 533f00e5546da0026bbe51e1c030ddd1c19fdd08 UNKNOWN
   * a2ee90c549332a7f17a7d9b2799849b77c08a8e6 UNKNOWN
   * 688381914087ff11acf2f8c9372dbf39afa0bd40 Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1035) 
   * fc2c138901e82ed5f3fbf363c2a0ce8140ade050 UNKNOWN
   * 6ed9e1cb1e5a191752d6d39c7a9d5d41840f0cdf Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1039) 
   * a510a7de1458bb996df5bbb341e8d010e4c91381 UNKNOWN
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] [HUDI-8141] Incremental Query with Completion Time [hudi]

Posted by "danny0405 (via GitHub)" <gi...@apache.org>.

danny0405 commented on code in PR #11947:
URL: https://github.com/apache/hudi/pull/11947#discussion_r1798655176


##########
hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/hudi/MergeOnReadIncrementalRelation.scala:
##########
@@ -186,12 +184,12 @@ trait HoodieIncrementalRelationTrait extends HoodieBaseRelation {
 
   protected def endTimestamp: String = optParams.getOrElse(
     DataSourceReadOptions.END_INSTANTTIME.key,
-    if (hollowCommitHandling == USE_TRANSITION_TIME) super.timeline.lastInstant().get.getCompletionTime
-    else super.timeline.lastInstant().get.getTimestamp)
-
-  protected def startInstantArchived: Boolean = super.timeline.isBeforeTimelineStarts(startTimestamp)
+    super.timeline.getInstantsAsStream.iterator().asScala.toList
+      .map(_.getCompletionTime)

Review Comment:
   Can we add a utility method `getLatestCompletionTime` on the timeline.



##########
hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/hudi/MergeOnReadIncrementalRelation.scala:
##########
@@ -186,12 +184,12 @@ trait HoodieIncrementalRelationTrait extends HoodieBaseRelation {
 
   protected def endTimestamp: String = optParams.getOrElse(
     DataSourceReadOptions.END_INSTANTTIME.key,
-    if (hollowCommitHandling == USE_TRANSITION_TIME) super.timeline.lastInstant().get.getCompletionTime
-    else super.timeline.lastInstant().get.getTimestamp)
-
-  protected def startInstantArchived: Boolean = super.timeline.isBeforeTimelineStarts(startTimestamp)
+    super.timeline.getInstantsAsStream.iterator().asScala.toList
+      .map(_.getCompletionTime)

Review Comment:
   Can we add an utility method `getLatestCompletionTime` on the timeline.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] [HUDI-8141] Incremental Query with Completion Time [hudi]

Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.

hudi-bot commented on PR #11947:
URL: https://github.com/apache/hudi/pull/11947#issuecomment-2423207251

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "090f256a315cbbf0c6a54d93cc2739dd693360e6",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=484",
       "triggerID" : "090f256a315cbbf0c6a54d93cc2739dd693360e6",
       "triggerType" : "PUSH"
     }, {
       "hash" : "0df5649013c1cf9e782fcbb67b8a2e7174f94512",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=985",
       "triggerID" : "0df5649013c1cf9e782fcbb67b8a2e7174f94512",
       "triggerType" : "PUSH"
     }, {
       "hash" : "d52e0128f409918644255abbe4376dc4a780bfc8",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=986",
       "triggerID" : "d52e0128f409918644255abbe4376dc4a780bfc8",
       "triggerType" : "PUSH"
     }, {
       "hash" : "9e668f65c6d4f73c76573236f8fa507dfb80edac",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "9e668f65c6d4f73c76573236f8fa507dfb80edac",
       "triggerType" : "PUSH"
     }, {
       "hash" : "f145131d524bf190986e5de600a326e32a27beca",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=987",
       "triggerID" : "f145131d524bf190986e5de600a326e32a27beca",
       "triggerType" : "PUSH"
     }, {
       "hash" : "0b008f807b42aa6fb2686922bb88d63e84a1f299",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=993",
       "triggerID" : "0b008f807b42aa6fb2686922bb88d63e84a1f299",
       "triggerType" : "PUSH"
     }, {
       "hash" : "65c01ad9f2b33d36d4890355feef91844438e732",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=995",
       "triggerID" : "65c01ad9f2b33d36d4890355feef91844438e732",
       "triggerType" : "PUSH"
     }, {
       "hash" : "955493fdc93842e8c99529cec35e39cecdbe9638",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=996",
       "triggerID" : "955493fdc93842e8c99529cec35e39cecdbe9638",
       "triggerType" : "PUSH"
     }, {
       "hash" : "533f00e5546da0026bbe51e1c030ddd1c19fdd08",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "533f00e5546da0026bbe51e1c030ddd1c19fdd08",
       "triggerType" : "PUSH"
     }, {
       "hash" : "05777be67e37f3a6a8eaa59e54d9e17e5fc6ba9b",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1016",
       "triggerID" : "05777be67e37f3a6a8eaa59e54d9e17e5fc6ba9b",
       "triggerType" : "PUSH"
     }, {
       "hash" : "a2ee90c549332a7f17a7d9b2799849b77c08a8e6",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "a2ee90c549332a7f17a7d9b2799849b77c08a8e6",
       "triggerType" : "PUSH"
     }, {
       "hash" : "05ce568b0b6ba14da9ca40d26b39ba478837d747",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1017",
       "triggerID" : "05ce568b0b6ba14da9ca40d26b39ba478837d747",
       "triggerType" : "PUSH"
     }, {
       "hash" : "840e3b33d39b1f86367bb6b0e65eadeb33de7765",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1025",
       "triggerID" : "840e3b33d39b1f86367bb6b0e65eadeb33de7765",
       "triggerType" : "PUSH"
     }, {
       "hash" : "8d2e2dd83f51d03e890d424a51d3e0e84e904a43",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1031",
       "triggerID" : "8d2e2dd83f51d03e890d424a51d3e0e84e904a43",
       "triggerType" : "PUSH"
     }, {
       "hash" : "688381914087ff11acf2f8c9372dbf39afa0bd40",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1035",
       "triggerID" : "688381914087ff11acf2f8c9372dbf39afa0bd40",
       "triggerType" : "PUSH"
     }, {
       "hash" : "fc2c138901e82ed5f3fbf363c2a0ce8140ade050",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "fc2c138901e82ed5f3fbf363c2a0ce8140ade050",
       "triggerType" : "PUSH"
     }, {
       "hash" : "6ed9e1cb1e5a191752d6d39c7a9d5d41840f0cdf",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1039",
       "triggerID" : "6ed9e1cb1e5a191752d6d39c7a9d5d41840f0cdf",
       "triggerType" : "PUSH"
     }, {
       "hash" : "a510a7de1458bb996df5bbb341e8d010e4c91381",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "a510a7de1458bb996df5bbb341e8d010e4c91381",
       "triggerType" : "PUSH"
     }, {
       "hash" : "49677397cd117228742205944606f7a789090376",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1042",
       "triggerID" : "49677397cd117228742205944606f7a789090376",
       "triggerType" : "PUSH"
     }, {
       "hash" : "1630db0a37422aed942ff280560a9d6329df5015",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1044",
       "triggerID" : "1630db0a37422aed942ff280560a9d6329df5015",
       "triggerType" : "PUSH"
     }, {
       "hash" : "f11c76301d4ab524db0af57aa831be3a898ea0fb",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1048",
       "triggerID" : "f11c76301d4ab524db0af57aa831be3a898ea0fb",
       "triggerType" : "PUSH"
     }, {
       "hash" : "d1a0dc4c2017853a7ba0c3302f886043a63f29da",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1050",
       "triggerID" : "d1a0dc4c2017853a7ba0c3302f886043a63f29da",
       "triggerType" : "PUSH"
     }, {
       "hash" : "6a4c15ec39ded1f90b113b272554217698a98e1c",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1087",
       "triggerID" : "6a4c15ec39ded1f90b113b272554217698a98e1c",
       "triggerType" : "PUSH"
     }, {
       "hash" : "288e730d645a5e58e50a87ef5f44ba11a8fb0533",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1092",
       "triggerID" : "288e730d645a5e58e50a87ef5f44ba11a8fb0533",
       "triggerType" : "PUSH"
     }, {
       "hash" : "70a5c21321a0da3a0be8d85e206c24befa7b89f1",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1095",
       "triggerID" : "70a5c21321a0da3a0be8d85e206c24befa7b89f1",
       "triggerType" : "PUSH"
     }, {
       "hash" : "f072a986114949c5e426ce64a5f5568fc718e25d",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "f072a986114949c5e426ce64a5f5568fc718e25d",
       "triggerType" : "PUSH"
     }, {
       "hash" : "45b27475b1e8806ec8c566d6f151f4bc79c8549d",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1096",
       "triggerID" : "45b27475b1e8806ec8c566d6f151f4bc79c8549d",
       "triggerType" : "PUSH"
     }, {
       "hash" : "3c62ef22f6cde28110e5eb2fca8760a008d77723",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "3c62ef22f6cde28110e5eb2fca8760a008d77723",
       "triggerType" : "PUSH"
     }, {
       "hash" : "dc785fa04e542b2ccc45494e0292f5153c7ecea8",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1097",
       "triggerID" : "dc785fa04e542b2ccc45494e0292f5153c7ecea8",
       "triggerType" : "PUSH"
     }, {
       "hash" : "585ddb8049b9b9ccb057027e265b08a1b586df8c",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "585ddb8049b9b9ccb057027e265b08a1b586df8c",
       "triggerType" : "PUSH"
     }, {
       "hash" : "b62ff8675beed2c4487c788b9ca1a0ad4271b913",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "b62ff8675beed2c4487c788b9ca1a0ad4271b913",
       "triggerType" : "PUSH"
     }, {
       "hash" : "2d46048a7402907f348cc96da2839ba1b3e4932a",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1099",
       "triggerID" : "2d46048a7402907f348cc96da2839ba1b3e4932a",
       "triggerType" : "PUSH"
     }, {
       "hash" : "004fb6ebf23f796a4b77650e0530f0f1145e8b91",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1101",
       "triggerID" : "004fb6ebf23f796a4b77650e0530f0f1145e8b91",
       "triggerType" : "PUSH"
     }, {
       "hash" : "af0c33333b5e9ae6d59f20a6db1d7c016ea015d3",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1165",
       "triggerID" : "af0c33333b5e9ae6d59f20a6db1d7c016ea015d3",
       "triggerType" : "PUSH"
     }, {
       "hash" : "da8111118d71306500229619d3b7996fcd2526e8",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "da8111118d71306500229619d3b7996fcd2526e8",
       "triggerType" : "PUSH"
     }, {
       "hash" : "83257e2b9bd3c426cf90e11003a5f244a750d7fa",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1167",
       "triggerID" : "83257e2b9bd3c426cf90e11003a5f244a750d7fa",
       "triggerType" : "PUSH"
     }, {
       "hash" : "e0de408f1b8153fdaaea4c87b27a7b69213b75b5",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1177",
       "triggerID" : "e0de408f1b8153fdaaea4c87b27a7b69213b75b5",
       "triggerType" : "PUSH"
     }, {
       "hash" : "3f7951f7ad7931f5dc0b977d7c33bc5429623535",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1180",
       "triggerID" : "3f7951f7ad7931f5dc0b977d7c33bc5429623535",
       "triggerType" : "PUSH"
     }, {
       "hash" : "9cd6f06a753b63630e8245d51338d2bf9ab86b84",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "9cd6f06a753b63630e8245d51338d2bf9ab86b84",
       "triggerType" : "PUSH"
     }, {
       "hash" : "7380b902731a92e0645c02397e14591a05798620",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "7380b902731a92e0645c02397e14591a05798620",
       "triggerType" : "PUSH"
     }, {
       "hash" : "0bd5cca01a541845081d5430d48b021d4bdf0969",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "0bd5cca01a541845081d5430d48b021d4bdf0969",
       "triggerType" : "PUSH"
     }, {
       "hash" : "8f4e175ef554f015e34e0968b656b5de714ab430",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1182",
       "triggerID" : "8f4e175ef554f015e34e0968b656b5de714ab430",
       "triggerType" : "PUSH"
     }, {
       "hash" : "2023f3c882d0c2f5b96e28d2f21812dc0c64aaee",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "2023f3c882d0c2f5b96e28d2f21812dc0c64aaee",
       "triggerType" : "PUSH"
     }, {
       "hash" : "8394acf060d5c3953dbd67cf930b9b1d3b92d8b5",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1198",
       "triggerID" : "8394acf060d5c3953dbd67cf930b9b1d3b92d8b5",
       "triggerType" : "PUSH"
     }, {
       "hash" : "7ec42c105971bff463e359019405a7f4e7947d86",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "7ec42c105971bff463e359019405a7f4e7947d86",
       "triggerType" : "PUSH"
     }, {
       "hash" : "4b930786a7b2de09ac1653cd884ee76a7d1b2ca5",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1202",
       "triggerID" : "4b930786a7b2de09ac1653cd884ee76a7d1b2ca5",
       "triggerType" : "PUSH"
     }, {
       "hash" : "0e9a648c4e03b5a74d6a0085831805041a9a26a2",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1216",
       "triggerID" : "0e9a648c4e03b5a74d6a0085831805041a9a26a2",
       "triggerType" : "PUSH"
     }, {
       "hash" : "326a8729a5d1dd88bc474a5c273aec6ca219f298",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "326a8729a5d1dd88bc474a5c273aec6ca219f298",
       "triggerType" : "PUSH"
     }, {
       "hash" : "cad2f761fc50fba28e2fb3595e4b1270e39ac48a",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1221",
       "triggerID" : "cad2f761fc50fba28e2fb3595e4b1270e39ac48a",
       "triggerType" : "PUSH"
     }, {
       "hash" : "d485a50aec34bc5707377afe96d696772832558e",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "d485a50aec34bc5707377afe96d696772832558e",
       "triggerType" : "PUSH"
     }, {
       "hash" : "38a98106f19fe4e51632a2c4289ec0f01805ba5e",
       "status" : "SUCCESS",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1224",
       "triggerID" : "38a98106f19fe4e51632a2c4289ec0f01805ba5e",
       "triggerType" : "PUSH"
     }, {
       "hash" : "7a6116cfbf4eba54e77194664225ed1c372d71be",
       "status" : "PENDING",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1250",
       "triggerID" : "7a6116cfbf4eba54e77194664225ed1c372d71be",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 9e668f65c6d4f73c76573236f8fa507dfb80edac UNKNOWN
   * 533f00e5546da0026bbe51e1c030ddd1c19fdd08 UNKNOWN
   * a2ee90c549332a7f17a7d9b2799849b77c08a8e6 UNKNOWN
   * fc2c138901e82ed5f3fbf363c2a0ce8140ade050 UNKNOWN
   * a510a7de1458bb996df5bbb341e8d010e4c91381 UNKNOWN
   * f072a986114949c5e426ce64a5f5568fc718e25d UNKNOWN
   * 3c62ef22f6cde28110e5eb2fca8760a008d77723 UNKNOWN
   * 585ddb8049b9b9ccb057027e265b08a1b586df8c UNKNOWN
   * b62ff8675beed2c4487c788b9ca1a0ad4271b913 UNKNOWN
   * da8111118d71306500229619d3b7996fcd2526e8 UNKNOWN
   * 9cd6f06a753b63630e8245d51338d2bf9ab86b84 UNKNOWN
   * 7380b902731a92e0645c02397e14591a05798620 UNKNOWN
   * 0bd5cca01a541845081d5430d48b021d4bdf0969 UNKNOWN
   * 2023f3c882d0c2f5b96e28d2f21812dc0c64aaee UNKNOWN
   * 7ec42c105971bff463e359019405a7f4e7947d86 UNKNOWN
   * 326a8729a5d1dd88bc474a5c273aec6ca219f298 UNKNOWN
   * d485a50aec34bc5707377afe96d696772832558e UNKNOWN
   * 38a98106f19fe4e51632a2c4289ec0f01805ba5e Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1224) 
   * 7a6116cfbf4eba54e77194664225ed1c372d71be Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1250) 
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] [HUDI-8141] Incremental Query with Completion Time [hudi]

Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.

hudi-bot commented on PR #11947:
URL: https://github.com/apache/hudi/pull/11947#issuecomment-2418786236

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "090f256a315cbbf0c6a54d93cc2739dd693360e6",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=484",
       "triggerID" : "090f256a315cbbf0c6a54d93cc2739dd693360e6",
       "triggerType" : "PUSH"
     }, {
       "hash" : "0df5649013c1cf9e782fcbb67b8a2e7174f94512",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=985",
       "triggerID" : "0df5649013c1cf9e782fcbb67b8a2e7174f94512",
       "triggerType" : "PUSH"
     }, {
       "hash" : "d52e0128f409918644255abbe4376dc4a780bfc8",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=986",
       "triggerID" : "d52e0128f409918644255abbe4376dc4a780bfc8",
       "triggerType" : "PUSH"
     }, {
       "hash" : "9e668f65c6d4f73c76573236f8fa507dfb80edac",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "9e668f65c6d4f73c76573236f8fa507dfb80edac",
       "triggerType" : "PUSH"
     }, {
       "hash" : "f145131d524bf190986e5de600a326e32a27beca",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=987",
       "triggerID" : "f145131d524bf190986e5de600a326e32a27beca",
       "triggerType" : "PUSH"
     }, {
       "hash" : "0b008f807b42aa6fb2686922bb88d63e84a1f299",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=993",
       "triggerID" : "0b008f807b42aa6fb2686922bb88d63e84a1f299",
       "triggerType" : "PUSH"
     }, {
       "hash" : "65c01ad9f2b33d36d4890355feef91844438e732",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=995",
       "triggerID" : "65c01ad9f2b33d36d4890355feef91844438e732",
       "triggerType" : "PUSH"
     }, {
       "hash" : "955493fdc93842e8c99529cec35e39cecdbe9638",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=996",
       "triggerID" : "955493fdc93842e8c99529cec35e39cecdbe9638",
       "triggerType" : "PUSH"
     }, {
       "hash" : "533f00e5546da0026bbe51e1c030ddd1c19fdd08",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "533f00e5546da0026bbe51e1c030ddd1c19fdd08",
       "triggerType" : "PUSH"
     }, {
       "hash" : "05777be67e37f3a6a8eaa59e54d9e17e5fc6ba9b",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1016",
       "triggerID" : "05777be67e37f3a6a8eaa59e54d9e17e5fc6ba9b",
       "triggerType" : "PUSH"
     }, {
       "hash" : "a2ee90c549332a7f17a7d9b2799849b77c08a8e6",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "a2ee90c549332a7f17a7d9b2799849b77c08a8e6",
       "triggerType" : "PUSH"
     }, {
       "hash" : "05ce568b0b6ba14da9ca40d26b39ba478837d747",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1017",
       "triggerID" : "05ce568b0b6ba14da9ca40d26b39ba478837d747",
       "triggerType" : "PUSH"
     }, {
       "hash" : "840e3b33d39b1f86367bb6b0e65eadeb33de7765",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1025",
       "triggerID" : "840e3b33d39b1f86367bb6b0e65eadeb33de7765",
       "triggerType" : "PUSH"
     }, {
       "hash" : "8d2e2dd83f51d03e890d424a51d3e0e84e904a43",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1031",
       "triggerID" : "8d2e2dd83f51d03e890d424a51d3e0e84e904a43",
       "triggerType" : "PUSH"
     }, {
       "hash" : "688381914087ff11acf2f8c9372dbf39afa0bd40",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1035",
       "triggerID" : "688381914087ff11acf2f8c9372dbf39afa0bd40",
       "triggerType" : "PUSH"
     }, {
       "hash" : "fc2c138901e82ed5f3fbf363c2a0ce8140ade050",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "fc2c138901e82ed5f3fbf363c2a0ce8140ade050",
       "triggerType" : "PUSH"
     }, {
       "hash" : "6ed9e1cb1e5a191752d6d39c7a9d5d41840f0cdf",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1039",
       "triggerID" : "6ed9e1cb1e5a191752d6d39c7a9d5d41840f0cdf",
       "triggerType" : "PUSH"
     }, {
       "hash" : "a510a7de1458bb996df5bbb341e8d010e4c91381",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "a510a7de1458bb996df5bbb341e8d010e4c91381",
       "triggerType" : "PUSH"
     }, {
       "hash" : "49677397cd117228742205944606f7a789090376",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1042",
       "triggerID" : "49677397cd117228742205944606f7a789090376",
       "triggerType" : "PUSH"
     }, {
       "hash" : "1630db0a37422aed942ff280560a9d6329df5015",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1044",
       "triggerID" : "1630db0a37422aed942ff280560a9d6329df5015",
       "triggerType" : "PUSH"
     }, {
       "hash" : "f11c76301d4ab524db0af57aa831be3a898ea0fb",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1048",
       "triggerID" : "f11c76301d4ab524db0af57aa831be3a898ea0fb",
       "triggerType" : "PUSH"
     }, {
       "hash" : "d1a0dc4c2017853a7ba0c3302f886043a63f29da",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1050",
       "triggerID" : "d1a0dc4c2017853a7ba0c3302f886043a63f29da",
       "triggerType" : "PUSH"
     }, {
       "hash" : "6a4c15ec39ded1f90b113b272554217698a98e1c",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1087",
       "triggerID" : "6a4c15ec39ded1f90b113b272554217698a98e1c",
       "triggerType" : "PUSH"
     }, {
       "hash" : "288e730d645a5e58e50a87ef5f44ba11a8fb0533",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1092",
       "triggerID" : "288e730d645a5e58e50a87ef5f44ba11a8fb0533",
       "triggerType" : "PUSH"
     }, {
       "hash" : "70a5c21321a0da3a0be8d85e206c24befa7b89f1",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1095",
       "triggerID" : "70a5c21321a0da3a0be8d85e206c24befa7b89f1",
       "triggerType" : "PUSH"
     }, {
       "hash" : "f072a986114949c5e426ce64a5f5568fc718e25d",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "f072a986114949c5e426ce64a5f5568fc718e25d",
       "triggerType" : "PUSH"
     }, {
       "hash" : "45b27475b1e8806ec8c566d6f151f4bc79c8549d",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1096",
       "triggerID" : "45b27475b1e8806ec8c566d6f151f4bc79c8549d",
       "triggerType" : "PUSH"
     }, {
       "hash" : "3c62ef22f6cde28110e5eb2fca8760a008d77723",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "3c62ef22f6cde28110e5eb2fca8760a008d77723",
       "triggerType" : "PUSH"
     }, {
       "hash" : "dc785fa04e542b2ccc45494e0292f5153c7ecea8",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1097",
       "triggerID" : "dc785fa04e542b2ccc45494e0292f5153c7ecea8",
       "triggerType" : "PUSH"
     }, {
       "hash" : "585ddb8049b9b9ccb057027e265b08a1b586df8c",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "585ddb8049b9b9ccb057027e265b08a1b586df8c",
       "triggerType" : "PUSH"
     }, {
       "hash" : "b62ff8675beed2c4487c788b9ca1a0ad4271b913",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "b62ff8675beed2c4487c788b9ca1a0ad4271b913",
       "triggerType" : "PUSH"
     }, {
       "hash" : "2d46048a7402907f348cc96da2839ba1b3e4932a",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1099",
       "triggerID" : "2d46048a7402907f348cc96da2839ba1b3e4932a",
       "triggerType" : "PUSH"
     }, {
       "hash" : "004fb6ebf23f796a4b77650e0530f0f1145e8b91",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1101",
       "triggerID" : "004fb6ebf23f796a4b77650e0530f0f1145e8b91",
       "triggerType" : "PUSH"
     }, {
       "hash" : "af0c33333b5e9ae6d59f20a6db1d7c016ea015d3",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1165",
       "triggerID" : "af0c33333b5e9ae6d59f20a6db1d7c016ea015d3",
       "triggerType" : "PUSH"
     }, {
       "hash" : "da8111118d71306500229619d3b7996fcd2526e8",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "da8111118d71306500229619d3b7996fcd2526e8",
       "triggerType" : "PUSH"
     }, {
       "hash" : "83257e2b9bd3c426cf90e11003a5f244a750d7fa",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1167",
       "triggerID" : "83257e2b9bd3c426cf90e11003a5f244a750d7fa",
       "triggerType" : "PUSH"
     }, {
       "hash" : "e0de408f1b8153fdaaea4c87b27a7b69213b75b5",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1177",
       "triggerID" : "e0de408f1b8153fdaaea4c87b27a7b69213b75b5",
       "triggerType" : "PUSH"
     }, {
       "hash" : "3f7951f7ad7931f5dc0b977d7c33bc5429623535",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1180",
       "triggerID" : "3f7951f7ad7931f5dc0b977d7c33bc5429623535",
       "triggerType" : "PUSH"
     }, {
       "hash" : "9cd6f06a753b63630e8245d51338d2bf9ab86b84",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "9cd6f06a753b63630e8245d51338d2bf9ab86b84",
       "triggerType" : "PUSH"
     }, {
       "hash" : "7380b902731a92e0645c02397e14591a05798620",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "7380b902731a92e0645c02397e14591a05798620",
       "triggerType" : "PUSH"
     }, {
       "hash" : "0bd5cca01a541845081d5430d48b021d4bdf0969",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "0bd5cca01a541845081d5430d48b021d4bdf0969",
       "triggerType" : "PUSH"
     }, {
       "hash" : "8f4e175ef554f015e34e0968b656b5de714ab430",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1182",
       "triggerID" : "8f4e175ef554f015e34e0968b656b5de714ab430",
       "triggerType" : "PUSH"
     }, {
       "hash" : "2023f3c882d0c2f5b96e28d2f21812dc0c64aaee",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "2023f3c882d0c2f5b96e28d2f21812dc0c64aaee",
       "triggerType" : "PUSH"
     }, {
       "hash" : "8394acf060d5c3953dbd67cf930b9b1d3b92d8b5",
       "status" : "SUCCESS",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1198",
       "triggerID" : "8394acf060d5c3953dbd67cf930b9b1d3b92d8b5",
       "triggerType" : "PUSH"
     }, {
       "hash" : "7ec42c105971bff463e359019405a7f4e7947d86",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "7ec42c105971bff463e359019405a7f4e7947d86",
       "triggerType" : "PUSH"
     }, {
       "hash" : "4b930786a7b2de09ac1653cd884ee76a7d1b2ca5",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "4b930786a7b2de09ac1653cd884ee76a7d1b2ca5",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 9e668f65c6d4f73c76573236f8fa507dfb80edac UNKNOWN
   * 533f00e5546da0026bbe51e1c030ddd1c19fdd08 UNKNOWN
   * a2ee90c549332a7f17a7d9b2799849b77c08a8e6 UNKNOWN
   * fc2c138901e82ed5f3fbf363c2a0ce8140ade050 UNKNOWN
   * a510a7de1458bb996df5bbb341e8d010e4c91381 UNKNOWN
   * f072a986114949c5e426ce64a5f5568fc718e25d UNKNOWN
   * 3c62ef22f6cde28110e5eb2fca8760a008d77723 UNKNOWN
   * 585ddb8049b9b9ccb057027e265b08a1b586df8c UNKNOWN
   * b62ff8675beed2c4487c788b9ca1a0ad4271b913 UNKNOWN
   * da8111118d71306500229619d3b7996fcd2526e8 UNKNOWN
   * 9cd6f06a753b63630e8245d51338d2bf9ab86b84 UNKNOWN
   * 7380b902731a92e0645c02397e14591a05798620 UNKNOWN
   * 0bd5cca01a541845081d5430d48b021d4bdf0969 UNKNOWN
   * 2023f3c882d0c2f5b96e28d2f21812dc0c64aaee UNKNOWN
   * 8394acf060d5c3953dbd67cf930b9b1d3b92d8b5 Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1198) 
   * 7ec42c105971bff463e359019405a7f4e7947d86 UNKNOWN
   * 4b930786a7b2de09ac1653cd884ee76a7d1b2ca5 UNKNOWN
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] [HUDI-8141] Incremental Query with Completion Time [hudi]

Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.

hudi-bot commented on PR #11947:
URL: https://github.com/apache/hudi/pull/11947#issuecomment-2421519370

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "090f256a315cbbf0c6a54d93cc2739dd693360e6",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=484",
       "triggerID" : "090f256a315cbbf0c6a54d93cc2739dd693360e6",
       "triggerType" : "PUSH"
     }, {
       "hash" : "0df5649013c1cf9e782fcbb67b8a2e7174f94512",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=985",
       "triggerID" : "0df5649013c1cf9e782fcbb67b8a2e7174f94512",
       "triggerType" : "PUSH"
     }, {
       "hash" : "d52e0128f409918644255abbe4376dc4a780bfc8",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=986",
       "triggerID" : "d52e0128f409918644255abbe4376dc4a780bfc8",
       "triggerType" : "PUSH"
     }, {
       "hash" : "9e668f65c6d4f73c76573236f8fa507dfb80edac",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "9e668f65c6d4f73c76573236f8fa507dfb80edac",
       "triggerType" : "PUSH"
     }, {
       "hash" : "f145131d524bf190986e5de600a326e32a27beca",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=987",
       "triggerID" : "f145131d524bf190986e5de600a326e32a27beca",
       "triggerType" : "PUSH"
     }, {
       "hash" : "0b008f807b42aa6fb2686922bb88d63e84a1f299",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=993",
       "triggerID" : "0b008f807b42aa6fb2686922bb88d63e84a1f299",
       "triggerType" : "PUSH"
     }, {
       "hash" : "65c01ad9f2b33d36d4890355feef91844438e732",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=995",
       "triggerID" : "65c01ad9f2b33d36d4890355feef91844438e732",
       "triggerType" : "PUSH"
     }, {
       "hash" : "955493fdc93842e8c99529cec35e39cecdbe9638",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=996",
       "triggerID" : "955493fdc93842e8c99529cec35e39cecdbe9638",
       "triggerType" : "PUSH"
     }, {
       "hash" : "533f00e5546da0026bbe51e1c030ddd1c19fdd08",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "533f00e5546da0026bbe51e1c030ddd1c19fdd08",
       "triggerType" : "PUSH"
     }, {
       "hash" : "05777be67e37f3a6a8eaa59e54d9e17e5fc6ba9b",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1016",
       "triggerID" : "05777be67e37f3a6a8eaa59e54d9e17e5fc6ba9b",
       "triggerType" : "PUSH"
     }, {
       "hash" : "a2ee90c549332a7f17a7d9b2799849b77c08a8e6",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "a2ee90c549332a7f17a7d9b2799849b77c08a8e6",
       "triggerType" : "PUSH"
     }, {
       "hash" : "05ce568b0b6ba14da9ca40d26b39ba478837d747",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1017",
       "triggerID" : "05ce568b0b6ba14da9ca40d26b39ba478837d747",
       "triggerType" : "PUSH"
     }, {
       "hash" : "840e3b33d39b1f86367bb6b0e65eadeb33de7765",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1025",
       "triggerID" : "840e3b33d39b1f86367bb6b0e65eadeb33de7765",
       "triggerType" : "PUSH"
     }, {
       "hash" : "8d2e2dd83f51d03e890d424a51d3e0e84e904a43",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1031",
       "triggerID" : "8d2e2dd83f51d03e890d424a51d3e0e84e904a43",
       "triggerType" : "PUSH"
     }, {
       "hash" : "688381914087ff11acf2f8c9372dbf39afa0bd40",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1035",
       "triggerID" : "688381914087ff11acf2f8c9372dbf39afa0bd40",
       "triggerType" : "PUSH"
     }, {
       "hash" : "fc2c138901e82ed5f3fbf363c2a0ce8140ade050",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "fc2c138901e82ed5f3fbf363c2a0ce8140ade050",
       "triggerType" : "PUSH"
     }, {
       "hash" : "6ed9e1cb1e5a191752d6d39c7a9d5d41840f0cdf",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1039",
       "triggerID" : "6ed9e1cb1e5a191752d6d39c7a9d5d41840f0cdf",
       "triggerType" : "PUSH"
     }, {
       "hash" : "a510a7de1458bb996df5bbb341e8d010e4c91381",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "a510a7de1458bb996df5bbb341e8d010e4c91381",
       "triggerType" : "PUSH"
     }, {
       "hash" : "49677397cd117228742205944606f7a789090376",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1042",
       "triggerID" : "49677397cd117228742205944606f7a789090376",
       "triggerType" : "PUSH"
     }, {
       "hash" : "1630db0a37422aed942ff280560a9d6329df5015",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1044",
       "triggerID" : "1630db0a37422aed942ff280560a9d6329df5015",
       "triggerType" : "PUSH"
     }, {
       "hash" : "f11c76301d4ab524db0af57aa831be3a898ea0fb",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1048",
       "triggerID" : "f11c76301d4ab524db0af57aa831be3a898ea0fb",
       "triggerType" : "PUSH"
     }, {
       "hash" : "d1a0dc4c2017853a7ba0c3302f886043a63f29da",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1050",
       "triggerID" : "d1a0dc4c2017853a7ba0c3302f886043a63f29da",
       "triggerType" : "PUSH"
     }, {
       "hash" : "6a4c15ec39ded1f90b113b272554217698a98e1c",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1087",
       "triggerID" : "6a4c15ec39ded1f90b113b272554217698a98e1c",
       "triggerType" : "PUSH"
     }, {
       "hash" : "288e730d645a5e58e50a87ef5f44ba11a8fb0533",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1092",
       "triggerID" : "288e730d645a5e58e50a87ef5f44ba11a8fb0533",
       "triggerType" : "PUSH"
     }, {
       "hash" : "70a5c21321a0da3a0be8d85e206c24befa7b89f1",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1095",
       "triggerID" : "70a5c21321a0da3a0be8d85e206c24befa7b89f1",
       "triggerType" : "PUSH"
     }, {
       "hash" : "f072a986114949c5e426ce64a5f5568fc718e25d",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "f072a986114949c5e426ce64a5f5568fc718e25d",
       "triggerType" : "PUSH"
     }, {
       "hash" : "45b27475b1e8806ec8c566d6f151f4bc79c8549d",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1096",
       "triggerID" : "45b27475b1e8806ec8c566d6f151f4bc79c8549d",
       "triggerType" : "PUSH"
     }, {
       "hash" : "3c62ef22f6cde28110e5eb2fca8760a008d77723",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "3c62ef22f6cde28110e5eb2fca8760a008d77723",
       "triggerType" : "PUSH"
     }, {
       "hash" : "dc785fa04e542b2ccc45494e0292f5153c7ecea8",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1097",
       "triggerID" : "dc785fa04e542b2ccc45494e0292f5153c7ecea8",
       "triggerType" : "PUSH"
     }, {
       "hash" : "585ddb8049b9b9ccb057027e265b08a1b586df8c",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "585ddb8049b9b9ccb057027e265b08a1b586df8c",
       "triggerType" : "PUSH"
     }, {
       "hash" : "b62ff8675beed2c4487c788b9ca1a0ad4271b913",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "b62ff8675beed2c4487c788b9ca1a0ad4271b913",
       "triggerType" : "PUSH"
     }, {
       "hash" : "2d46048a7402907f348cc96da2839ba1b3e4932a",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1099",
       "triggerID" : "2d46048a7402907f348cc96da2839ba1b3e4932a",
       "triggerType" : "PUSH"
     }, {
       "hash" : "004fb6ebf23f796a4b77650e0530f0f1145e8b91",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1101",
       "triggerID" : "004fb6ebf23f796a4b77650e0530f0f1145e8b91",
       "triggerType" : "PUSH"
     }, {
       "hash" : "af0c33333b5e9ae6d59f20a6db1d7c016ea015d3",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1165",
       "triggerID" : "af0c33333b5e9ae6d59f20a6db1d7c016ea015d3",
       "triggerType" : "PUSH"
     }, {
       "hash" : "da8111118d71306500229619d3b7996fcd2526e8",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "da8111118d71306500229619d3b7996fcd2526e8",
       "triggerType" : "PUSH"
     }, {
       "hash" : "83257e2b9bd3c426cf90e11003a5f244a750d7fa",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1167",
       "triggerID" : "83257e2b9bd3c426cf90e11003a5f244a750d7fa",
       "triggerType" : "PUSH"
     }, {
       "hash" : "e0de408f1b8153fdaaea4c87b27a7b69213b75b5",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1177",
       "triggerID" : "e0de408f1b8153fdaaea4c87b27a7b69213b75b5",
       "triggerType" : "PUSH"
     }, {
       "hash" : "3f7951f7ad7931f5dc0b977d7c33bc5429623535",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1180",
       "triggerID" : "3f7951f7ad7931f5dc0b977d7c33bc5429623535",
       "triggerType" : "PUSH"
     }, {
       "hash" : "9cd6f06a753b63630e8245d51338d2bf9ab86b84",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "9cd6f06a753b63630e8245d51338d2bf9ab86b84",
       "triggerType" : "PUSH"
     }, {
       "hash" : "7380b902731a92e0645c02397e14591a05798620",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "7380b902731a92e0645c02397e14591a05798620",
       "triggerType" : "PUSH"
     }, {
       "hash" : "0bd5cca01a541845081d5430d48b021d4bdf0969",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "0bd5cca01a541845081d5430d48b021d4bdf0969",
       "triggerType" : "PUSH"
     }, {
       "hash" : "8f4e175ef554f015e34e0968b656b5de714ab430",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1182",
       "triggerID" : "8f4e175ef554f015e34e0968b656b5de714ab430",
       "triggerType" : "PUSH"
     }, {
       "hash" : "2023f3c882d0c2f5b96e28d2f21812dc0c64aaee",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "2023f3c882d0c2f5b96e28d2f21812dc0c64aaee",
       "triggerType" : "PUSH"
     }, {
       "hash" : "8394acf060d5c3953dbd67cf930b9b1d3b92d8b5",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1198",
       "triggerID" : "8394acf060d5c3953dbd67cf930b9b1d3b92d8b5",
       "triggerType" : "PUSH"
     }, {
       "hash" : "7ec42c105971bff463e359019405a7f4e7947d86",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "7ec42c105971bff463e359019405a7f4e7947d86",
       "triggerType" : "PUSH"
     }, {
       "hash" : "4b930786a7b2de09ac1653cd884ee76a7d1b2ca5",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1202",
       "triggerID" : "4b930786a7b2de09ac1653cd884ee76a7d1b2ca5",
       "triggerType" : "PUSH"
     }, {
       "hash" : "0e9a648c4e03b5a74d6a0085831805041a9a26a2",
       "status" : "SUCCESS",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1216",
       "triggerID" : "0e9a648c4e03b5a74d6a0085831805041a9a26a2",
       "triggerType" : "PUSH"
     }, {
       "hash" : "326a8729a5d1dd88bc474a5c273aec6ca219f298",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "326a8729a5d1dd88bc474a5c273aec6ca219f298",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 9e668f65c6d4f73c76573236f8fa507dfb80edac UNKNOWN
   * 533f00e5546da0026bbe51e1c030ddd1c19fdd08 UNKNOWN
   * a2ee90c549332a7f17a7d9b2799849b77c08a8e6 UNKNOWN
   * fc2c138901e82ed5f3fbf363c2a0ce8140ade050 UNKNOWN
   * a510a7de1458bb996df5bbb341e8d010e4c91381 UNKNOWN
   * f072a986114949c5e426ce64a5f5568fc718e25d UNKNOWN
   * 3c62ef22f6cde28110e5eb2fca8760a008d77723 UNKNOWN
   * 585ddb8049b9b9ccb057027e265b08a1b586df8c UNKNOWN
   * b62ff8675beed2c4487c788b9ca1a0ad4271b913 UNKNOWN
   * da8111118d71306500229619d3b7996fcd2526e8 UNKNOWN
   * 9cd6f06a753b63630e8245d51338d2bf9ab86b84 UNKNOWN
   * 7380b902731a92e0645c02397e14591a05798620 UNKNOWN
   * 0bd5cca01a541845081d5430d48b021d4bdf0969 UNKNOWN
   * 2023f3c882d0c2f5b96e28d2f21812dc0c64aaee UNKNOWN
   * 7ec42c105971bff463e359019405a7f4e7947d86 UNKNOWN
   * 0e9a648c4e03b5a74d6a0085831805041a9a26a2 Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1216) 
   * 326a8729a5d1dd88bc474a5c273aec6ca219f298 UNKNOWN
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] [HUDI-8141] Incremental Query with Completion Time [hudi]

Posted by "yihua (via GitHub)" <gi...@apache.org>.

yihua commented on code in PR #11947:
URL: https://github.com/apache/hudi/pull/11947#discussion_r1805924151


##########
hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/spark/sql/hudi/streaming/HoodieStreamSource.scala:
##########
@@ -186,11 +160,11 @@ class HoodieStreamSource(
     }
   }
 
-  private def startCommitTime(startOffset: HoodieSourceOffset): String = {
+  private def startCompletionTime(startOffset: HoodieSourceOffset): String = {
     startOffset match {
-      case INIT_OFFSET => startOffset.commitTime
-      case HoodieSourceOffset(commitTime) =>
-        commitTime
+      case INIT_OFFSET => startOffset.completionTime
+      case HoodieSourceOffset(completionTime) =>
+        HoodieInstantTimeGenerator.instantTimePlusMillis(completionTime, 1)

Review Comment:
   I understand your point now.  I've changed the `IncrementalRelation`, `MergeOnReadIncremetnalRelation`, and `CDCRelation` to take `rangeType` as an argument so it can be dynamically changed based on different scenarios.  Now the logic in `HoodieStreamSource` is easy to understand, without using `HoodieInstantTimeGenerator#instantTimePlusMillis`.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] [HUDI-8141] Incremental Query with Completion Time [hudi]

Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.

hudi-bot commented on PR #11947:
URL: https://github.com/apache/hudi/pull/11947#issuecomment-2418630107

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "090f256a315cbbf0c6a54d93cc2739dd693360e6",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=484",
       "triggerID" : "090f256a315cbbf0c6a54d93cc2739dd693360e6",
       "triggerType" : "PUSH"
     }, {
       "hash" : "0df5649013c1cf9e782fcbb67b8a2e7174f94512",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=985",
       "triggerID" : "0df5649013c1cf9e782fcbb67b8a2e7174f94512",
       "triggerType" : "PUSH"
     }, {
       "hash" : "d52e0128f409918644255abbe4376dc4a780bfc8",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=986",
       "triggerID" : "d52e0128f409918644255abbe4376dc4a780bfc8",
       "triggerType" : "PUSH"
     }, {
       "hash" : "9e668f65c6d4f73c76573236f8fa507dfb80edac",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "9e668f65c6d4f73c76573236f8fa507dfb80edac",
       "triggerType" : "PUSH"
     }, {
       "hash" : "f145131d524bf190986e5de600a326e32a27beca",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=987",
       "triggerID" : "f145131d524bf190986e5de600a326e32a27beca",
       "triggerType" : "PUSH"
     }, {
       "hash" : "0b008f807b42aa6fb2686922bb88d63e84a1f299",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=993",
       "triggerID" : "0b008f807b42aa6fb2686922bb88d63e84a1f299",
       "triggerType" : "PUSH"
     }, {
       "hash" : "65c01ad9f2b33d36d4890355feef91844438e732",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=995",
       "triggerID" : "65c01ad9f2b33d36d4890355feef91844438e732",
       "triggerType" : "PUSH"
     }, {
       "hash" : "955493fdc93842e8c99529cec35e39cecdbe9638",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=996",
       "triggerID" : "955493fdc93842e8c99529cec35e39cecdbe9638",
       "triggerType" : "PUSH"
     }, {
       "hash" : "533f00e5546da0026bbe51e1c030ddd1c19fdd08",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "533f00e5546da0026bbe51e1c030ddd1c19fdd08",
       "triggerType" : "PUSH"
     }, {
       "hash" : "05777be67e37f3a6a8eaa59e54d9e17e5fc6ba9b",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1016",
       "triggerID" : "05777be67e37f3a6a8eaa59e54d9e17e5fc6ba9b",
       "triggerType" : "PUSH"
     }, {
       "hash" : "a2ee90c549332a7f17a7d9b2799849b77c08a8e6",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "a2ee90c549332a7f17a7d9b2799849b77c08a8e6",
       "triggerType" : "PUSH"
     }, {
       "hash" : "05ce568b0b6ba14da9ca40d26b39ba478837d747",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1017",
       "triggerID" : "05ce568b0b6ba14da9ca40d26b39ba478837d747",
       "triggerType" : "PUSH"
     }, {
       "hash" : "840e3b33d39b1f86367bb6b0e65eadeb33de7765",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1025",
       "triggerID" : "840e3b33d39b1f86367bb6b0e65eadeb33de7765",
       "triggerType" : "PUSH"
     }, {
       "hash" : "8d2e2dd83f51d03e890d424a51d3e0e84e904a43",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1031",
       "triggerID" : "8d2e2dd83f51d03e890d424a51d3e0e84e904a43",
       "triggerType" : "PUSH"
     }, {
       "hash" : "688381914087ff11acf2f8c9372dbf39afa0bd40",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1035",
       "triggerID" : "688381914087ff11acf2f8c9372dbf39afa0bd40",
       "triggerType" : "PUSH"
     }, {
       "hash" : "fc2c138901e82ed5f3fbf363c2a0ce8140ade050",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "fc2c138901e82ed5f3fbf363c2a0ce8140ade050",
       "triggerType" : "PUSH"
     }, {
       "hash" : "6ed9e1cb1e5a191752d6d39c7a9d5d41840f0cdf",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1039",
       "triggerID" : "6ed9e1cb1e5a191752d6d39c7a9d5d41840f0cdf",
       "triggerType" : "PUSH"
     }, {
       "hash" : "a510a7de1458bb996df5bbb341e8d010e4c91381",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "a510a7de1458bb996df5bbb341e8d010e4c91381",
       "triggerType" : "PUSH"
     }, {
       "hash" : "49677397cd117228742205944606f7a789090376",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1042",
       "triggerID" : "49677397cd117228742205944606f7a789090376",
       "triggerType" : "PUSH"
     }, {
       "hash" : "1630db0a37422aed942ff280560a9d6329df5015",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1044",
       "triggerID" : "1630db0a37422aed942ff280560a9d6329df5015",
       "triggerType" : "PUSH"
     }, {
       "hash" : "f11c76301d4ab524db0af57aa831be3a898ea0fb",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1048",
       "triggerID" : "f11c76301d4ab524db0af57aa831be3a898ea0fb",
       "triggerType" : "PUSH"
     }, {
       "hash" : "d1a0dc4c2017853a7ba0c3302f886043a63f29da",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1050",
       "triggerID" : "d1a0dc4c2017853a7ba0c3302f886043a63f29da",
       "triggerType" : "PUSH"
     }, {
       "hash" : "6a4c15ec39ded1f90b113b272554217698a98e1c",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1087",
       "triggerID" : "6a4c15ec39ded1f90b113b272554217698a98e1c",
       "triggerType" : "PUSH"
     }, {
       "hash" : "288e730d645a5e58e50a87ef5f44ba11a8fb0533",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1092",
       "triggerID" : "288e730d645a5e58e50a87ef5f44ba11a8fb0533",
       "triggerType" : "PUSH"
     }, {
       "hash" : "70a5c21321a0da3a0be8d85e206c24befa7b89f1",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1095",
       "triggerID" : "70a5c21321a0da3a0be8d85e206c24befa7b89f1",
       "triggerType" : "PUSH"
     }, {
       "hash" : "f072a986114949c5e426ce64a5f5568fc718e25d",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "f072a986114949c5e426ce64a5f5568fc718e25d",
       "triggerType" : "PUSH"
     }, {
       "hash" : "45b27475b1e8806ec8c566d6f151f4bc79c8549d",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1096",
       "triggerID" : "45b27475b1e8806ec8c566d6f151f4bc79c8549d",
       "triggerType" : "PUSH"
     }, {
       "hash" : "3c62ef22f6cde28110e5eb2fca8760a008d77723",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "3c62ef22f6cde28110e5eb2fca8760a008d77723",
       "triggerType" : "PUSH"
     }, {
       "hash" : "dc785fa04e542b2ccc45494e0292f5153c7ecea8",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1097",
       "triggerID" : "dc785fa04e542b2ccc45494e0292f5153c7ecea8",
       "triggerType" : "PUSH"
     }, {
       "hash" : "585ddb8049b9b9ccb057027e265b08a1b586df8c",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "585ddb8049b9b9ccb057027e265b08a1b586df8c",
       "triggerType" : "PUSH"
     }, {
       "hash" : "b62ff8675beed2c4487c788b9ca1a0ad4271b913",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "b62ff8675beed2c4487c788b9ca1a0ad4271b913",
       "triggerType" : "PUSH"
     }, {
       "hash" : "2d46048a7402907f348cc96da2839ba1b3e4932a",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1099",
       "triggerID" : "2d46048a7402907f348cc96da2839ba1b3e4932a",
       "triggerType" : "PUSH"
     }, {
       "hash" : "004fb6ebf23f796a4b77650e0530f0f1145e8b91",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1101",
       "triggerID" : "004fb6ebf23f796a4b77650e0530f0f1145e8b91",
       "triggerType" : "PUSH"
     }, {
       "hash" : "af0c33333b5e9ae6d59f20a6db1d7c016ea015d3",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1165",
       "triggerID" : "af0c33333b5e9ae6d59f20a6db1d7c016ea015d3",
       "triggerType" : "PUSH"
     }, {
       "hash" : "da8111118d71306500229619d3b7996fcd2526e8",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "da8111118d71306500229619d3b7996fcd2526e8",
       "triggerType" : "PUSH"
     }, {
       "hash" : "83257e2b9bd3c426cf90e11003a5f244a750d7fa",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1167",
       "triggerID" : "83257e2b9bd3c426cf90e11003a5f244a750d7fa",
       "triggerType" : "PUSH"
     }, {
       "hash" : "e0de408f1b8153fdaaea4c87b27a7b69213b75b5",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1177",
       "triggerID" : "e0de408f1b8153fdaaea4c87b27a7b69213b75b5",
       "triggerType" : "PUSH"
     }, {
       "hash" : "3f7951f7ad7931f5dc0b977d7c33bc5429623535",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1180",
       "triggerID" : "3f7951f7ad7931f5dc0b977d7c33bc5429623535",
       "triggerType" : "PUSH"
     }, {
       "hash" : "9cd6f06a753b63630e8245d51338d2bf9ab86b84",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "9cd6f06a753b63630e8245d51338d2bf9ab86b84",
       "triggerType" : "PUSH"
     }, {
       "hash" : "7380b902731a92e0645c02397e14591a05798620",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "7380b902731a92e0645c02397e14591a05798620",
       "triggerType" : "PUSH"
     }, {
       "hash" : "0bd5cca01a541845081d5430d48b021d4bdf0969",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "0bd5cca01a541845081d5430d48b021d4bdf0969",
       "triggerType" : "PUSH"
     }, {
       "hash" : "8f4e175ef554f015e34e0968b656b5de714ab430",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1182",
       "triggerID" : "8f4e175ef554f015e34e0968b656b5de714ab430",
       "triggerType" : "PUSH"
     }, {
       "hash" : "2023f3c882d0c2f5b96e28d2f21812dc0c64aaee",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "2023f3c882d0c2f5b96e28d2f21812dc0c64aaee",
       "triggerType" : "PUSH"
     }, {
       "hash" : "8394acf060d5c3953dbd67cf930b9b1d3b92d8b5",
       "status" : "SUCCESS",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1198",
       "triggerID" : "8394acf060d5c3953dbd67cf930b9b1d3b92d8b5",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 9e668f65c6d4f73c76573236f8fa507dfb80edac UNKNOWN
   * 533f00e5546da0026bbe51e1c030ddd1c19fdd08 UNKNOWN
   * a2ee90c549332a7f17a7d9b2799849b77c08a8e6 UNKNOWN
   * fc2c138901e82ed5f3fbf363c2a0ce8140ade050 UNKNOWN
   * a510a7de1458bb996df5bbb341e8d010e4c91381 UNKNOWN
   * f072a986114949c5e426ce64a5f5568fc718e25d UNKNOWN
   * 3c62ef22f6cde28110e5eb2fca8760a008d77723 UNKNOWN
   * 585ddb8049b9b9ccb057027e265b08a1b586df8c UNKNOWN
   * b62ff8675beed2c4487c788b9ca1a0ad4271b913 UNKNOWN
   * da8111118d71306500229619d3b7996fcd2526e8 UNKNOWN
   * 9cd6f06a753b63630e8245d51338d2bf9ab86b84 UNKNOWN
   * 7380b902731a92e0645c02397e14591a05798620 UNKNOWN
   * 0bd5cca01a541845081d5430d48b021d4bdf0969 UNKNOWN
   * 2023f3c882d0c2f5b96e28d2f21812dc0c64aaee UNKNOWN
   * 8394acf060d5c3953dbd67cf930b9b1d3b92d8b5 Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1198) 
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] [HUDI-8141] Incremental Query with Completion Time [hudi]

Posted by "yihua (via GitHub)" <gi...@apache.org>.

yihua commented on code in PR #11947:
URL: https://github.com/apache/hudi/pull/11947#discussion_r1805757843


##########
hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/spark/sql/hudi/streaming/HoodieStreamSource.scala:
##########
@@ -186,11 +160,11 @@ class HoodieStreamSource(
     }
   }
 
-  private def startCommitTime(startOffset: HoodieSourceOffset): String = {
+  private def startCompletionTime(startOffset: HoodieSourceOffset): String = {
     startOffset match {
-      case INIT_OFFSET => startOffset.commitTime
-      case HoodieSourceOffset(commitTime) =>
-        commitTime
+      case INIT_OFFSET => startOffset.completionTime
+      case HoodieSourceOffset(completionTime) =>
+        HoodieInstantTimeGenerator.instantTimePlusMillis(completionTime, 1)

Review Comment:
   Yes in the sense of `OPEN_CLOSED` range type.  However,`CLOSED_CLOSED` range type is used in the Spark incremental relation.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] [HUDI-8141] Incremental Query with Completion Time [hudi]

Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.

hudi-bot commented on PR #11947:
URL: https://github.com/apache/hudi/pull/11947#issuecomment-2421837115

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "090f256a315cbbf0c6a54d93cc2739dd693360e6",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=484",
       "triggerID" : "090f256a315cbbf0c6a54d93cc2739dd693360e6",
       "triggerType" : "PUSH"
     }, {
       "hash" : "0df5649013c1cf9e782fcbb67b8a2e7174f94512",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=985",
       "triggerID" : "0df5649013c1cf9e782fcbb67b8a2e7174f94512",
       "triggerType" : "PUSH"
     }, {
       "hash" : "d52e0128f409918644255abbe4376dc4a780bfc8",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=986",
       "triggerID" : "d52e0128f409918644255abbe4376dc4a780bfc8",
       "triggerType" : "PUSH"
     }, {
       "hash" : "9e668f65c6d4f73c76573236f8fa507dfb80edac",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "9e668f65c6d4f73c76573236f8fa507dfb80edac",
       "triggerType" : "PUSH"
     }, {
       "hash" : "f145131d524bf190986e5de600a326e32a27beca",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=987",
       "triggerID" : "f145131d524bf190986e5de600a326e32a27beca",
       "triggerType" : "PUSH"
     }, {
       "hash" : "0b008f807b42aa6fb2686922bb88d63e84a1f299",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=993",
       "triggerID" : "0b008f807b42aa6fb2686922bb88d63e84a1f299",
       "triggerType" : "PUSH"
     }, {
       "hash" : "65c01ad9f2b33d36d4890355feef91844438e732",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=995",
       "triggerID" : "65c01ad9f2b33d36d4890355feef91844438e732",
       "triggerType" : "PUSH"
     }, {
       "hash" : "955493fdc93842e8c99529cec35e39cecdbe9638",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=996",
       "triggerID" : "955493fdc93842e8c99529cec35e39cecdbe9638",
       "triggerType" : "PUSH"
     }, {
       "hash" : "533f00e5546da0026bbe51e1c030ddd1c19fdd08",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "533f00e5546da0026bbe51e1c030ddd1c19fdd08",
       "triggerType" : "PUSH"
     }, {
       "hash" : "05777be67e37f3a6a8eaa59e54d9e17e5fc6ba9b",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1016",
       "triggerID" : "05777be67e37f3a6a8eaa59e54d9e17e5fc6ba9b",
       "triggerType" : "PUSH"
     }, {
       "hash" : "a2ee90c549332a7f17a7d9b2799849b77c08a8e6",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "a2ee90c549332a7f17a7d9b2799849b77c08a8e6",
       "triggerType" : "PUSH"
     }, {
       "hash" : "05ce568b0b6ba14da9ca40d26b39ba478837d747",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1017",
       "triggerID" : "05ce568b0b6ba14da9ca40d26b39ba478837d747",
       "triggerType" : "PUSH"
     }, {
       "hash" : "840e3b33d39b1f86367bb6b0e65eadeb33de7765",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1025",
       "triggerID" : "840e3b33d39b1f86367bb6b0e65eadeb33de7765",
       "triggerType" : "PUSH"
     }, {
       "hash" : "8d2e2dd83f51d03e890d424a51d3e0e84e904a43",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1031",
       "triggerID" : "8d2e2dd83f51d03e890d424a51d3e0e84e904a43",
       "triggerType" : "PUSH"
     }, {
       "hash" : "688381914087ff11acf2f8c9372dbf39afa0bd40",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1035",
       "triggerID" : "688381914087ff11acf2f8c9372dbf39afa0bd40",
       "triggerType" : "PUSH"
     }, {
       "hash" : "fc2c138901e82ed5f3fbf363c2a0ce8140ade050",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "fc2c138901e82ed5f3fbf363c2a0ce8140ade050",
       "triggerType" : "PUSH"
     }, {
       "hash" : "6ed9e1cb1e5a191752d6d39c7a9d5d41840f0cdf",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1039",
       "triggerID" : "6ed9e1cb1e5a191752d6d39c7a9d5d41840f0cdf",
       "triggerType" : "PUSH"
     }, {
       "hash" : "a510a7de1458bb996df5bbb341e8d010e4c91381",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "a510a7de1458bb996df5bbb341e8d010e4c91381",
       "triggerType" : "PUSH"
     }, {
       "hash" : "49677397cd117228742205944606f7a789090376",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1042",
       "triggerID" : "49677397cd117228742205944606f7a789090376",
       "triggerType" : "PUSH"
     }, {
       "hash" : "1630db0a37422aed942ff280560a9d6329df5015",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1044",
       "triggerID" : "1630db0a37422aed942ff280560a9d6329df5015",
       "triggerType" : "PUSH"
     }, {
       "hash" : "f11c76301d4ab524db0af57aa831be3a898ea0fb",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1048",
       "triggerID" : "f11c76301d4ab524db0af57aa831be3a898ea0fb",
       "triggerType" : "PUSH"
     }, {
       "hash" : "d1a0dc4c2017853a7ba0c3302f886043a63f29da",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1050",
       "triggerID" : "d1a0dc4c2017853a7ba0c3302f886043a63f29da",
       "triggerType" : "PUSH"
     }, {
       "hash" : "6a4c15ec39ded1f90b113b272554217698a98e1c",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1087",
       "triggerID" : "6a4c15ec39ded1f90b113b272554217698a98e1c",
       "triggerType" : "PUSH"
     }, {
       "hash" : "288e730d645a5e58e50a87ef5f44ba11a8fb0533",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1092",
       "triggerID" : "288e730d645a5e58e50a87ef5f44ba11a8fb0533",
       "triggerType" : "PUSH"
     }, {
       "hash" : "70a5c21321a0da3a0be8d85e206c24befa7b89f1",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1095",
       "triggerID" : "70a5c21321a0da3a0be8d85e206c24befa7b89f1",
       "triggerType" : "PUSH"
     }, {
       "hash" : "f072a986114949c5e426ce64a5f5568fc718e25d",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "f072a986114949c5e426ce64a5f5568fc718e25d",
       "triggerType" : "PUSH"
     }, {
       "hash" : "45b27475b1e8806ec8c566d6f151f4bc79c8549d",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1096",
       "triggerID" : "45b27475b1e8806ec8c566d6f151f4bc79c8549d",
       "triggerType" : "PUSH"
     }, {
       "hash" : "3c62ef22f6cde28110e5eb2fca8760a008d77723",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "3c62ef22f6cde28110e5eb2fca8760a008d77723",
       "triggerType" : "PUSH"
     }, {
       "hash" : "dc785fa04e542b2ccc45494e0292f5153c7ecea8",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1097",
       "triggerID" : "dc785fa04e542b2ccc45494e0292f5153c7ecea8",
       "triggerType" : "PUSH"
     }, {
       "hash" : "585ddb8049b9b9ccb057027e265b08a1b586df8c",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "585ddb8049b9b9ccb057027e265b08a1b586df8c",
       "triggerType" : "PUSH"
     }, {
       "hash" : "b62ff8675beed2c4487c788b9ca1a0ad4271b913",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "b62ff8675beed2c4487c788b9ca1a0ad4271b913",
       "triggerType" : "PUSH"
     }, {
       "hash" : "2d46048a7402907f348cc96da2839ba1b3e4932a",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1099",
       "triggerID" : "2d46048a7402907f348cc96da2839ba1b3e4932a",
       "triggerType" : "PUSH"
     }, {
       "hash" : "004fb6ebf23f796a4b77650e0530f0f1145e8b91",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1101",
       "triggerID" : "004fb6ebf23f796a4b77650e0530f0f1145e8b91",
       "triggerType" : "PUSH"
     }, {
       "hash" : "af0c33333b5e9ae6d59f20a6db1d7c016ea015d3",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1165",
       "triggerID" : "af0c33333b5e9ae6d59f20a6db1d7c016ea015d3",
       "triggerType" : "PUSH"
     }, {
       "hash" : "da8111118d71306500229619d3b7996fcd2526e8",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "da8111118d71306500229619d3b7996fcd2526e8",
       "triggerType" : "PUSH"
     }, {
       "hash" : "83257e2b9bd3c426cf90e11003a5f244a750d7fa",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1167",
       "triggerID" : "83257e2b9bd3c426cf90e11003a5f244a750d7fa",
       "triggerType" : "PUSH"
     }, {
       "hash" : "e0de408f1b8153fdaaea4c87b27a7b69213b75b5",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1177",
       "triggerID" : "e0de408f1b8153fdaaea4c87b27a7b69213b75b5",
       "triggerType" : "PUSH"
     }, {
       "hash" : "3f7951f7ad7931f5dc0b977d7c33bc5429623535",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1180",
       "triggerID" : "3f7951f7ad7931f5dc0b977d7c33bc5429623535",
       "triggerType" : "PUSH"
     }, {
       "hash" : "9cd6f06a753b63630e8245d51338d2bf9ab86b84",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "9cd6f06a753b63630e8245d51338d2bf9ab86b84",
       "triggerType" : "PUSH"
     }, {
       "hash" : "7380b902731a92e0645c02397e14591a05798620",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "7380b902731a92e0645c02397e14591a05798620",
       "triggerType" : "PUSH"
     }, {
       "hash" : "0bd5cca01a541845081d5430d48b021d4bdf0969",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "0bd5cca01a541845081d5430d48b021d4bdf0969",
       "triggerType" : "PUSH"
     }, {
       "hash" : "8f4e175ef554f015e34e0968b656b5de714ab430",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1182",
       "triggerID" : "8f4e175ef554f015e34e0968b656b5de714ab430",
       "triggerType" : "PUSH"
     }, {
       "hash" : "2023f3c882d0c2f5b96e28d2f21812dc0c64aaee",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "2023f3c882d0c2f5b96e28d2f21812dc0c64aaee",
       "triggerType" : "PUSH"
     }, {
       "hash" : "8394acf060d5c3953dbd67cf930b9b1d3b92d8b5",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1198",
       "triggerID" : "8394acf060d5c3953dbd67cf930b9b1d3b92d8b5",
       "triggerType" : "PUSH"
     }, {
       "hash" : "7ec42c105971bff463e359019405a7f4e7947d86",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "7ec42c105971bff463e359019405a7f4e7947d86",
       "triggerType" : "PUSH"
     }, {
       "hash" : "4b930786a7b2de09ac1653cd884ee76a7d1b2ca5",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1202",
       "triggerID" : "4b930786a7b2de09ac1653cd884ee76a7d1b2ca5",
       "triggerType" : "PUSH"
     }, {
       "hash" : "0e9a648c4e03b5a74d6a0085831805041a9a26a2",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1216",
       "triggerID" : "0e9a648c4e03b5a74d6a0085831805041a9a26a2",
       "triggerType" : "PUSH"
     }, {
       "hash" : "326a8729a5d1dd88bc474a5c273aec6ca219f298",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "326a8729a5d1dd88bc474a5c273aec6ca219f298",
       "triggerType" : "PUSH"
     }, {
       "hash" : "cad2f761fc50fba28e2fb3595e4b1270e39ac48a",
       "status" : "CANCELED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1221",
       "triggerID" : "cad2f761fc50fba28e2fb3595e4b1270e39ac48a",
       "triggerType" : "PUSH"
     }, {
       "hash" : "d485a50aec34bc5707377afe96d696772832558e",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "d485a50aec34bc5707377afe96d696772832558e",
       "triggerType" : "PUSH"
     }, {
       "hash" : "38a98106f19fe4e51632a2c4289ec0f01805ba5e",
       "status" : "PENDING",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1224",
       "triggerID" : "38a98106f19fe4e51632a2c4289ec0f01805ba5e",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 9e668f65c6d4f73c76573236f8fa507dfb80edac UNKNOWN
   * 533f00e5546da0026bbe51e1c030ddd1c19fdd08 UNKNOWN
   * a2ee90c549332a7f17a7d9b2799849b77c08a8e6 UNKNOWN
   * fc2c138901e82ed5f3fbf363c2a0ce8140ade050 UNKNOWN
   * a510a7de1458bb996df5bbb341e8d010e4c91381 UNKNOWN
   * f072a986114949c5e426ce64a5f5568fc718e25d UNKNOWN
   * 3c62ef22f6cde28110e5eb2fca8760a008d77723 UNKNOWN
   * 585ddb8049b9b9ccb057027e265b08a1b586df8c UNKNOWN
   * b62ff8675beed2c4487c788b9ca1a0ad4271b913 UNKNOWN
   * da8111118d71306500229619d3b7996fcd2526e8 UNKNOWN
   * 9cd6f06a753b63630e8245d51338d2bf9ab86b84 UNKNOWN
   * 7380b902731a92e0645c02397e14591a05798620 UNKNOWN
   * 0bd5cca01a541845081d5430d48b021d4bdf0969 UNKNOWN
   * 2023f3c882d0c2f5b96e28d2f21812dc0c64aaee UNKNOWN
   * 7ec42c105971bff463e359019405a7f4e7947d86 UNKNOWN
   * 326a8729a5d1dd88bc474a5c273aec6ca219f298 UNKNOWN
   * cad2f761fc50fba28e2fb3595e4b1270e39ac48a Azure: [CANCELED](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1221) 
   * d485a50aec34bc5707377afe96d696772832558e UNKNOWN
   * 38a98106f19fe4e51632a2c4289ec0f01805ba5e Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1224) 
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] [HUDI-8141] Always use completion time to find begin Instant for incremental source [hudi]

Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.

hudi-bot commented on PR #11947:
URL: https://github.com/apache/hudi/pull/11947#issuecomment-2403898852

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "090f256a315cbbf0c6a54d93cc2739dd693360e6",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=484",
       "triggerID" : "090f256a315cbbf0c6a54d93cc2739dd693360e6",
       "triggerType" : "PUSH"
     }, {
       "hash" : "0df5649013c1cf9e782fcbb67b8a2e7174f94512",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=985",
       "triggerID" : "0df5649013c1cf9e782fcbb67b8a2e7174f94512",
       "triggerType" : "PUSH"
     }, {
       "hash" : "d52e0128f409918644255abbe4376dc4a780bfc8",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=986",
       "triggerID" : "d52e0128f409918644255abbe4376dc4a780bfc8",
       "triggerType" : "PUSH"
     }, {
       "hash" : "9e668f65c6d4f73c76573236f8fa507dfb80edac",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "9e668f65c6d4f73c76573236f8fa507dfb80edac",
       "triggerType" : "PUSH"
     }, {
       "hash" : "f145131d524bf190986e5de600a326e32a27beca",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=987",
       "triggerID" : "f145131d524bf190986e5de600a326e32a27beca",
       "triggerType" : "PUSH"
     }, {
       "hash" : "0b008f807b42aa6fb2686922bb88d63e84a1f299",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=993",
       "triggerID" : "0b008f807b42aa6fb2686922bb88d63e84a1f299",
       "triggerType" : "PUSH"
     }, {
       "hash" : "65c01ad9f2b33d36d4890355feef91844438e732",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=995",
       "triggerID" : "65c01ad9f2b33d36d4890355feef91844438e732",
       "triggerType" : "PUSH"
     }, {
       "hash" : "955493fdc93842e8c99529cec35e39cecdbe9638",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=996",
       "triggerID" : "955493fdc93842e8c99529cec35e39cecdbe9638",
       "triggerType" : "PUSH"
     }, {
       "hash" : "533f00e5546da0026bbe51e1c030ddd1c19fdd08",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "533f00e5546da0026bbe51e1c030ddd1c19fdd08",
       "triggerType" : "PUSH"
     }, {
       "hash" : "05777be67e37f3a6a8eaa59e54d9e17e5fc6ba9b",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1016",
       "triggerID" : "05777be67e37f3a6a8eaa59e54d9e17e5fc6ba9b",
       "triggerType" : "PUSH"
     }, {
       "hash" : "a2ee90c549332a7f17a7d9b2799849b77c08a8e6",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "a2ee90c549332a7f17a7d9b2799849b77c08a8e6",
       "triggerType" : "PUSH"
     }, {
       "hash" : "05ce568b0b6ba14da9ca40d26b39ba478837d747",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1017",
       "triggerID" : "05ce568b0b6ba14da9ca40d26b39ba478837d747",
       "triggerType" : "PUSH"
     }, {
       "hash" : "840e3b33d39b1f86367bb6b0e65eadeb33de7765",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1025",
       "triggerID" : "840e3b33d39b1f86367bb6b0e65eadeb33de7765",
       "triggerType" : "PUSH"
     }, {
       "hash" : "8d2e2dd83f51d03e890d424a51d3e0e84e904a43",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1031",
       "triggerID" : "8d2e2dd83f51d03e890d424a51d3e0e84e904a43",
       "triggerType" : "PUSH"
     }, {
       "hash" : "688381914087ff11acf2f8c9372dbf39afa0bd40",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1035",
       "triggerID" : "688381914087ff11acf2f8c9372dbf39afa0bd40",
       "triggerType" : "PUSH"
     }, {
       "hash" : "fc2c138901e82ed5f3fbf363c2a0ce8140ade050",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "fc2c138901e82ed5f3fbf363c2a0ce8140ade050",
       "triggerType" : "PUSH"
     }, {
       "hash" : "6ed9e1cb1e5a191752d6d39c7a9d5d41840f0cdf",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1039",
       "triggerID" : "6ed9e1cb1e5a191752d6d39c7a9d5d41840f0cdf",
       "triggerType" : "PUSH"
     }, {
       "hash" : "a510a7de1458bb996df5bbb341e8d010e4c91381",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "a510a7de1458bb996df5bbb341e8d010e4c91381",
       "triggerType" : "PUSH"
     }, {
       "hash" : "49677397cd117228742205944606f7a789090376",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1042",
       "triggerID" : "49677397cd117228742205944606f7a789090376",
       "triggerType" : "PUSH"
     }, {
       "hash" : "1630db0a37422aed942ff280560a9d6329df5015",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1044",
       "triggerID" : "1630db0a37422aed942ff280560a9d6329df5015",
       "triggerType" : "PUSH"
     }, {
       "hash" : "f11c76301d4ab524db0af57aa831be3a898ea0fb",
       "status" : "CANCELED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1048",
       "triggerID" : "f11c76301d4ab524db0af57aa831be3a898ea0fb",
       "triggerType" : "PUSH"
     }, {
       "hash" : "d1a0dc4c2017853a7ba0c3302f886043a63f29da",
       "status" : "PENDING",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1050",
       "triggerID" : "d1a0dc4c2017853a7ba0c3302f886043a63f29da",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 9e668f65c6d4f73c76573236f8fa507dfb80edac UNKNOWN
   * 533f00e5546da0026bbe51e1c030ddd1c19fdd08 UNKNOWN
   * a2ee90c549332a7f17a7d9b2799849b77c08a8e6 UNKNOWN
   * fc2c138901e82ed5f3fbf363c2a0ce8140ade050 UNKNOWN
   * a510a7de1458bb996df5bbb341e8d010e4c91381 UNKNOWN
   * f11c76301d4ab524db0af57aa831be3a898ea0fb Azure: [CANCELED](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1048) 
   * d1a0dc4c2017853a7ba0c3302f886043a63f29da Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1050) 
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] [HUDI-8141] Incremental Query with Completion Time [hudi]

Posted by "CTTY (via GitHub)" <gi...@apache.org>.

CTTY commented on code in PR #11947:
URL: https://github.com/apache/hudi/pull/11947#discussion_r1797749269


##########
hudi-utilities/src/main/java/org/apache/hudi/utilities/sources/HoodieIncrSource.java:
##########
@@ -212,35 +257,43 @@ public Pair<Option<Dataset<Row>>, String> fetchNextBatch(Option<String> lastCkpt
           .collect(Collectors.toMap(Pair::getLeft, Pair::getRight));
       reader = reader.options(optionsMap);
     }
+    boolean shouldFullScan =
+        queryContext.getActiveTimeline().isBeforeTimelineStartsByCompletionTime(startTime)
+        && missingCheckpointStrategy == MissingCheckpointStrategy.READ_UPTO_LATEST_COMMIT;
     Dataset<Row> source;
-    // Do Incr pull. Set end instant if available
-    if (queryInfo.isIncremental()) {
-      source = reader
-          .options(readOpts)
-          .option(QUERY_TYPE().key(), QUERY_TYPE_INCREMENTAL_OPT_VAL())
-          .option(BEGIN_INSTANTTIME().key(), queryInfo.getStartInstant())
-          .option(END_INSTANTTIME().key(), queryInfo.getEndInstant())
-          .option(INCREMENTAL_FALLBACK_TO_FULL_TABLE_SCAN().key(),
-              props.getString(INCREMENTAL_FALLBACK_TO_FULL_TABLE_SCAN().key(),
-                  INCREMENTAL_FALLBACK_TO_FULL_TABLE_SCAN().defaultValue()))
-          .option(INCREMENTAL_READ_HANDLE_HOLLOW_COMMIT().key(), handlingMode.name())
-          .load(srcPath);
-    } else {
-      // if checkpoint is missing from source table, and if strategy is set to READ_UPTO_LATEST_COMMIT, we have to issue snapshot query
+    if (instantRange.isEmpty() || shouldFullScan) {
+      // snapshot query
       Dataset<Row> snapshot = reader
           .options(readOpts)
           .option(DataSourceReadOptions.QUERY_TYPE().key(), DataSourceReadOptions.QUERY_TYPE_SNAPSHOT_OPT_VAL())
           .load(srcPath);
       if (snapshotLoadQuerySplitter.isPresent()) {
-        queryInfo = snapshotLoadQuerySplitter.get().getNextCheckpoint(snapshot, queryInfo, sourceProfileSupplier);
+        queryContext = snapshotLoadQuerySplitter.get().getNextCheckpoint(snapshot, queryContext, sourceProfileSupplier);
+        // update endTime/next checkpoint
+        endTime = queryContext.getEndInstant().orElse(queryContext.getLastInstant());
       }
+      Set<String> validInstants = new HashSet<>(queryContext.getInstants());
+      snapshot = queryContext.getPredicateFilter().map(snapshot::filter).orElse(snapshot);
       source = snapshot
           // add filtering so that only interested records are returned.
-          .filter(String.format("%s > '%s'", HoodieRecord.COMMIT_TIME_METADATA_FIELD,
-              queryInfo.getStartInstant()))
+          // completion time comparison uses ( , ], but when comparing start time we need to use [, ]
+          .filter(String.format("%s >= '%s'", HoodieRecord.COMMIT_TIME_METADATA_FIELD,
+              queryContext.getStartInstant().get()))
           .filter(String.format("%s <= '%s'", HoodieRecord.COMMIT_TIME_METADATA_FIELD,
-              queryInfo.getEndInstant()));
-      source = queryInfo.getPredicateFilter().map(source::filter).orElse(source);
+              queryContext.getEndInstant().orElse(queryContext.getLastInstant())))
+          .filter((Row row) -> validInstants.contains(row.getAs(HoodieRecord.COMMIT_TIME_METADATA_FIELD)));

Review Comment:
   This cannot be pushed down to any existing Hudi relation,because it's running a full table scan. This predicate has to be written like this because of the limitation of using Java. Maybe we could add a scala util like `JavaScalaConverters` to make this predicate more elegant? 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] [HUDI-8141] Incremental Query with Completion Time [hudi]

Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.

hudi-bot commented on PR #11947:
URL: https://github.com/apache/hudi/pull/11947#issuecomment-2408802776

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "090f256a315cbbf0c6a54d93cc2739dd693360e6",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=484",
       "triggerID" : "090f256a315cbbf0c6a54d93cc2739dd693360e6",
       "triggerType" : "PUSH"
     }, {
       "hash" : "0df5649013c1cf9e782fcbb67b8a2e7174f94512",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=985",
       "triggerID" : "0df5649013c1cf9e782fcbb67b8a2e7174f94512",
       "triggerType" : "PUSH"
     }, {
       "hash" : "d52e0128f409918644255abbe4376dc4a780bfc8",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=986",
       "triggerID" : "d52e0128f409918644255abbe4376dc4a780bfc8",
       "triggerType" : "PUSH"
     }, {
       "hash" : "9e668f65c6d4f73c76573236f8fa507dfb80edac",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "9e668f65c6d4f73c76573236f8fa507dfb80edac",
       "triggerType" : "PUSH"
     }, {
       "hash" : "f145131d524bf190986e5de600a326e32a27beca",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=987",
       "triggerID" : "f145131d524bf190986e5de600a326e32a27beca",
       "triggerType" : "PUSH"
     }, {
       "hash" : "0b008f807b42aa6fb2686922bb88d63e84a1f299",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=993",
       "triggerID" : "0b008f807b42aa6fb2686922bb88d63e84a1f299",
       "triggerType" : "PUSH"
     }, {
       "hash" : "65c01ad9f2b33d36d4890355feef91844438e732",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=995",
       "triggerID" : "65c01ad9f2b33d36d4890355feef91844438e732",
       "triggerType" : "PUSH"
     }, {
       "hash" : "955493fdc93842e8c99529cec35e39cecdbe9638",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=996",
       "triggerID" : "955493fdc93842e8c99529cec35e39cecdbe9638",
       "triggerType" : "PUSH"
     }, {
       "hash" : "533f00e5546da0026bbe51e1c030ddd1c19fdd08",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "533f00e5546da0026bbe51e1c030ddd1c19fdd08",
       "triggerType" : "PUSH"
     }, {
       "hash" : "05777be67e37f3a6a8eaa59e54d9e17e5fc6ba9b",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1016",
       "triggerID" : "05777be67e37f3a6a8eaa59e54d9e17e5fc6ba9b",
       "triggerType" : "PUSH"
     }, {
       "hash" : "a2ee90c549332a7f17a7d9b2799849b77c08a8e6",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "a2ee90c549332a7f17a7d9b2799849b77c08a8e6",
       "triggerType" : "PUSH"
     }, {
       "hash" : "05ce568b0b6ba14da9ca40d26b39ba478837d747",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1017",
       "triggerID" : "05ce568b0b6ba14da9ca40d26b39ba478837d747",
       "triggerType" : "PUSH"
     }, {
       "hash" : "840e3b33d39b1f86367bb6b0e65eadeb33de7765",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1025",
       "triggerID" : "840e3b33d39b1f86367bb6b0e65eadeb33de7765",
       "triggerType" : "PUSH"
     }, {
       "hash" : "8d2e2dd83f51d03e890d424a51d3e0e84e904a43",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1031",
       "triggerID" : "8d2e2dd83f51d03e890d424a51d3e0e84e904a43",
       "triggerType" : "PUSH"
     }, {
       "hash" : "688381914087ff11acf2f8c9372dbf39afa0bd40",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1035",
       "triggerID" : "688381914087ff11acf2f8c9372dbf39afa0bd40",
       "triggerType" : "PUSH"
     }, {
       "hash" : "fc2c138901e82ed5f3fbf363c2a0ce8140ade050",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "fc2c138901e82ed5f3fbf363c2a0ce8140ade050",
       "triggerType" : "PUSH"
     }, {
       "hash" : "6ed9e1cb1e5a191752d6d39c7a9d5d41840f0cdf",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1039",
       "triggerID" : "6ed9e1cb1e5a191752d6d39c7a9d5d41840f0cdf",
       "triggerType" : "PUSH"
     }, {
       "hash" : "a510a7de1458bb996df5bbb341e8d010e4c91381",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "a510a7de1458bb996df5bbb341e8d010e4c91381",
       "triggerType" : "PUSH"
     }, {
       "hash" : "49677397cd117228742205944606f7a789090376",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1042",
       "triggerID" : "49677397cd117228742205944606f7a789090376",
       "triggerType" : "PUSH"
     }, {
       "hash" : "1630db0a37422aed942ff280560a9d6329df5015",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1044",
       "triggerID" : "1630db0a37422aed942ff280560a9d6329df5015",
       "triggerType" : "PUSH"
     }, {
       "hash" : "f11c76301d4ab524db0af57aa831be3a898ea0fb",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1048",
       "triggerID" : "f11c76301d4ab524db0af57aa831be3a898ea0fb",
       "triggerType" : "PUSH"
     }, {
       "hash" : "d1a0dc4c2017853a7ba0c3302f886043a63f29da",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1050",
       "triggerID" : "d1a0dc4c2017853a7ba0c3302f886043a63f29da",
       "triggerType" : "PUSH"
     }, {
       "hash" : "6a4c15ec39ded1f90b113b272554217698a98e1c",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1087",
       "triggerID" : "6a4c15ec39ded1f90b113b272554217698a98e1c",
       "triggerType" : "PUSH"
     }, {
       "hash" : "288e730d645a5e58e50a87ef5f44ba11a8fb0533",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1092",
       "triggerID" : "288e730d645a5e58e50a87ef5f44ba11a8fb0533",
       "triggerType" : "PUSH"
     }, {
       "hash" : "70a5c21321a0da3a0be8d85e206c24befa7b89f1",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1095",
       "triggerID" : "70a5c21321a0da3a0be8d85e206c24befa7b89f1",
       "triggerType" : "PUSH"
     }, {
       "hash" : "f072a986114949c5e426ce64a5f5568fc718e25d",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "f072a986114949c5e426ce64a5f5568fc718e25d",
       "triggerType" : "PUSH"
     }, {
       "hash" : "45b27475b1e8806ec8c566d6f151f4bc79c8549d",
       "status" : "CANCELED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1096",
       "triggerID" : "45b27475b1e8806ec8c566d6f151f4bc79c8549d",
       "triggerType" : "PUSH"
     }, {
       "hash" : "3c62ef22f6cde28110e5eb2fca8760a008d77723",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "3c62ef22f6cde28110e5eb2fca8760a008d77723",
       "triggerType" : "PUSH"
     }, {
       "hash" : "dc785fa04e542b2ccc45494e0292f5153c7ecea8",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "dc785fa04e542b2ccc45494e0292f5153c7ecea8",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 9e668f65c6d4f73c76573236f8fa507dfb80edac UNKNOWN
   * 533f00e5546da0026bbe51e1c030ddd1c19fdd08 UNKNOWN
   * a2ee90c549332a7f17a7d9b2799849b77c08a8e6 UNKNOWN
   * fc2c138901e82ed5f3fbf363c2a0ce8140ade050 UNKNOWN
   * a510a7de1458bb996df5bbb341e8d010e4c91381 UNKNOWN
   * f072a986114949c5e426ce64a5f5568fc718e25d UNKNOWN
   * 45b27475b1e8806ec8c566d6f151f4bc79c8549d Azure: [CANCELED](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1096) 
   * 3c62ef22f6cde28110e5eb2fca8760a008d77723 UNKNOWN
   * dc785fa04e542b2ccc45494e0292f5153c7ecea8 UNKNOWN
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] [HUDI-8141] Incremental Query with Completion Time [hudi]

Posted by "yihua (via GitHub)" <gi...@apache.org>.

yihua commented on code in PR #11947:
URL: https://github.com/apache/hudi/pull/11947#discussion_r1797779486


##########
hudi-common/src/main/java/org/apache/hudi/common/table/timeline/TimelineUtils.java:
##########
@@ -308,6 +309,38 @@ public static HoodieTimeline getCommitsTimelineAfter(
     return timelineSinceLastSync;
   }
 
+  public static HoodieTimeline getCommitsTimeLineAfterByCompletionTimeRange(

Review Comment:
   Also, the data files added by the commits in the archived timeline may have already been deleted by the cleaner, so incremental query may not be able to read the files, and has to go through full scan approach.



##########
hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/hudi/IncrementalRelation.scala:
##########
@@ -90,24 +78,25 @@ class IncrementalRelation(val sqlContext: SQLContext,
     throw new HoodieException("Incremental queries are not supported when meta fields are disabled")
   }
 
+  private val queryContext: IncrementalQueryAnalyzer.QueryContext =
+    IncrementalQueryAnalyzer.builder()
+      .metaClient(metaClient)
+      .startTime(optParams(DataSourceReadOptions.BEGIN_INSTANTTIME.key))
+      .endTime(optParams.getOrElse(DataSourceReadOptions.END_INSTANTTIME.key, null))
+      .rangeType(InstantRange.RangeType.OPEN_CLOSED)
+      .limit(optParams.getOrElse(
+        DataSourceReadOptions.INCREMENTAL_LIMIT.key,
+        DataSourceReadOptions.INCREMENTAL_LIMIT.defaultValue).toInt)

Review Comment:
   Is this for Hudi streamer only?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] [HUDI-8141] Incremental Query with Completion Time [hudi]

Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.

hudi-bot commented on PR #11947:
URL: https://github.com/apache/hudi/pull/11947#issuecomment-2408827096

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "090f256a315cbbf0c6a54d93cc2739dd693360e6",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=484",
       "triggerID" : "090f256a315cbbf0c6a54d93cc2739dd693360e6",
       "triggerType" : "PUSH"
     }, {
       "hash" : "0df5649013c1cf9e782fcbb67b8a2e7174f94512",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=985",
       "triggerID" : "0df5649013c1cf9e782fcbb67b8a2e7174f94512",
       "triggerType" : "PUSH"
     }, {
       "hash" : "d52e0128f409918644255abbe4376dc4a780bfc8",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=986",
       "triggerID" : "d52e0128f409918644255abbe4376dc4a780bfc8",
       "triggerType" : "PUSH"
     }, {
       "hash" : "9e668f65c6d4f73c76573236f8fa507dfb80edac",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "9e668f65c6d4f73c76573236f8fa507dfb80edac",
       "triggerType" : "PUSH"
     }, {
       "hash" : "f145131d524bf190986e5de600a326e32a27beca",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=987",
       "triggerID" : "f145131d524bf190986e5de600a326e32a27beca",
       "triggerType" : "PUSH"
     }, {
       "hash" : "0b008f807b42aa6fb2686922bb88d63e84a1f299",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=993",
       "triggerID" : "0b008f807b42aa6fb2686922bb88d63e84a1f299",
       "triggerType" : "PUSH"
     }, {
       "hash" : "65c01ad9f2b33d36d4890355feef91844438e732",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=995",
       "triggerID" : "65c01ad9f2b33d36d4890355feef91844438e732",
       "triggerType" : "PUSH"
     }, {
       "hash" : "955493fdc93842e8c99529cec35e39cecdbe9638",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=996",
       "triggerID" : "955493fdc93842e8c99529cec35e39cecdbe9638",
       "triggerType" : "PUSH"
     }, {
       "hash" : "533f00e5546da0026bbe51e1c030ddd1c19fdd08",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "533f00e5546da0026bbe51e1c030ddd1c19fdd08",
       "triggerType" : "PUSH"
     }, {
       "hash" : "05777be67e37f3a6a8eaa59e54d9e17e5fc6ba9b",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1016",
       "triggerID" : "05777be67e37f3a6a8eaa59e54d9e17e5fc6ba9b",
       "triggerType" : "PUSH"
     }, {
       "hash" : "a2ee90c549332a7f17a7d9b2799849b77c08a8e6",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "a2ee90c549332a7f17a7d9b2799849b77c08a8e6",
       "triggerType" : "PUSH"
     }, {
       "hash" : "05ce568b0b6ba14da9ca40d26b39ba478837d747",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1017",
       "triggerID" : "05ce568b0b6ba14da9ca40d26b39ba478837d747",
       "triggerType" : "PUSH"
     }, {
       "hash" : "840e3b33d39b1f86367bb6b0e65eadeb33de7765",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1025",
       "triggerID" : "840e3b33d39b1f86367bb6b0e65eadeb33de7765",
       "triggerType" : "PUSH"
     }, {
       "hash" : "8d2e2dd83f51d03e890d424a51d3e0e84e904a43",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1031",
       "triggerID" : "8d2e2dd83f51d03e890d424a51d3e0e84e904a43",
       "triggerType" : "PUSH"
     }, {
       "hash" : "688381914087ff11acf2f8c9372dbf39afa0bd40",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1035",
       "triggerID" : "688381914087ff11acf2f8c9372dbf39afa0bd40",
       "triggerType" : "PUSH"
     }, {
       "hash" : "fc2c138901e82ed5f3fbf363c2a0ce8140ade050",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "fc2c138901e82ed5f3fbf363c2a0ce8140ade050",
       "triggerType" : "PUSH"
     }, {
       "hash" : "6ed9e1cb1e5a191752d6d39c7a9d5d41840f0cdf",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1039",
       "triggerID" : "6ed9e1cb1e5a191752d6d39c7a9d5d41840f0cdf",
       "triggerType" : "PUSH"
     }, {
       "hash" : "a510a7de1458bb996df5bbb341e8d010e4c91381",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "a510a7de1458bb996df5bbb341e8d010e4c91381",
       "triggerType" : "PUSH"
     }, {
       "hash" : "49677397cd117228742205944606f7a789090376",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1042",
       "triggerID" : "49677397cd117228742205944606f7a789090376",
       "triggerType" : "PUSH"
     }, {
       "hash" : "1630db0a37422aed942ff280560a9d6329df5015",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1044",
       "triggerID" : "1630db0a37422aed942ff280560a9d6329df5015",
       "triggerType" : "PUSH"
     }, {
       "hash" : "f11c76301d4ab524db0af57aa831be3a898ea0fb",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1048",
       "triggerID" : "f11c76301d4ab524db0af57aa831be3a898ea0fb",
       "triggerType" : "PUSH"
     }, {
       "hash" : "d1a0dc4c2017853a7ba0c3302f886043a63f29da",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1050",
       "triggerID" : "d1a0dc4c2017853a7ba0c3302f886043a63f29da",
       "triggerType" : "PUSH"
     }, {
       "hash" : "6a4c15ec39ded1f90b113b272554217698a98e1c",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1087",
       "triggerID" : "6a4c15ec39ded1f90b113b272554217698a98e1c",
       "triggerType" : "PUSH"
     }, {
       "hash" : "288e730d645a5e58e50a87ef5f44ba11a8fb0533",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1092",
       "triggerID" : "288e730d645a5e58e50a87ef5f44ba11a8fb0533",
       "triggerType" : "PUSH"
     }, {
       "hash" : "70a5c21321a0da3a0be8d85e206c24befa7b89f1",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1095",
       "triggerID" : "70a5c21321a0da3a0be8d85e206c24befa7b89f1",
       "triggerType" : "PUSH"
     }, {
       "hash" : "f072a986114949c5e426ce64a5f5568fc718e25d",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "f072a986114949c5e426ce64a5f5568fc718e25d",
       "triggerType" : "PUSH"
     }, {
       "hash" : "45b27475b1e8806ec8c566d6f151f4bc79c8549d",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1096",
       "triggerID" : "45b27475b1e8806ec8c566d6f151f4bc79c8549d",
       "triggerType" : "PUSH"
     }, {
       "hash" : "3c62ef22f6cde28110e5eb2fca8760a008d77723",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "3c62ef22f6cde28110e5eb2fca8760a008d77723",
       "triggerType" : "PUSH"
     }, {
       "hash" : "dc785fa04e542b2ccc45494e0292f5153c7ecea8",
       "status" : "FAILURE",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1097",
       "triggerID" : "dc785fa04e542b2ccc45494e0292f5153c7ecea8",
       "triggerType" : "PUSH"
     }, {
       "hash" : "585ddb8049b9b9ccb057027e265b08a1b586df8c",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "585ddb8049b9b9ccb057027e265b08a1b586df8c",
       "triggerType" : "PUSH"
     }, {
       "hash" : "b62ff8675beed2c4487c788b9ca1a0ad4271b913",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "b62ff8675beed2c4487c788b9ca1a0ad4271b913",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 9e668f65c6d4f73c76573236f8fa507dfb80edac UNKNOWN
   * 533f00e5546da0026bbe51e1c030ddd1c19fdd08 UNKNOWN
   * a2ee90c549332a7f17a7d9b2799849b77c08a8e6 UNKNOWN
   * fc2c138901e82ed5f3fbf363c2a0ce8140ade050 UNKNOWN
   * a510a7de1458bb996df5bbb341e8d010e4c91381 UNKNOWN
   * f072a986114949c5e426ce64a5f5568fc718e25d UNKNOWN
   * 3c62ef22f6cde28110e5eb2fca8760a008d77723 UNKNOWN
   * dc785fa04e542b2ccc45494e0292f5153c7ecea8 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1097) 
   * 585ddb8049b9b9ccb057027e265b08a1b586df8c UNKNOWN
   * b62ff8675beed2c4487c788b9ca1a0ad4271b913 UNKNOWN
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] [HUDI-8141] Incremental Query with Completion Time [hudi]

Posted by "danny0405 (via GitHub)" <gi...@apache.org>.

danny0405 commented on code in PR #11947:
URL: https://github.com/apache/hudi/pull/11947#discussion_r1798659033


##########
hudi-utilities/src/main/java/org/apache/hudi/utilities/sources/helpers/IncrSourceHelper.java:
##########
@@ -174,6 +170,64 @@ public static QueryInfo generateQueryInfo(JavaSparkContext jssc, String srcBaseP
     }
   }
 
+  public static IncrementalQueryAnalyzer getIncrementalQueryAnalyzer(
+      JavaSparkContext jssc,
+      String srcPath,
+      Option<String> lastCkptStr,
+      MissingCheckpointStrategy missingCheckpointStrategy,
+      int numInstantsFromConfig,
+      Option<SourceProfile<Integer>> latestSourceProfile) {
+    HoodieTableMetaClient metaClient = HoodieTableMetaClient.builder()
+        .setConf(HadoopFSUtils.getStorageConfWithCopy(jssc.hadoopConfiguration()))
+        .setBasePath(srcPath)
+        .setLoadActiveTimelineOnLoad(true)
+        .build();
+
+    String startTime;
+    if (lastCkptStr.isPresent() && !lastCkptStr.get().isEmpty()) {
+      startTime = lastCkptStr.get();
+    } else if (missingCheckpointStrategy != null) {
+      switch (missingCheckpointStrategy) {
+        case READ_UPTO_LATEST_COMMIT:
+          startTime = DEFAULT_BEGIN_TIMESTAMP;
+          // disrespect numInstantsFromConfig when reading up to latest
+          numInstantsFromConfig = -1;
+          break;
+        case READ_LATEST:
+          Option<HoodieInstant> lastInstant = metaClient
+              .getCommitsAndCompactionTimeline()
+              .filterCompletedInstants()
+              .lastInstant();
+          startTime = lastInstant
+              .map(hoodieInstant -> instantTimeMinusMillis(hoodieInstant.getCompletionTime(), 1))

Review Comment:
   Use the `closed_xxx` range type instead.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] [HUDI-8141] Incremental Query with Completion Time [hudi]

Posted by "danny0405 (via GitHub)" <gi...@apache.org>.

danny0405 commented on code in PR #11947:
URL: https://github.com/apache/hudi/pull/11947#discussion_r1798653934


##########
hudi-common/src/main/java/org/apache/hudi/common/table/timeline/HoodieDefaultTimeline.java:
##########
@@ -629,16 +649,23 @@ private Set<String> getOrCreatePendingClusteringInstantSet() {
   /**
    * Returns the first non savepoint commit on the timeline.
    */
-  private static Option<HoodieInstant> findFirstNonSavepointCommit(List<HoodieInstant> instants) {
+  private static Option<HoodieInstant> findFirstNonSavepointCommit(
+      List<HoodieInstant> instants,
+      boolean byCompletionTime) {

Review Comment:
   Can we pass around a func like `() -> HoodieInstant:getCompletionTime` instead of a flag.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] [HUDI-8141] Incremental Query with Completion Time [hudi]

Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.

hudi-bot commented on PR #11947:
URL: https://github.com/apache/hudi/pull/11947#issuecomment-2418085747

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "090f256a315cbbf0c6a54d93cc2739dd693360e6",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=484",
       "triggerID" : "090f256a315cbbf0c6a54d93cc2739dd693360e6",
       "triggerType" : "PUSH"
     }, {
       "hash" : "0df5649013c1cf9e782fcbb67b8a2e7174f94512",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=985",
       "triggerID" : "0df5649013c1cf9e782fcbb67b8a2e7174f94512",
       "triggerType" : "PUSH"
     }, {
       "hash" : "d52e0128f409918644255abbe4376dc4a780bfc8",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=986",
       "triggerID" : "d52e0128f409918644255abbe4376dc4a780bfc8",
       "triggerType" : "PUSH"
     }, {
       "hash" : "9e668f65c6d4f73c76573236f8fa507dfb80edac",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "9e668f65c6d4f73c76573236f8fa507dfb80edac",
       "triggerType" : "PUSH"
     }, {
       "hash" : "f145131d524bf190986e5de600a326e32a27beca",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=987",
       "triggerID" : "f145131d524bf190986e5de600a326e32a27beca",
       "triggerType" : "PUSH"
     }, {
       "hash" : "0b008f807b42aa6fb2686922bb88d63e84a1f299",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=993",
       "triggerID" : "0b008f807b42aa6fb2686922bb88d63e84a1f299",
       "triggerType" : "PUSH"
     }, {
       "hash" : "65c01ad9f2b33d36d4890355feef91844438e732",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=995",
       "triggerID" : "65c01ad9f2b33d36d4890355feef91844438e732",
       "triggerType" : "PUSH"
     }, {
       "hash" : "955493fdc93842e8c99529cec35e39cecdbe9638",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=996",
       "triggerID" : "955493fdc93842e8c99529cec35e39cecdbe9638",
       "triggerType" : "PUSH"
     }, {
       "hash" : "533f00e5546da0026bbe51e1c030ddd1c19fdd08",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "533f00e5546da0026bbe51e1c030ddd1c19fdd08",
       "triggerType" : "PUSH"
     }, {
       "hash" : "05777be67e37f3a6a8eaa59e54d9e17e5fc6ba9b",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1016",
       "triggerID" : "05777be67e37f3a6a8eaa59e54d9e17e5fc6ba9b",
       "triggerType" : "PUSH"
     }, {
       "hash" : "a2ee90c549332a7f17a7d9b2799849b77c08a8e6",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "a2ee90c549332a7f17a7d9b2799849b77c08a8e6",
       "triggerType" : "PUSH"
     }, {
       "hash" : "05ce568b0b6ba14da9ca40d26b39ba478837d747",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1017",
       "triggerID" : "05ce568b0b6ba14da9ca40d26b39ba478837d747",
       "triggerType" : "PUSH"
     }, {
       "hash" : "840e3b33d39b1f86367bb6b0e65eadeb33de7765",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1025",
       "triggerID" : "840e3b33d39b1f86367bb6b0e65eadeb33de7765",
       "triggerType" : "PUSH"
     }, {
       "hash" : "8d2e2dd83f51d03e890d424a51d3e0e84e904a43",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1031",
       "triggerID" : "8d2e2dd83f51d03e890d424a51d3e0e84e904a43",
       "triggerType" : "PUSH"
     }, {
       "hash" : "688381914087ff11acf2f8c9372dbf39afa0bd40",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1035",
       "triggerID" : "688381914087ff11acf2f8c9372dbf39afa0bd40",
       "triggerType" : "PUSH"
     }, {
       "hash" : "fc2c138901e82ed5f3fbf363c2a0ce8140ade050",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "fc2c138901e82ed5f3fbf363c2a0ce8140ade050",
       "triggerType" : "PUSH"
     }, {
       "hash" : "6ed9e1cb1e5a191752d6d39c7a9d5d41840f0cdf",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1039",
       "triggerID" : "6ed9e1cb1e5a191752d6d39c7a9d5d41840f0cdf",
       "triggerType" : "PUSH"
     }, {
       "hash" : "a510a7de1458bb996df5bbb341e8d010e4c91381",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "a510a7de1458bb996df5bbb341e8d010e4c91381",
       "triggerType" : "PUSH"
     }, {
       "hash" : "49677397cd117228742205944606f7a789090376",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1042",
       "triggerID" : "49677397cd117228742205944606f7a789090376",
       "triggerType" : "PUSH"
     }, {
       "hash" : "1630db0a37422aed942ff280560a9d6329df5015",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1044",
       "triggerID" : "1630db0a37422aed942ff280560a9d6329df5015",
       "triggerType" : "PUSH"
     }, {
       "hash" : "f11c76301d4ab524db0af57aa831be3a898ea0fb",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1048",
       "triggerID" : "f11c76301d4ab524db0af57aa831be3a898ea0fb",
       "triggerType" : "PUSH"
     }, {
       "hash" : "d1a0dc4c2017853a7ba0c3302f886043a63f29da",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1050",
       "triggerID" : "d1a0dc4c2017853a7ba0c3302f886043a63f29da",
       "triggerType" : "PUSH"
     }, {
       "hash" : "6a4c15ec39ded1f90b113b272554217698a98e1c",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1087",
       "triggerID" : "6a4c15ec39ded1f90b113b272554217698a98e1c",
       "triggerType" : "PUSH"
     }, {
       "hash" : "288e730d645a5e58e50a87ef5f44ba11a8fb0533",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1092",
       "triggerID" : "288e730d645a5e58e50a87ef5f44ba11a8fb0533",
       "triggerType" : "PUSH"
     }, {
       "hash" : "70a5c21321a0da3a0be8d85e206c24befa7b89f1",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1095",
       "triggerID" : "70a5c21321a0da3a0be8d85e206c24befa7b89f1",
       "triggerType" : "PUSH"
     }, {
       "hash" : "f072a986114949c5e426ce64a5f5568fc718e25d",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "f072a986114949c5e426ce64a5f5568fc718e25d",
       "triggerType" : "PUSH"
     }, {
       "hash" : "45b27475b1e8806ec8c566d6f151f4bc79c8549d",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1096",
       "triggerID" : "45b27475b1e8806ec8c566d6f151f4bc79c8549d",
       "triggerType" : "PUSH"
     }, {
       "hash" : "3c62ef22f6cde28110e5eb2fca8760a008d77723",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "3c62ef22f6cde28110e5eb2fca8760a008d77723",
       "triggerType" : "PUSH"
     }, {
       "hash" : "dc785fa04e542b2ccc45494e0292f5153c7ecea8",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1097",
       "triggerID" : "dc785fa04e542b2ccc45494e0292f5153c7ecea8",
       "triggerType" : "PUSH"
     }, {
       "hash" : "585ddb8049b9b9ccb057027e265b08a1b586df8c",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "585ddb8049b9b9ccb057027e265b08a1b586df8c",
       "triggerType" : "PUSH"
     }, {
       "hash" : "b62ff8675beed2c4487c788b9ca1a0ad4271b913",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "b62ff8675beed2c4487c788b9ca1a0ad4271b913",
       "triggerType" : "PUSH"
     }, {
       "hash" : "2d46048a7402907f348cc96da2839ba1b3e4932a",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1099",
       "triggerID" : "2d46048a7402907f348cc96da2839ba1b3e4932a",
       "triggerType" : "PUSH"
     }, {
       "hash" : "004fb6ebf23f796a4b77650e0530f0f1145e8b91",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1101",
       "triggerID" : "004fb6ebf23f796a4b77650e0530f0f1145e8b91",
       "triggerType" : "PUSH"
     }, {
       "hash" : "af0c33333b5e9ae6d59f20a6db1d7c016ea015d3",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1165",
       "triggerID" : "af0c33333b5e9ae6d59f20a6db1d7c016ea015d3",
       "triggerType" : "PUSH"
     }, {
       "hash" : "da8111118d71306500229619d3b7996fcd2526e8",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "da8111118d71306500229619d3b7996fcd2526e8",
       "triggerType" : "PUSH"
     }, {
       "hash" : "83257e2b9bd3c426cf90e11003a5f244a750d7fa",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1167",
       "triggerID" : "83257e2b9bd3c426cf90e11003a5f244a750d7fa",
       "triggerType" : "PUSH"
     }, {
       "hash" : "e0de408f1b8153fdaaea4c87b27a7b69213b75b5",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1177",
       "triggerID" : "e0de408f1b8153fdaaea4c87b27a7b69213b75b5",
       "triggerType" : "PUSH"
     }, {
       "hash" : "3f7951f7ad7931f5dc0b977d7c33bc5429623535",
       "status" : "FAILURE",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1180",
       "triggerID" : "3f7951f7ad7931f5dc0b977d7c33bc5429623535",
       "triggerType" : "PUSH"
     }, {
       "hash" : "9cd6f06a753b63630e8245d51338d2bf9ab86b84",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "9cd6f06a753b63630e8245d51338d2bf9ab86b84",
       "triggerType" : "PUSH"
     }, {
       "hash" : "7380b902731a92e0645c02397e14591a05798620",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "7380b902731a92e0645c02397e14591a05798620",
       "triggerType" : "PUSH"
     }, {
       "hash" : "0bd5cca01a541845081d5430d48b021d4bdf0969",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "0bd5cca01a541845081d5430d48b021d4bdf0969",
       "triggerType" : "PUSH"
     }, {
       "hash" : "8f4e175ef554f015e34e0968b656b5de714ab430",
       "status" : "PENDING",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1182",
       "triggerID" : "8f4e175ef554f015e34e0968b656b5de714ab430",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 9e668f65c6d4f73c76573236f8fa507dfb80edac UNKNOWN
   * 533f00e5546da0026bbe51e1c030ddd1c19fdd08 UNKNOWN
   * a2ee90c549332a7f17a7d9b2799849b77c08a8e6 UNKNOWN
   * fc2c138901e82ed5f3fbf363c2a0ce8140ade050 UNKNOWN
   * a510a7de1458bb996df5bbb341e8d010e4c91381 UNKNOWN
   * f072a986114949c5e426ce64a5f5568fc718e25d UNKNOWN
   * 3c62ef22f6cde28110e5eb2fca8760a008d77723 UNKNOWN
   * 585ddb8049b9b9ccb057027e265b08a1b586df8c UNKNOWN
   * b62ff8675beed2c4487c788b9ca1a0ad4271b913 UNKNOWN
   * da8111118d71306500229619d3b7996fcd2526e8 UNKNOWN
   * 3f7951f7ad7931f5dc0b977d7c33bc5429623535 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1180) 
   * 9cd6f06a753b63630e8245d51338d2bf9ab86b84 UNKNOWN
   * 7380b902731a92e0645c02397e14591a05798620 UNKNOWN
   * 0bd5cca01a541845081d5430d48b021d4bdf0969 UNKNOWN
   * 8f4e175ef554f015e34e0968b656b5de714ab430 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1182) 
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] [HUDI-8141] Incremental Query with Completion Time [hudi]

Posted by "yihua (via GitHub)" <gi...@apache.org>.

yihua commented on code in PR #11947:
URL: https://github.com/apache/hudi/pull/11947#discussion_r1802140310


##########
hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/hudi/MergeOnReadIncrementalRelation.scala:
##########
@@ -201,55 +199,42 @@ trait HoodieIncrementalRelationTrait extends HoodieBaseRelation {
     val fallbackToFullTableScan = optParams.getOrElse(DataSourceReadOptions.INCREMENTAL_FALLBACK_TO_FULL_TABLE_SCAN.key,
       DataSourceReadOptions.INCREMENTAL_FALLBACK_TO_FULL_TABLE_SCAN.defaultValue).toBoolean
 
-    fallbackToFullTableScan && (startInstantArchived || endInstantArchived
+    fallbackToFullTableScan && (startInstantArchived
       || affectedFilesInCommits.asScala.exists(fileStatus => !metaClient.getStorage.exists(fileStatus.getPath)))
   }
 
-  protected lazy val includedCommits: immutable.Seq[HoodieInstant] = {
-    if (!startInstantArchived || !endInstantArchived) {
-      // If endTimestamp commit is not archived, will filter instants
-      // before endTimestamp.
-      if (hollowCommitHandling == USE_TRANSITION_TIME) {
-        super.timeline.findInstantsInRangeByCompletionTime(startTimestamp, endTimestamp).getInstants.asScala.toList
-      } else {
-        super.timeline.findInstantsInRange(startTimestamp, endTimestamp).getInstants.asScala.toList
-      }
-    } else {
-      super.timeline.getInstants.asScala.toList
-    }
-  }
+  protected lazy val queryContext: IncrementalQueryAnalyzer.QueryContext =
+    IncrementalQueryAnalyzer.builder()
+      .metaClient(metaClient)
+      .startTime(optParams(DataSourceReadOptions.BEGIN_INSTANTTIME.key))
+      .endTime(optParams.getOrElse(DataSourceReadOptions.END_INSTANTTIME.key, null))
+      .rangeType(InstantRange.RangeType.OPEN_CLOSED)
+      .build()
+      .analyze()
+
+  protected lazy val includedCommits: immutable.Seq[HoodieInstant] = List.concat(
+    queryContext.getArchivedInstants.asScala,

Review Comment:
   Same here, `HoodieInstant` is required while `instants` contains timestamp only.  We can change the type of `instants` if needed.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] [HUDI-8141] Incremental Query with Completion Time [hudi]

Posted by "CTTY (via GitHub)" <gi...@apache.org>.

CTTY commented on code in PR #11947:
URL: https://github.com/apache/hudi/pull/11947#discussion_r1797956169


##########
hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/hudi/MergeOnReadIncrementalRelation.scala:
##########
@@ -205,51 +204,46 @@ trait HoodieIncrementalRelationTrait extends HoodieBaseRelation {
       || affectedFilesInCommits.asScala.exists(fileStatus => !metaClient.getStorage.exists(fileStatus.getPath)))
   }
 
-  protected lazy val includedCommits: immutable.Seq[HoodieInstant] = {
-    if (!startInstantArchived || !endInstantArchived) {
-      // If endTimestamp commit is not archived, will filter instants
-      // before endTimestamp.
-      if (hollowCommitHandling == USE_TRANSITION_TIME) {
-        super.timeline.findInstantsInRangeByCompletionTime(startTimestamp, endTimestamp).getInstants.asScala.toList
-      } else {
-        super.timeline.findInstantsInRange(startTimestamp, endTimestamp).getInstants.asScala.toList
-      }
-    } else {
-      super.timeline.getInstants.asScala.toList
-    }
-  }
+  protected lazy val queryContext: IncrementalQueryAnalyzer.QueryContext =
+    IncrementalQueryAnalyzer.builder()
+      .metaClient(metaClient)
+      .startTime(optParams(DataSourceReadOptions.BEGIN_INSTANTTIME.key))
+      .endTime(optParams.getOrElse(DataSourceReadOptions.END_INSTANTTIME.key, null))
+      .rangeType(InstantRange.RangeType.OPEN_CLOSED)
+      .limit(optParams.getOrElse(
+        DataSourceReadOptions.INCREMENTAL_LIMIT.key,
+        DataSourceReadOptions.INCREMENTAL_LIMIT.defaultValue).toInt)
+      .build()
+      .analyze()
+
+  protected lazy val includedCommits: immutable.Seq[HoodieInstant] = List.concat(
+    queryContext.getArchivedInstants.asScala,
+    queryContext.getActiveInstants.asScala)
 
   protected lazy val commitsMetadata = includedCommits.map(getCommitMetadata(_, super.timeline)).asJava
 
   protected lazy val affectedFilesInCommits: java.util.List[StoragePathInfo] = {
     listAffectedFilesForCommits(conf, metaClient.getBasePath, commitsMetadata)
   }
 
-  protected lazy val (includeStartTime, startTs) = if (startInstantArchived) {
-    (false, startTimestamp)

Review Comment:
   I think the variable name `startInstantArchived` is not very accurate. It's actually checking if the `startTimestamp` passed in is archived. So in this case, if the `startTimestamp` is archived, Hudi will use the first commit in the `includedCommits` as the range start, which still falls in original time range, so it should be included
   



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] [HUDI-8141] Incremental Query with Completion Time [hudi]

Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.

hudi-bot commented on PR #11947:
URL: https://github.com/apache/hudi/pull/11947#issuecomment-2418019921

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "090f256a315cbbf0c6a54d93cc2739dd693360e6",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=484",
       "triggerID" : "090f256a315cbbf0c6a54d93cc2739dd693360e6",
       "triggerType" : "PUSH"
     }, {
       "hash" : "0df5649013c1cf9e782fcbb67b8a2e7174f94512",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=985",
       "triggerID" : "0df5649013c1cf9e782fcbb67b8a2e7174f94512",
       "triggerType" : "PUSH"
     }, {
       "hash" : "d52e0128f409918644255abbe4376dc4a780bfc8",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=986",
       "triggerID" : "d52e0128f409918644255abbe4376dc4a780bfc8",
       "triggerType" : "PUSH"
     }, {
       "hash" : "9e668f65c6d4f73c76573236f8fa507dfb80edac",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "9e668f65c6d4f73c76573236f8fa507dfb80edac",
       "triggerType" : "PUSH"
     }, {
       "hash" : "f145131d524bf190986e5de600a326e32a27beca",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=987",
       "triggerID" : "f145131d524bf190986e5de600a326e32a27beca",
       "triggerType" : "PUSH"
     }, {
       "hash" : "0b008f807b42aa6fb2686922bb88d63e84a1f299",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=993",
       "triggerID" : "0b008f807b42aa6fb2686922bb88d63e84a1f299",
       "triggerType" : "PUSH"
     }, {
       "hash" : "65c01ad9f2b33d36d4890355feef91844438e732",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=995",
       "triggerID" : "65c01ad9f2b33d36d4890355feef91844438e732",
       "triggerType" : "PUSH"
     }, {
       "hash" : "955493fdc93842e8c99529cec35e39cecdbe9638",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=996",
       "triggerID" : "955493fdc93842e8c99529cec35e39cecdbe9638",
       "triggerType" : "PUSH"
     }, {
       "hash" : "533f00e5546da0026bbe51e1c030ddd1c19fdd08",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "533f00e5546da0026bbe51e1c030ddd1c19fdd08",
       "triggerType" : "PUSH"
     }, {
       "hash" : "05777be67e37f3a6a8eaa59e54d9e17e5fc6ba9b",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1016",
       "triggerID" : "05777be67e37f3a6a8eaa59e54d9e17e5fc6ba9b",
       "triggerType" : "PUSH"
     }, {
       "hash" : "a2ee90c549332a7f17a7d9b2799849b77c08a8e6",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "a2ee90c549332a7f17a7d9b2799849b77c08a8e6",
       "triggerType" : "PUSH"
     }, {
       "hash" : "05ce568b0b6ba14da9ca40d26b39ba478837d747",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1017",
       "triggerID" : "05ce568b0b6ba14da9ca40d26b39ba478837d747",
       "triggerType" : "PUSH"
     }, {
       "hash" : "840e3b33d39b1f86367bb6b0e65eadeb33de7765",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1025",
       "triggerID" : "840e3b33d39b1f86367bb6b0e65eadeb33de7765",
       "triggerType" : "PUSH"
     }, {
       "hash" : "8d2e2dd83f51d03e890d424a51d3e0e84e904a43",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1031",
       "triggerID" : "8d2e2dd83f51d03e890d424a51d3e0e84e904a43",
       "triggerType" : "PUSH"
     }, {
       "hash" : "688381914087ff11acf2f8c9372dbf39afa0bd40",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1035",
       "triggerID" : "688381914087ff11acf2f8c9372dbf39afa0bd40",
       "triggerType" : "PUSH"
     }, {
       "hash" : "fc2c138901e82ed5f3fbf363c2a0ce8140ade050",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "fc2c138901e82ed5f3fbf363c2a0ce8140ade050",
       "triggerType" : "PUSH"
     }, {
       "hash" : "6ed9e1cb1e5a191752d6d39c7a9d5d41840f0cdf",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1039",
       "triggerID" : "6ed9e1cb1e5a191752d6d39c7a9d5d41840f0cdf",
       "triggerType" : "PUSH"
     }, {
       "hash" : "a510a7de1458bb996df5bbb341e8d010e4c91381",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "a510a7de1458bb996df5bbb341e8d010e4c91381",
       "triggerType" : "PUSH"
     }, {
       "hash" : "49677397cd117228742205944606f7a789090376",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1042",
       "triggerID" : "49677397cd117228742205944606f7a789090376",
       "triggerType" : "PUSH"
     }, {
       "hash" : "1630db0a37422aed942ff280560a9d6329df5015",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1044",
       "triggerID" : "1630db0a37422aed942ff280560a9d6329df5015",
       "triggerType" : "PUSH"
     }, {
       "hash" : "f11c76301d4ab524db0af57aa831be3a898ea0fb",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1048",
       "triggerID" : "f11c76301d4ab524db0af57aa831be3a898ea0fb",
       "triggerType" : "PUSH"
     }, {
       "hash" : "d1a0dc4c2017853a7ba0c3302f886043a63f29da",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1050",
       "triggerID" : "d1a0dc4c2017853a7ba0c3302f886043a63f29da",
       "triggerType" : "PUSH"
     }, {
       "hash" : "6a4c15ec39ded1f90b113b272554217698a98e1c",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1087",
       "triggerID" : "6a4c15ec39ded1f90b113b272554217698a98e1c",
       "triggerType" : "PUSH"
     }, {
       "hash" : "288e730d645a5e58e50a87ef5f44ba11a8fb0533",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1092",
       "triggerID" : "288e730d645a5e58e50a87ef5f44ba11a8fb0533",
       "triggerType" : "PUSH"
     }, {
       "hash" : "70a5c21321a0da3a0be8d85e206c24befa7b89f1",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1095",
       "triggerID" : "70a5c21321a0da3a0be8d85e206c24befa7b89f1",
       "triggerType" : "PUSH"
     }, {
       "hash" : "f072a986114949c5e426ce64a5f5568fc718e25d",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "f072a986114949c5e426ce64a5f5568fc718e25d",
       "triggerType" : "PUSH"
     }, {
       "hash" : "45b27475b1e8806ec8c566d6f151f4bc79c8549d",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1096",
       "triggerID" : "45b27475b1e8806ec8c566d6f151f4bc79c8549d",
       "triggerType" : "PUSH"
     }, {
       "hash" : "3c62ef22f6cde28110e5eb2fca8760a008d77723",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "3c62ef22f6cde28110e5eb2fca8760a008d77723",
       "triggerType" : "PUSH"
     }, {
       "hash" : "dc785fa04e542b2ccc45494e0292f5153c7ecea8",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1097",
       "triggerID" : "dc785fa04e542b2ccc45494e0292f5153c7ecea8",
       "triggerType" : "PUSH"
     }, {
       "hash" : "585ddb8049b9b9ccb057027e265b08a1b586df8c",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "585ddb8049b9b9ccb057027e265b08a1b586df8c",
       "triggerType" : "PUSH"
     }, {
       "hash" : "b62ff8675beed2c4487c788b9ca1a0ad4271b913",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "b62ff8675beed2c4487c788b9ca1a0ad4271b913",
       "triggerType" : "PUSH"
     }, {
       "hash" : "2d46048a7402907f348cc96da2839ba1b3e4932a",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1099",
       "triggerID" : "2d46048a7402907f348cc96da2839ba1b3e4932a",
       "triggerType" : "PUSH"
     }, {
       "hash" : "004fb6ebf23f796a4b77650e0530f0f1145e8b91",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1101",
       "triggerID" : "004fb6ebf23f796a4b77650e0530f0f1145e8b91",
       "triggerType" : "PUSH"
     }, {
       "hash" : "af0c33333b5e9ae6d59f20a6db1d7c016ea015d3",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1165",
       "triggerID" : "af0c33333b5e9ae6d59f20a6db1d7c016ea015d3",
       "triggerType" : "PUSH"
     }, {
       "hash" : "da8111118d71306500229619d3b7996fcd2526e8",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "da8111118d71306500229619d3b7996fcd2526e8",
       "triggerType" : "PUSH"
     }, {
       "hash" : "83257e2b9bd3c426cf90e11003a5f244a750d7fa",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1167",
       "triggerID" : "83257e2b9bd3c426cf90e11003a5f244a750d7fa",
       "triggerType" : "PUSH"
     }, {
       "hash" : "e0de408f1b8153fdaaea4c87b27a7b69213b75b5",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1177",
       "triggerID" : "e0de408f1b8153fdaaea4c87b27a7b69213b75b5",
       "triggerType" : "PUSH"
     }, {
       "hash" : "3f7951f7ad7931f5dc0b977d7c33bc5429623535",
       "status" : "FAILURE",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1180",
       "triggerID" : "3f7951f7ad7931f5dc0b977d7c33bc5429623535",
       "triggerType" : "PUSH"
     }, {
       "hash" : "9cd6f06a753b63630e8245d51338d2bf9ab86b84",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "9cd6f06a753b63630e8245d51338d2bf9ab86b84",
       "triggerType" : "PUSH"
     }, {
       "hash" : "7380b902731a92e0645c02397e14591a05798620",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "7380b902731a92e0645c02397e14591a05798620",
       "triggerType" : "PUSH"
     }, {
       "hash" : "0bd5cca01a541845081d5430d48b021d4bdf0969",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "0bd5cca01a541845081d5430d48b021d4bdf0969",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 9e668f65c6d4f73c76573236f8fa507dfb80edac UNKNOWN
   * 533f00e5546da0026bbe51e1c030ddd1c19fdd08 UNKNOWN
   * a2ee90c549332a7f17a7d9b2799849b77c08a8e6 UNKNOWN
   * fc2c138901e82ed5f3fbf363c2a0ce8140ade050 UNKNOWN
   * a510a7de1458bb996df5bbb341e8d010e4c91381 UNKNOWN
   * f072a986114949c5e426ce64a5f5568fc718e25d UNKNOWN
   * 3c62ef22f6cde28110e5eb2fca8760a008d77723 UNKNOWN
   * 585ddb8049b9b9ccb057027e265b08a1b586df8c UNKNOWN
   * b62ff8675beed2c4487c788b9ca1a0ad4271b913 UNKNOWN
   * da8111118d71306500229619d3b7996fcd2526e8 UNKNOWN
   * 3f7951f7ad7931f5dc0b977d7c33bc5429623535 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1180) 
   * 9cd6f06a753b63630e8245d51338d2bf9ab86b84 UNKNOWN
   * 7380b902731a92e0645c02397e14591a05798620 UNKNOWN
   * 0bd5cca01a541845081d5430d48b021d4bdf0969 UNKNOWN
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] [HUDI-8141] Incremental Query with Completion Time [hudi]

Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.

hudi-bot commented on PR #11947:
URL: https://github.com/apache/hudi/pull/11947#issuecomment-2421705188

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "090f256a315cbbf0c6a54d93cc2739dd693360e6",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=484",
       "triggerID" : "090f256a315cbbf0c6a54d93cc2739dd693360e6",
       "triggerType" : "PUSH"
     }, {
       "hash" : "0df5649013c1cf9e782fcbb67b8a2e7174f94512",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=985",
       "triggerID" : "0df5649013c1cf9e782fcbb67b8a2e7174f94512",
       "triggerType" : "PUSH"
     }, {
       "hash" : "d52e0128f409918644255abbe4376dc4a780bfc8",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=986",
       "triggerID" : "d52e0128f409918644255abbe4376dc4a780bfc8",
       "triggerType" : "PUSH"
     }, {
       "hash" : "9e668f65c6d4f73c76573236f8fa507dfb80edac",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "9e668f65c6d4f73c76573236f8fa507dfb80edac",
       "triggerType" : "PUSH"
     }, {
       "hash" : "f145131d524bf190986e5de600a326e32a27beca",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=987",
       "triggerID" : "f145131d524bf190986e5de600a326e32a27beca",
       "triggerType" : "PUSH"
     }, {
       "hash" : "0b008f807b42aa6fb2686922bb88d63e84a1f299",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=993",
       "triggerID" : "0b008f807b42aa6fb2686922bb88d63e84a1f299",
       "triggerType" : "PUSH"
     }, {
       "hash" : "65c01ad9f2b33d36d4890355feef91844438e732",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=995",
       "triggerID" : "65c01ad9f2b33d36d4890355feef91844438e732",
       "triggerType" : "PUSH"
     }, {
       "hash" : "955493fdc93842e8c99529cec35e39cecdbe9638",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=996",
       "triggerID" : "955493fdc93842e8c99529cec35e39cecdbe9638",
       "triggerType" : "PUSH"
     }, {
       "hash" : "533f00e5546da0026bbe51e1c030ddd1c19fdd08",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "533f00e5546da0026bbe51e1c030ddd1c19fdd08",
       "triggerType" : "PUSH"
     }, {
       "hash" : "05777be67e37f3a6a8eaa59e54d9e17e5fc6ba9b",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1016",
       "triggerID" : "05777be67e37f3a6a8eaa59e54d9e17e5fc6ba9b",
       "triggerType" : "PUSH"
     }, {
       "hash" : "a2ee90c549332a7f17a7d9b2799849b77c08a8e6",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "a2ee90c549332a7f17a7d9b2799849b77c08a8e6",
       "triggerType" : "PUSH"
     }, {
       "hash" : "05ce568b0b6ba14da9ca40d26b39ba478837d747",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1017",
       "triggerID" : "05ce568b0b6ba14da9ca40d26b39ba478837d747",
       "triggerType" : "PUSH"
     }, {
       "hash" : "840e3b33d39b1f86367bb6b0e65eadeb33de7765",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1025",
       "triggerID" : "840e3b33d39b1f86367bb6b0e65eadeb33de7765",
       "triggerType" : "PUSH"
     }, {
       "hash" : "8d2e2dd83f51d03e890d424a51d3e0e84e904a43",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1031",
       "triggerID" : "8d2e2dd83f51d03e890d424a51d3e0e84e904a43",
       "triggerType" : "PUSH"
     }, {
       "hash" : "688381914087ff11acf2f8c9372dbf39afa0bd40",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1035",
       "triggerID" : "688381914087ff11acf2f8c9372dbf39afa0bd40",
       "triggerType" : "PUSH"
     }, {
       "hash" : "fc2c138901e82ed5f3fbf363c2a0ce8140ade050",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "fc2c138901e82ed5f3fbf363c2a0ce8140ade050",
       "triggerType" : "PUSH"
     }, {
       "hash" : "6ed9e1cb1e5a191752d6d39c7a9d5d41840f0cdf",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1039",
       "triggerID" : "6ed9e1cb1e5a191752d6d39c7a9d5d41840f0cdf",
       "triggerType" : "PUSH"
     }, {
       "hash" : "a510a7de1458bb996df5bbb341e8d010e4c91381",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "a510a7de1458bb996df5bbb341e8d010e4c91381",
       "triggerType" : "PUSH"
     }, {
       "hash" : "49677397cd117228742205944606f7a789090376",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1042",
       "triggerID" : "49677397cd117228742205944606f7a789090376",
       "triggerType" : "PUSH"
     }, {
       "hash" : "1630db0a37422aed942ff280560a9d6329df5015",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1044",
       "triggerID" : "1630db0a37422aed942ff280560a9d6329df5015",
       "triggerType" : "PUSH"
     }, {
       "hash" : "f11c76301d4ab524db0af57aa831be3a898ea0fb",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1048",
       "triggerID" : "f11c76301d4ab524db0af57aa831be3a898ea0fb",
       "triggerType" : "PUSH"
     }, {
       "hash" : "d1a0dc4c2017853a7ba0c3302f886043a63f29da",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1050",
       "triggerID" : "d1a0dc4c2017853a7ba0c3302f886043a63f29da",
       "triggerType" : "PUSH"
     }, {
       "hash" : "6a4c15ec39ded1f90b113b272554217698a98e1c",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1087",
       "triggerID" : "6a4c15ec39ded1f90b113b272554217698a98e1c",
       "triggerType" : "PUSH"
     }, {
       "hash" : "288e730d645a5e58e50a87ef5f44ba11a8fb0533",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1092",
       "triggerID" : "288e730d645a5e58e50a87ef5f44ba11a8fb0533",
       "triggerType" : "PUSH"
     }, {
       "hash" : "70a5c21321a0da3a0be8d85e206c24befa7b89f1",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1095",
       "triggerID" : "70a5c21321a0da3a0be8d85e206c24befa7b89f1",
       "triggerType" : "PUSH"
     }, {
       "hash" : "f072a986114949c5e426ce64a5f5568fc718e25d",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "f072a986114949c5e426ce64a5f5568fc718e25d",
       "triggerType" : "PUSH"
     }, {
       "hash" : "45b27475b1e8806ec8c566d6f151f4bc79c8549d",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1096",
       "triggerID" : "45b27475b1e8806ec8c566d6f151f4bc79c8549d",
       "triggerType" : "PUSH"
     }, {
       "hash" : "3c62ef22f6cde28110e5eb2fca8760a008d77723",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "3c62ef22f6cde28110e5eb2fca8760a008d77723",
       "triggerType" : "PUSH"
     }, {
       "hash" : "dc785fa04e542b2ccc45494e0292f5153c7ecea8",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1097",
       "triggerID" : "dc785fa04e542b2ccc45494e0292f5153c7ecea8",
       "triggerType" : "PUSH"
     }, {
       "hash" : "585ddb8049b9b9ccb057027e265b08a1b586df8c",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "585ddb8049b9b9ccb057027e265b08a1b586df8c",
       "triggerType" : "PUSH"
     }, {
       "hash" : "b62ff8675beed2c4487c788b9ca1a0ad4271b913",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "b62ff8675beed2c4487c788b9ca1a0ad4271b913",
       "triggerType" : "PUSH"
     }, {
       "hash" : "2d46048a7402907f348cc96da2839ba1b3e4932a",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1099",
       "triggerID" : "2d46048a7402907f348cc96da2839ba1b3e4932a",
       "triggerType" : "PUSH"
     }, {
       "hash" : "004fb6ebf23f796a4b77650e0530f0f1145e8b91",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1101",
       "triggerID" : "004fb6ebf23f796a4b77650e0530f0f1145e8b91",
       "triggerType" : "PUSH"
     }, {
       "hash" : "af0c33333b5e9ae6d59f20a6db1d7c016ea015d3",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1165",
       "triggerID" : "af0c33333b5e9ae6d59f20a6db1d7c016ea015d3",
       "triggerType" : "PUSH"
     }, {
       "hash" : "da8111118d71306500229619d3b7996fcd2526e8",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "da8111118d71306500229619d3b7996fcd2526e8",
       "triggerType" : "PUSH"
     }, {
       "hash" : "83257e2b9bd3c426cf90e11003a5f244a750d7fa",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1167",
       "triggerID" : "83257e2b9bd3c426cf90e11003a5f244a750d7fa",
       "triggerType" : "PUSH"
     }, {
       "hash" : "e0de408f1b8153fdaaea4c87b27a7b69213b75b5",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1177",
       "triggerID" : "e0de408f1b8153fdaaea4c87b27a7b69213b75b5",
       "triggerType" : "PUSH"
     }, {
       "hash" : "3f7951f7ad7931f5dc0b977d7c33bc5429623535",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1180",
       "triggerID" : "3f7951f7ad7931f5dc0b977d7c33bc5429623535",
       "triggerType" : "PUSH"
     }, {
       "hash" : "9cd6f06a753b63630e8245d51338d2bf9ab86b84",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "9cd6f06a753b63630e8245d51338d2bf9ab86b84",
       "triggerType" : "PUSH"
     }, {
       "hash" : "7380b902731a92e0645c02397e14591a05798620",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "7380b902731a92e0645c02397e14591a05798620",
       "triggerType" : "PUSH"
     }, {
       "hash" : "0bd5cca01a541845081d5430d48b021d4bdf0969",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "0bd5cca01a541845081d5430d48b021d4bdf0969",
       "triggerType" : "PUSH"
     }, {
       "hash" : "8f4e175ef554f015e34e0968b656b5de714ab430",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1182",
       "triggerID" : "8f4e175ef554f015e34e0968b656b5de714ab430",
       "triggerType" : "PUSH"
     }, {
       "hash" : "2023f3c882d0c2f5b96e28d2f21812dc0c64aaee",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "2023f3c882d0c2f5b96e28d2f21812dc0c64aaee",
       "triggerType" : "PUSH"
     }, {
       "hash" : "8394acf060d5c3953dbd67cf930b9b1d3b92d8b5",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1198",
       "triggerID" : "8394acf060d5c3953dbd67cf930b9b1d3b92d8b5",
       "triggerType" : "PUSH"
     }, {
       "hash" : "7ec42c105971bff463e359019405a7f4e7947d86",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "7ec42c105971bff463e359019405a7f4e7947d86",
       "triggerType" : "PUSH"
     }, {
       "hash" : "4b930786a7b2de09ac1653cd884ee76a7d1b2ca5",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1202",
       "triggerID" : "4b930786a7b2de09ac1653cd884ee76a7d1b2ca5",
       "triggerType" : "PUSH"
     }, {
       "hash" : "0e9a648c4e03b5a74d6a0085831805041a9a26a2",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1216",
       "triggerID" : "0e9a648c4e03b5a74d6a0085831805041a9a26a2",
       "triggerType" : "PUSH"
     }, {
       "hash" : "326a8729a5d1dd88bc474a5c273aec6ca219f298",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "326a8729a5d1dd88bc474a5c273aec6ca219f298",
       "triggerType" : "PUSH"
     }, {
       "hash" : "cad2f761fc50fba28e2fb3595e4b1270e39ac48a",
       "status" : "CANCELED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1221",
       "triggerID" : "cad2f761fc50fba28e2fb3595e4b1270e39ac48a",
       "triggerType" : "PUSH"
     }, {
       "hash" : "d485a50aec34bc5707377afe96d696772832558e",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "d485a50aec34bc5707377afe96d696772832558e",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 9e668f65c6d4f73c76573236f8fa507dfb80edac UNKNOWN
   * 533f00e5546da0026bbe51e1c030ddd1c19fdd08 UNKNOWN
   * a2ee90c549332a7f17a7d9b2799849b77c08a8e6 UNKNOWN
   * fc2c138901e82ed5f3fbf363c2a0ce8140ade050 UNKNOWN
   * a510a7de1458bb996df5bbb341e8d010e4c91381 UNKNOWN
   * f072a986114949c5e426ce64a5f5568fc718e25d UNKNOWN
   * 3c62ef22f6cde28110e5eb2fca8760a008d77723 UNKNOWN
   * 585ddb8049b9b9ccb057027e265b08a1b586df8c UNKNOWN
   * b62ff8675beed2c4487c788b9ca1a0ad4271b913 UNKNOWN
   * da8111118d71306500229619d3b7996fcd2526e8 UNKNOWN
   * 9cd6f06a753b63630e8245d51338d2bf9ab86b84 UNKNOWN
   * 7380b902731a92e0645c02397e14591a05798620 UNKNOWN
   * 0bd5cca01a541845081d5430d48b021d4bdf0969 UNKNOWN
   * 2023f3c882d0c2f5b96e28d2f21812dc0c64aaee UNKNOWN
   * 7ec42c105971bff463e359019405a7f4e7947d86 UNKNOWN
   * 326a8729a5d1dd88bc474a5c273aec6ca219f298 UNKNOWN
   * cad2f761fc50fba28e2fb3595e4b1270e39ac48a Azure: [CANCELED](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1221) 
   * d485a50aec34bc5707377afe96d696772832558e UNKNOWN
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] [HUDI-8141] Incremental Query with Completion Time [hudi]

Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.

hudi-bot commented on PR #11947:
URL: https://github.com/apache/hudi/pull/11947#issuecomment-2423321647

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "090f256a315cbbf0c6a54d93cc2739dd693360e6",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=484",
       "triggerID" : "090f256a315cbbf0c6a54d93cc2739dd693360e6",
       "triggerType" : "PUSH"
     }, {
       "hash" : "0df5649013c1cf9e782fcbb67b8a2e7174f94512",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=985",
       "triggerID" : "0df5649013c1cf9e782fcbb67b8a2e7174f94512",
       "triggerType" : "PUSH"
     }, {
       "hash" : "d52e0128f409918644255abbe4376dc4a780bfc8",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=986",
       "triggerID" : "d52e0128f409918644255abbe4376dc4a780bfc8",
       "triggerType" : "PUSH"
     }, {
       "hash" : "9e668f65c6d4f73c76573236f8fa507dfb80edac",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "9e668f65c6d4f73c76573236f8fa507dfb80edac",
       "triggerType" : "PUSH"
     }, {
       "hash" : "f145131d524bf190986e5de600a326e32a27beca",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=987",
       "triggerID" : "f145131d524bf190986e5de600a326e32a27beca",
       "triggerType" : "PUSH"
     }, {
       "hash" : "0b008f807b42aa6fb2686922bb88d63e84a1f299",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=993",
       "triggerID" : "0b008f807b42aa6fb2686922bb88d63e84a1f299",
       "triggerType" : "PUSH"
     }, {
       "hash" : "65c01ad9f2b33d36d4890355feef91844438e732",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=995",
       "triggerID" : "65c01ad9f2b33d36d4890355feef91844438e732",
       "triggerType" : "PUSH"
     }, {
       "hash" : "955493fdc93842e8c99529cec35e39cecdbe9638",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=996",
       "triggerID" : "955493fdc93842e8c99529cec35e39cecdbe9638",
       "triggerType" : "PUSH"
     }, {
       "hash" : "533f00e5546da0026bbe51e1c030ddd1c19fdd08",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "533f00e5546da0026bbe51e1c030ddd1c19fdd08",
       "triggerType" : "PUSH"
     }, {
       "hash" : "05777be67e37f3a6a8eaa59e54d9e17e5fc6ba9b",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1016",
       "triggerID" : "05777be67e37f3a6a8eaa59e54d9e17e5fc6ba9b",
       "triggerType" : "PUSH"
     }, {
       "hash" : "a2ee90c549332a7f17a7d9b2799849b77c08a8e6",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "a2ee90c549332a7f17a7d9b2799849b77c08a8e6",
       "triggerType" : "PUSH"
     }, {
       "hash" : "05ce568b0b6ba14da9ca40d26b39ba478837d747",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1017",
       "triggerID" : "05ce568b0b6ba14da9ca40d26b39ba478837d747",
       "triggerType" : "PUSH"
     }, {
       "hash" : "840e3b33d39b1f86367bb6b0e65eadeb33de7765",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1025",
       "triggerID" : "840e3b33d39b1f86367bb6b0e65eadeb33de7765",
       "triggerType" : "PUSH"
     }, {
       "hash" : "8d2e2dd83f51d03e890d424a51d3e0e84e904a43",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1031",
       "triggerID" : "8d2e2dd83f51d03e890d424a51d3e0e84e904a43",
       "triggerType" : "PUSH"
     }, {
       "hash" : "688381914087ff11acf2f8c9372dbf39afa0bd40",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1035",
       "triggerID" : "688381914087ff11acf2f8c9372dbf39afa0bd40",
       "triggerType" : "PUSH"
     }, {
       "hash" : "fc2c138901e82ed5f3fbf363c2a0ce8140ade050",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "fc2c138901e82ed5f3fbf363c2a0ce8140ade050",
       "triggerType" : "PUSH"
     }, {
       "hash" : "6ed9e1cb1e5a191752d6d39c7a9d5d41840f0cdf",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1039",
       "triggerID" : "6ed9e1cb1e5a191752d6d39c7a9d5d41840f0cdf",
       "triggerType" : "PUSH"
     }, {
       "hash" : "a510a7de1458bb996df5bbb341e8d010e4c91381",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "a510a7de1458bb996df5bbb341e8d010e4c91381",
       "triggerType" : "PUSH"
     }, {
       "hash" : "49677397cd117228742205944606f7a789090376",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1042",
       "triggerID" : "49677397cd117228742205944606f7a789090376",
       "triggerType" : "PUSH"
     }, {
       "hash" : "1630db0a37422aed942ff280560a9d6329df5015",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1044",
       "triggerID" : "1630db0a37422aed942ff280560a9d6329df5015",
       "triggerType" : "PUSH"
     }, {
       "hash" : "f11c76301d4ab524db0af57aa831be3a898ea0fb",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1048",
       "triggerID" : "f11c76301d4ab524db0af57aa831be3a898ea0fb",
       "triggerType" : "PUSH"
     }, {
       "hash" : "d1a0dc4c2017853a7ba0c3302f886043a63f29da",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1050",
       "triggerID" : "d1a0dc4c2017853a7ba0c3302f886043a63f29da",
       "triggerType" : "PUSH"
     }, {
       "hash" : "6a4c15ec39ded1f90b113b272554217698a98e1c",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1087",
       "triggerID" : "6a4c15ec39ded1f90b113b272554217698a98e1c",
       "triggerType" : "PUSH"
     }, {
       "hash" : "288e730d645a5e58e50a87ef5f44ba11a8fb0533",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1092",
       "triggerID" : "288e730d645a5e58e50a87ef5f44ba11a8fb0533",
       "triggerType" : "PUSH"
     }, {
       "hash" : "70a5c21321a0da3a0be8d85e206c24befa7b89f1",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1095",
       "triggerID" : "70a5c21321a0da3a0be8d85e206c24befa7b89f1",
       "triggerType" : "PUSH"
     }, {
       "hash" : "f072a986114949c5e426ce64a5f5568fc718e25d",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "f072a986114949c5e426ce64a5f5568fc718e25d",
       "triggerType" : "PUSH"
     }, {
       "hash" : "45b27475b1e8806ec8c566d6f151f4bc79c8549d",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1096",
       "triggerID" : "45b27475b1e8806ec8c566d6f151f4bc79c8549d",
       "triggerType" : "PUSH"
     }, {
       "hash" : "3c62ef22f6cde28110e5eb2fca8760a008d77723",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "3c62ef22f6cde28110e5eb2fca8760a008d77723",
       "triggerType" : "PUSH"
     }, {
       "hash" : "dc785fa04e542b2ccc45494e0292f5153c7ecea8",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1097",
       "triggerID" : "dc785fa04e542b2ccc45494e0292f5153c7ecea8",
       "triggerType" : "PUSH"
     }, {
       "hash" : "585ddb8049b9b9ccb057027e265b08a1b586df8c",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "585ddb8049b9b9ccb057027e265b08a1b586df8c",
       "triggerType" : "PUSH"
     }, {
       "hash" : "b62ff8675beed2c4487c788b9ca1a0ad4271b913",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "b62ff8675beed2c4487c788b9ca1a0ad4271b913",
       "triggerType" : "PUSH"
     }, {
       "hash" : "2d46048a7402907f348cc96da2839ba1b3e4932a",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1099",
       "triggerID" : "2d46048a7402907f348cc96da2839ba1b3e4932a",
       "triggerType" : "PUSH"
     }, {
       "hash" : "004fb6ebf23f796a4b77650e0530f0f1145e8b91",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1101",
       "triggerID" : "004fb6ebf23f796a4b77650e0530f0f1145e8b91",
       "triggerType" : "PUSH"
     }, {
       "hash" : "af0c33333b5e9ae6d59f20a6db1d7c016ea015d3",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1165",
       "triggerID" : "af0c33333b5e9ae6d59f20a6db1d7c016ea015d3",
       "triggerType" : "PUSH"
     }, {
       "hash" : "da8111118d71306500229619d3b7996fcd2526e8",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "da8111118d71306500229619d3b7996fcd2526e8",
       "triggerType" : "PUSH"
     }, {
       "hash" : "83257e2b9bd3c426cf90e11003a5f244a750d7fa",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1167",
       "triggerID" : "83257e2b9bd3c426cf90e11003a5f244a750d7fa",
       "triggerType" : "PUSH"
     }, {
       "hash" : "e0de408f1b8153fdaaea4c87b27a7b69213b75b5",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1177",
       "triggerID" : "e0de408f1b8153fdaaea4c87b27a7b69213b75b5",
       "triggerType" : "PUSH"
     }, {
       "hash" : "3f7951f7ad7931f5dc0b977d7c33bc5429623535",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1180",
       "triggerID" : "3f7951f7ad7931f5dc0b977d7c33bc5429623535",
       "triggerType" : "PUSH"
     }, {
       "hash" : "9cd6f06a753b63630e8245d51338d2bf9ab86b84",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "9cd6f06a753b63630e8245d51338d2bf9ab86b84",
       "triggerType" : "PUSH"
     }, {
       "hash" : "7380b902731a92e0645c02397e14591a05798620",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "7380b902731a92e0645c02397e14591a05798620",
       "triggerType" : "PUSH"
     }, {
       "hash" : "0bd5cca01a541845081d5430d48b021d4bdf0969",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "0bd5cca01a541845081d5430d48b021d4bdf0969",
       "triggerType" : "PUSH"
     }, {
       "hash" : "8f4e175ef554f015e34e0968b656b5de714ab430",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1182",
       "triggerID" : "8f4e175ef554f015e34e0968b656b5de714ab430",
       "triggerType" : "PUSH"
     }, {
       "hash" : "2023f3c882d0c2f5b96e28d2f21812dc0c64aaee",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "2023f3c882d0c2f5b96e28d2f21812dc0c64aaee",
       "triggerType" : "PUSH"
     }, {
       "hash" : "8394acf060d5c3953dbd67cf930b9b1d3b92d8b5",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1198",
       "triggerID" : "8394acf060d5c3953dbd67cf930b9b1d3b92d8b5",
       "triggerType" : "PUSH"
     }, {
       "hash" : "7ec42c105971bff463e359019405a7f4e7947d86",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "7ec42c105971bff463e359019405a7f4e7947d86",
       "triggerType" : "PUSH"
     }, {
       "hash" : "4b930786a7b2de09ac1653cd884ee76a7d1b2ca5",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1202",
       "triggerID" : "4b930786a7b2de09ac1653cd884ee76a7d1b2ca5",
       "triggerType" : "PUSH"
     }, {
       "hash" : "0e9a648c4e03b5a74d6a0085831805041a9a26a2",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1216",
       "triggerID" : "0e9a648c4e03b5a74d6a0085831805041a9a26a2",
       "triggerType" : "PUSH"
     }, {
       "hash" : "326a8729a5d1dd88bc474a5c273aec6ca219f298",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "326a8729a5d1dd88bc474a5c273aec6ca219f298",
       "triggerType" : "PUSH"
     }, {
       "hash" : "cad2f761fc50fba28e2fb3595e4b1270e39ac48a",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1221",
       "triggerID" : "cad2f761fc50fba28e2fb3595e4b1270e39ac48a",
       "triggerType" : "PUSH"
     }, {
       "hash" : "d485a50aec34bc5707377afe96d696772832558e",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "d485a50aec34bc5707377afe96d696772832558e",
       "triggerType" : "PUSH"
     }, {
       "hash" : "38a98106f19fe4e51632a2c4289ec0f01805ba5e",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1224",
       "triggerID" : "38a98106f19fe4e51632a2c4289ec0f01805ba5e",
       "triggerType" : "PUSH"
     }, {
       "hash" : "7a6116cfbf4eba54e77194664225ed1c372d71be",
       "status" : "FAILURE",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1250",
       "triggerID" : "7a6116cfbf4eba54e77194664225ed1c372d71be",
       "triggerType" : "PUSH"
     }, {
       "hash" : "df721d8b5f610e156430f46b1a704310c70a4725",
       "status" : "PENDING",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1254",
       "triggerID" : "df721d8b5f610e156430f46b1a704310c70a4725",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 9e668f65c6d4f73c76573236f8fa507dfb80edac UNKNOWN
   * 533f00e5546da0026bbe51e1c030ddd1c19fdd08 UNKNOWN
   * a2ee90c549332a7f17a7d9b2799849b77c08a8e6 UNKNOWN
   * fc2c138901e82ed5f3fbf363c2a0ce8140ade050 UNKNOWN
   * a510a7de1458bb996df5bbb341e8d010e4c91381 UNKNOWN
   * f072a986114949c5e426ce64a5f5568fc718e25d UNKNOWN
   * 3c62ef22f6cde28110e5eb2fca8760a008d77723 UNKNOWN
   * 585ddb8049b9b9ccb057027e265b08a1b586df8c UNKNOWN
   * b62ff8675beed2c4487c788b9ca1a0ad4271b913 UNKNOWN
   * da8111118d71306500229619d3b7996fcd2526e8 UNKNOWN
   * 9cd6f06a753b63630e8245d51338d2bf9ab86b84 UNKNOWN
   * 7380b902731a92e0645c02397e14591a05798620 UNKNOWN
   * 0bd5cca01a541845081d5430d48b021d4bdf0969 UNKNOWN
   * 2023f3c882d0c2f5b96e28d2f21812dc0c64aaee UNKNOWN
   * 7ec42c105971bff463e359019405a7f4e7947d86 UNKNOWN
   * 326a8729a5d1dd88bc474a5c273aec6ca219f298 UNKNOWN
   * d485a50aec34bc5707377afe96d696772832558e UNKNOWN
   * 7a6116cfbf4eba54e77194664225ed1c372d71be Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1250) 
   * df721d8b5f610e156430f46b1a704310c70a4725 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1254) 
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] [HUDI-8141] Incremental Query with Completion Time [hudi]

Posted by "yihua (via GitHub)" <gi...@apache.org>.

yihua commented on code in PR #11947:
URL: https://github.com/apache/hudi/pull/11947#discussion_r1804238956


##########
hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/spark/sql/hudi/streaming/HoodieStreamSource.scala:
##########
@@ -186,11 +160,11 @@ class HoodieStreamSource(
     }
   }
 
-  private def startCommitTime(startOffset: HoodieSourceOffset): String = {
+  private def startCompletionTime(startOffset: HoodieSourceOffset): String = {
     startOffset match {
-      case INIT_OFFSET => startOffset.commitTime
-      case HoodieSourceOffset(commitTime) =>
-        commitTime
+      case INIT_OFFSET => startOffset.completionTime
+      case HoodieSourceOffset(completionTime) =>
+        HoodieInstantTimeGenerator.instantTimePlusMillis(completionTime, 1)

Review Comment:
   This is to skip the open range start based on the last checkpoint of the stream source.  I'm trying to see if there a better and efficient way to get the completion time of next commit.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] [HUDI-8141] Incremental Query with Completion Time [hudi]

Posted by "yihua (via GitHub)" <gi...@apache.org>.

yihua commented on code in PR #11947:
URL: https://github.com/apache/hudi/pull/11947#discussion_r1804271104


##########
hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/hudi/IncrementalRelation.scala:
##########
@@ -90,24 +78,24 @@ class IncrementalRelation(val sqlContext: SQLContext,
     throw new HoodieException("Incremental queries are not supported when meta fields are disabled")
   }
 
+  private val queryContext: IncrementalQueryAnalyzer.QueryContext =
+    IncrementalQueryAnalyzer.builder()
+      .metaClient(metaClient)
+      .startTime(optParams(DataSourceReadOptions.BEGIN_INSTANTTIME.key))
+      .endTime(optParams.getOrElse(DataSourceReadOptions.END_INSTANTTIME.key, null))
+      .rangeType(InstantRange.RangeType.OPEN_CLOSED)
+      .build()
+      .analyze()
+
+  private val commitsToReturn = List.concat(
+    queryContext.getArchivedInstants.asScala,
+    queryContext.getActiveInstants.asScala)

Review Comment:
   This is fixed now by adding a new API in `QueryContext` to get a list of `HoodieInstant`.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] [HUDI-8141] Incremental Query with Completion Time [hudi]

Posted by "danny0405 (via GitHub)" <gi...@apache.org>.

danny0405 commented on code in PR #11947:
URL: https://github.com/apache/hudi/pull/11947#discussion_r1805710712


##########
hudi-utilities/src/main/java/org/apache/hudi/utilities/sources/HoodieIncrSource.java:
##########
@@ -212,35 +209,58 @@ public Pair<Option<Dataset<Row>>, String> fetchNextBatch(Option<String> lastCkpt
           .collect(Collectors.toMap(Pair::getLeft, Pair::getRight));
       reader = reader.options(optionsMap);
     }
+
+    boolean shouldFullScan =
+        missingCheckpointStrategy == MissingCheckpointStrategy.READ_UPTO_LATEST_COMMIT
+            && queryContext.getActiveTimeline()
+            .isBeforeTimelineStartsByCompletionTime(analyzer.getBeginCompletionTime().get());
     Dataset<Row> source;
-    // Do Incr pull. Set end instant if available
-    if (queryInfo.isIncremental()) {
-      source = reader
-          .options(readOpts)
-          .option(QUERY_TYPE().key(), QUERY_TYPE_INCREMENTAL_OPT_VAL())
-          .option(BEGIN_INSTANTTIME().key(), queryInfo.getStartInstant())
-          .option(END_INSTANTTIME().key(), queryInfo.getEndInstant())
-          .option(INCREMENTAL_FALLBACK_TO_FULL_TABLE_SCAN().key(),
-              props.getString(INCREMENTAL_FALLBACK_TO_FULL_TABLE_SCAN().key(),
-                  INCREMENTAL_FALLBACK_TO_FULL_TABLE_SCAN().defaultValue()))
-          .option(INCREMENTAL_READ_HANDLE_HOLLOW_COMMIT().key(), handlingMode.name())
-          .load(srcPath);
-    } else {
-      // if checkpoint is missing from source table, and if strategy is set to READ_UPTO_LATEST_COMMIT, we have to issue snapshot query
+    if (instantRange.isEmpty() || shouldFullScan) {
+      // snapshot query
       Dataset<Row> snapshot = reader
           .options(readOpts)
           .option(DataSourceReadOptions.QUERY_TYPE().key(), DataSourceReadOptions.QUERY_TYPE_SNAPSHOT_OPT_VAL())
           .load(srcPath);
+
+      Option<String> predicate = Option.empty();
+      List<String> instantTimeList = queryContext.getInstantTimeList();
       if (snapshotLoadQuerySplitter.isPresent()) {
-        queryInfo = snapshotLoadQuerySplitter.get().getNextCheckpoint(snapshot, queryInfo, sourceProfileSupplier);
+        Option<SnapshotLoadQuerySplitter.CheckpointWithPredicates> newCheckpointAndPredicate =
+            snapshotLoadQuerySplitter.get().getNextCheckpoint(snapshot, queryContext, sourceProfileSupplier);
+        if (newCheckpointAndPredicate.isPresent()) {
+          endCompletionTime = newCheckpointAndPredicate.get().endInstant;
+          predicate = Option.of(newCheckpointAndPredicate.get().predicateFilter);
+          instantTimeList = queryContext.getInstants().stream()
+              .filter(instant -> HoodieTimeline.compareTimestamps(
+                  instant.getCompletionTime(), HoodieTimeline.LESSER_THAN_OR_EQUALS, newCheckpointAndPredicate.get().endInstant))
+              .map(HoodieInstant::getTimestamp)
+              .collect(Collectors.toList());
+        } else {
+          endCompletionTime = queryContext.getEndInstant().orElse(queryContext.getLastInstant());
+        }
       }
+
+      snapshot = predicate.map(snapshot::filter).orElse(snapshot);
       source = snapshot
           // add filtering so that only interested records are returned.
-          .filter(String.format("%s > '%s'", HoodieRecord.COMMIT_TIME_METADATA_FIELD,
-              queryInfo.getStartInstant()))
-          .filter(String.format("%s <= '%s'", HoodieRecord.COMMIT_TIME_METADATA_FIELD,
-              queryInfo.getEndInstant()));
-      source = queryInfo.getPredicateFilter().map(source::filter).orElse(source);
+          .filter(String.format("%s IN ('%s')", HoodieRecord.COMMIT_TIME_METADATA_FIELD,
+              String.join("','", instantTimeList)));
+    } else {
+      // normal incremental query
+      String beginCompletionTime = queryContext.getInstants().stream()
+          .map(HoodieInstant::getCompletionTime)

Review Comment:
   Use completion time comparator?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] [HUDI-8141] Incremental Query with Completion Time [hudi]

Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.

hudi-bot commented on PR #11947:
URL: https://github.com/apache/hudi/pull/11947#issuecomment-2421229093

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "090f256a315cbbf0c6a54d93cc2739dd693360e6",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=484",
       "triggerID" : "090f256a315cbbf0c6a54d93cc2739dd693360e6",
       "triggerType" : "PUSH"
     }, {
       "hash" : "0df5649013c1cf9e782fcbb67b8a2e7174f94512",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=985",
       "triggerID" : "0df5649013c1cf9e782fcbb67b8a2e7174f94512",
       "triggerType" : "PUSH"
     }, {
       "hash" : "d52e0128f409918644255abbe4376dc4a780bfc8",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=986",
       "triggerID" : "d52e0128f409918644255abbe4376dc4a780bfc8",
       "triggerType" : "PUSH"
     }, {
       "hash" : "9e668f65c6d4f73c76573236f8fa507dfb80edac",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "9e668f65c6d4f73c76573236f8fa507dfb80edac",
       "triggerType" : "PUSH"
     }, {
       "hash" : "f145131d524bf190986e5de600a326e32a27beca",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=987",
       "triggerID" : "f145131d524bf190986e5de600a326e32a27beca",
       "triggerType" : "PUSH"
     }, {
       "hash" : "0b008f807b42aa6fb2686922bb88d63e84a1f299",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=993",
       "triggerID" : "0b008f807b42aa6fb2686922bb88d63e84a1f299",
       "triggerType" : "PUSH"
     }, {
       "hash" : "65c01ad9f2b33d36d4890355feef91844438e732",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=995",
       "triggerID" : "65c01ad9f2b33d36d4890355feef91844438e732",
       "triggerType" : "PUSH"
     }, {
       "hash" : "955493fdc93842e8c99529cec35e39cecdbe9638",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=996",
       "triggerID" : "955493fdc93842e8c99529cec35e39cecdbe9638",
       "triggerType" : "PUSH"
     }, {
       "hash" : "533f00e5546da0026bbe51e1c030ddd1c19fdd08",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "533f00e5546da0026bbe51e1c030ddd1c19fdd08",
       "triggerType" : "PUSH"
     }, {
       "hash" : "05777be67e37f3a6a8eaa59e54d9e17e5fc6ba9b",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1016",
       "triggerID" : "05777be67e37f3a6a8eaa59e54d9e17e5fc6ba9b",
       "triggerType" : "PUSH"
     }, {
       "hash" : "a2ee90c549332a7f17a7d9b2799849b77c08a8e6",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "a2ee90c549332a7f17a7d9b2799849b77c08a8e6",
       "triggerType" : "PUSH"
     }, {
       "hash" : "05ce568b0b6ba14da9ca40d26b39ba478837d747",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1017",
       "triggerID" : "05ce568b0b6ba14da9ca40d26b39ba478837d747",
       "triggerType" : "PUSH"
     }, {
       "hash" : "840e3b33d39b1f86367bb6b0e65eadeb33de7765",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1025",
       "triggerID" : "840e3b33d39b1f86367bb6b0e65eadeb33de7765",
       "triggerType" : "PUSH"
     }, {
       "hash" : "8d2e2dd83f51d03e890d424a51d3e0e84e904a43",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1031",
       "triggerID" : "8d2e2dd83f51d03e890d424a51d3e0e84e904a43",
       "triggerType" : "PUSH"
     }, {
       "hash" : "688381914087ff11acf2f8c9372dbf39afa0bd40",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1035",
       "triggerID" : "688381914087ff11acf2f8c9372dbf39afa0bd40",
       "triggerType" : "PUSH"
     }, {
       "hash" : "fc2c138901e82ed5f3fbf363c2a0ce8140ade050",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "fc2c138901e82ed5f3fbf363c2a0ce8140ade050",
       "triggerType" : "PUSH"
     }, {
       "hash" : "6ed9e1cb1e5a191752d6d39c7a9d5d41840f0cdf",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1039",
       "triggerID" : "6ed9e1cb1e5a191752d6d39c7a9d5d41840f0cdf",
       "triggerType" : "PUSH"
     }, {
       "hash" : "a510a7de1458bb996df5bbb341e8d010e4c91381",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "a510a7de1458bb996df5bbb341e8d010e4c91381",
       "triggerType" : "PUSH"
     }, {
       "hash" : "49677397cd117228742205944606f7a789090376",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1042",
       "triggerID" : "49677397cd117228742205944606f7a789090376",
       "triggerType" : "PUSH"
     }, {
       "hash" : "1630db0a37422aed942ff280560a9d6329df5015",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1044",
       "triggerID" : "1630db0a37422aed942ff280560a9d6329df5015",
       "triggerType" : "PUSH"
     }, {
       "hash" : "f11c76301d4ab524db0af57aa831be3a898ea0fb",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1048",
       "triggerID" : "f11c76301d4ab524db0af57aa831be3a898ea0fb",
       "triggerType" : "PUSH"
     }, {
       "hash" : "d1a0dc4c2017853a7ba0c3302f886043a63f29da",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1050",
       "triggerID" : "d1a0dc4c2017853a7ba0c3302f886043a63f29da",
       "triggerType" : "PUSH"
     }, {
       "hash" : "6a4c15ec39ded1f90b113b272554217698a98e1c",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1087",
       "triggerID" : "6a4c15ec39ded1f90b113b272554217698a98e1c",
       "triggerType" : "PUSH"
     }, {
       "hash" : "288e730d645a5e58e50a87ef5f44ba11a8fb0533",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1092",
       "triggerID" : "288e730d645a5e58e50a87ef5f44ba11a8fb0533",
       "triggerType" : "PUSH"
     }, {
       "hash" : "70a5c21321a0da3a0be8d85e206c24befa7b89f1",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1095",
       "triggerID" : "70a5c21321a0da3a0be8d85e206c24befa7b89f1",
       "triggerType" : "PUSH"
     }, {
       "hash" : "f072a986114949c5e426ce64a5f5568fc718e25d",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "f072a986114949c5e426ce64a5f5568fc718e25d",
       "triggerType" : "PUSH"
     }, {
       "hash" : "45b27475b1e8806ec8c566d6f151f4bc79c8549d",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1096",
       "triggerID" : "45b27475b1e8806ec8c566d6f151f4bc79c8549d",
       "triggerType" : "PUSH"
     }, {
       "hash" : "3c62ef22f6cde28110e5eb2fca8760a008d77723",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "3c62ef22f6cde28110e5eb2fca8760a008d77723",
       "triggerType" : "PUSH"
     }, {
       "hash" : "dc785fa04e542b2ccc45494e0292f5153c7ecea8",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1097",
       "triggerID" : "dc785fa04e542b2ccc45494e0292f5153c7ecea8",
       "triggerType" : "PUSH"
     }, {
       "hash" : "585ddb8049b9b9ccb057027e265b08a1b586df8c",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "585ddb8049b9b9ccb057027e265b08a1b586df8c",
       "triggerType" : "PUSH"
     }, {
       "hash" : "b62ff8675beed2c4487c788b9ca1a0ad4271b913",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "b62ff8675beed2c4487c788b9ca1a0ad4271b913",
       "triggerType" : "PUSH"
     }, {
       "hash" : "2d46048a7402907f348cc96da2839ba1b3e4932a",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1099",
       "triggerID" : "2d46048a7402907f348cc96da2839ba1b3e4932a",
       "triggerType" : "PUSH"
     }, {
       "hash" : "004fb6ebf23f796a4b77650e0530f0f1145e8b91",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1101",
       "triggerID" : "004fb6ebf23f796a4b77650e0530f0f1145e8b91",
       "triggerType" : "PUSH"
     }, {
       "hash" : "af0c33333b5e9ae6d59f20a6db1d7c016ea015d3",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1165",
       "triggerID" : "af0c33333b5e9ae6d59f20a6db1d7c016ea015d3",
       "triggerType" : "PUSH"
     }, {
       "hash" : "da8111118d71306500229619d3b7996fcd2526e8",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "da8111118d71306500229619d3b7996fcd2526e8",
       "triggerType" : "PUSH"
     }, {
       "hash" : "83257e2b9bd3c426cf90e11003a5f244a750d7fa",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1167",
       "triggerID" : "83257e2b9bd3c426cf90e11003a5f244a750d7fa",
       "triggerType" : "PUSH"
     }, {
       "hash" : "e0de408f1b8153fdaaea4c87b27a7b69213b75b5",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1177",
       "triggerID" : "e0de408f1b8153fdaaea4c87b27a7b69213b75b5",
       "triggerType" : "PUSH"
     }, {
       "hash" : "3f7951f7ad7931f5dc0b977d7c33bc5429623535",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1180",
       "triggerID" : "3f7951f7ad7931f5dc0b977d7c33bc5429623535",
       "triggerType" : "PUSH"
     }, {
       "hash" : "9cd6f06a753b63630e8245d51338d2bf9ab86b84",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "9cd6f06a753b63630e8245d51338d2bf9ab86b84",
       "triggerType" : "PUSH"
     }, {
       "hash" : "7380b902731a92e0645c02397e14591a05798620",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "7380b902731a92e0645c02397e14591a05798620",
       "triggerType" : "PUSH"
     }, {
       "hash" : "0bd5cca01a541845081d5430d48b021d4bdf0969",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "0bd5cca01a541845081d5430d48b021d4bdf0969",
       "triggerType" : "PUSH"
     }, {
       "hash" : "8f4e175ef554f015e34e0968b656b5de714ab430",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1182",
       "triggerID" : "8f4e175ef554f015e34e0968b656b5de714ab430",
       "triggerType" : "PUSH"
     }, {
       "hash" : "2023f3c882d0c2f5b96e28d2f21812dc0c64aaee",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "2023f3c882d0c2f5b96e28d2f21812dc0c64aaee",
       "triggerType" : "PUSH"
     }, {
       "hash" : "8394acf060d5c3953dbd67cf930b9b1d3b92d8b5",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1198",
       "triggerID" : "8394acf060d5c3953dbd67cf930b9b1d3b92d8b5",
       "triggerType" : "PUSH"
     }, {
       "hash" : "7ec42c105971bff463e359019405a7f4e7947d86",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "7ec42c105971bff463e359019405a7f4e7947d86",
       "triggerType" : "PUSH"
     }, {
       "hash" : "4b930786a7b2de09ac1653cd884ee76a7d1b2ca5",
       "status" : "SUCCESS",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1202",
       "triggerID" : "4b930786a7b2de09ac1653cd884ee76a7d1b2ca5",
       "triggerType" : "PUSH"
     }, {
       "hash" : "0e9a648c4e03b5a74d6a0085831805041a9a26a2",
       "status" : "PENDING",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1216",
       "triggerID" : "0e9a648c4e03b5a74d6a0085831805041a9a26a2",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 9e668f65c6d4f73c76573236f8fa507dfb80edac UNKNOWN
   * 533f00e5546da0026bbe51e1c030ddd1c19fdd08 UNKNOWN
   * a2ee90c549332a7f17a7d9b2799849b77c08a8e6 UNKNOWN
   * fc2c138901e82ed5f3fbf363c2a0ce8140ade050 UNKNOWN
   * a510a7de1458bb996df5bbb341e8d010e4c91381 UNKNOWN
   * f072a986114949c5e426ce64a5f5568fc718e25d UNKNOWN
   * 3c62ef22f6cde28110e5eb2fca8760a008d77723 UNKNOWN
   * 585ddb8049b9b9ccb057027e265b08a1b586df8c UNKNOWN
   * b62ff8675beed2c4487c788b9ca1a0ad4271b913 UNKNOWN
   * da8111118d71306500229619d3b7996fcd2526e8 UNKNOWN
   * 9cd6f06a753b63630e8245d51338d2bf9ab86b84 UNKNOWN
   * 7380b902731a92e0645c02397e14591a05798620 UNKNOWN
   * 0bd5cca01a541845081d5430d48b021d4bdf0969 UNKNOWN
   * 2023f3c882d0c2f5b96e28d2f21812dc0c64aaee UNKNOWN
   * 7ec42c105971bff463e359019405a7f4e7947d86 UNKNOWN
   * 4b930786a7b2de09ac1653cd884ee76a7d1b2ca5 Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1202) 
   * 0e9a648c4e03b5a74d6a0085831805041a9a26a2 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1216) 
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] [HUDI-8141] Incremental Query with Completion Time [hudi]

Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.

hudi-bot commented on PR #11947:
URL: https://github.com/apache/hudi/pull/11947#issuecomment-2418515835

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "090f256a315cbbf0c6a54d93cc2739dd693360e6",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=484",
       "triggerID" : "090f256a315cbbf0c6a54d93cc2739dd693360e6",
       "triggerType" : "PUSH"
     }, {
       "hash" : "0df5649013c1cf9e782fcbb67b8a2e7174f94512",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=985",
       "triggerID" : "0df5649013c1cf9e782fcbb67b8a2e7174f94512",
       "triggerType" : "PUSH"
     }, {
       "hash" : "d52e0128f409918644255abbe4376dc4a780bfc8",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=986",
       "triggerID" : "d52e0128f409918644255abbe4376dc4a780bfc8",
       "triggerType" : "PUSH"
     }, {
       "hash" : "9e668f65c6d4f73c76573236f8fa507dfb80edac",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "9e668f65c6d4f73c76573236f8fa507dfb80edac",
       "triggerType" : "PUSH"
     }, {
       "hash" : "f145131d524bf190986e5de600a326e32a27beca",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=987",
       "triggerID" : "f145131d524bf190986e5de600a326e32a27beca",
       "triggerType" : "PUSH"
     }, {
       "hash" : "0b008f807b42aa6fb2686922bb88d63e84a1f299",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=993",
       "triggerID" : "0b008f807b42aa6fb2686922bb88d63e84a1f299",
       "triggerType" : "PUSH"
     }, {
       "hash" : "65c01ad9f2b33d36d4890355feef91844438e732",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=995",
       "triggerID" : "65c01ad9f2b33d36d4890355feef91844438e732",
       "triggerType" : "PUSH"
     }, {
       "hash" : "955493fdc93842e8c99529cec35e39cecdbe9638",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=996",
       "triggerID" : "955493fdc93842e8c99529cec35e39cecdbe9638",
       "triggerType" : "PUSH"
     }, {
       "hash" : "533f00e5546da0026bbe51e1c030ddd1c19fdd08",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "533f00e5546da0026bbe51e1c030ddd1c19fdd08",
       "triggerType" : "PUSH"
     }, {
       "hash" : "05777be67e37f3a6a8eaa59e54d9e17e5fc6ba9b",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1016",
       "triggerID" : "05777be67e37f3a6a8eaa59e54d9e17e5fc6ba9b",
       "triggerType" : "PUSH"
     }, {
       "hash" : "a2ee90c549332a7f17a7d9b2799849b77c08a8e6",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "a2ee90c549332a7f17a7d9b2799849b77c08a8e6",
       "triggerType" : "PUSH"
     }, {
       "hash" : "05ce568b0b6ba14da9ca40d26b39ba478837d747",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1017",
       "triggerID" : "05ce568b0b6ba14da9ca40d26b39ba478837d747",
       "triggerType" : "PUSH"
     }, {
       "hash" : "840e3b33d39b1f86367bb6b0e65eadeb33de7765",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1025",
       "triggerID" : "840e3b33d39b1f86367bb6b0e65eadeb33de7765",
       "triggerType" : "PUSH"
     }, {
       "hash" : "8d2e2dd83f51d03e890d424a51d3e0e84e904a43",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1031",
       "triggerID" : "8d2e2dd83f51d03e890d424a51d3e0e84e904a43",
       "triggerType" : "PUSH"
     }, {
       "hash" : "688381914087ff11acf2f8c9372dbf39afa0bd40",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1035",
       "triggerID" : "688381914087ff11acf2f8c9372dbf39afa0bd40",
       "triggerType" : "PUSH"
     }, {
       "hash" : "fc2c138901e82ed5f3fbf363c2a0ce8140ade050",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "fc2c138901e82ed5f3fbf363c2a0ce8140ade050",
       "triggerType" : "PUSH"
     }, {
       "hash" : "6ed9e1cb1e5a191752d6d39c7a9d5d41840f0cdf",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1039",
       "triggerID" : "6ed9e1cb1e5a191752d6d39c7a9d5d41840f0cdf",
       "triggerType" : "PUSH"
     }, {
       "hash" : "a510a7de1458bb996df5bbb341e8d010e4c91381",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "a510a7de1458bb996df5bbb341e8d010e4c91381",
       "triggerType" : "PUSH"
     }, {
       "hash" : "49677397cd117228742205944606f7a789090376",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1042",
       "triggerID" : "49677397cd117228742205944606f7a789090376",
       "triggerType" : "PUSH"
     }, {
       "hash" : "1630db0a37422aed942ff280560a9d6329df5015",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1044",
       "triggerID" : "1630db0a37422aed942ff280560a9d6329df5015",
       "triggerType" : "PUSH"
     }, {
       "hash" : "f11c76301d4ab524db0af57aa831be3a898ea0fb",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1048",
       "triggerID" : "f11c76301d4ab524db0af57aa831be3a898ea0fb",
       "triggerType" : "PUSH"
     }, {
       "hash" : "d1a0dc4c2017853a7ba0c3302f886043a63f29da",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1050",
       "triggerID" : "d1a0dc4c2017853a7ba0c3302f886043a63f29da",
       "triggerType" : "PUSH"
     }, {
       "hash" : "6a4c15ec39ded1f90b113b272554217698a98e1c",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1087",
       "triggerID" : "6a4c15ec39ded1f90b113b272554217698a98e1c",
       "triggerType" : "PUSH"
     }, {
       "hash" : "288e730d645a5e58e50a87ef5f44ba11a8fb0533",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1092",
       "triggerID" : "288e730d645a5e58e50a87ef5f44ba11a8fb0533",
       "triggerType" : "PUSH"
     }, {
       "hash" : "70a5c21321a0da3a0be8d85e206c24befa7b89f1",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1095",
       "triggerID" : "70a5c21321a0da3a0be8d85e206c24befa7b89f1",
       "triggerType" : "PUSH"
     }, {
       "hash" : "f072a986114949c5e426ce64a5f5568fc718e25d",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "f072a986114949c5e426ce64a5f5568fc718e25d",
       "triggerType" : "PUSH"
     }, {
       "hash" : "45b27475b1e8806ec8c566d6f151f4bc79c8549d",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1096",
       "triggerID" : "45b27475b1e8806ec8c566d6f151f4bc79c8549d",
       "triggerType" : "PUSH"
     }, {
       "hash" : "3c62ef22f6cde28110e5eb2fca8760a008d77723",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "3c62ef22f6cde28110e5eb2fca8760a008d77723",
       "triggerType" : "PUSH"
     }, {
       "hash" : "dc785fa04e542b2ccc45494e0292f5153c7ecea8",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1097",
       "triggerID" : "dc785fa04e542b2ccc45494e0292f5153c7ecea8",
       "triggerType" : "PUSH"
     }, {
       "hash" : "585ddb8049b9b9ccb057027e265b08a1b586df8c",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "585ddb8049b9b9ccb057027e265b08a1b586df8c",
       "triggerType" : "PUSH"
     }, {
       "hash" : "b62ff8675beed2c4487c788b9ca1a0ad4271b913",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "b62ff8675beed2c4487c788b9ca1a0ad4271b913",
       "triggerType" : "PUSH"
     }, {
       "hash" : "2d46048a7402907f348cc96da2839ba1b3e4932a",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1099",
       "triggerID" : "2d46048a7402907f348cc96da2839ba1b3e4932a",
       "triggerType" : "PUSH"
     }, {
       "hash" : "004fb6ebf23f796a4b77650e0530f0f1145e8b91",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1101",
       "triggerID" : "004fb6ebf23f796a4b77650e0530f0f1145e8b91",
       "triggerType" : "PUSH"
     }, {
       "hash" : "af0c33333b5e9ae6d59f20a6db1d7c016ea015d3",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1165",
       "triggerID" : "af0c33333b5e9ae6d59f20a6db1d7c016ea015d3",
       "triggerType" : "PUSH"
     }, {
       "hash" : "da8111118d71306500229619d3b7996fcd2526e8",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "da8111118d71306500229619d3b7996fcd2526e8",
       "triggerType" : "PUSH"
     }, {
       "hash" : "83257e2b9bd3c426cf90e11003a5f244a750d7fa",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1167",
       "triggerID" : "83257e2b9bd3c426cf90e11003a5f244a750d7fa",
       "triggerType" : "PUSH"
     }, {
       "hash" : "e0de408f1b8153fdaaea4c87b27a7b69213b75b5",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1177",
       "triggerID" : "e0de408f1b8153fdaaea4c87b27a7b69213b75b5",
       "triggerType" : "PUSH"
     }, {
       "hash" : "3f7951f7ad7931f5dc0b977d7c33bc5429623535",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1180",
       "triggerID" : "3f7951f7ad7931f5dc0b977d7c33bc5429623535",
       "triggerType" : "PUSH"
     }, {
       "hash" : "9cd6f06a753b63630e8245d51338d2bf9ab86b84",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "9cd6f06a753b63630e8245d51338d2bf9ab86b84",
       "triggerType" : "PUSH"
     }, {
       "hash" : "7380b902731a92e0645c02397e14591a05798620",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "7380b902731a92e0645c02397e14591a05798620",
       "triggerType" : "PUSH"
     }, {
       "hash" : "0bd5cca01a541845081d5430d48b021d4bdf0969",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "0bd5cca01a541845081d5430d48b021d4bdf0969",
       "triggerType" : "PUSH"
     }, {
       "hash" : "8f4e175ef554f015e34e0968b656b5de714ab430",
       "status" : "SUCCESS",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1182",
       "triggerID" : "8f4e175ef554f015e34e0968b656b5de714ab430",
       "triggerType" : "PUSH"
     }, {
       "hash" : "2023f3c882d0c2f5b96e28d2f21812dc0c64aaee",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "2023f3c882d0c2f5b96e28d2f21812dc0c64aaee",
       "triggerType" : "PUSH"
     }, {
       "hash" : "8394acf060d5c3953dbd67cf930b9b1d3b92d8b5",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "8394acf060d5c3953dbd67cf930b9b1d3b92d8b5",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 9e668f65c6d4f73c76573236f8fa507dfb80edac UNKNOWN
   * 533f00e5546da0026bbe51e1c030ddd1c19fdd08 UNKNOWN
   * a2ee90c549332a7f17a7d9b2799849b77c08a8e6 UNKNOWN
   * fc2c138901e82ed5f3fbf363c2a0ce8140ade050 UNKNOWN
   * a510a7de1458bb996df5bbb341e8d010e4c91381 UNKNOWN
   * f072a986114949c5e426ce64a5f5568fc718e25d UNKNOWN
   * 3c62ef22f6cde28110e5eb2fca8760a008d77723 UNKNOWN
   * 585ddb8049b9b9ccb057027e265b08a1b586df8c UNKNOWN
   * b62ff8675beed2c4487c788b9ca1a0ad4271b913 UNKNOWN
   * da8111118d71306500229619d3b7996fcd2526e8 UNKNOWN
   * 9cd6f06a753b63630e8245d51338d2bf9ab86b84 UNKNOWN
   * 7380b902731a92e0645c02397e14591a05798620 UNKNOWN
   * 0bd5cca01a541845081d5430d48b021d4bdf0969 UNKNOWN
   * 8f4e175ef554f015e34e0968b656b5de714ab430 Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1182) 
   * 2023f3c882d0c2f5b96e28d2f21812dc0c64aaee UNKNOWN
   * 8394acf060d5c3953dbd67cf930b9b1d3b92d8b5 UNKNOWN
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] [HUDI-8141] Incremental Query with Completion Time [hudi]

Posted by "yihua (via GitHub)" <gi...@apache.org>.

yihua commented on code in PR #11947:
URL: https://github.com/apache/hudi/pull/11947#discussion_r1804233787


##########
hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/hudi/DataSourceOptions.scala:
##########
@@ -108,7 +108,7 @@ object DataSourceReadOptions {
     .key("hoodie.datasource.read.begin.instanttime")
     .noDefaultValue()
     .withDocumentation("Required when `" + QUERY_TYPE.key() + "` is set to `" + QUERY_TYPE_INCREMENTAL_OPT_VAL + "`. Represents the instant time to start incrementally pulling data from. The instanttime here need not necessarily "
-      + "correspond to an instant on the timeline. New data written with an instant_time > BEGIN_INSTANTTIME are fetched out. "
+      + "correspond to an instant on the timeline. New data written with an instant_time >= BEGIN_INSTANTTIME are fetched out. "

Review Comment:
   Fixed.  I also renamed the variable and revised the docs as a whole.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] [HUDI-8141] Incremental Query with Completion Time [hudi]

Posted by "yihua (via GitHub)" <gi...@apache.org>.

yihua commented on code in PR #11947:
URL: https://github.com/apache/hudi/pull/11947#discussion_r1805926619


##########
hudi-utilities/src/main/java/org/apache/hudi/utilities/sources/SnapshotLoadQuerySplitter.java:
##########
@@ -118,6 +124,32 @@ public QueryInfo getNextCheckpoint(Dataset<Row> df, QueryInfo queryInfo, Option<
         .orElse(queryInfo);
   }
 
+  public Option<CheckpointWithPredicates> getNextCheckpoint(Dataset<Row> df, QueryContext queryContext,
+                                                            Option<SourceProfileSupplier> sourceProfileSupplier) {
+    // the start instant would be included into the final query result. So we need to get
+    // a strictly lower timestamp to have query splitter include the start instant
+    Option<CheckpointWithPredicates> nextCheckpointWithPredicates =
+        getNextCheckpointWithPredicates(df, instantTimeMinusMillis(queryContext.getBeginInstant().get(), 1));
+    if (nextCheckpointWithPredicates.isPresent()) {
+      // getNextCheckpointWithPredicates is based on instant times,
+      // so we need to translate the instant time to the completion time
+      String endInstantTime = nextCheckpointWithPredicates.get().getEndInstant();
+      Option<String> endCompletionTime = Option.fromJavaOptional(queryContext.getInstants().stream()
+          .filter(instant -> endInstantTime.equals(instant.getTimestamp()))

Review Comment:
   This logic is no longer needed and removed.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] [HUDI-8141] Incremental Query with Completion Time [hudi]

Posted by "danny0405 (via GitHub)" <gi...@apache.org>.

danny0405 commented on code in PR #11947:
URL: https://github.com/apache/hudi/pull/11947#discussion_r1806010130


##########
hudi-utilities/src/main/java/org/apache/hudi/utilities/sources/helpers/IncrSourceHelper.java:
##########
@@ -174,6 +168,63 @@ public static QueryInfo generateQueryInfo(JavaSparkContext jssc, String srcBaseP
     }
   }
 
+  public static IncrementalQueryAnalyzer getIncrementalQueryAnalyzer(
+      JavaSparkContext jssc,
+      String srcPath,
+      Option<String> lastCkptStr,
+      MissingCheckpointStrategy missingCheckpointStrategy,
+      int numInstantsFromConfig,
+      Option<SourceProfile<Integer>> latestSourceProfile) {
+    HoodieTableMetaClient metaClient = HoodieTableMetaClient.builder()
+        .setConf(HadoopFSUtils.getStorageConfWithCopy(jssc.hadoopConfiguration()))
+        .setBasePath(srcPath)
+        .setLoadActiveTimelineOnLoad(true)
+        .build();
+
+    String beginCompletionTime;
+    RangeType rangeType;
+
+    if (lastCkptStr.isPresent() && !lastCkptStr.get().isEmpty()) {
+      beginCompletionTime = lastCkptStr.get();
+      rangeType = RangeType.OPEN_CLOSED;
+    } else if (missingCheckpointStrategy != null) {
+      rangeType = RangeType.CLOSED_CLOSED;
+      switch (missingCheckpointStrategy) {
+        case READ_UPTO_LATEST_COMMIT:
+          beginCompletionTime = DEFAULT_BEGIN_TIMESTAMP;
+          // disrespect numInstantsFromConfig when reading up to latest
+          numInstantsFromConfig = -1;
+          break;
+        case READ_LATEST:
+          // rely on IncrementalQueryAnalyzer to use the latest completed instant
+          beginCompletionTime = null;
+          break;
+        default:
+          throw new IllegalArgumentException("Unknown missing checkpoint strategy: " + missingCheckpointStrategy);
+      }
+    } else {
+      throw new IllegalArgumentException("Missing begin instant for incremental pull. For reading from latest "

Review Comment:
   Missing start completion time for ...



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] [HUDI-8141] WIP:Always use completion time to find begin Instant for incremental source [hudi]

Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.

hudi-bot commented on PR #11947:
URL: https://github.com/apache/hudi/pull/11947#issuecomment-2402964463

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "090f256a315cbbf0c6a54d93cc2739dd693360e6",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=484",
       "triggerID" : "090f256a315cbbf0c6a54d93cc2739dd693360e6",
       "triggerType" : "PUSH"
     }, {
       "hash" : "0df5649013c1cf9e782fcbb67b8a2e7174f94512",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=985",
       "triggerID" : "0df5649013c1cf9e782fcbb67b8a2e7174f94512",
       "triggerType" : "PUSH"
     }, {
       "hash" : "d52e0128f409918644255abbe4376dc4a780bfc8",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=986",
       "triggerID" : "d52e0128f409918644255abbe4376dc4a780bfc8",
       "triggerType" : "PUSH"
     }, {
       "hash" : "9e668f65c6d4f73c76573236f8fa507dfb80edac",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "9e668f65c6d4f73c76573236f8fa507dfb80edac",
       "triggerType" : "PUSH"
     }, {
       "hash" : "f145131d524bf190986e5de600a326e32a27beca",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=987",
       "triggerID" : "f145131d524bf190986e5de600a326e32a27beca",
       "triggerType" : "PUSH"
     }, {
       "hash" : "0b008f807b42aa6fb2686922bb88d63e84a1f299",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=993",
       "triggerID" : "0b008f807b42aa6fb2686922bb88d63e84a1f299",
       "triggerType" : "PUSH"
     }, {
       "hash" : "65c01ad9f2b33d36d4890355feef91844438e732",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=995",
       "triggerID" : "65c01ad9f2b33d36d4890355feef91844438e732",
       "triggerType" : "PUSH"
     }, {
       "hash" : "955493fdc93842e8c99529cec35e39cecdbe9638",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=996",
       "triggerID" : "955493fdc93842e8c99529cec35e39cecdbe9638",
       "triggerType" : "PUSH"
     }, {
       "hash" : "533f00e5546da0026bbe51e1c030ddd1c19fdd08",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "533f00e5546da0026bbe51e1c030ddd1c19fdd08",
       "triggerType" : "PUSH"
     }, {
       "hash" : "05777be67e37f3a6a8eaa59e54d9e17e5fc6ba9b",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1016",
       "triggerID" : "05777be67e37f3a6a8eaa59e54d9e17e5fc6ba9b",
       "triggerType" : "PUSH"
     }, {
       "hash" : "a2ee90c549332a7f17a7d9b2799849b77c08a8e6",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "a2ee90c549332a7f17a7d9b2799849b77c08a8e6",
       "triggerType" : "PUSH"
     }, {
       "hash" : "05ce568b0b6ba14da9ca40d26b39ba478837d747",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1017",
       "triggerID" : "05ce568b0b6ba14da9ca40d26b39ba478837d747",
       "triggerType" : "PUSH"
     }, {
       "hash" : "840e3b33d39b1f86367bb6b0e65eadeb33de7765",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1025",
       "triggerID" : "840e3b33d39b1f86367bb6b0e65eadeb33de7765",
       "triggerType" : "PUSH"
     }, {
       "hash" : "8d2e2dd83f51d03e890d424a51d3e0e84e904a43",
       "status" : "FAILURE",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1031",
       "triggerID" : "8d2e2dd83f51d03e890d424a51d3e0e84e904a43",
       "triggerType" : "PUSH"
     }, {
       "hash" : "688381914087ff11acf2f8c9372dbf39afa0bd40",
       "status" : "PENDING",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1035",
       "triggerID" : "688381914087ff11acf2f8c9372dbf39afa0bd40",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 9e668f65c6d4f73c76573236f8fa507dfb80edac UNKNOWN
   * 533f00e5546da0026bbe51e1c030ddd1c19fdd08 UNKNOWN
   * a2ee90c549332a7f17a7d9b2799849b77c08a8e6 UNKNOWN
   * 8d2e2dd83f51d03e890d424a51d3e0e84e904a43 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1031) 
   * 688381914087ff11acf2f8c9372dbf39afa0bd40 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1035) 
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] [HUDI-8141] WIP:Always use completion time to find begin Instant for incremental source [hudi]

Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.

hudi-bot commented on PR #11947:
URL: https://github.com/apache/hudi/pull/11947#issuecomment-2401490606

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "090f256a315cbbf0c6a54d93cc2739dd693360e6",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=484",
       "triggerID" : "090f256a315cbbf0c6a54d93cc2739dd693360e6",
       "triggerType" : "PUSH"
     }, {
       "hash" : "0df5649013c1cf9e782fcbb67b8a2e7174f94512",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=985",
       "triggerID" : "0df5649013c1cf9e782fcbb67b8a2e7174f94512",
       "triggerType" : "PUSH"
     }, {
       "hash" : "d52e0128f409918644255abbe4376dc4a780bfc8",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=986",
       "triggerID" : "d52e0128f409918644255abbe4376dc4a780bfc8",
       "triggerType" : "PUSH"
     }, {
       "hash" : "9e668f65c6d4f73c76573236f8fa507dfb80edac",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "9e668f65c6d4f73c76573236f8fa507dfb80edac",
       "triggerType" : "PUSH"
     }, {
       "hash" : "f145131d524bf190986e5de600a326e32a27beca",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=987",
       "triggerID" : "f145131d524bf190986e5de600a326e32a27beca",
       "triggerType" : "PUSH"
     }, {
       "hash" : "0b008f807b42aa6fb2686922bb88d63e84a1f299",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=993",
       "triggerID" : "0b008f807b42aa6fb2686922bb88d63e84a1f299",
       "triggerType" : "PUSH"
     }, {
       "hash" : "65c01ad9f2b33d36d4890355feef91844438e732",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=995",
       "triggerID" : "65c01ad9f2b33d36d4890355feef91844438e732",
       "triggerType" : "PUSH"
     }, {
       "hash" : "955493fdc93842e8c99529cec35e39cecdbe9638",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=996",
       "triggerID" : "955493fdc93842e8c99529cec35e39cecdbe9638",
       "triggerType" : "PUSH"
     }, {
       "hash" : "533f00e5546da0026bbe51e1c030ddd1c19fdd08",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "533f00e5546da0026bbe51e1c030ddd1c19fdd08",
       "triggerType" : "PUSH"
     }, {
       "hash" : "05777be67e37f3a6a8eaa59e54d9e17e5fc6ba9b",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1016",
       "triggerID" : "05777be67e37f3a6a8eaa59e54d9e17e5fc6ba9b",
       "triggerType" : "PUSH"
     }, {
       "hash" : "a2ee90c549332a7f17a7d9b2799849b77c08a8e6",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "a2ee90c549332a7f17a7d9b2799849b77c08a8e6",
       "triggerType" : "PUSH"
     }, {
       "hash" : "05ce568b0b6ba14da9ca40d26b39ba478837d747",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1017",
       "triggerID" : "05ce568b0b6ba14da9ca40d26b39ba478837d747",
       "triggerType" : "PUSH"
     }, {
       "hash" : "840e3b33d39b1f86367bb6b0e65eadeb33de7765",
       "status" : "FAILURE",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1025",
       "triggerID" : "840e3b33d39b1f86367bb6b0e65eadeb33de7765",
       "triggerType" : "PUSH"
     }, {
       "hash" : "8d2e2dd83f51d03e890d424a51d3e0e84e904a43",
       "status" : "PENDING",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1031",
       "triggerID" : "8d2e2dd83f51d03e890d424a51d3e0e84e904a43",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 9e668f65c6d4f73c76573236f8fa507dfb80edac UNKNOWN
   * 533f00e5546da0026bbe51e1c030ddd1c19fdd08 UNKNOWN
   * a2ee90c549332a7f17a7d9b2799849b77c08a8e6 UNKNOWN
   * 840e3b33d39b1f86367bb6b0e65eadeb33de7765 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1025) 
   * 8d2e2dd83f51d03e890d424a51d3e0e84e904a43 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1031) 
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] [HUDI-8141] WIP:Always use completion time to find begin Instant for incremental source [hudi]

Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.

hudi-bot commented on PR #11947:
URL: https://github.com/apache/hudi/pull/11947#issuecomment-2403785893

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "090f256a315cbbf0c6a54d93cc2739dd693360e6",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=484",
       "triggerID" : "090f256a315cbbf0c6a54d93cc2739dd693360e6",
       "triggerType" : "PUSH"
     }, {
       "hash" : "0df5649013c1cf9e782fcbb67b8a2e7174f94512",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=985",
       "triggerID" : "0df5649013c1cf9e782fcbb67b8a2e7174f94512",
       "triggerType" : "PUSH"
     }, {
       "hash" : "d52e0128f409918644255abbe4376dc4a780bfc8",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=986",
       "triggerID" : "d52e0128f409918644255abbe4376dc4a780bfc8",
       "triggerType" : "PUSH"
     }, {
       "hash" : "9e668f65c6d4f73c76573236f8fa507dfb80edac",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "9e668f65c6d4f73c76573236f8fa507dfb80edac",
       "triggerType" : "PUSH"
     }, {
       "hash" : "f145131d524bf190986e5de600a326e32a27beca",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=987",
       "triggerID" : "f145131d524bf190986e5de600a326e32a27beca",
       "triggerType" : "PUSH"
     }, {
       "hash" : "0b008f807b42aa6fb2686922bb88d63e84a1f299",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=993",
       "triggerID" : "0b008f807b42aa6fb2686922bb88d63e84a1f299",
       "triggerType" : "PUSH"
     }, {
       "hash" : "65c01ad9f2b33d36d4890355feef91844438e732",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=995",
       "triggerID" : "65c01ad9f2b33d36d4890355feef91844438e732",
       "triggerType" : "PUSH"
     }, {
       "hash" : "955493fdc93842e8c99529cec35e39cecdbe9638",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=996",
       "triggerID" : "955493fdc93842e8c99529cec35e39cecdbe9638",
       "triggerType" : "PUSH"
     }, {
       "hash" : "533f00e5546da0026bbe51e1c030ddd1c19fdd08",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "533f00e5546da0026bbe51e1c030ddd1c19fdd08",
       "triggerType" : "PUSH"
     }, {
       "hash" : "05777be67e37f3a6a8eaa59e54d9e17e5fc6ba9b",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1016",
       "triggerID" : "05777be67e37f3a6a8eaa59e54d9e17e5fc6ba9b",
       "triggerType" : "PUSH"
     }, {
       "hash" : "a2ee90c549332a7f17a7d9b2799849b77c08a8e6",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "a2ee90c549332a7f17a7d9b2799849b77c08a8e6",
       "triggerType" : "PUSH"
     }, {
       "hash" : "05ce568b0b6ba14da9ca40d26b39ba478837d747",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1017",
       "triggerID" : "05ce568b0b6ba14da9ca40d26b39ba478837d747",
       "triggerType" : "PUSH"
     }, {
       "hash" : "840e3b33d39b1f86367bb6b0e65eadeb33de7765",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1025",
       "triggerID" : "840e3b33d39b1f86367bb6b0e65eadeb33de7765",
       "triggerType" : "PUSH"
     }, {
       "hash" : "8d2e2dd83f51d03e890d424a51d3e0e84e904a43",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1031",
       "triggerID" : "8d2e2dd83f51d03e890d424a51d3e0e84e904a43",
       "triggerType" : "PUSH"
     }, {
       "hash" : "688381914087ff11acf2f8c9372dbf39afa0bd40",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1035",
       "triggerID" : "688381914087ff11acf2f8c9372dbf39afa0bd40",
       "triggerType" : "PUSH"
     }, {
       "hash" : "fc2c138901e82ed5f3fbf363c2a0ce8140ade050",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "fc2c138901e82ed5f3fbf363c2a0ce8140ade050",
       "triggerType" : "PUSH"
     }, {
       "hash" : "6ed9e1cb1e5a191752d6d39c7a9d5d41840f0cdf",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1039",
       "triggerID" : "6ed9e1cb1e5a191752d6d39c7a9d5d41840f0cdf",
       "triggerType" : "PUSH"
     }, {
       "hash" : "a510a7de1458bb996df5bbb341e8d010e4c91381",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "a510a7de1458bb996df5bbb341e8d010e4c91381",
       "triggerType" : "PUSH"
     }, {
       "hash" : "49677397cd117228742205944606f7a789090376",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1042",
       "triggerID" : "49677397cd117228742205944606f7a789090376",
       "triggerType" : "PUSH"
     }, {
       "hash" : "1630db0a37422aed942ff280560a9d6329df5015",
       "status" : "CANCELED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1044",
       "triggerID" : "1630db0a37422aed942ff280560a9d6329df5015",
       "triggerType" : "PUSH"
     }, {
       "hash" : "f11c76301d4ab524db0af57aa831be3a898ea0fb",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "f11c76301d4ab524db0af57aa831be3a898ea0fb",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 9e668f65c6d4f73c76573236f8fa507dfb80edac UNKNOWN
   * 533f00e5546da0026bbe51e1c030ddd1c19fdd08 UNKNOWN
   * a2ee90c549332a7f17a7d9b2799849b77c08a8e6 UNKNOWN
   * fc2c138901e82ed5f3fbf363c2a0ce8140ade050 UNKNOWN
   * a510a7de1458bb996df5bbb341e8d010e4c91381 UNKNOWN
   * 1630db0a37422aed942ff280560a9d6329df5015 Azure: [CANCELED](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1044) 
   * f11c76301d4ab524db0af57aa831be3a898ea0fb UNKNOWN
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] [HUDI-8141] Incremental Query with Completion Time [hudi]

Posted by "danny0405 (via GitHub)" <gi...@apache.org>.

danny0405 commented on code in PR #11947:
URL: https://github.com/apache/hudi/pull/11947#discussion_r1798656711


##########
hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/spark/sql/hudi/streaming/HoodieStreamSource.scala:
##########
@@ -114,19 +99,14 @@ class HoodieStreamSource(
 
   private def getLatestOffset: Option[HoodieSourceOffset] = {
     metaClient.reloadActiveTimeline()
-    val filteredTimeline = handleHollowCommitIfNeeded(
-      metaClient.getActiveTimeline.filterCompletedInstants(), metaClient, hollowCommitHandling)
+    val filteredTimeline = metaClient.getActiveTimeline.filterCompletedInstants()
     filteredTimeline match {
       case activeInstants if !activeInstants.empty() =>
-        val timestamp = if (hollowCommitHandling == USE_TRANSITION_TIME) {
-          activeInstants.getInstantsOrderedByCompletionTime
-            .skip(activeInstants.countInstants() - 1)
-            .findFirst()
-            .get()
-            .getCompletionTime
-        } else {
-          activeInstants.lastInstant().get().getTimestamp
-        }
+        val timestamp = activeInstants.getInstantsOrderedByCompletionTime
+          .skip(activeInstants.countInstants() - 1)

Review Comment:
   add `getLatestCompletionTime` method on timeline.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] [HUDI-8141] Incremental Query with Completion Time [hudi]

Posted by "yihua (via GitHub)" <gi...@apache.org>.

yihua commented on code in PR #11947:
URL: https://github.com/apache/hudi/pull/11947#discussion_r1797488190


##########
hudi-utilities/src/main/java/org/apache/hudi/utilities/sources/HoodieIncrSource.java:
##########
@@ -175,33 +181,72 @@ public Pair<Option<Dataset<Row>>, String> fetchNextBatch(Option<String> lastCkpt
     IncrSourceHelper.MissingCheckpointStrategy missingCheckpointStrategy =
         (containsConfigProperty(props, HoodieIncrSourceConfig.MISSING_CHECKPOINT_STRATEGY))
             ? IncrSourceHelper.MissingCheckpointStrategy.valueOf(
-            getStringWithAltKeys(props, HoodieIncrSourceConfig.MISSING_CHECKPOINT_STRATEGY))
+                getStringWithAltKeys(props, HoodieIncrSourceConfig.MISSING_CHECKPOINT_STRATEGY))
             : null;
     if (readLatestOnMissingCkpt) {
       missingCheckpointStrategy = IncrSourceHelper.MissingCheckpointStrategy.READ_LATEST;
     }
 
-    // Use begin Instant if set and non-empty
-    Option<String> beginInstant =
-        lastCkptStr.isPresent() ? lastCkptStr.get().isEmpty() ? Option.empty() : lastCkptStr : Option.empty();
+    HoodieTableMetaClient metaClient = HoodieTableMetaClient.builder()
+        .setConf(HadoopFSUtils.getStorageConfWithCopy(sparkContext.hadoopConfiguration()))
+        .setBasePath(srcPath)
+        .setLoadActiveTimelineOnLoad(true)
+        .build();
 
+    int numInstantsFromConfig = getIntWithAltKeys(props, HoodieIncrSourceConfig.NUM_INSTANTS_PER_FETCH);
+
+    String startTime;
+    if (lastCkptStr.isPresent() && !lastCkptStr.get().isEmpty()) {

Review Comment:
   Shall we wrap the logic here of determining the completion time to resume from and other info into a method and still reuse `QueryInfo` by changing what it contains?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] [HUDI-8141] Incremental Query with Completion Time [hudi]

Posted by "yihua (via GitHub)" <gi...@apache.org>.

yihua commented on code in PR #11947:
URL: https://github.com/apache/hudi/pull/11947#discussion_r1797780051


##########
hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/hudi/IncrementalRelation.scala:
##########
@@ -90,24 +78,25 @@ class IncrementalRelation(val sqlContext: SQLContext,
     throw new HoodieException("Incremental queries are not supported when meta fields are disabled")
   }
 
+  private val queryContext: IncrementalQueryAnalyzer.QueryContext =
+    IncrementalQueryAnalyzer.builder()
+      .metaClient(metaClient)
+      .startTime(optParams(DataSourceReadOptions.BEGIN_INSTANTTIME.key))
+      .endTime(optParams.getOrElse(DataSourceReadOptions.END_INSTANTTIME.key, null))
+      .rangeType(InstantRange.RangeType.OPEN_CLOSED)
+      .limit(optParams.getOrElse(
+        DataSourceReadOptions.INCREMENTAL_LIMIT.key,
+        DataSourceReadOptions.INCREMENTAL_LIMIT.defaultValue).toInt)
+      .build()
+      .analyze()
+
+  private val commitsToReturn = List.concat(

Review Comment:
   Are these commits sorted by completion time?  There are logic in deriving the schema that relies on the ordering to get the latest table schema.



##########
hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/hudi/IncrementalRelation.scala:
##########
@@ -90,24 +78,25 @@ class IncrementalRelation(val sqlContext: SQLContext,
     throw new HoodieException("Incremental queries are not supported when meta fields are disabled")
   }
 
+  private val queryContext: IncrementalQueryAnalyzer.QueryContext =
+    IncrementalQueryAnalyzer.builder()
+      .metaClient(metaClient)
+      .startTime(optParams(DataSourceReadOptions.BEGIN_INSTANTTIME.key))
+      .endTime(optParams.getOrElse(DataSourceReadOptions.END_INSTANTTIME.key, null))

Review Comment:
   So for incremental query in Spark, these begin and end instant time represent completion time.



##########
hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/hudi/IncrementalRelation.scala:
##########
@@ -220,15 +209,12 @@ class IncrementalRelation(val sqlContext: SQLContext,
 
       val sOpts = optParams.filter(p => !p._1.equalsIgnoreCase("path"))
 
-      val startInstantTime = optParams(DataSourceReadOptions.BEGIN_INSTANTTIME.key)
+      val startInstantTime = queryContext.getStartInstant.get()
       val startInstantArchived = commitTimeline.isBeforeTimelineStarts(startInstantTime)
-      val endInstantTime = optParams.getOrElse(DataSourceReadOptions.END_INSTANTTIME.key(), lastInstant.getTimestamp)
+      val endInstantTime = queryContext.getEndInstant.get()

Review Comment:
   `startInstantArchived` and `endInstantArchived` can be avoided as `queryContext` already indicates whether there are instants in the archived timeline to pull.



##########
hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/hudi/IncrementalRelation.scala:
##########
@@ -90,24 +78,25 @@ class IncrementalRelation(val sqlContext: SQLContext,
     throw new HoodieException("Incremental queries are not supported when meta fields are disabled")
   }
 
+  private val queryContext: IncrementalQueryAnalyzer.QueryContext =
+    IncrementalQueryAnalyzer.builder()
+      .metaClient(metaClient)
+      .startTime(optParams(DataSourceReadOptions.BEGIN_INSTANTTIME.key))
+      .endTime(optParams.getOrElse(DataSourceReadOptions.END_INSTANTTIME.key, null))
+      .rangeType(InstantRange.RangeType.OPEN_CLOSED)
+      .limit(optParams.getOrElse(
+        DataSourceReadOptions.INCREMENTAL_LIMIT.key,
+        DataSourceReadOptions.INCREMENTAL_LIMIT.defaultValue).toInt)
+      .build()
+      .analyze()
+
+  private val commitsToReturn = List.concat(
+    queryContext.getArchivedInstants.asScala,
+    queryContext.getActiveInstants.asScala)
+
   private val useEndInstantSchema = optParams.getOrElse(INCREMENTAL_READ_SCHEMA_USE_END_INSTANTTIME.key,
     INCREMENTAL_READ_SCHEMA_USE_END_INSTANTTIME.defaultValue).toBoolean
 
-  private val lastInstant = commitTimeline.lastInstant().get()
-
-  private val commitsTimelineToReturn = {
-    if (hollowCommitHandling == USE_TRANSITION_TIME) {
-      commitTimeline.findInstantsInRangeByCompletionTime(
-        optParams(DataSourceReadOptions.BEGIN_INSTANTTIME.key),
-        optParams.getOrElse(DataSourceReadOptions.END_INSTANTTIME.key(), lastInstant.getCompletionTime))
-    } else {
-      commitTimeline.findInstantsInRange(
-        optParams(DataSourceReadOptions.BEGIN_INSTANTTIME.key),
-        optParams.getOrElse(DataSourceReadOptions.END_INSTANTTIME.key(), lastInstant.getTimestamp))
-    }
-  }
-  private val commitsToReturn = commitsTimelineToReturn.getInstantsAsStream.iterator().asScala.toList

Review Comment:
   Let's file a Jira to track how to differentiate the completion time for 0.x vs 1.0 tables.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] [HUDI-8141] Incremental Query with Completion Time [hudi]

Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.

hudi-bot commented on PR #11947:
URL: https://github.com/apache/hudi/pull/11947#issuecomment-2408323449

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "090f256a315cbbf0c6a54d93cc2739dd693360e6",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=484",
       "triggerID" : "090f256a315cbbf0c6a54d93cc2739dd693360e6",
       "triggerType" : "PUSH"
     }, {
       "hash" : "0df5649013c1cf9e782fcbb67b8a2e7174f94512",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=985",
       "triggerID" : "0df5649013c1cf9e782fcbb67b8a2e7174f94512",
       "triggerType" : "PUSH"
     }, {
       "hash" : "d52e0128f409918644255abbe4376dc4a780bfc8",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=986",
       "triggerID" : "d52e0128f409918644255abbe4376dc4a780bfc8",
       "triggerType" : "PUSH"
     }, {
       "hash" : "9e668f65c6d4f73c76573236f8fa507dfb80edac",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "9e668f65c6d4f73c76573236f8fa507dfb80edac",
       "triggerType" : "PUSH"
     }, {
       "hash" : "f145131d524bf190986e5de600a326e32a27beca",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=987",
       "triggerID" : "f145131d524bf190986e5de600a326e32a27beca",
       "triggerType" : "PUSH"
     }, {
       "hash" : "0b008f807b42aa6fb2686922bb88d63e84a1f299",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=993",
       "triggerID" : "0b008f807b42aa6fb2686922bb88d63e84a1f299",
       "triggerType" : "PUSH"
     }, {
       "hash" : "65c01ad9f2b33d36d4890355feef91844438e732",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=995",
       "triggerID" : "65c01ad9f2b33d36d4890355feef91844438e732",
       "triggerType" : "PUSH"
     }, {
       "hash" : "955493fdc93842e8c99529cec35e39cecdbe9638",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=996",
       "triggerID" : "955493fdc93842e8c99529cec35e39cecdbe9638",
       "triggerType" : "PUSH"
     }, {
       "hash" : "533f00e5546da0026bbe51e1c030ddd1c19fdd08",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "533f00e5546da0026bbe51e1c030ddd1c19fdd08",
       "triggerType" : "PUSH"
     }, {
       "hash" : "05777be67e37f3a6a8eaa59e54d9e17e5fc6ba9b",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1016",
       "triggerID" : "05777be67e37f3a6a8eaa59e54d9e17e5fc6ba9b",
       "triggerType" : "PUSH"
     }, {
       "hash" : "a2ee90c549332a7f17a7d9b2799849b77c08a8e6",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "a2ee90c549332a7f17a7d9b2799849b77c08a8e6",
       "triggerType" : "PUSH"
     }, {
       "hash" : "05ce568b0b6ba14da9ca40d26b39ba478837d747",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1017",
       "triggerID" : "05ce568b0b6ba14da9ca40d26b39ba478837d747",
       "triggerType" : "PUSH"
     }, {
       "hash" : "840e3b33d39b1f86367bb6b0e65eadeb33de7765",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1025",
       "triggerID" : "840e3b33d39b1f86367bb6b0e65eadeb33de7765",
       "triggerType" : "PUSH"
     }, {
       "hash" : "8d2e2dd83f51d03e890d424a51d3e0e84e904a43",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1031",
       "triggerID" : "8d2e2dd83f51d03e890d424a51d3e0e84e904a43",
       "triggerType" : "PUSH"
     }, {
       "hash" : "688381914087ff11acf2f8c9372dbf39afa0bd40",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1035",
       "triggerID" : "688381914087ff11acf2f8c9372dbf39afa0bd40",
       "triggerType" : "PUSH"
     }, {
       "hash" : "fc2c138901e82ed5f3fbf363c2a0ce8140ade050",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "fc2c138901e82ed5f3fbf363c2a0ce8140ade050",
       "triggerType" : "PUSH"
     }, {
       "hash" : "6ed9e1cb1e5a191752d6d39c7a9d5d41840f0cdf",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1039",
       "triggerID" : "6ed9e1cb1e5a191752d6d39c7a9d5d41840f0cdf",
       "triggerType" : "PUSH"
     }, {
       "hash" : "a510a7de1458bb996df5bbb341e8d010e4c91381",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "a510a7de1458bb996df5bbb341e8d010e4c91381",
       "triggerType" : "PUSH"
     }, {
       "hash" : "49677397cd117228742205944606f7a789090376",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1042",
       "triggerID" : "49677397cd117228742205944606f7a789090376",
       "triggerType" : "PUSH"
     }, {
       "hash" : "1630db0a37422aed942ff280560a9d6329df5015",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1044",
       "triggerID" : "1630db0a37422aed942ff280560a9d6329df5015",
       "triggerType" : "PUSH"
     }, {
       "hash" : "f11c76301d4ab524db0af57aa831be3a898ea0fb",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1048",
       "triggerID" : "f11c76301d4ab524db0af57aa831be3a898ea0fb",
       "triggerType" : "PUSH"
     }, {
       "hash" : "d1a0dc4c2017853a7ba0c3302f886043a63f29da",
       "status" : "FAILURE",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1050",
       "triggerID" : "d1a0dc4c2017853a7ba0c3302f886043a63f29da",
       "triggerType" : "PUSH"
     }, {
       "hash" : "6a4c15ec39ded1f90b113b272554217698a98e1c",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "6a4c15ec39ded1f90b113b272554217698a98e1c",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 9e668f65c6d4f73c76573236f8fa507dfb80edac UNKNOWN
   * 533f00e5546da0026bbe51e1c030ddd1c19fdd08 UNKNOWN
   * a2ee90c549332a7f17a7d9b2799849b77c08a8e6 UNKNOWN
   * fc2c138901e82ed5f3fbf363c2a0ce8140ade050 UNKNOWN
   * a510a7de1458bb996df5bbb341e8d010e4c91381 UNKNOWN
   * d1a0dc4c2017853a7ba0c3302f886043a63f29da Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1050) 
   * 6a4c15ec39ded1f90b113b272554217698a98e1c UNKNOWN
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] [HUDI-8141] Incremental Query with Completion Time [hudi]

Posted by "CTTY (via GitHub)" <gi...@apache.org>.

CTTY commented on code in PR #11947:
URL: https://github.com/apache/hudi/pull/11947#discussion_r1797907016


##########
hudi-spark-datasource/hudi-spark/src/test/scala/org/apache/hudi/functional/TestCOWDataSourceStorage.scala:
##########
@@ -185,7 +188,7 @@ class TestCOWDataSourceStorage extends SparkClientFunctionalTestHarness {
     val hoodieIncViewDF1 = spark.read.format("org.apache.hudi")
       .option(DataSourceReadOptions.QUERY_TYPE.key, DataSourceReadOptions.QUERY_TYPE_INCREMENTAL_OPT_VAL)
       .option(DataSourceReadOptions.BEGIN_INSTANTTIME.key, "000")
-      .option(DataSourceReadOptions.END_INSTANTTIME.key, firstCommit)
+      .option(DataSourceReadOptions.END_INSTANTTIME.key, completionTime1)

Review Comment:
   I'll add a new start completion time in this test to avoid confusion: 
   ```
   val beforeCompletionTime1 = HoodieInstantTimeGenerator.instantTimeMinusMillis(completionTime1, 1)
   ```



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] [HUDI-8141] Incremental Query with Completion Time [hudi]

Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.

hudi-bot commented on PR #11947:
URL: https://github.com/apache/hudi/pull/11947#issuecomment-2408785779

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "090f256a315cbbf0c6a54d93cc2739dd693360e6",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=484",
       "triggerID" : "090f256a315cbbf0c6a54d93cc2739dd693360e6",
       "triggerType" : "PUSH"
     }, {
       "hash" : "0df5649013c1cf9e782fcbb67b8a2e7174f94512",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=985",
       "triggerID" : "0df5649013c1cf9e782fcbb67b8a2e7174f94512",
       "triggerType" : "PUSH"
     }, {
       "hash" : "d52e0128f409918644255abbe4376dc4a780bfc8",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=986",
       "triggerID" : "d52e0128f409918644255abbe4376dc4a780bfc8",
       "triggerType" : "PUSH"
     }, {
       "hash" : "9e668f65c6d4f73c76573236f8fa507dfb80edac",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "9e668f65c6d4f73c76573236f8fa507dfb80edac",
       "triggerType" : "PUSH"
     }, {
       "hash" : "f145131d524bf190986e5de600a326e32a27beca",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=987",
       "triggerID" : "f145131d524bf190986e5de600a326e32a27beca",
       "triggerType" : "PUSH"
     }, {
       "hash" : "0b008f807b42aa6fb2686922bb88d63e84a1f299",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=993",
       "triggerID" : "0b008f807b42aa6fb2686922bb88d63e84a1f299",
       "triggerType" : "PUSH"
     }, {
       "hash" : "65c01ad9f2b33d36d4890355feef91844438e732",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=995",
       "triggerID" : "65c01ad9f2b33d36d4890355feef91844438e732",
       "triggerType" : "PUSH"
     }, {
       "hash" : "955493fdc93842e8c99529cec35e39cecdbe9638",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=996",
       "triggerID" : "955493fdc93842e8c99529cec35e39cecdbe9638",
       "triggerType" : "PUSH"
     }, {
       "hash" : "533f00e5546da0026bbe51e1c030ddd1c19fdd08",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "533f00e5546da0026bbe51e1c030ddd1c19fdd08",
       "triggerType" : "PUSH"
     }, {
       "hash" : "05777be67e37f3a6a8eaa59e54d9e17e5fc6ba9b",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1016",
       "triggerID" : "05777be67e37f3a6a8eaa59e54d9e17e5fc6ba9b",
       "triggerType" : "PUSH"
     }, {
       "hash" : "a2ee90c549332a7f17a7d9b2799849b77c08a8e6",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "a2ee90c549332a7f17a7d9b2799849b77c08a8e6",
       "triggerType" : "PUSH"
     }, {
       "hash" : "05ce568b0b6ba14da9ca40d26b39ba478837d747",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1017",
       "triggerID" : "05ce568b0b6ba14da9ca40d26b39ba478837d747",
       "triggerType" : "PUSH"
     }, {
       "hash" : "840e3b33d39b1f86367bb6b0e65eadeb33de7765",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1025",
       "triggerID" : "840e3b33d39b1f86367bb6b0e65eadeb33de7765",
       "triggerType" : "PUSH"
     }, {
       "hash" : "8d2e2dd83f51d03e890d424a51d3e0e84e904a43",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1031",
       "triggerID" : "8d2e2dd83f51d03e890d424a51d3e0e84e904a43",
       "triggerType" : "PUSH"
     }, {
       "hash" : "688381914087ff11acf2f8c9372dbf39afa0bd40",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1035",
       "triggerID" : "688381914087ff11acf2f8c9372dbf39afa0bd40",
       "triggerType" : "PUSH"
     }, {
       "hash" : "fc2c138901e82ed5f3fbf363c2a0ce8140ade050",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "fc2c138901e82ed5f3fbf363c2a0ce8140ade050",
       "triggerType" : "PUSH"
     }, {
       "hash" : "6ed9e1cb1e5a191752d6d39c7a9d5d41840f0cdf",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1039",
       "triggerID" : "6ed9e1cb1e5a191752d6d39c7a9d5d41840f0cdf",
       "triggerType" : "PUSH"
     }, {
       "hash" : "a510a7de1458bb996df5bbb341e8d010e4c91381",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "a510a7de1458bb996df5bbb341e8d010e4c91381",
       "triggerType" : "PUSH"
     }, {
       "hash" : "49677397cd117228742205944606f7a789090376",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1042",
       "triggerID" : "49677397cd117228742205944606f7a789090376",
       "triggerType" : "PUSH"
     }, {
       "hash" : "1630db0a37422aed942ff280560a9d6329df5015",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1044",
       "triggerID" : "1630db0a37422aed942ff280560a9d6329df5015",
       "triggerType" : "PUSH"
     }, {
       "hash" : "f11c76301d4ab524db0af57aa831be3a898ea0fb",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1048",
       "triggerID" : "f11c76301d4ab524db0af57aa831be3a898ea0fb",
       "triggerType" : "PUSH"
     }, {
       "hash" : "d1a0dc4c2017853a7ba0c3302f886043a63f29da",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1050",
       "triggerID" : "d1a0dc4c2017853a7ba0c3302f886043a63f29da",
       "triggerType" : "PUSH"
     }, {
       "hash" : "6a4c15ec39ded1f90b113b272554217698a98e1c",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1087",
       "triggerID" : "6a4c15ec39ded1f90b113b272554217698a98e1c",
       "triggerType" : "PUSH"
     }, {
       "hash" : "288e730d645a5e58e50a87ef5f44ba11a8fb0533",
       "status" : "FAILURE",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1092",
       "triggerID" : "288e730d645a5e58e50a87ef5f44ba11a8fb0533",
       "triggerType" : "PUSH"
     }, {
       "hash" : "70a5c21321a0da3a0be8d85e206c24befa7b89f1",
       "status" : "PENDING",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1095",
       "triggerID" : "70a5c21321a0da3a0be8d85e206c24befa7b89f1",
       "triggerType" : "PUSH"
     }, {
       "hash" : "f072a986114949c5e426ce64a5f5568fc718e25d",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "f072a986114949c5e426ce64a5f5568fc718e25d",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 9e668f65c6d4f73c76573236f8fa507dfb80edac UNKNOWN
   * 533f00e5546da0026bbe51e1c030ddd1c19fdd08 UNKNOWN
   * a2ee90c549332a7f17a7d9b2799849b77c08a8e6 UNKNOWN
   * fc2c138901e82ed5f3fbf363c2a0ce8140ade050 UNKNOWN
   * a510a7de1458bb996df5bbb341e8d010e4c91381 UNKNOWN
   * 288e730d645a5e58e50a87ef5f44ba11a8fb0533 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1092) 
   * 70a5c21321a0da3a0be8d85e206c24befa7b89f1 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1095) 
   * f072a986114949c5e426ce64a5f5568fc718e25d UNKNOWN
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] [HUDI-8141] Incremental Query with Completion Time [hudi]

Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.

hudi-bot commented on PR #11947:
URL: https://github.com/apache/hudi/pull/11947#issuecomment-2408835975

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "090f256a315cbbf0c6a54d93cc2739dd693360e6",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=484",
       "triggerID" : "090f256a315cbbf0c6a54d93cc2739dd693360e6",
       "triggerType" : "PUSH"
     }, {
       "hash" : "0df5649013c1cf9e782fcbb67b8a2e7174f94512",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=985",
       "triggerID" : "0df5649013c1cf9e782fcbb67b8a2e7174f94512",
       "triggerType" : "PUSH"
     }, {
       "hash" : "d52e0128f409918644255abbe4376dc4a780bfc8",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=986",
       "triggerID" : "d52e0128f409918644255abbe4376dc4a780bfc8",
       "triggerType" : "PUSH"
     }, {
       "hash" : "9e668f65c6d4f73c76573236f8fa507dfb80edac",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "9e668f65c6d4f73c76573236f8fa507dfb80edac",
       "triggerType" : "PUSH"
     }, {
       "hash" : "f145131d524bf190986e5de600a326e32a27beca",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=987",
       "triggerID" : "f145131d524bf190986e5de600a326e32a27beca",
       "triggerType" : "PUSH"
     }, {
       "hash" : "0b008f807b42aa6fb2686922bb88d63e84a1f299",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=993",
       "triggerID" : "0b008f807b42aa6fb2686922bb88d63e84a1f299",
       "triggerType" : "PUSH"
     }, {
       "hash" : "65c01ad9f2b33d36d4890355feef91844438e732",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=995",
       "triggerID" : "65c01ad9f2b33d36d4890355feef91844438e732",
       "triggerType" : "PUSH"
     }, {
       "hash" : "955493fdc93842e8c99529cec35e39cecdbe9638",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=996",
       "triggerID" : "955493fdc93842e8c99529cec35e39cecdbe9638",
       "triggerType" : "PUSH"
     }, {
       "hash" : "533f00e5546da0026bbe51e1c030ddd1c19fdd08",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "533f00e5546da0026bbe51e1c030ddd1c19fdd08",
       "triggerType" : "PUSH"
     }, {
       "hash" : "05777be67e37f3a6a8eaa59e54d9e17e5fc6ba9b",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1016",
       "triggerID" : "05777be67e37f3a6a8eaa59e54d9e17e5fc6ba9b",
       "triggerType" : "PUSH"
     }, {
       "hash" : "a2ee90c549332a7f17a7d9b2799849b77c08a8e6",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "a2ee90c549332a7f17a7d9b2799849b77c08a8e6",
       "triggerType" : "PUSH"
     }, {
       "hash" : "05ce568b0b6ba14da9ca40d26b39ba478837d747",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1017",
       "triggerID" : "05ce568b0b6ba14da9ca40d26b39ba478837d747",
       "triggerType" : "PUSH"
     }, {
       "hash" : "840e3b33d39b1f86367bb6b0e65eadeb33de7765",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1025",
       "triggerID" : "840e3b33d39b1f86367bb6b0e65eadeb33de7765",
       "triggerType" : "PUSH"
     }, {
       "hash" : "8d2e2dd83f51d03e890d424a51d3e0e84e904a43",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1031",
       "triggerID" : "8d2e2dd83f51d03e890d424a51d3e0e84e904a43",
       "triggerType" : "PUSH"
     }, {
       "hash" : "688381914087ff11acf2f8c9372dbf39afa0bd40",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1035",
       "triggerID" : "688381914087ff11acf2f8c9372dbf39afa0bd40",
       "triggerType" : "PUSH"
     }, {
       "hash" : "fc2c138901e82ed5f3fbf363c2a0ce8140ade050",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "fc2c138901e82ed5f3fbf363c2a0ce8140ade050",
       "triggerType" : "PUSH"
     }, {
       "hash" : "6ed9e1cb1e5a191752d6d39c7a9d5d41840f0cdf",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1039",
       "triggerID" : "6ed9e1cb1e5a191752d6d39c7a9d5d41840f0cdf",
       "triggerType" : "PUSH"
     }, {
       "hash" : "a510a7de1458bb996df5bbb341e8d010e4c91381",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "a510a7de1458bb996df5bbb341e8d010e4c91381",
       "triggerType" : "PUSH"
     }, {
       "hash" : "49677397cd117228742205944606f7a789090376",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1042",
       "triggerID" : "49677397cd117228742205944606f7a789090376",
       "triggerType" : "PUSH"
     }, {
       "hash" : "1630db0a37422aed942ff280560a9d6329df5015",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1044",
       "triggerID" : "1630db0a37422aed942ff280560a9d6329df5015",
       "triggerType" : "PUSH"
     }, {
       "hash" : "f11c76301d4ab524db0af57aa831be3a898ea0fb",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1048",
       "triggerID" : "f11c76301d4ab524db0af57aa831be3a898ea0fb",
       "triggerType" : "PUSH"
     }, {
       "hash" : "d1a0dc4c2017853a7ba0c3302f886043a63f29da",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1050",
       "triggerID" : "d1a0dc4c2017853a7ba0c3302f886043a63f29da",
       "triggerType" : "PUSH"
     }, {
       "hash" : "6a4c15ec39ded1f90b113b272554217698a98e1c",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1087",
       "triggerID" : "6a4c15ec39ded1f90b113b272554217698a98e1c",
       "triggerType" : "PUSH"
     }, {
       "hash" : "288e730d645a5e58e50a87ef5f44ba11a8fb0533",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1092",
       "triggerID" : "288e730d645a5e58e50a87ef5f44ba11a8fb0533",
       "triggerType" : "PUSH"
     }, {
       "hash" : "70a5c21321a0da3a0be8d85e206c24befa7b89f1",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1095",
       "triggerID" : "70a5c21321a0da3a0be8d85e206c24befa7b89f1",
       "triggerType" : "PUSH"
     }, {
       "hash" : "f072a986114949c5e426ce64a5f5568fc718e25d",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "f072a986114949c5e426ce64a5f5568fc718e25d",
       "triggerType" : "PUSH"
     }, {
       "hash" : "45b27475b1e8806ec8c566d6f151f4bc79c8549d",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1096",
       "triggerID" : "45b27475b1e8806ec8c566d6f151f4bc79c8549d",
       "triggerType" : "PUSH"
     }, {
       "hash" : "3c62ef22f6cde28110e5eb2fca8760a008d77723",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "3c62ef22f6cde28110e5eb2fca8760a008d77723",
       "triggerType" : "PUSH"
     }, {
       "hash" : "dc785fa04e542b2ccc45494e0292f5153c7ecea8",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1097",
       "triggerID" : "dc785fa04e542b2ccc45494e0292f5153c7ecea8",
       "triggerType" : "PUSH"
     }, {
       "hash" : "585ddb8049b9b9ccb057027e265b08a1b586df8c",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "585ddb8049b9b9ccb057027e265b08a1b586df8c",
       "triggerType" : "PUSH"
     }, {
       "hash" : "b62ff8675beed2c4487c788b9ca1a0ad4271b913",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "b62ff8675beed2c4487c788b9ca1a0ad4271b913",
       "triggerType" : "PUSH"
     }, {
       "hash" : "2d46048a7402907f348cc96da2839ba1b3e4932a",
       "status" : "CANCELED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1099",
       "triggerID" : "2d46048a7402907f348cc96da2839ba1b3e4932a",
       "triggerType" : "PUSH"
     }, {
       "hash" : "004fb6ebf23f796a4b77650e0530f0f1145e8b91",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "004fb6ebf23f796a4b77650e0530f0f1145e8b91",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 9e668f65c6d4f73c76573236f8fa507dfb80edac UNKNOWN
   * 533f00e5546da0026bbe51e1c030ddd1c19fdd08 UNKNOWN
   * a2ee90c549332a7f17a7d9b2799849b77c08a8e6 UNKNOWN
   * fc2c138901e82ed5f3fbf363c2a0ce8140ade050 UNKNOWN
   * a510a7de1458bb996df5bbb341e8d010e4c91381 UNKNOWN
   * f072a986114949c5e426ce64a5f5568fc718e25d UNKNOWN
   * 3c62ef22f6cde28110e5eb2fca8760a008d77723 UNKNOWN
   * 585ddb8049b9b9ccb057027e265b08a1b586df8c UNKNOWN
   * b62ff8675beed2c4487c788b9ca1a0ad4271b913 UNKNOWN
   * 2d46048a7402907f348cc96da2839ba1b3e4932a Azure: [CANCELED](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1099) 
   * 004fb6ebf23f796a4b77650e0530f0f1145e8b91 UNKNOWN
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] [HUDI-8141] WIP:Always use completion time to find begin Instant for incremental source [hudi]

Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.

hudi-bot commented on PR #11947:
URL: https://github.com/apache/hudi/pull/11947#issuecomment-2351154203

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "090f256a315cbbf0c6a54d93cc2739dd693360e6",
       "status" : "PENDING",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=484",
       "triggerID" : "090f256a315cbbf0c6a54d93cc2739dd693360e6",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 090f256a315cbbf0c6a54d93cc2739dd693360e6 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=484) 
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] [HUDI-8141] Incremental Query with Completion Time [hudi]

Posted by "yihua (via GitHub)" <gi...@apache.org>.

yihua commented on code in PR #11947:
URL: https://github.com/apache/hudi/pull/11947#discussion_r1805755727


##########
hudi-utilities/src/main/java/org/apache/hudi/utilities/sources/HoodieIncrSource.java:
##########
@@ -212,35 +209,58 @@ public Pair<Option<Dataset<Row>>, String> fetchNextBatch(Option<String> lastCkpt
           .collect(Collectors.toMap(Pair::getLeft, Pair::getRight));
       reader = reader.options(optionsMap);
     }
+
+    boolean shouldFullScan =
+        missingCheckpointStrategy == MissingCheckpointStrategy.READ_UPTO_LATEST_COMMIT
+            && queryContext.getActiveTimeline()
+            .isBeforeTimelineStartsByCompletionTime(analyzer.getBeginCompletionTime().get());
     Dataset<Row> source;
-    // Do Incr pull. Set end instant if available
-    if (queryInfo.isIncremental()) {
-      source = reader
-          .options(readOpts)
-          .option(QUERY_TYPE().key(), QUERY_TYPE_INCREMENTAL_OPT_VAL())
-          .option(BEGIN_INSTANTTIME().key(), queryInfo.getStartInstant())
-          .option(END_INSTANTTIME().key(), queryInfo.getEndInstant())
-          .option(INCREMENTAL_FALLBACK_TO_FULL_TABLE_SCAN().key(),
-              props.getString(INCREMENTAL_FALLBACK_TO_FULL_TABLE_SCAN().key(),
-                  INCREMENTAL_FALLBACK_TO_FULL_TABLE_SCAN().defaultValue()))
-          .option(INCREMENTAL_READ_HANDLE_HOLLOW_COMMIT().key(), handlingMode.name())
-          .load(srcPath);
-    } else {
-      // if checkpoint is missing from source table, and if strategy is set to READ_UPTO_LATEST_COMMIT, we have to issue snapshot query
+    if (instantRange.isEmpty() || shouldFullScan) {
+      // snapshot query
       Dataset<Row> snapshot = reader
           .options(readOpts)
           .option(DataSourceReadOptions.QUERY_TYPE().key(), DataSourceReadOptions.QUERY_TYPE_SNAPSHOT_OPT_VAL())
           .load(srcPath);
+
+      Option<String> predicate = Option.empty();
+      List<String> instantTimeList = queryContext.getInstantTimeList();
       if (snapshotLoadQuerySplitter.isPresent()) {
-        queryInfo = snapshotLoadQuerySplitter.get().getNextCheckpoint(snapshot, queryInfo, sourceProfileSupplier);
+        Option<SnapshotLoadQuerySplitter.CheckpointWithPredicates> newCheckpointAndPredicate =
+            snapshotLoadQuerySplitter.get().getNextCheckpoint(snapshot, queryContext, sourceProfileSupplier);
+        if (newCheckpointAndPredicate.isPresent()) {
+          endCompletionTime = newCheckpointAndPredicate.get().endInstant;
+          predicate = Option.of(newCheckpointAndPredicate.get().predicateFilter);
+          instantTimeList = queryContext.getInstants().stream()
+              .filter(instant -> HoodieTimeline.compareTimestamps(
+                  instant.getCompletionTime(), HoodieTimeline.LESSER_THAN_OR_EQUALS, newCheckpointAndPredicate.get().endInstant))
+              .map(HoodieInstant::getTimestamp)
+              .collect(Collectors.toList());
+        } else {
+          endCompletionTime = queryContext.getEndInstant().orElse(queryContext.getLastInstant());
+        }
       }
+
+      snapshot = predicate.map(snapshot::filter).orElse(snapshot);
       source = snapshot
           // add filtering so that only interested records are returned.
-          .filter(String.format("%s > '%s'", HoodieRecord.COMMIT_TIME_METADATA_FIELD,
-              queryInfo.getStartInstant()))
-          .filter(String.format("%s <= '%s'", HoodieRecord.COMMIT_TIME_METADATA_FIELD,
-              queryInfo.getEndInstant()));
-      source = queryInfo.getPredicateFilter().map(source::filter).orElse(source);
+          .filter(String.format("%s IN ('%s')", HoodieRecord.COMMIT_TIME_METADATA_FIELD,
+              String.join("','", instantTimeList)));
+    } else {
+      // normal incremental query
+      String beginCompletionTime = queryContext.getInstants().stream()
+          .map(HoodieInstant::getCompletionTime)

Review Comment:
   Fixed.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] [HUDI-8141] Incremental Query with Completion Time [hudi]

Posted by "CTTY (via GitHub)" <gi...@apache.org>.

CTTY commented on code in PR #11947:
URL: https://github.com/apache/hudi/pull/11947#discussion_r1797777809


##########
hudi-common/src/main/java/org/apache/hudi/common/table/read/IncrementalQueryAnalyzer.java:
##########
@@ -471,5 +500,20 @@ public HoodieTimeline getActiveTimeline() {
     public @Nullable HoodieTimeline getArchivedTimeline() {
       return archivedTimeline;
     }
+
+    public Option<String> getPredicateFilter() {
+      return predicateFilter;
+    }
+
+    public QueryContext withUpdatedEndInstant(String endInstant) {
+      return new QueryContext(
+          startInstant.get(),
+          endInstant,
+          instants,
+          archivedInstants,
+          activeInstants,
+          activeTimeline,archivedTimeline,
+          null);
+    }

Review Comment:
   Makes sense. I'll remove this method



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] [HUDI-8141] Incremental Query with Completion Time [hudi]

Posted by "CTTY (via GitHub)" <gi...@apache.org>.

CTTY commented on code in PR #11947:
URL: https://github.com/apache/hudi/pull/11947#discussion_r1797968979


##########
hudi-common/src/test/java/org/apache/hudi/common/testutils/HoodieTestDataGenerator.java:
##########
@@ -681,6 +681,10 @@ public List<HoodieRecord> generateInserts(String instantTime, Integer n) {
     return generateInserts(instantTime, n, false);
   }
 
+  public List<HoodieRecord> generateInserts(Long instantTime, Integer n) {
+    return generateInserts(String.valueOf(instantTime), n, false);
+  }

Review Comment:
   I've removed all new test methods in this class



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] [HUDI-8141] Incremental Query with Completion Time [hudi]

Posted by "CTTY (via GitHub)" <gi...@apache.org>.

CTTY commented on code in PR #11947:
URL: https://github.com/apache/hudi/pull/11947#discussion_r1798107264


##########
hudi-spark-datasource/hudi-spark/src/test/scala/org/apache/hudi/functional/TestStreamingSource.scala:
##########
@@ -205,13 +205,10 @@ class TestStreamingSource extends StreamTest {
       addData(tablePath, Seq(("2", "a1", "11", "001")))
       addData(tablePath, Seq(("3", "a1", "12", "002")))
 
-      val timestamp = if (handlingMode == USE_TRANSITION_TIME) {
+      val timestamp =

Review Comment:
   Filed JIRA: https://issues.apache.org/jira/browse/HUDI-8358



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] [HUDI-8141] Incremental Query with Completion Time [hudi]

Posted by "yihua (via GitHub)" <gi...@apache.org>.

yihua commented on code in PR #11947:
URL: https://github.com/apache/hudi/pull/11947#discussion_r1802125316


##########
hudi-common/src/main/java/org/apache/hudi/common/table/timeline/HoodieDefaultTimeline.java:
##########
@@ -629,16 +649,23 @@ private Set<String> getOrCreatePendingClusteringInstantSet() {
   /**
    * Returns the first non savepoint commit on the timeline.
    */
-  private static Option<HoodieInstant> findFirstNonSavepointCommit(List<HoodieInstant> instants) {
+  private static Option<HoodieInstant> findFirstNonSavepointCommit(
+      List<HoodieInstant> instants,
+      boolean byCompletionTime) {

Review Comment:
   Fixed.  I also improved the logic to be simpler.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] [HUDI-8141] Incremental Query with Completion Time [hudi]

Posted by "danny0405 (via GitHub)" <gi...@apache.org>.

danny0405 commented on code in PR #11947:
URL: https://github.com/apache/hudi/pull/11947#discussion_r1802199649


##########
hudi-common/src/main/java/org/apache/hudi/common/table/timeline/CompletionTimeQueryView.java:
##########
@@ -189,46 +193,78 @@ public Option<String> getCompletionTime(String beginTime) {
    * <p>By default, assumes there is at most 1 day time of duration for an instant to accelerate the queries.
    *
    * @param timeline The timeline.
-   * @param rangeStart   The query range start completion time.
-   * @param rangeEnd     The query range end completion time.
+   * @param rangeStartCompletionTime   The query range start completion time.
+   * @param rangeEndCompletionTime     The query range end completion time.
    * @param rangeType    The range type.
    *
    * @return The sorted instant time list.
    */
   public List<String> getStartTimes(
       HoodieTimeline timeline,
-      Option<String> rangeStart,
-      Option<String> rangeEnd,
+      Option<String> rangeStartCompletionTime,

Review Comment:
   Let's do it in this PR.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] [HUDI-8141] Incremental Query with Completion Time [hudi]

Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.

hudi-bot commented on PR #11947:
URL: https://github.com/apache/hudi/pull/11947#issuecomment-2416114805

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "090f256a315cbbf0c6a54d93cc2739dd693360e6",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=484",
       "triggerID" : "090f256a315cbbf0c6a54d93cc2739dd693360e6",
       "triggerType" : "PUSH"
     }, {
       "hash" : "0df5649013c1cf9e782fcbb67b8a2e7174f94512",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=985",
       "triggerID" : "0df5649013c1cf9e782fcbb67b8a2e7174f94512",
       "triggerType" : "PUSH"
     }, {
       "hash" : "d52e0128f409918644255abbe4376dc4a780bfc8",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=986",
       "triggerID" : "d52e0128f409918644255abbe4376dc4a780bfc8",
       "triggerType" : "PUSH"
     }, {
       "hash" : "9e668f65c6d4f73c76573236f8fa507dfb80edac",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "9e668f65c6d4f73c76573236f8fa507dfb80edac",
       "triggerType" : "PUSH"
     }, {
       "hash" : "f145131d524bf190986e5de600a326e32a27beca",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=987",
       "triggerID" : "f145131d524bf190986e5de600a326e32a27beca",
       "triggerType" : "PUSH"
     }, {
       "hash" : "0b008f807b42aa6fb2686922bb88d63e84a1f299",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=993",
       "triggerID" : "0b008f807b42aa6fb2686922bb88d63e84a1f299",
       "triggerType" : "PUSH"
     }, {
       "hash" : "65c01ad9f2b33d36d4890355feef91844438e732",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=995",
       "triggerID" : "65c01ad9f2b33d36d4890355feef91844438e732",
       "triggerType" : "PUSH"
     }, {
       "hash" : "955493fdc93842e8c99529cec35e39cecdbe9638",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=996",
       "triggerID" : "955493fdc93842e8c99529cec35e39cecdbe9638",
       "triggerType" : "PUSH"
     }, {
       "hash" : "533f00e5546da0026bbe51e1c030ddd1c19fdd08",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "533f00e5546da0026bbe51e1c030ddd1c19fdd08",
       "triggerType" : "PUSH"
     }, {
       "hash" : "05777be67e37f3a6a8eaa59e54d9e17e5fc6ba9b",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1016",
       "triggerID" : "05777be67e37f3a6a8eaa59e54d9e17e5fc6ba9b",
       "triggerType" : "PUSH"
     }, {
       "hash" : "a2ee90c549332a7f17a7d9b2799849b77c08a8e6",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "a2ee90c549332a7f17a7d9b2799849b77c08a8e6",
       "triggerType" : "PUSH"
     }, {
       "hash" : "05ce568b0b6ba14da9ca40d26b39ba478837d747",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1017",
       "triggerID" : "05ce568b0b6ba14da9ca40d26b39ba478837d747",
       "triggerType" : "PUSH"
     }, {
       "hash" : "840e3b33d39b1f86367bb6b0e65eadeb33de7765",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1025",
       "triggerID" : "840e3b33d39b1f86367bb6b0e65eadeb33de7765",
       "triggerType" : "PUSH"
     }, {
       "hash" : "8d2e2dd83f51d03e890d424a51d3e0e84e904a43",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1031",
       "triggerID" : "8d2e2dd83f51d03e890d424a51d3e0e84e904a43",
       "triggerType" : "PUSH"
     }, {
       "hash" : "688381914087ff11acf2f8c9372dbf39afa0bd40",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1035",
       "triggerID" : "688381914087ff11acf2f8c9372dbf39afa0bd40",
       "triggerType" : "PUSH"
     }, {
       "hash" : "fc2c138901e82ed5f3fbf363c2a0ce8140ade050",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "fc2c138901e82ed5f3fbf363c2a0ce8140ade050",
       "triggerType" : "PUSH"
     }, {
       "hash" : "6ed9e1cb1e5a191752d6d39c7a9d5d41840f0cdf",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1039",
       "triggerID" : "6ed9e1cb1e5a191752d6d39c7a9d5d41840f0cdf",
       "triggerType" : "PUSH"
     }, {
       "hash" : "a510a7de1458bb996df5bbb341e8d010e4c91381",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "a510a7de1458bb996df5bbb341e8d010e4c91381",
       "triggerType" : "PUSH"
     }, {
       "hash" : "49677397cd117228742205944606f7a789090376",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1042",
       "triggerID" : "49677397cd117228742205944606f7a789090376",
       "triggerType" : "PUSH"
     }, {
       "hash" : "1630db0a37422aed942ff280560a9d6329df5015",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1044",
       "triggerID" : "1630db0a37422aed942ff280560a9d6329df5015",
       "triggerType" : "PUSH"
     }, {
       "hash" : "f11c76301d4ab524db0af57aa831be3a898ea0fb",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1048",
       "triggerID" : "f11c76301d4ab524db0af57aa831be3a898ea0fb",
       "triggerType" : "PUSH"
     }, {
       "hash" : "d1a0dc4c2017853a7ba0c3302f886043a63f29da",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1050",
       "triggerID" : "d1a0dc4c2017853a7ba0c3302f886043a63f29da",
       "triggerType" : "PUSH"
     }, {
       "hash" : "6a4c15ec39ded1f90b113b272554217698a98e1c",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1087",
       "triggerID" : "6a4c15ec39ded1f90b113b272554217698a98e1c",
       "triggerType" : "PUSH"
     }, {
       "hash" : "288e730d645a5e58e50a87ef5f44ba11a8fb0533",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1092",
       "triggerID" : "288e730d645a5e58e50a87ef5f44ba11a8fb0533",
       "triggerType" : "PUSH"
     }, {
       "hash" : "70a5c21321a0da3a0be8d85e206c24befa7b89f1",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1095",
       "triggerID" : "70a5c21321a0da3a0be8d85e206c24befa7b89f1",
       "triggerType" : "PUSH"
     }, {
       "hash" : "f072a986114949c5e426ce64a5f5568fc718e25d",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "f072a986114949c5e426ce64a5f5568fc718e25d",
       "triggerType" : "PUSH"
     }, {
       "hash" : "45b27475b1e8806ec8c566d6f151f4bc79c8549d",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1096",
       "triggerID" : "45b27475b1e8806ec8c566d6f151f4bc79c8549d",
       "triggerType" : "PUSH"
     }, {
       "hash" : "3c62ef22f6cde28110e5eb2fca8760a008d77723",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "3c62ef22f6cde28110e5eb2fca8760a008d77723",
       "triggerType" : "PUSH"
     }, {
       "hash" : "dc785fa04e542b2ccc45494e0292f5153c7ecea8",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1097",
       "triggerID" : "dc785fa04e542b2ccc45494e0292f5153c7ecea8",
       "triggerType" : "PUSH"
     }, {
       "hash" : "585ddb8049b9b9ccb057027e265b08a1b586df8c",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "585ddb8049b9b9ccb057027e265b08a1b586df8c",
       "triggerType" : "PUSH"
     }, {
       "hash" : "b62ff8675beed2c4487c788b9ca1a0ad4271b913",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "b62ff8675beed2c4487c788b9ca1a0ad4271b913",
       "triggerType" : "PUSH"
     }, {
       "hash" : "2d46048a7402907f348cc96da2839ba1b3e4932a",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1099",
       "triggerID" : "2d46048a7402907f348cc96da2839ba1b3e4932a",
       "triggerType" : "PUSH"
     }, {
       "hash" : "004fb6ebf23f796a4b77650e0530f0f1145e8b91",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1101",
       "triggerID" : "004fb6ebf23f796a4b77650e0530f0f1145e8b91",
       "triggerType" : "PUSH"
     }, {
       "hash" : "af0c33333b5e9ae6d59f20a6db1d7c016ea015d3",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1165",
       "triggerID" : "af0c33333b5e9ae6d59f20a6db1d7c016ea015d3",
       "triggerType" : "PUSH"
     }, {
       "hash" : "da8111118d71306500229619d3b7996fcd2526e8",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "da8111118d71306500229619d3b7996fcd2526e8",
       "triggerType" : "PUSH"
     }, {
       "hash" : "83257e2b9bd3c426cf90e11003a5f244a750d7fa",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1167",
       "triggerID" : "83257e2b9bd3c426cf90e11003a5f244a750d7fa",
       "triggerType" : "PUSH"
     }, {
       "hash" : "e0de408f1b8153fdaaea4c87b27a7b69213b75b5",
       "status" : "CANCELED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1177",
       "triggerID" : "e0de408f1b8153fdaaea4c87b27a7b69213b75b5",
       "triggerType" : "PUSH"
     }, {
       "hash" : "3f7951f7ad7931f5dc0b977d7c33bc5429623535",
       "status" : "PENDING",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1180",
       "triggerID" : "3f7951f7ad7931f5dc0b977d7c33bc5429623535",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 9e668f65c6d4f73c76573236f8fa507dfb80edac UNKNOWN
   * 533f00e5546da0026bbe51e1c030ddd1c19fdd08 UNKNOWN
   * a2ee90c549332a7f17a7d9b2799849b77c08a8e6 UNKNOWN
   * fc2c138901e82ed5f3fbf363c2a0ce8140ade050 UNKNOWN
   * a510a7de1458bb996df5bbb341e8d010e4c91381 UNKNOWN
   * f072a986114949c5e426ce64a5f5568fc718e25d UNKNOWN
   * 3c62ef22f6cde28110e5eb2fca8760a008d77723 UNKNOWN
   * 585ddb8049b9b9ccb057027e265b08a1b586df8c UNKNOWN
   * b62ff8675beed2c4487c788b9ca1a0ad4271b913 UNKNOWN
   * da8111118d71306500229619d3b7996fcd2526e8 UNKNOWN
   * e0de408f1b8153fdaaea4c87b27a7b69213b75b5 Azure: [CANCELED](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1177) 
   * 3f7951f7ad7931f5dc0b977d7c33bc5429623535 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1180) 
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] [HUDI-8141] Incremental Query with Completion Time [hudi]

Posted by "yihua (via GitHub)" <gi...@apache.org>.

yihua commented on code in PR #11947:
URL: https://github.com/apache/hudi/pull/11947#discussion_r1802082655


##########
hudi-spark-datasource/hudi-spark/src/test/scala/org/apache/hudi/functional/TestCOWDataSource.scala:
##########
@@ -65,7 +65,6 @@ import org.junit.jupiter.params.provider.{CsvSource, EnumSource, ValueSource}
 import java.sql.{Date, Timestamp}
 import java.util.concurrent.{CountDownLatch, TimeUnit}
 import java.util.function.Consumer
-

Review Comment:
   Nit: keep import grouping



##########
hudi-spark-datasource/hudi-spark/src/test/scala/org/apache/hudi/functional/TestCOWDataSourceStorage.scala:
##########
@@ -185,7 +188,7 @@ class TestCOWDataSourceStorage extends SparkClientFunctionalTestHarness {
     val hoodieIncViewDF1 = spark.read.format("org.apache.hudi")
       .option(DataSourceReadOptions.QUERY_TYPE.key, DataSourceReadOptions.QUERY_TYPE_INCREMENTAL_OPT_VAL)
       .option(DataSourceReadOptions.BEGIN_INSTANTTIME.key, "000")
-      .option(DataSourceReadOptions.END_INSTANTTIME.key, firstCommit)
+      .option(DataSourceReadOptions.END_INSTANTTIME.key, completionTime1)

Review Comment:
   Sg



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] [HUDI-8141] Incremental Query with Completion Time [hudi]

Posted by "yihua (via GitHub)" <gi...@apache.org>.

yihua commented on code in PR #11947:
URL: https://github.com/apache/hudi/pull/11947#discussion_r1802132570


##########
hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/hudi/IncrementalRelation.scala:
##########
@@ -90,24 +78,24 @@ class IncrementalRelation(val sqlContext: SQLContext,
     throw new HoodieException("Incremental queries are not supported when meta fields are disabled")
   }
 
+  private val queryContext: IncrementalQueryAnalyzer.QueryContext =
+    IncrementalQueryAnalyzer.builder()
+      .metaClient(metaClient)
+      .startTime(optParams(DataSourceReadOptions.BEGIN_INSTANTTIME.key))
+      .endTime(optParams.getOrElse(DataSourceReadOptions.END_INSTANTTIME.key, null))
+      .rangeType(InstantRange.RangeType.OPEN_CLOSED)

Review Comment:
   In Spark the `hoodie.datasource.read.begin.instanttime` is exclusive, being treated as the last checkpoint just like `HoodieStreamer`'s last checkpoint of the source.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] [HUDI-8141] Incremental Query with Completion Time [hudi]

Posted by "danny0405 (via GitHub)" <gi...@apache.org>.

danny0405 commented on code in PR #11947:
URL: https://github.com/apache/hudi/pull/11947#discussion_r1802202562


##########
hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/hudi/IncrementalRelation.scala:
##########
@@ -305,14 +286,17 @@ class IncrementalRelation(val sqlContext: SQLContext,
   }
 
   private def fullTableScanDataFrame(startInstantTime: String, endInstantTime: String): DataFrame = {
+    val commitTimesToReturn = commitsToReturn.map(_.getTimestamp)
     val hudiDF = sqlContext.read
       .format("hudi_v1")
       .schema(usedSchema)
       .load(basePath.toString)
-      .filter(String.format("%s > '%s'", HoodieRecord.COMMIT_TIME_METADATA_FIELD, //Notice the > in place of >= because we are working with optParam instead of first commit > optParam
+      // using closed_closed range here since we are using start instant from QueryContext
+      .filter(String.format("%s >= '%s'", HoodieRecord.COMMIT_TIME_METADATA_FIELD,
         startInstantTime))
       .filter(String.format("%s <= '%s'", HoodieRecord.COMMIT_TIME_METADATA_FIELD,

Review Comment:
   > IN predicate should also enable predicate pushdown
   
   In predicate is indeed not very friendly for predicates pushdown, let's make it clear for this.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] [HUDI-8141] Incremental Query with Completion Time [hudi]

Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.

hudi-bot commented on PR #11947:
URL: https://github.com/apache/hudi/pull/11947#issuecomment-2415549692

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "090f256a315cbbf0c6a54d93cc2739dd693360e6",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=484",
       "triggerID" : "090f256a315cbbf0c6a54d93cc2739dd693360e6",
       "triggerType" : "PUSH"
     }, {
       "hash" : "0df5649013c1cf9e782fcbb67b8a2e7174f94512",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=985",
       "triggerID" : "0df5649013c1cf9e782fcbb67b8a2e7174f94512",
       "triggerType" : "PUSH"
     }, {
       "hash" : "d52e0128f409918644255abbe4376dc4a780bfc8",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=986",
       "triggerID" : "d52e0128f409918644255abbe4376dc4a780bfc8",
       "triggerType" : "PUSH"
     }, {
       "hash" : "9e668f65c6d4f73c76573236f8fa507dfb80edac",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "9e668f65c6d4f73c76573236f8fa507dfb80edac",
       "triggerType" : "PUSH"
     }, {
       "hash" : "f145131d524bf190986e5de600a326e32a27beca",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=987",
       "triggerID" : "f145131d524bf190986e5de600a326e32a27beca",
       "triggerType" : "PUSH"
     }, {
       "hash" : "0b008f807b42aa6fb2686922bb88d63e84a1f299",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=993",
       "triggerID" : "0b008f807b42aa6fb2686922bb88d63e84a1f299",
       "triggerType" : "PUSH"
     }, {
       "hash" : "65c01ad9f2b33d36d4890355feef91844438e732",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=995",
       "triggerID" : "65c01ad9f2b33d36d4890355feef91844438e732",
       "triggerType" : "PUSH"
     }, {
       "hash" : "955493fdc93842e8c99529cec35e39cecdbe9638",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=996",
       "triggerID" : "955493fdc93842e8c99529cec35e39cecdbe9638",
       "triggerType" : "PUSH"
     }, {
       "hash" : "533f00e5546da0026bbe51e1c030ddd1c19fdd08",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "533f00e5546da0026bbe51e1c030ddd1c19fdd08",
       "triggerType" : "PUSH"
     }, {
       "hash" : "05777be67e37f3a6a8eaa59e54d9e17e5fc6ba9b",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1016",
       "triggerID" : "05777be67e37f3a6a8eaa59e54d9e17e5fc6ba9b",
       "triggerType" : "PUSH"
     }, {
       "hash" : "a2ee90c549332a7f17a7d9b2799849b77c08a8e6",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "a2ee90c549332a7f17a7d9b2799849b77c08a8e6",
       "triggerType" : "PUSH"
     }, {
       "hash" : "05ce568b0b6ba14da9ca40d26b39ba478837d747",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1017",
       "triggerID" : "05ce568b0b6ba14da9ca40d26b39ba478837d747",
       "triggerType" : "PUSH"
     }, {
       "hash" : "840e3b33d39b1f86367bb6b0e65eadeb33de7765",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1025",
       "triggerID" : "840e3b33d39b1f86367bb6b0e65eadeb33de7765",
       "triggerType" : "PUSH"
     }, {
       "hash" : "8d2e2dd83f51d03e890d424a51d3e0e84e904a43",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1031",
       "triggerID" : "8d2e2dd83f51d03e890d424a51d3e0e84e904a43",
       "triggerType" : "PUSH"
     }, {
       "hash" : "688381914087ff11acf2f8c9372dbf39afa0bd40",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1035",
       "triggerID" : "688381914087ff11acf2f8c9372dbf39afa0bd40",
       "triggerType" : "PUSH"
     }, {
       "hash" : "fc2c138901e82ed5f3fbf363c2a0ce8140ade050",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "fc2c138901e82ed5f3fbf363c2a0ce8140ade050",
       "triggerType" : "PUSH"
     }, {
       "hash" : "6ed9e1cb1e5a191752d6d39c7a9d5d41840f0cdf",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1039",
       "triggerID" : "6ed9e1cb1e5a191752d6d39c7a9d5d41840f0cdf",
       "triggerType" : "PUSH"
     }, {
       "hash" : "a510a7de1458bb996df5bbb341e8d010e4c91381",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "a510a7de1458bb996df5bbb341e8d010e4c91381",
       "triggerType" : "PUSH"
     }, {
       "hash" : "49677397cd117228742205944606f7a789090376",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1042",
       "triggerID" : "49677397cd117228742205944606f7a789090376",
       "triggerType" : "PUSH"
     }, {
       "hash" : "1630db0a37422aed942ff280560a9d6329df5015",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1044",
       "triggerID" : "1630db0a37422aed942ff280560a9d6329df5015",
       "triggerType" : "PUSH"
     }, {
       "hash" : "f11c76301d4ab524db0af57aa831be3a898ea0fb",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1048",
       "triggerID" : "f11c76301d4ab524db0af57aa831be3a898ea0fb",
       "triggerType" : "PUSH"
     }, {
       "hash" : "d1a0dc4c2017853a7ba0c3302f886043a63f29da",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1050",
       "triggerID" : "d1a0dc4c2017853a7ba0c3302f886043a63f29da",
       "triggerType" : "PUSH"
     }, {
       "hash" : "6a4c15ec39ded1f90b113b272554217698a98e1c",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1087",
       "triggerID" : "6a4c15ec39ded1f90b113b272554217698a98e1c",
       "triggerType" : "PUSH"
     }, {
       "hash" : "288e730d645a5e58e50a87ef5f44ba11a8fb0533",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1092",
       "triggerID" : "288e730d645a5e58e50a87ef5f44ba11a8fb0533",
       "triggerType" : "PUSH"
     }, {
       "hash" : "70a5c21321a0da3a0be8d85e206c24befa7b89f1",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1095",
       "triggerID" : "70a5c21321a0da3a0be8d85e206c24befa7b89f1",
       "triggerType" : "PUSH"
     }, {
       "hash" : "f072a986114949c5e426ce64a5f5568fc718e25d",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "f072a986114949c5e426ce64a5f5568fc718e25d",
       "triggerType" : "PUSH"
     }, {
       "hash" : "45b27475b1e8806ec8c566d6f151f4bc79c8549d",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1096",
       "triggerID" : "45b27475b1e8806ec8c566d6f151f4bc79c8549d",
       "triggerType" : "PUSH"
     }, {
       "hash" : "3c62ef22f6cde28110e5eb2fca8760a008d77723",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "3c62ef22f6cde28110e5eb2fca8760a008d77723",
       "triggerType" : "PUSH"
     }, {
       "hash" : "dc785fa04e542b2ccc45494e0292f5153c7ecea8",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1097",
       "triggerID" : "dc785fa04e542b2ccc45494e0292f5153c7ecea8",
       "triggerType" : "PUSH"
     }, {
       "hash" : "585ddb8049b9b9ccb057027e265b08a1b586df8c",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "585ddb8049b9b9ccb057027e265b08a1b586df8c",
       "triggerType" : "PUSH"
     }, {
       "hash" : "b62ff8675beed2c4487c788b9ca1a0ad4271b913",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "b62ff8675beed2c4487c788b9ca1a0ad4271b913",
       "triggerType" : "PUSH"
     }, {
       "hash" : "2d46048a7402907f348cc96da2839ba1b3e4932a",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1099",
       "triggerID" : "2d46048a7402907f348cc96da2839ba1b3e4932a",
       "triggerType" : "PUSH"
     }, {
       "hash" : "004fb6ebf23f796a4b77650e0530f0f1145e8b91",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1101",
       "triggerID" : "004fb6ebf23f796a4b77650e0530f0f1145e8b91",
       "triggerType" : "PUSH"
     }, {
       "hash" : "af0c33333b5e9ae6d59f20a6db1d7c016ea015d3",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1165",
       "triggerID" : "af0c33333b5e9ae6d59f20a6db1d7c016ea015d3",
       "triggerType" : "PUSH"
     }, {
       "hash" : "da8111118d71306500229619d3b7996fcd2526e8",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "da8111118d71306500229619d3b7996fcd2526e8",
       "triggerType" : "PUSH"
     }, {
       "hash" : "83257e2b9bd3c426cf90e11003a5f244a750d7fa",
       "status" : "FAILURE",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1167",
       "triggerID" : "83257e2b9bd3c426cf90e11003a5f244a750d7fa",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 9e668f65c6d4f73c76573236f8fa507dfb80edac UNKNOWN
   * 533f00e5546da0026bbe51e1c030ddd1c19fdd08 UNKNOWN
   * a2ee90c549332a7f17a7d9b2799849b77c08a8e6 UNKNOWN
   * fc2c138901e82ed5f3fbf363c2a0ce8140ade050 UNKNOWN
   * a510a7de1458bb996df5bbb341e8d010e4c91381 UNKNOWN
   * f072a986114949c5e426ce64a5f5568fc718e25d UNKNOWN
   * 3c62ef22f6cde28110e5eb2fca8760a008d77723 UNKNOWN
   * 585ddb8049b9b9ccb057027e265b08a1b586df8c UNKNOWN
   * b62ff8675beed2c4487c788b9ca1a0ad4271b913 UNKNOWN
   * da8111118d71306500229619d3b7996fcd2526e8 UNKNOWN
   * 83257e2b9bd3c426cf90e11003a5f244a750d7fa Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1167) 
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] [HUDI-8141] Incremental Query with Completion Time [hudi]

Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.

hudi-bot commented on PR #11947:
URL: https://github.com/apache/hudi/pull/11947#issuecomment-2415913040

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "090f256a315cbbf0c6a54d93cc2739dd693360e6",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=484",
       "triggerID" : "090f256a315cbbf0c6a54d93cc2739dd693360e6",
       "triggerType" : "PUSH"
     }, {
       "hash" : "0df5649013c1cf9e782fcbb67b8a2e7174f94512",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=985",
       "triggerID" : "0df5649013c1cf9e782fcbb67b8a2e7174f94512",
       "triggerType" : "PUSH"
     }, {
       "hash" : "d52e0128f409918644255abbe4376dc4a780bfc8",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=986",
       "triggerID" : "d52e0128f409918644255abbe4376dc4a780bfc8",
       "triggerType" : "PUSH"
     }, {
       "hash" : "9e668f65c6d4f73c76573236f8fa507dfb80edac",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "9e668f65c6d4f73c76573236f8fa507dfb80edac",
       "triggerType" : "PUSH"
     }, {
       "hash" : "f145131d524bf190986e5de600a326e32a27beca",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=987",
       "triggerID" : "f145131d524bf190986e5de600a326e32a27beca",
       "triggerType" : "PUSH"
     }, {
       "hash" : "0b008f807b42aa6fb2686922bb88d63e84a1f299",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=993",
       "triggerID" : "0b008f807b42aa6fb2686922bb88d63e84a1f299",
       "triggerType" : "PUSH"
     }, {
       "hash" : "65c01ad9f2b33d36d4890355feef91844438e732",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=995",
       "triggerID" : "65c01ad9f2b33d36d4890355feef91844438e732",
       "triggerType" : "PUSH"
     }, {
       "hash" : "955493fdc93842e8c99529cec35e39cecdbe9638",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=996",
       "triggerID" : "955493fdc93842e8c99529cec35e39cecdbe9638",
       "triggerType" : "PUSH"
     }, {
       "hash" : "533f00e5546da0026bbe51e1c030ddd1c19fdd08",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "533f00e5546da0026bbe51e1c030ddd1c19fdd08",
       "triggerType" : "PUSH"
     }, {
       "hash" : "05777be67e37f3a6a8eaa59e54d9e17e5fc6ba9b",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1016",
       "triggerID" : "05777be67e37f3a6a8eaa59e54d9e17e5fc6ba9b",
       "triggerType" : "PUSH"
     }, {
       "hash" : "a2ee90c549332a7f17a7d9b2799849b77c08a8e6",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "a2ee90c549332a7f17a7d9b2799849b77c08a8e6",
       "triggerType" : "PUSH"
     }, {
       "hash" : "05ce568b0b6ba14da9ca40d26b39ba478837d747",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1017",
       "triggerID" : "05ce568b0b6ba14da9ca40d26b39ba478837d747",
       "triggerType" : "PUSH"
     }, {
       "hash" : "840e3b33d39b1f86367bb6b0e65eadeb33de7765",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1025",
       "triggerID" : "840e3b33d39b1f86367bb6b0e65eadeb33de7765",
       "triggerType" : "PUSH"
     }, {
       "hash" : "8d2e2dd83f51d03e890d424a51d3e0e84e904a43",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1031",
       "triggerID" : "8d2e2dd83f51d03e890d424a51d3e0e84e904a43",
       "triggerType" : "PUSH"
     }, {
       "hash" : "688381914087ff11acf2f8c9372dbf39afa0bd40",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1035",
       "triggerID" : "688381914087ff11acf2f8c9372dbf39afa0bd40",
       "triggerType" : "PUSH"
     }, {
       "hash" : "fc2c138901e82ed5f3fbf363c2a0ce8140ade050",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "fc2c138901e82ed5f3fbf363c2a0ce8140ade050",
       "triggerType" : "PUSH"
     }, {
       "hash" : "6ed9e1cb1e5a191752d6d39c7a9d5d41840f0cdf",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1039",
       "triggerID" : "6ed9e1cb1e5a191752d6d39c7a9d5d41840f0cdf",
       "triggerType" : "PUSH"
     }, {
       "hash" : "a510a7de1458bb996df5bbb341e8d010e4c91381",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "a510a7de1458bb996df5bbb341e8d010e4c91381",
       "triggerType" : "PUSH"
     }, {
       "hash" : "49677397cd117228742205944606f7a789090376",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1042",
       "triggerID" : "49677397cd117228742205944606f7a789090376",
       "triggerType" : "PUSH"
     }, {
       "hash" : "1630db0a37422aed942ff280560a9d6329df5015",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1044",
       "triggerID" : "1630db0a37422aed942ff280560a9d6329df5015",
       "triggerType" : "PUSH"
     }, {
       "hash" : "f11c76301d4ab524db0af57aa831be3a898ea0fb",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1048",
       "triggerID" : "f11c76301d4ab524db0af57aa831be3a898ea0fb",
       "triggerType" : "PUSH"
     }, {
       "hash" : "d1a0dc4c2017853a7ba0c3302f886043a63f29da",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1050",
       "triggerID" : "d1a0dc4c2017853a7ba0c3302f886043a63f29da",
       "triggerType" : "PUSH"
     }, {
       "hash" : "6a4c15ec39ded1f90b113b272554217698a98e1c",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1087",
       "triggerID" : "6a4c15ec39ded1f90b113b272554217698a98e1c",
       "triggerType" : "PUSH"
     }, {
       "hash" : "288e730d645a5e58e50a87ef5f44ba11a8fb0533",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1092",
       "triggerID" : "288e730d645a5e58e50a87ef5f44ba11a8fb0533",
       "triggerType" : "PUSH"
     }, {
       "hash" : "70a5c21321a0da3a0be8d85e206c24befa7b89f1",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1095",
       "triggerID" : "70a5c21321a0da3a0be8d85e206c24befa7b89f1",
       "triggerType" : "PUSH"
     }, {
       "hash" : "f072a986114949c5e426ce64a5f5568fc718e25d",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "f072a986114949c5e426ce64a5f5568fc718e25d",
       "triggerType" : "PUSH"
     }, {
       "hash" : "45b27475b1e8806ec8c566d6f151f4bc79c8549d",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1096",
       "triggerID" : "45b27475b1e8806ec8c566d6f151f4bc79c8549d",
       "triggerType" : "PUSH"
     }, {
       "hash" : "3c62ef22f6cde28110e5eb2fca8760a008d77723",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "3c62ef22f6cde28110e5eb2fca8760a008d77723",
       "triggerType" : "PUSH"
     }, {
       "hash" : "dc785fa04e542b2ccc45494e0292f5153c7ecea8",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1097",
       "triggerID" : "dc785fa04e542b2ccc45494e0292f5153c7ecea8",
       "triggerType" : "PUSH"
     }, {
       "hash" : "585ddb8049b9b9ccb057027e265b08a1b586df8c",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "585ddb8049b9b9ccb057027e265b08a1b586df8c",
       "triggerType" : "PUSH"
     }, {
       "hash" : "b62ff8675beed2c4487c788b9ca1a0ad4271b913",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "b62ff8675beed2c4487c788b9ca1a0ad4271b913",
       "triggerType" : "PUSH"
     }, {
       "hash" : "2d46048a7402907f348cc96da2839ba1b3e4932a",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1099",
       "triggerID" : "2d46048a7402907f348cc96da2839ba1b3e4932a",
       "triggerType" : "PUSH"
     }, {
       "hash" : "004fb6ebf23f796a4b77650e0530f0f1145e8b91",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1101",
       "triggerID" : "004fb6ebf23f796a4b77650e0530f0f1145e8b91",
       "triggerType" : "PUSH"
     }, {
       "hash" : "af0c33333b5e9ae6d59f20a6db1d7c016ea015d3",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1165",
       "triggerID" : "af0c33333b5e9ae6d59f20a6db1d7c016ea015d3",
       "triggerType" : "PUSH"
     }, {
       "hash" : "da8111118d71306500229619d3b7996fcd2526e8",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "da8111118d71306500229619d3b7996fcd2526e8",
       "triggerType" : "PUSH"
     }, {
       "hash" : "83257e2b9bd3c426cf90e11003a5f244a750d7fa",
       "status" : "FAILURE",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1167",
       "triggerID" : "83257e2b9bd3c426cf90e11003a5f244a750d7fa",
       "triggerType" : "PUSH"
     }, {
       "hash" : "e0de408f1b8153fdaaea4c87b27a7b69213b75b5",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "e0de408f1b8153fdaaea4c87b27a7b69213b75b5",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 9e668f65c6d4f73c76573236f8fa507dfb80edac UNKNOWN
   * 533f00e5546da0026bbe51e1c030ddd1c19fdd08 UNKNOWN
   * a2ee90c549332a7f17a7d9b2799849b77c08a8e6 UNKNOWN
   * fc2c138901e82ed5f3fbf363c2a0ce8140ade050 UNKNOWN
   * a510a7de1458bb996df5bbb341e8d010e4c91381 UNKNOWN
   * f072a986114949c5e426ce64a5f5568fc718e25d UNKNOWN
   * 3c62ef22f6cde28110e5eb2fca8760a008d77723 UNKNOWN
   * 585ddb8049b9b9ccb057027e265b08a1b586df8c UNKNOWN
   * b62ff8675beed2c4487c788b9ca1a0ad4271b913 UNKNOWN
   * da8111118d71306500229619d3b7996fcd2526e8 UNKNOWN
   * 83257e2b9bd3c426cf90e11003a5f244a750d7fa Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1167) 
   * e0de408f1b8153fdaaea4c87b27a7b69213b75b5 UNKNOWN
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] [HUDI-8141] Incremental Query with Completion Time [hudi]

Posted by "yihua (via GitHub)" <gi...@apache.org>.

yihua commented on code in PR #11947:
URL: https://github.com/apache/hudi/pull/11947#discussion_r1802553652


##########
hudi-common/src/main/java/org/apache/hudi/common/table/read/IncrementalQueryAnalyzer.java:
##########
@@ -376,14 +382,16 @@ private QueryContext(
         List<HoodieInstant> archivedInstants,
         List<HoodieInstant> activeInstants,
         HoodieTimeline activeTimeline,
-        @Nullable HoodieTimeline archivedTimeline) {
+        @Nullable HoodieTimeline archivedTimeline,
+        @Nullable String predicateFilter) {
       this.startInstant = Option.ofNullable(startInstant);
       this.endInstant = Option.ofNullable(endInstant);
       this.archivedInstants = archivedInstants;
       this.activeInstants = activeInstants;
       this.activeTimeline = activeTimeline;
       this.archivedTimeline = archivedTimeline;
       this.instants = instants;
+      this.predicateFilter = Option.ofNullable(predicateFilter);

Review Comment:
   Makes sense.  The predicate is decoupled from `QueryContext`.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] [HUDI-8141] Incremental Query with Completion Time [hudi]

Posted by "yihua (via GitHub)" <gi...@apache.org>.

yihua commented on code in PR #11947:
URL: https://github.com/apache/hudi/pull/11947#discussion_r1805924992


##########
hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/spark/sql/hudi/streaming/HoodieStreamSource.scala:
##########
@@ -186,11 +160,11 @@ class HoodieStreamSource(
     }
   }
 
-  private def startCommitTime(startOffset: HoodieSourceOffset): String = {
+  private def startCompletionTime(startOffset: HoodieSourceOffset): String = {
     startOffset match {
-      case INIT_OFFSET => startOffset.commitTime
-      case HoodieSourceOffset(commitTime) =>
-        commitTime
+      case INIT_OFFSET => startOffset.completionTime
+      case HoodieSourceOffset(completionTime) =>
+        HoodieInstantTimeGenerator.instantTimePlusMillis(completionTime, 1)

Review Comment:
   One question I have is for `CDCRelation`. The default range type is hardcoded to `OPEN_CLOSED` before this PR.  I've maintained that.  Let me know if I need to change it too.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] [HUDI-8141] Incremental Query with Completion Time [hudi]

Posted by "yihua (via GitHub)" <gi...@apache.org>.

yihua commented on code in PR #11947:
URL: https://github.com/apache/hudi/pull/11947#discussion_r1805926310


##########
hudi-utilities/src/main/java/org/apache/hudi/utilities/sources/SnapshotLoadQuerySplitter.java:
##########
@@ -118,6 +124,32 @@ public QueryInfo getNextCheckpoint(Dataset<Row> df, QueryInfo queryInfo, Option<
         .orElse(queryInfo);
   }
 
+  public Option<CheckpointWithPredicates> getNextCheckpoint(Dataset<Row> df, QueryContext queryContext,
+                                                            Option<SourceProfileSupplier> sourceProfileSupplier) {
+    // the start instant would be included into the final query result. So we need to get
+    // a strictly lower timestamp to have query splitter include the start instant
+    Option<CheckpointWithPredicates> nextCheckpointWithPredicates =
+        getNextCheckpointWithPredicates(df, instantTimeMinusMillis(queryContext.getBeginInstant().get(), 1));

Review Comment:
   The `getNextCheckpointWithPredicates` API in `SnapshotLoadQuerySplitter` is changed to take `QueryContext` and return completion time as the checkpoint directly, so all such logic of conversion between instant time and completion is no longer needed.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] [HUDI-8141] Incremental Query with Completion Time [hudi]

Posted by "danny0405 (via GitHub)" <gi...@apache.org>.

danny0405 commented on code in PR #11947:
URL: https://github.com/apache/hudi/pull/11947#discussion_r1806004445


##########
hudi-utilities/src/main/java/org/apache/hudi/utilities/sources/HoodieIncrSource.java:
##########
@@ -212,35 +208,58 @@ public Pair<Option<Dataset<Row>>, String> fetchNextBatch(Option<String> lastCkpt
           .collect(Collectors.toMap(Pair::getLeft, Pair::getRight));
       reader = reader.options(optionsMap);
     }
+
+    boolean shouldFullScan =
+        missingCheckpointStrategy == MissingCheckpointStrategy.READ_UPTO_LATEST_COMMIT
+            && queryContext.getActiveTimeline()
+            .isBeforeTimelineStartsByCompletionTime(analyzer.getStartCompletionTime().get());
     Dataset<Row> source;
-    // Do Incr pull. Set end instant if available
-    if (queryInfo.isIncremental()) {
-      source = reader
-          .options(readOpts)
-          .option(QUERY_TYPE().key(), QUERY_TYPE_INCREMENTAL_OPT_VAL())
-          .option(BEGIN_INSTANTTIME().key(), queryInfo.getStartInstant())
-          .option(END_INSTANTTIME().key(), queryInfo.getEndInstant())
-          .option(INCREMENTAL_FALLBACK_TO_FULL_TABLE_SCAN().key(),
-              props.getString(INCREMENTAL_FALLBACK_TO_FULL_TABLE_SCAN().key(),
-                  INCREMENTAL_FALLBACK_TO_FULL_TABLE_SCAN().defaultValue()))
-          .option(INCREMENTAL_READ_HANDLE_HOLLOW_COMMIT().key(), handlingMode.name())
-          .load(srcPath);
-    } else {
-      // if checkpoint is missing from source table, and if strategy is set to READ_UPTO_LATEST_COMMIT, we have to issue snapshot query
+    if (instantRange.isEmpty() || shouldFullScan) {
+      // snapshot query
       Dataset<Row> snapshot = reader
           .options(readOpts)
           .option(DataSourceReadOptions.QUERY_TYPE().key(), DataSourceReadOptions.QUERY_TYPE_SNAPSHOT_OPT_VAL())
           .load(srcPath);
+
+      Option<String> predicate = Option.empty();
+      List<String> instantTimeList = queryContext.getInstantTimeList();
       if (snapshotLoadQuerySplitter.isPresent()) {
-        queryInfo = snapshotLoadQuerySplitter.get().getNextCheckpoint(snapshot, queryInfo, sourceProfileSupplier);
+        Option<SnapshotLoadQuerySplitter.CheckpointWithPredicates> newCheckpointAndPredicate =
+            snapshotLoadQuerySplitter.get().getNextCheckpointWithPredicates(snapshot, queryContext);
+        if (newCheckpointAndPredicate.isPresent()) {
+          endCompletionTime = newCheckpointAndPredicate.get().endInstant;
+          predicate = Option.of(newCheckpointAndPredicate.get().predicateFilter);
+          instantTimeList = queryContext.getInstants().stream()
+              .filter(instant -> HoodieTimeline.compareTimestamps(
+                  instant.getCompletionTime(), HoodieTimeline.LESSER_THAN_OR_EQUALS, newCheckpointAndPredicate.get().endInstant))
+              .map(HoodieInstant::getTimestamp)
+              .collect(Collectors.toList());
+        } else {
+          endCompletionTime = queryContext.getMaxCompletionTime();
+        }
       }
+
+      snapshot = predicate.map(snapshot::filter).orElse(snapshot);
       source = snapshot
           // add filtering so that only interested records are returned.
-          .filter(String.format("%s > '%s'", HoodieRecord.COMMIT_TIME_METADATA_FIELD,
-              queryInfo.getStartInstant()))
-          .filter(String.format("%s <= '%s'", HoodieRecord.COMMIT_TIME_METADATA_FIELD,
-              queryInfo.getEndInstant()));
-      source = queryInfo.getPredicateFilter().map(source::filter).orElse(source);
+          .filter(String.format("%s IN ('%s')", HoodieRecord.COMMIT_TIME_METADATA_FIELD,
+              String.join("','", instantTimeList)));
+    } else {
+      // normal incremental query
+      String beginCompletionTime = queryContext.getInstants().stream()
+          .min(HoodieInstant.COMPLETION_TIME_COMPARATOR)
+          .map(HoodieInstant::getCompletionTime)
+          .get();

Review Comment:
   Isn't the start completion time already specified in the analyzer?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] [HUDI-8141] WIP:Always use completion time to find begin Instant for incremental source [hudi]

Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.

hudi-bot commented on PR #11947:
URL: https://github.com/apache/hudi/pull/11947#issuecomment-2403740932

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "090f256a315cbbf0c6a54d93cc2739dd693360e6",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=484",
       "triggerID" : "090f256a315cbbf0c6a54d93cc2739dd693360e6",
       "triggerType" : "PUSH"
     }, {
       "hash" : "0df5649013c1cf9e782fcbb67b8a2e7174f94512",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=985",
       "triggerID" : "0df5649013c1cf9e782fcbb67b8a2e7174f94512",
       "triggerType" : "PUSH"
     }, {
       "hash" : "d52e0128f409918644255abbe4376dc4a780bfc8",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=986",
       "triggerID" : "d52e0128f409918644255abbe4376dc4a780bfc8",
       "triggerType" : "PUSH"
     }, {
       "hash" : "9e668f65c6d4f73c76573236f8fa507dfb80edac",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "9e668f65c6d4f73c76573236f8fa507dfb80edac",
       "triggerType" : "PUSH"
     }, {
       "hash" : "f145131d524bf190986e5de600a326e32a27beca",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=987",
       "triggerID" : "f145131d524bf190986e5de600a326e32a27beca",
       "triggerType" : "PUSH"
     }, {
       "hash" : "0b008f807b42aa6fb2686922bb88d63e84a1f299",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=993",
       "triggerID" : "0b008f807b42aa6fb2686922bb88d63e84a1f299",
       "triggerType" : "PUSH"
     }, {
       "hash" : "65c01ad9f2b33d36d4890355feef91844438e732",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=995",
       "triggerID" : "65c01ad9f2b33d36d4890355feef91844438e732",
       "triggerType" : "PUSH"
     }, {
       "hash" : "955493fdc93842e8c99529cec35e39cecdbe9638",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=996",
       "triggerID" : "955493fdc93842e8c99529cec35e39cecdbe9638",
       "triggerType" : "PUSH"
     }, {
       "hash" : "533f00e5546da0026bbe51e1c030ddd1c19fdd08",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "533f00e5546da0026bbe51e1c030ddd1c19fdd08",
       "triggerType" : "PUSH"
     }, {
       "hash" : "05777be67e37f3a6a8eaa59e54d9e17e5fc6ba9b",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1016",
       "triggerID" : "05777be67e37f3a6a8eaa59e54d9e17e5fc6ba9b",
       "triggerType" : "PUSH"
     }, {
       "hash" : "a2ee90c549332a7f17a7d9b2799849b77c08a8e6",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "a2ee90c549332a7f17a7d9b2799849b77c08a8e6",
       "triggerType" : "PUSH"
     }, {
       "hash" : "05ce568b0b6ba14da9ca40d26b39ba478837d747",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1017",
       "triggerID" : "05ce568b0b6ba14da9ca40d26b39ba478837d747",
       "triggerType" : "PUSH"
     }, {
       "hash" : "840e3b33d39b1f86367bb6b0e65eadeb33de7765",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1025",
       "triggerID" : "840e3b33d39b1f86367bb6b0e65eadeb33de7765",
       "triggerType" : "PUSH"
     }, {
       "hash" : "8d2e2dd83f51d03e890d424a51d3e0e84e904a43",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1031",
       "triggerID" : "8d2e2dd83f51d03e890d424a51d3e0e84e904a43",
       "triggerType" : "PUSH"
     }, {
       "hash" : "688381914087ff11acf2f8c9372dbf39afa0bd40",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1035",
       "triggerID" : "688381914087ff11acf2f8c9372dbf39afa0bd40",
       "triggerType" : "PUSH"
     }, {
       "hash" : "fc2c138901e82ed5f3fbf363c2a0ce8140ade050",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "fc2c138901e82ed5f3fbf363c2a0ce8140ade050",
       "triggerType" : "PUSH"
     }, {
       "hash" : "6ed9e1cb1e5a191752d6d39c7a9d5d41840f0cdf",
       "status" : "CANCELED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1039",
       "triggerID" : "6ed9e1cb1e5a191752d6d39c7a9d5d41840f0cdf",
       "triggerType" : "PUSH"
     }, {
       "hash" : "a510a7de1458bb996df5bbb341e8d010e4c91381",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "a510a7de1458bb996df5bbb341e8d010e4c91381",
       "triggerType" : "PUSH"
     }, {
       "hash" : "49677397cd117228742205944606f7a789090376",
       "status" : "PENDING",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1042",
       "triggerID" : "49677397cd117228742205944606f7a789090376",
       "triggerType" : "PUSH"
     }, {
       "hash" : "1630db0a37422aed942ff280560a9d6329df5015",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "1630db0a37422aed942ff280560a9d6329df5015",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 9e668f65c6d4f73c76573236f8fa507dfb80edac UNKNOWN
   * 533f00e5546da0026bbe51e1c030ddd1c19fdd08 UNKNOWN
   * a2ee90c549332a7f17a7d9b2799849b77c08a8e6 UNKNOWN
   * fc2c138901e82ed5f3fbf363c2a0ce8140ade050 UNKNOWN
   * 6ed9e1cb1e5a191752d6d39c7a9d5d41840f0cdf Azure: [CANCELED](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1039) 
   * a510a7de1458bb996df5bbb341e8d010e4c91381 UNKNOWN
   * 49677397cd117228742205944606f7a789090376 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1042) 
   * 1630db0a37422aed942ff280560a9d6329df5015 UNKNOWN
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] [HUDI-8141] Incremental Query with Completion Time [hudi]

Posted by "CTTY (via GitHub)" <gi...@apache.org>.

CTTY commented on code in PR #11947:
URL: https://github.com/apache/hudi/pull/11947#discussion_r1797784717


##########
hudi-common/src/main/java/org/apache/hudi/common/table/timeline/TimelineUtils.java:
##########
@@ -308,6 +309,38 @@ public static HoodieTimeline getCommitsTimelineAfter(
     return timelineSinceLastSync;
   }
 
+  public static HoodieTimeline getCommitsTimeLineAfterByCompletionTimeRange(
+      HoodieTableMetaClient metaClient,
+      String exclusiveStartCompletionTime,
+      String inclusiveEndCompletionTime) {
+    HoodieDefaultTimeline writeTimeline = metaClient.getActiveTimeline().getWriteTimeline();
+
+    HoodieDefaultTimeline timeline;
+    if (writeTimeline.isBeforeTimelineStartsByCompletionTime(exclusiveStartCompletionTime)) {
+      // need to load archived timeline as well
+      try (CompletionTimeQueryView view = new CompletionTimeQueryView(metaClient)) {
+        List<String> instants = view.getStartTimes(
+            exclusiveStartCompletionTime,
+            inclusiveEndCompletionTime,
+            RangeType.OPEN_CLOSED);
+        if (instants.isEmpty()) {
+          LOG.warn("Did not find any instant when getting timeline, using write timeline for incremental read");
+          timeline = writeTimeline;
+        } else {
+          timeline = metaClient.getArchivedTimeline(instants.get(0))
+              .mergeTimeline(writeTimeline)
+              .mergeTimeline(writeTimeline);

Review Comment:
   Yes, we only need to merge once here. Will fix this typo



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] [HUDI-8141] Incremental Query with Completion Time [hudi]

Posted by "yihua (via GitHub)" <gi...@apache.org>.

yihua commented on code in PR #11947:
URL: https://github.com/apache/hudi/pull/11947#discussion_r1797785153


##########
hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/hudi/IncrementalRelation.scala:
##########
@@ -309,7 +292,8 @@ class IncrementalRelation(val sqlContext: SQLContext,
       .format("hudi_v1")
       .schema(usedSchema)
       .load(basePath.toString)
-      .filter(String.format("%s > '%s'", HoodieRecord.COMMIT_TIME_METADATA_FIELD, //Notice the > in place of >= because we are working with optParam instead of first commit > optParam
+      // using closed_closed range here since we are using start instant from QueryContext
+      .filter(String.format("%s >= '%s'", HoodieRecord.COMMIT_TIME_METADATA_FIELD,

Review Comment:
   Should full scann also use `commitTimesToReturn` for specific filtering?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] [HUDI-8141] Incremental Query with Completion Time [hudi]

Posted by "danny0405 (via GitHub)" <gi...@apache.org>.

danny0405 commented on code in PR #11947:
URL: https://github.com/apache/hudi/pull/11947#discussion_r1798654878


##########
hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/hudi/IncrementalRelation.scala:
##########
@@ -305,14 +286,17 @@ class IncrementalRelation(val sqlContext: SQLContext,
   }
 
   private def fullTableScanDataFrame(startInstantTime: String, endInstantTime: String): DataFrame = {
+    val commitTimesToReturn = commitsToReturn.map(_.getTimestamp)
     val hudiDF = sqlContext.read
       .format("hudi_v1")
       .schema(usedSchema)
       .load(basePath.toString)
-      .filter(String.format("%s > '%s'", HoodieRecord.COMMIT_TIME_METADATA_FIELD, //Notice the > in place of >= because we are working with optParam instead of first commit > optParam
+      // using closed_closed range here since we are using start instant from QueryContext
+      .filter(String.format("%s >= '%s'", HoodieRecord.COMMIT_TIME_METADATA_FIELD,
         startInstantTime))
       .filter(String.format("%s <= '%s'", HoodieRecord.COMMIT_TIME_METADATA_FIELD,

Review Comment:
   do we still need the `<=` and `>=` predicates because we already have the `in` predicate.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] [HUDI-8141] Incremental Query with Completion Time [hudi]

Posted by "danny0405 (via GitHub)" <gi...@apache.org>.

danny0405 commented on code in PR #11947:
URL: https://github.com/apache/hudi/pull/11947#discussion_r1798654551


##########
hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/hudi/IncrementalRelation.scala:
##########
@@ -90,24 +78,24 @@ class IncrementalRelation(val sqlContext: SQLContext,
     throw new HoodieException("Incremental queries are not supported when meta fields are disabled")
   }
 
+  private val queryContext: IncrementalQueryAnalyzer.QueryContext =
+    IncrementalQueryAnalyzer.builder()
+      .metaClient(metaClient)
+      .startTime(optParams(DataSourceReadOptions.BEGIN_INSTANTTIME.key))
+      .endTime(optParams.getOrElse(DataSourceReadOptions.END_INSTANTTIME.key, null))
+      .rangeType(InstantRange.RangeType.OPEN_CLOSED)
+      .build()
+      .analyze()
+
+  private val commitsToReturn = List.concat(
+    queryContext.getArchivedInstants.asScala,
+    queryContext.getActiveInstants.asScala)

Review Comment:
   The `instants` is already a concatenation of archived and active instants.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] [HUDI-8141] Incremental Query with Completion Time [hudi]

Posted by "CTTY (via GitHub)" <gi...@apache.org>.

CTTY commented on code in PR #11947:
URL: https://github.com/apache/hudi/pull/11947#discussion_r1798104950


##########
hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/hudi/IncrementalRelation.scala:
##########
@@ -90,24 +78,25 @@ class IncrementalRelation(val sqlContext: SQLContext,
     throw new HoodieException("Incremental queries are not supported when meta fields are disabled")
   }
 
+  private val queryContext: IncrementalQueryAnalyzer.QueryContext =
+    IncrementalQueryAnalyzer.builder()
+      .metaClient(metaClient)
+      .startTime(optParams(DataSourceReadOptions.BEGIN_INSTANTTIME.key))
+      .endTime(optParams.getOrElse(DataSourceReadOptions.END_INSTANTTIME.key, null))

Review Comment:
   JIRA: https://issues.apache.org/jira/browse/HUDI-8356



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] [HUDI-8141] Incremental Query with Completion Time [hudi]

Posted by "CTTY (via GitHub)" <gi...@apache.org>.

CTTY commented on code in PR #11947:
URL: https://github.com/apache/hudi/pull/11947#discussion_r1797840340


##########
hudi-spark-datasource/hudi-spark/src/main/java/org/apache/hudi/HoodieDataSourceHelpers.java:
##########
@@ -74,6 +74,15 @@ public static List<String> listCommitsSince(HoodieStorage storage, String basePa
         .map(HoodieInstant::getTimestamp).collect(Collectors.toList());
   }
 
+  public static List<String> listCompletionTimeSince(FileSystem fs, String basePath,

Review Comment:
   This will be called in integtest script: `docker/demo/sparksql-incremental.commands`



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] [HUDI-8141] Incremental Query with Completion Time [hudi]

Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.

hudi-bot commented on PR #11947:
URL: https://github.com/apache/hudi/pull/11947#issuecomment-2408796865

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "090f256a315cbbf0c6a54d93cc2739dd693360e6",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=484",
       "triggerID" : "090f256a315cbbf0c6a54d93cc2739dd693360e6",
       "triggerType" : "PUSH"
     }, {
       "hash" : "0df5649013c1cf9e782fcbb67b8a2e7174f94512",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=985",
       "triggerID" : "0df5649013c1cf9e782fcbb67b8a2e7174f94512",
       "triggerType" : "PUSH"
     }, {
       "hash" : "d52e0128f409918644255abbe4376dc4a780bfc8",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=986",
       "triggerID" : "d52e0128f409918644255abbe4376dc4a780bfc8",
       "triggerType" : "PUSH"
     }, {
       "hash" : "9e668f65c6d4f73c76573236f8fa507dfb80edac",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "9e668f65c6d4f73c76573236f8fa507dfb80edac",
       "triggerType" : "PUSH"
     }, {
       "hash" : "f145131d524bf190986e5de600a326e32a27beca",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=987",
       "triggerID" : "f145131d524bf190986e5de600a326e32a27beca",
       "triggerType" : "PUSH"
     }, {
       "hash" : "0b008f807b42aa6fb2686922bb88d63e84a1f299",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=993",
       "triggerID" : "0b008f807b42aa6fb2686922bb88d63e84a1f299",
       "triggerType" : "PUSH"
     }, {
       "hash" : "65c01ad9f2b33d36d4890355feef91844438e732",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=995",
       "triggerID" : "65c01ad9f2b33d36d4890355feef91844438e732",
       "triggerType" : "PUSH"
     }, {
       "hash" : "955493fdc93842e8c99529cec35e39cecdbe9638",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=996",
       "triggerID" : "955493fdc93842e8c99529cec35e39cecdbe9638",
       "triggerType" : "PUSH"
     }, {
       "hash" : "533f00e5546da0026bbe51e1c030ddd1c19fdd08",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "533f00e5546da0026bbe51e1c030ddd1c19fdd08",
       "triggerType" : "PUSH"
     }, {
       "hash" : "05777be67e37f3a6a8eaa59e54d9e17e5fc6ba9b",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1016",
       "triggerID" : "05777be67e37f3a6a8eaa59e54d9e17e5fc6ba9b",
       "triggerType" : "PUSH"
     }, {
       "hash" : "a2ee90c549332a7f17a7d9b2799849b77c08a8e6",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "a2ee90c549332a7f17a7d9b2799849b77c08a8e6",
       "triggerType" : "PUSH"
     }, {
       "hash" : "05ce568b0b6ba14da9ca40d26b39ba478837d747",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1017",
       "triggerID" : "05ce568b0b6ba14da9ca40d26b39ba478837d747",
       "triggerType" : "PUSH"
     }, {
       "hash" : "840e3b33d39b1f86367bb6b0e65eadeb33de7765",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1025",
       "triggerID" : "840e3b33d39b1f86367bb6b0e65eadeb33de7765",
       "triggerType" : "PUSH"
     }, {
       "hash" : "8d2e2dd83f51d03e890d424a51d3e0e84e904a43",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1031",
       "triggerID" : "8d2e2dd83f51d03e890d424a51d3e0e84e904a43",
       "triggerType" : "PUSH"
     }, {
       "hash" : "688381914087ff11acf2f8c9372dbf39afa0bd40",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1035",
       "triggerID" : "688381914087ff11acf2f8c9372dbf39afa0bd40",
       "triggerType" : "PUSH"
     }, {
       "hash" : "fc2c138901e82ed5f3fbf363c2a0ce8140ade050",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "fc2c138901e82ed5f3fbf363c2a0ce8140ade050",
       "triggerType" : "PUSH"
     }, {
       "hash" : "6ed9e1cb1e5a191752d6d39c7a9d5d41840f0cdf",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1039",
       "triggerID" : "6ed9e1cb1e5a191752d6d39c7a9d5d41840f0cdf",
       "triggerType" : "PUSH"
     }, {
       "hash" : "a510a7de1458bb996df5bbb341e8d010e4c91381",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "a510a7de1458bb996df5bbb341e8d010e4c91381",
       "triggerType" : "PUSH"
     }, {
       "hash" : "49677397cd117228742205944606f7a789090376",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1042",
       "triggerID" : "49677397cd117228742205944606f7a789090376",
       "triggerType" : "PUSH"
     }, {
       "hash" : "1630db0a37422aed942ff280560a9d6329df5015",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1044",
       "triggerID" : "1630db0a37422aed942ff280560a9d6329df5015",
       "triggerType" : "PUSH"
     }, {
       "hash" : "f11c76301d4ab524db0af57aa831be3a898ea0fb",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1048",
       "triggerID" : "f11c76301d4ab524db0af57aa831be3a898ea0fb",
       "triggerType" : "PUSH"
     }, {
       "hash" : "d1a0dc4c2017853a7ba0c3302f886043a63f29da",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1050",
       "triggerID" : "d1a0dc4c2017853a7ba0c3302f886043a63f29da",
       "triggerType" : "PUSH"
     }, {
       "hash" : "6a4c15ec39ded1f90b113b272554217698a98e1c",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1087",
       "triggerID" : "6a4c15ec39ded1f90b113b272554217698a98e1c",
       "triggerType" : "PUSH"
     }, {
       "hash" : "288e730d645a5e58e50a87ef5f44ba11a8fb0533",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1092",
       "triggerID" : "288e730d645a5e58e50a87ef5f44ba11a8fb0533",
       "triggerType" : "PUSH"
     }, {
       "hash" : "70a5c21321a0da3a0be8d85e206c24befa7b89f1",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1095",
       "triggerID" : "70a5c21321a0da3a0be8d85e206c24befa7b89f1",
       "triggerType" : "PUSH"
     }, {
       "hash" : "f072a986114949c5e426ce64a5f5568fc718e25d",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "f072a986114949c5e426ce64a5f5568fc718e25d",
       "triggerType" : "PUSH"
     }, {
       "hash" : "45b27475b1e8806ec8c566d6f151f4bc79c8549d",
       "status" : "CANCELED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1096",
       "triggerID" : "45b27475b1e8806ec8c566d6f151f4bc79c8549d",
       "triggerType" : "PUSH"
     }, {
       "hash" : "3c62ef22f6cde28110e5eb2fca8760a008d77723",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "3c62ef22f6cde28110e5eb2fca8760a008d77723",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 9e668f65c6d4f73c76573236f8fa507dfb80edac UNKNOWN
   * 533f00e5546da0026bbe51e1c030ddd1c19fdd08 UNKNOWN
   * a2ee90c549332a7f17a7d9b2799849b77c08a8e6 UNKNOWN
   * fc2c138901e82ed5f3fbf363c2a0ce8140ade050 UNKNOWN
   * a510a7de1458bb996df5bbb341e8d010e4c91381 UNKNOWN
   * f072a986114949c5e426ce64a5f5568fc718e25d UNKNOWN
   * 45b27475b1e8806ec8c566d6f151f4bc79c8549d Azure: [CANCELED](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1096) 
   * 3c62ef22f6cde28110e5eb2fca8760a008d77723 UNKNOWN
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] [HUDI-8141] Incremental Query with Completion Time [hudi]

Posted by "CTTY (via GitHub)" <gi...@apache.org>.

CTTY commented on code in PR #11947:
URL: https://github.com/apache/hudi/pull/11947#discussion_r1797780363


##########
hudi-common/src/main/java/org/apache/hudi/common/table/timeline/HoodieDefaultTimeline.java:
##########
@@ -532,6 +532,13 @@ public boolean isBeforeTimelineStarts(String instant) {
         && compareTimestamps(instant, LESSER_THAN, firstNonSavepointCommit.get().getTimestamp());
   }
 
+  @Override
+  public boolean isBeforeTimelineStartsByCompletionTime(String completionTime) {
+    Option<HoodieInstant> firstNonSavepointCommit = getFirstNonSavepointCommit();

Review Comment:
   Good catch! Will fix this



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] [HUDI-8141] Incremental Query with Completion Time [hudi]

Posted by "yihua (via GitHub)" <gi...@apache.org>.

yihua commented on code in PR #11947:
URL: https://github.com/apache/hudi/pull/11947#discussion_r1797860501


##########
hudi-spark-datasource/hudi-spark/src/test/scala/org/apache/hudi/functional/TestMORDataSource.scala:
##########
@@ -114,21 +114,24 @@ class TestMORDataSource extends HoodieSparkClientTestBase with SparkDatasetMixin
       .mode(SaveMode.Overwrite)
       .save(basePath)
     assertTrue(HoodieDataSourceHelpers.hasNewCommits(storage, basePath, "000"))
+    val df1CompletionTime = HoodieDataSourceHelpers.latestCommitCompletionTime(storage, basePath)

Review Comment:
   rename to `commit1CompletionTime`; same for other variables.



##########
hudi-spark-datasource/hudi-spark/src/test/scala/org/apache/hudi/functional/TestSparkDataSource.scala:
##########
@@ -138,7 +140,7 @@ class TestSparkDataSource extends SparkClientFunctionalTestHarness {
     val hoodieIncViewDf1 = spark.read.format("org.apache.hudi")
       .option(DataSourceReadOptions.QUERY_TYPE.key, DataSourceReadOptions.QUERY_TYPE_INCREMENTAL_OPT_VAL)
       .option(DataSourceReadOptions.BEGIN_INSTANTTIME.key, "000")
-      .option(DataSourceReadOptions.END_INSTANTTIME.key, firstCommit)
+      .option(DataSourceReadOptions.END_INSTANTTIME.key, commitCompletionTime1)

Review Comment:
   Should the begin instant time be changed too?



##########
hudi-spark-datasource/hudi-spark/src/test/scala/org/apache/hudi/functional/TestStreamingSource.scala:
##########
@@ -205,13 +205,10 @@ class TestStreamingSource extends StreamTest {
       addData(tablePath, Seq(("2", "a1", "11", "001")))
       addData(tablePath, Seq(("3", "a1", "12", "002")))
 
-      val timestamp = if (handlingMode == USE_TRANSITION_TIME) {
+      val timestamp =

Review Comment:
   Similar here.  Should we add a test case where the smallest completion time in the active timeline does not have the smallest instant time to validate the correctness of the incremental query logic?



##########
hudi-spark-datasource/hudi-spark/src/test/scala/org/apache/hudi/functional/TestStreamingSource.scala:
##########
@@ -46,7 +46,7 @@ class TestStreamingSource extends StreamTest {
   )
   private val columns = Seq("id", "name", "price", "ts")
 
-  val handlingMode: HollowCommitHandling = BLOCK
+  val handlingMode: HollowCommitHandling = USE_TRANSITION_TIME

Review Comment:
   Should this be removed now?  Or do we want to keep it for incremental pulls on 0.x tables?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] [HUDI-8141] WIP:Always use completion time to find begin Instant for incremental source [hudi]

Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.

hudi-bot commented on PR #11947:
URL: https://github.com/apache/hudi/pull/11947#issuecomment-2403683621

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "090f256a315cbbf0c6a54d93cc2739dd693360e6",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=484",
       "triggerID" : "090f256a315cbbf0c6a54d93cc2739dd693360e6",
       "triggerType" : "PUSH"
     }, {
       "hash" : "0df5649013c1cf9e782fcbb67b8a2e7174f94512",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=985",
       "triggerID" : "0df5649013c1cf9e782fcbb67b8a2e7174f94512",
       "triggerType" : "PUSH"
     }, {
       "hash" : "d52e0128f409918644255abbe4376dc4a780bfc8",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=986",
       "triggerID" : "d52e0128f409918644255abbe4376dc4a780bfc8",
       "triggerType" : "PUSH"
     }, {
       "hash" : "9e668f65c6d4f73c76573236f8fa507dfb80edac",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "9e668f65c6d4f73c76573236f8fa507dfb80edac",
       "triggerType" : "PUSH"
     }, {
       "hash" : "f145131d524bf190986e5de600a326e32a27beca",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=987",
       "triggerID" : "f145131d524bf190986e5de600a326e32a27beca",
       "triggerType" : "PUSH"
     }, {
       "hash" : "0b008f807b42aa6fb2686922bb88d63e84a1f299",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=993",
       "triggerID" : "0b008f807b42aa6fb2686922bb88d63e84a1f299",
       "triggerType" : "PUSH"
     }, {
       "hash" : "65c01ad9f2b33d36d4890355feef91844438e732",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=995",
       "triggerID" : "65c01ad9f2b33d36d4890355feef91844438e732",
       "triggerType" : "PUSH"
     }, {
       "hash" : "955493fdc93842e8c99529cec35e39cecdbe9638",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=996",
       "triggerID" : "955493fdc93842e8c99529cec35e39cecdbe9638",
       "triggerType" : "PUSH"
     }, {
       "hash" : "533f00e5546da0026bbe51e1c030ddd1c19fdd08",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "533f00e5546da0026bbe51e1c030ddd1c19fdd08",
       "triggerType" : "PUSH"
     }, {
       "hash" : "05777be67e37f3a6a8eaa59e54d9e17e5fc6ba9b",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1016",
       "triggerID" : "05777be67e37f3a6a8eaa59e54d9e17e5fc6ba9b",
       "triggerType" : "PUSH"
     }, {
       "hash" : "a2ee90c549332a7f17a7d9b2799849b77c08a8e6",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "a2ee90c549332a7f17a7d9b2799849b77c08a8e6",
       "triggerType" : "PUSH"
     }, {
       "hash" : "05ce568b0b6ba14da9ca40d26b39ba478837d747",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1017",
       "triggerID" : "05ce568b0b6ba14da9ca40d26b39ba478837d747",
       "triggerType" : "PUSH"
     }, {
       "hash" : "840e3b33d39b1f86367bb6b0e65eadeb33de7765",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1025",
       "triggerID" : "840e3b33d39b1f86367bb6b0e65eadeb33de7765",
       "triggerType" : "PUSH"
     }, {
       "hash" : "8d2e2dd83f51d03e890d424a51d3e0e84e904a43",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1031",
       "triggerID" : "8d2e2dd83f51d03e890d424a51d3e0e84e904a43",
       "triggerType" : "PUSH"
     }, {
       "hash" : "688381914087ff11acf2f8c9372dbf39afa0bd40",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1035",
       "triggerID" : "688381914087ff11acf2f8c9372dbf39afa0bd40",
       "triggerType" : "PUSH"
     }, {
       "hash" : "fc2c138901e82ed5f3fbf363c2a0ce8140ade050",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "fc2c138901e82ed5f3fbf363c2a0ce8140ade050",
       "triggerType" : "PUSH"
     }, {
       "hash" : "6ed9e1cb1e5a191752d6d39c7a9d5d41840f0cdf",
       "status" : "CANCELED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1039",
       "triggerID" : "6ed9e1cb1e5a191752d6d39c7a9d5d41840f0cdf",
       "triggerType" : "PUSH"
     }, {
       "hash" : "a510a7de1458bb996df5bbb341e8d010e4c91381",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "a510a7de1458bb996df5bbb341e8d010e4c91381",
       "triggerType" : "PUSH"
     }, {
       "hash" : "49677397cd117228742205944606f7a789090376",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "49677397cd117228742205944606f7a789090376",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 9e668f65c6d4f73c76573236f8fa507dfb80edac UNKNOWN
   * 533f00e5546da0026bbe51e1c030ddd1c19fdd08 UNKNOWN
   * a2ee90c549332a7f17a7d9b2799849b77c08a8e6 UNKNOWN
   * fc2c138901e82ed5f3fbf363c2a0ce8140ade050 UNKNOWN
   * 6ed9e1cb1e5a191752d6d39c7a9d5d41840f0cdf Azure: [CANCELED](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1039) 
   * a510a7de1458bb996df5bbb341e8d010e4c91381 UNKNOWN
   * 49677397cd117228742205944606f7a789090376 UNKNOWN
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] [HUDI-8141] Incremental Query with Completion Time [hudi]

Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.

hudi-bot commented on PR #11947:
URL: https://github.com/apache/hudi/pull/11947#issuecomment-2415341449

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "090f256a315cbbf0c6a54d93cc2739dd693360e6",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=484",
       "triggerID" : "090f256a315cbbf0c6a54d93cc2739dd693360e6",
       "triggerType" : "PUSH"
     }, {
       "hash" : "0df5649013c1cf9e782fcbb67b8a2e7174f94512",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=985",
       "triggerID" : "0df5649013c1cf9e782fcbb67b8a2e7174f94512",
       "triggerType" : "PUSH"
     }, {
       "hash" : "d52e0128f409918644255abbe4376dc4a780bfc8",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=986",
       "triggerID" : "d52e0128f409918644255abbe4376dc4a780bfc8",
       "triggerType" : "PUSH"
     }, {
       "hash" : "9e668f65c6d4f73c76573236f8fa507dfb80edac",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "9e668f65c6d4f73c76573236f8fa507dfb80edac",
       "triggerType" : "PUSH"
     }, {
       "hash" : "f145131d524bf190986e5de600a326e32a27beca",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=987",
       "triggerID" : "f145131d524bf190986e5de600a326e32a27beca",
       "triggerType" : "PUSH"
     }, {
       "hash" : "0b008f807b42aa6fb2686922bb88d63e84a1f299",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=993",
       "triggerID" : "0b008f807b42aa6fb2686922bb88d63e84a1f299",
       "triggerType" : "PUSH"
     }, {
       "hash" : "65c01ad9f2b33d36d4890355feef91844438e732",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=995",
       "triggerID" : "65c01ad9f2b33d36d4890355feef91844438e732",
       "triggerType" : "PUSH"
     }, {
       "hash" : "955493fdc93842e8c99529cec35e39cecdbe9638",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=996",
       "triggerID" : "955493fdc93842e8c99529cec35e39cecdbe9638",
       "triggerType" : "PUSH"
     }, {
       "hash" : "533f00e5546da0026bbe51e1c030ddd1c19fdd08",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "533f00e5546da0026bbe51e1c030ddd1c19fdd08",
       "triggerType" : "PUSH"
     }, {
       "hash" : "05777be67e37f3a6a8eaa59e54d9e17e5fc6ba9b",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1016",
       "triggerID" : "05777be67e37f3a6a8eaa59e54d9e17e5fc6ba9b",
       "triggerType" : "PUSH"
     }, {
       "hash" : "a2ee90c549332a7f17a7d9b2799849b77c08a8e6",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "a2ee90c549332a7f17a7d9b2799849b77c08a8e6",
       "triggerType" : "PUSH"
     }, {
       "hash" : "05ce568b0b6ba14da9ca40d26b39ba478837d747",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1017",
       "triggerID" : "05ce568b0b6ba14da9ca40d26b39ba478837d747",
       "triggerType" : "PUSH"
     }, {
       "hash" : "840e3b33d39b1f86367bb6b0e65eadeb33de7765",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1025",
       "triggerID" : "840e3b33d39b1f86367bb6b0e65eadeb33de7765",
       "triggerType" : "PUSH"
     }, {
       "hash" : "8d2e2dd83f51d03e890d424a51d3e0e84e904a43",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1031",
       "triggerID" : "8d2e2dd83f51d03e890d424a51d3e0e84e904a43",
       "triggerType" : "PUSH"
     }, {
       "hash" : "688381914087ff11acf2f8c9372dbf39afa0bd40",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1035",
       "triggerID" : "688381914087ff11acf2f8c9372dbf39afa0bd40",
       "triggerType" : "PUSH"
     }, {
       "hash" : "fc2c138901e82ed5f3fbf363c2a0ce8140ade050",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "fc2c138901e82ed5f3fbf363c2a0ce8140ade050",
       "triggerType" : "PUSH"
     }, {
       "hash" : "6ed9e1cb1e5a191752d6d39c7a9d5d41840f0cdf",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1039",
       "triggerID" : "6ed9e1cb1e5a191752d6d39c7a9d5d41840f0cdf",
       "triggerType" : "PUSH"
     }, {
       "hash" : "a510a7de1458bb996df5bbb341e8d010e4c91381",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "a510a7de1458bb996df5bbb341e8d010e4c91381",
       "triggerType" : "PUSH"
     }, {
       "hash" : "49677397cd117228742205944606f7a789090376",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1042",
       "triggerID" : "49677397cd117228742205944606f7a789090376",
       "triggerType" : "PUSH"
     }, {
       "hash" : "1630db0a37422aed942ff280560a9d6329df5015",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1044",
       "triggerID" : "1630db0a37422aed942ff280560a9d6329df5015",
       "triggerType" : "PUSH"
     }, {
       "hash" : "f11c76301d4ab524db0af57aa831be3a898ea0fb",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1048",
       "triggerID" : "f11c76301d4ab524db0af57aa831be3a898ea0fb",
       "triggerType" : "PUSH"
     }, {
       "hash" : "d1a0dc4c2017853a7ba0c3302f886043a63f29da",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1050",
       "triggerID" : "d1a0dc4c2017853a7ba0c3302f886043a63f29da",
       "triggerType" : "PUSH"
     }, {
       "hash" : "6a4c15ec39ded1f90b113b272554217698a98e1c",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1087",
       "triggerID" : "6a4c15ec39ded1f90b113b272554217698a98e1c",
       "triggerType" : "PUSH"
     }, {
       "hash" : "288e730d645a5e58e50a87ef5f44ba11a8fb0533",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1092",
       "triggerID" : "288e730d645a5e58e50a87ef5f44ba11a8fb0533",
       "triggerType" : "PUSH"
     }, {
       "hash" : "70a5c21321a0da3a0be8d85e206c24befa7b89f1",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1095",
       "triggerID" : "70a5c21321a0da3a0be8d85e206c24befa7b89f1",
       "triggerType" : "PUSH"
     }, {
       "hash" : "f072a986114949c5e426ce64a5f5568fc718e25d",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "f072a986114949c5e426ce64a5f5568fc718e25d",
       "triggerType" : "PUSH"
     }, {
       "hash" : "45b27475b1e8806ec8c566d6f151f4bc79c8549d",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1096",
       "triggerID" : "45b27475b1e8806ec8c566d6f151f4bc79c8549d",
       "triggerType" : "PUSH"
     }, {
       "hash" : "3c62ef22f6cde28110e5eb2fca8760a008d77723",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "3c62ef22f6cde28110e5eb2fca8760a008d77723",
       "triggerType" : "PUSH"
     }, {
       "hash" : "dc785fa04e542b2ccc45494e0292f5153c7ecea8",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1097",
       "triggerID" : "dc785fa04e542b2ccc45494e0292f5153c7ecea8",
       "triggerType" : "PUSH"
     }, {
       "hash" : "585ddb8049b9b9ccb057027e265b08a1b586df8c",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "585ddb8049b9b9ccb057027e265b08a1b586df8c",
       "triggerType" : "PUSH"
     }, {
       "hash" : "b62ff8675beed2c4487c788b9ca1a0ad4271b913",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "b62ff8675beed2c4487c788b9ca1a0ad4271b913",
       "triggerType" : "PUSH"
     }, {
       "hash" : "2d46048a7402907f348cc96da2839ba1b3e4932a",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1099",
       "triggerID" : "2d46048a7402907f348cc96da2839ba1b3e4932a",
       "triggerType" : "PUSH"
     }, {
       "hash" : "004fb6ebf23f796a4b77650e0530f0f1145e8b91",
       "status" : "SUCCESS",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1101",
       "triggerID" : "004fb6ebf23f796a4b77650e0530f0f1145e8b91",
       "triggerType" : "PUSH"
     }, {
       "hash" : "af0c33333b5e9ae6d59f20a6db1d7c016ea015d3",
       "status" : "PENDING",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1165",
       "triggerID" : "af0c33333b5e9ae6d59f20a6db1d7c016ea015d3",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 9e668f65c6d4f73c76573236f8fa507dfb80edac UNKNOWN
   * 533f00e5546da0026bbe51e1c030ddd1c19fdd08 UNKNOWN
   * a2ee90c549332a7f17a7d9b2799849b77c08a8e6 UNKNOWN
   * fc2c138901e82ed5f3fbf363c2a0ce8140ade050 UNKNOWN
   * a510a7de1458bb996df5bbb341e8d010e4c91381 UNKNOWN
   * f072a986114949c5e426ce64a5f5568fc718e25d UNKNOWN
   * 3c62ef22f6cde28110e5eb2fca8760a008d77723 UNKNOWN
   * 585ddb8049b9b9ccb057027e265b08a1b586df8c UNKNOWN
   * b62ff8675beed2c4487c788b9ca1a0ad4271b913 UNKNOWN
   * 004fb6ebf23f796a4b77650e0530f0f1145e8b91 Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1101) 
   * af0c33333b5e9ae6d59f20a6db1d7c016ea015d3 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1165) 
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] [HUDI-8141] Incremental Query with Completion Time [hudi]

Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.

hudi-bot commented on PR #11947:
URL: https://github.com/apache/hudi/pull/11947#issuecomment-2417519136

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "090f256a315cbbf0c6a54d93cc2739dd693360e6",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=484",
       "triggerID" : "090f256a315cbbf0c6a54d93cc2739dd693360e6",
       "triggerType" : "PUSH"
     }, {
       "hash" : "0df5649013c1cf9e782fcbb67b8a2e7174f94512",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=985",
       "triggerID" : "0df5649013c1cf9e782fcbb67b8a2e7174f94512",
       "triggerType" : "PUSH"
     }, {
       "hash" : "d52e0128f409918644255abbe4376dc4a780bfc8",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=986",
       "triggerID" : "d52e0128f409918644255abbe4376dc4a780bfc8",
       "triggerType" : "PUSH"
     }, {
       "hash" : "9e668f65c6d4f73c76573236f8fa507dfb80edac",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "9e668f65c6d4f73c76573236f8fa507dfb80edac",
       "triggerType" : "PUSH"
     }, {
       "hash" : "f145131d524bf190986e5de600a326e32a27beca",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=987",
       "triggerID" : "f145131d524bf190986e5de600a326e32a27beca",
       "triggerType" : "PUSH"
     }, {
       "hash" : "0b008f807b42aa6fb2686922bb88d63e84a1f299",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=993",
       "triggerID" : "0b008f807b42aa6fb2686922bb88d63e84a1f299",
       "triggerType" : "PUSH"
     }, {
       "hash" : "65c01ad9f2b33d36d4890355feef91844438e732",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=995",
       "triggerID" : "65c01ad9f2b33d36d4890355feef91844438e732",
       "triggerType" : "PUSH"
     }, {
       "hash" : "955493fdc93842e8c99529cec35e39cecdbe9638",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=996",
       "triggerID" : "955493fdc93842e8c99529cec35e39cecdbe9638",
       "triggerType" : "PUSH"
     }, {
       "hash" : "533f00e5546da0026bbe51e1c030ddd1c19fdd08",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "533f00e5546da0026bbe51e1c030ddd1c19fdd08",
       "triggerType" : "PUSH"
     }, {
       "hash" : "05777be67e37f3a6a8eaa59e54d9e17e5fc6ba9b",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1016",
       "triggerID" : "05777be67e37f3a6a8eaa59e54d9e17e5fc6ba9b",
       "triggerType" : "PUSH"
     }, {
       "hash" : "a2ee90c549332a7f17a7d9b2799849b77c08a8e6",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "a2ee90c549332a7f17a7d9b2799849b77c08a8e6",
       "triggerType" : "PUSH"
     }, {
       "hash" : "05ce568b0b6ba14da9ca40d26b39ba478837d747",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1017",
       "triggerID" : "05ce568b0b6ba14da9ca40d26b39ba478837d747",
       "triggerType" : "PUSH"
     }, {
       "hash" : "840e3b33d39b1f86367bb6b0e65eadeb33de7765",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1025",
       "triggerID" : "840e3b33d39b1f86367bb6b0e65eadeb33de7765",
       "triggerType" : "PUSH"
     }, {
       "hash" : "8d2e2dd83f51d03e890d424a51d3e0e84e904a43",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1031",
       "triggerID" : "8d2e2dd83f51d03e890d424a51d3e0e84e904a43",
       "triggerType" : "PUSH"
     }, {
       "hash" : "688381914087ff11acf2f8c9372dbf39afa0bd40",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1035",
       "triggerID" : "688381914087ff11acf2f8c9372dbf39afa0bd40",
       "triggerType" : "PUSH"
     }, {
       "hash" : "fc2c138901e82ed5f3fbf363c2a0ce8140ade050",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "fc2c138901e82ed5f3fbf363c2a0ce8140ade050",
       "triggerType" : "PUSH"
     }, {
       "hash" : "6ed9e1cb1e5a191752d6d39c7a9d5d41840f0cdf",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1039",
       "triggerID" : "6ed9e1cb1e5a191752d6d39c7a9d5d41840f0cdf",
       "triggerType" : "PUSH"
     }, {
       "hash" : "a510a7de1458bb996df5bbb341e8d010e4c91381",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "a510a7de1458bb996df5bbb341e8d010e4c91381",
       "triggerType" : "PUSH"
     }, {
       "hash" : "49677397cd117228742205944606f7a789090376",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1042",
       "triggerID" : "49677397cd117228742205944606f7a789090376",
       "triggerType" : "PUSH"
     }, {
       "hash" : "1630db0a37422aed942ff280560a9d6329df5015",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1044",
       "triggerID" : "1630db0a37422aed942ff280560a9d6329df5015",
       "triggerType" : "PUSH"
     }, {
       "hash" : "f11c76301d4ab524db0af57aa831be3a898ea0fb",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1048",
       "triggerID" : "f11c76301d4ab524db0af57aa831be3a898ea0fb",
       "triggerType" : "PUSH"
     }, {
       "hash" : "d1a0dc4c2017853a7ba0c3302f886043a63f29da",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1050",
       "triggerID" : "d1a0dc4c2017853a7ba0c3302f886043a63f29da",
       "triggerType" : "PUSH"
     }, {
       "hash" : "6a4c15ec39ded1f90b113b272554217698a98e1c",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1087",
       "triggerID" : "6a4c15ec39ded1f90b113b272554217698a98e1c",
       "triggerType" : "PUSH"
     }, {
       "hash" : "288e730d645a5e58e50a87ef5f44ba11a8fb0533",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1092",
       "triggerID" : "288e730d645a5e58e50a87ef5f44ba11a8fb0533",
       "triggerType" : "PUSH"
     }, {
       "hash" : "70a5c21321a0da3a0be8d85e206c24befa7b89f1",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1095",
       "triggerID" : "70a5c21321a0da3a0be8d85e206c24befa7b89f1",
       "triggerType" : "PUSH"
     }, {
       "hash" : "f072a986114949c5e426ce64a5f5568fc718e25d",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "f072a986114949c5e426ce64a5f5568fc718e25d",
       "triggerType" : "PUSH"
     }, {
       "hash" : "45b27475b1e8806ec8c566d6f151f4bc79c8549d",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1096",
       "triggerID" : "45b27475b1e8806ec8c566d6f151f4bc79c8549d",
       "triggerType" : "PUSH"
     }, {
       "hash" : "3c62ef22f6cde28110e5eb2fca8760a008d77723",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "3c62ef22f6cde28110e5eb2fca8760a008d77723",
       "triggerType" : "PUSH"
     }, {
       "hash" : "dc785fa04e542b2ccc45494e0292f5153c7ecea8",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1097",
       "triggerID" : "dc785fa04e542b2ccc45494e0292f5153c7ecea8",
       "triggerType" : "PUSH"
     }, {
       "hash" : "585ddb8049b9b9ccb057027e265b08a1b586df8c",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "585ddb8049b9b9ccb057027e265b08a1b586df8c",
       "triggerType" : "PUSH"
     }, {
       "hash" : "b62ff8675beed2c4487c788b9ca1a0ad4271b913",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "b62ff8675beed2c4487c788b9ca1a0ad4271b913",
       "triggerType" : "PUSH"
     }, {
       "hash" : "2d46048a7402907f348cc96da2839ba1b3e4932a",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1099",
       "triggerID" : "2d46048a7402907f348cc96da2839ba1b3e4932a",
       "triggerType" : "PUSH"
     }, {
       "hash" : "004fb6ebf23f796a4b77650e0530f0f1145e8b91",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1101",
       "triggerID" : "004fb6ebf23f796a4b77650e0530f0f1145e8b91",
       "triggerType" : "PUSH"
     }, {
       "hash" : "af0c33333b5e9ae6d59f20a6db1d7c016ea015d3",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1165",
       "triggerID" : "af0c33333b5e9ae6d59f20a6db1d7c016ea015d3",
       "triggerType" : "PUSH"
     }, {
       "hash" : "da8111118d71306500229619d3b7996fcd2526e8",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "da8111118d71306500229619d3b7996fcd2526e8",
       "triggerType" : "PUSH"
     }, {
       "hash" : "83257e2b9bd3c426cf90e11003a5f244a750d7fa",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1167",
       "triggerID" : "83257e2b9bd3c426cf90e11003a5f244a750d7fa",
       "triggerType" : "PUSH"
     }, {
       "hash" : "e0de408f1b8153fdaaea4c87b27a7b69213b75b5",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1177",
       "triggerID" : "e0de408f1b8153fdaaea4c87b27a7b69213b75b5",
       "triggerType" : "PUSH"
     }, {
       "hash" : "3f7951f7ad7931f5dc0b977d7c33bc5429623535",
       "status" : "FAILURE",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1180",
       "triggerID" : "3f7951f7ad7931f5dc0b977d7c33bc5429623535",
       "triggerType" : "PUSH"
     }, {
       "hash" : "9cd6f06a753b63630e8245d51338d2bf9ab86b84",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "9cd6f06a753b63630e8245d51338d2bf9ab86b84",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 9e668f65c6d4f73c76573236f8fa507dfb80edac UNKNOWN
   * 533f00e5546da0026bbe51e1c030ddd1c19fdd08 UNKNOWN
   * a2ee90c549332a7f17a7d9b2799849b77c08a8e6 UNKNOWN
   * fc2c138901e82ed5f3fbf363c2a0ce8140ade050 UNKNOWN
   * a510a7de1458bb996df5bbb341e8d010e4c91381 UNKNOWN
   * f072a986114949c5e426ce64a5f5568fc718e25d UNKNOWN
   * 3c62ef22f6cde28110e5eb2fca8760a008d77723 UNKNOWN
   * 585ddb8049b9b9ccb057027e265b08a1b586df8c UNKNOWN
   * b62ff8675beed2c4487c788b9ca1a0ad4271b913 UNKNOWN
   * da8111118d71306500229619d3b7996fcd2526e8 UNKNOWN
   * 3f7951f7ad7931f5dc0b977d7c33bc5429623535 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1180) 
   * 9cd6f06a753b63630e8245d51338d2bf9ab86b84 UNKNOWN
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] [HUDI-8141] Incremental Query with Completion Time [hudi]

Posted by "yihua (via GitHub)" <gi...@apache.org>.

yihua commented on code in PR #11947:
URL: https://github.com/apache/hudi/pull/11947#discussion_r1802567703


##########
hudi-utilities/src/main/java/org/apache/hudi/utilities/sources/helpers/IncrSourceHelper.java:
##########
@@ -174,6 +170,64 @@ public static QueryInfo generateQueryInfo(JavaSparkContext jssc, String srcBaseP
     }
   }
 
+  public static IncrementalQueryAnalyzer getIncrementalQueryAnalyzer(
+      JavaSparkContext jssc,
+      String srcPath,
+      Option<String> lastCkptStr,
+      MissingCheckpointStrategy missingCheckpointStrategy,
+      int numInstantsFromConfig,
+      Option<SourceProfile<Integer>> latestSourceProfile) {
+    HoodieTableMetaClient metaClient = HoodieTableMetaClient.builder()
+        .setConf(HadoopFSUtils.getStorageConfWithCopy(jssc.hadoopConfiguration()))
+        .setBasePath(srcPath)
+        .setLoadActiveTimelineOnLoad(true)
+        .build();
+
+    String startTime;
+    if (lastCkptStr.isPresent() && !lastCkptStr.get().isEmpty()) {
+      startTime = lastCkptStr.get();
+    } else if (missingCheckpointStrategy != null) {
+      switch (missingCheckpointStrategy) {
+        case READ_UPTO_LATEST_COMMIT:
+          startTime = DEFAULT_BEGIN_TIMESTAMP;
+          // disrespect numInstantsFromConfig when reading up to latest
+          numInstantsFromConfig = -1;
+          break;
+        case READ_LATEST:
+          Option<HoodieInstant> lastInstant = metaClient
+              .getCommitsAndCompactionTimeline()
+              .filterCompletedInstants()
+              .lastInstant();
+          startTime = lastInstant
+              .map(hoodieInstant -> instantTimeMinusMillis(hoodieInstant.getCompletionTime(), 1))

Review Comment:
   Makes sense.  I've fixed `IncrSourceHelper#getIncrementalQueryAnalyzer` based on the semantics we discussed.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] [HUDI-8141] Incremental Query with Completion Time [hudi]

Posted by "danny0405 (via GitHub)" <gi...@apache.org>.

danny0405 commented on code in PR #11947:
URL: https://github.com/apache/hudi/pull/11947#discussion_r1804196938


##########
hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/spark/sql/hudi/streaming/HoodieStreamSource.scala:
##########
@@ -186,11 +160,11 @@ class HoodieStreamSource(
     }
   }
 
-  private def startCommitTime(startOffset: HoodieSourceOffset): String = {
+  private def startCompletionTime(startOffset: HoodieSourceOffset): String = {
     startOffset match {
-      case INIT_OFFSET => startOffset.commitTime
-      case HoodieSourceOffset(commitTime) =>
-        commitTime
+      case INIT_OFFSET => startOffset.completionTime
+      case HoodieSourceOffset(completionTime) =>
+        HoodieInstantTimeGenerator.instantTimePlusMillis(completionTime, 1)

Review Comment:
   why plus 1 millisecond?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] [HUDI-8141] Incremental Query with Completion Time [hudi]

Posted by "yihua (via GitHub)" <gi...@apache.org>.

yihua commented on code in PR #11947:
URL: https://github.com/apache/hudi/pull/11947#discussion_r1802105757


##########
hudi-common/src/main/java/org/apache/hudi/common/table/timeline/CompletionTimeQueryView.java:
##########
@@ -189,46 +193,78 @@ public Option<String> getCompletionTime(String beginTime) {
    * <p>By default, assumes there is at most 1 day time of duration for an instant to accelerate the queries.
    *
    * @param timeline The timeline.
-   * @param rangeStart   The query range start completion time.
-   * @param rangeEnd     The query range end completion time.
+   * @param rangeStartCompletionTime   The query range start completion time.
+   * @param rangeEndCompletionTime     The query range end completion time.
    * @param rangeType    The range type.
    *
    * @return The sorted instant time list.
    */
   public List<String> getStartTimes(
       HoodieTimeline timeline,
-      Option<String> rangeStart,
-      Option<String> rangeEnd,
+      Option<String> rangeStartCompletionTime,

Review Comment:
   Yes, this is a better name.  I'll fix this one and other places changed in this PR.  I have filed a follow-up to make the naming consistent across the code base: HUDI-8373.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] [HUDI-8141] Incremental Query with Completion Time [hudi]

Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.

hudi-bot commented on PR #11947:
URL: https://github.com/apache/hudi/pull/11947#issuecomment-2416050987

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "090f256a315cbbf0c6a54d93cc2739dd693360e6",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=484",
       "triggerID" : "090f256a315cbbf0c6a54d93cc2739dd693360e6",
       "triggerType" : "PUSH"
     }, {
       "hash" : "0df5649013c1cf9e782fcbb67b8a2e7174f94512",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=985",
       "triggerID" : "0df5649013c1cf9e782fcbb67b8a2e7174f94512",
       "triggerType" : "PUSH"
     }, {
       "hash" : "d52e0128f409918644255abbe4376dc4a780bfc8",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=986",
       "triggerID" : "d52e0128f409918644255abbe4376dc4a780bfc8",
       "triggerType" : "PUSH"
     }, {
       "hash" : "9e668f65c6d4f73c76573236f8fa507dfb80edac",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "9e668f65c6d4f73c76573236f8fa507dfb80edac",
       "triggerType" : "PUSH"
     }, {
       "hash" : "f145131d524bf190986e5de600a326e32a27beca",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=987",
       "triggerID" : "f145131d524bf190986e5de600a326e32a27beca",
       "triggerType" : "PUSH"
     }, {
       "hash" : "0b008f807b42aa6fb2686922bb88d63e84a1f299",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=993",
       "triggerID" : "0b008f807b42aa6fb2686922bb88d63e84a1f299",
       "triggerType" : "PUSH"
     }, {
       "hash" : "65c01ad9f2b33d36d4890355feef91844438e732",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=995",
       "triggerID" : "65c01ad9f2b33d36d4890355feef91844438e732",
       "triggerType" : "PUSH"
     }, {
       "hash" : "955493fdc93842e8c99529cec35e39cecdbe9638",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=996",
       "triggerID" : "955493fdc93842e8c99529cec35e39cecdbe9638",
       "triggerType" : "PUSH"
     }, {
       "hash" : "533f00e5546da0026bbe51e1c030ddd1c19fdd08",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "533f00e5546da0026bbe51e1c030ddd1c19fdd08",
       "triggerType" : "PUSH"
     }, {
       "hash" : "05777be67e37f3a6a8eaa59e54d9e17e5fc6ba9b",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1016",
       "triggerID" : "05777be67e37f3a6a8eaa59e54d9e17e5fc6ba9b",
       "triggerType" : "PUSH"
     }, {
       "hash" : "a2ee90c549332a7f17a7d9b2799849b77c08a8e6",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "a2ee90c549332a7f17a7d9b2799849b77c08a8e6",
       "triggerType" : "PUSH"
     }, {
       "hash" : "05ce568b0b6ba14da9ca40d26b39ba478837d747",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1017",
       "triggerID" : "05ce568b0b6ba14da9ca40d26b39ba478837d747",
       "triggerType" : "PUSH"
     }, {
       "hash" : "840e3b33d39b1f86367bb6b0e65eadeb33de7765",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1025",
       "triggerID" : "840e3b33d39b1f86367bb6b0e65eadeb33de7765",
       "triggerType" : "PUSH"
     }, {
       "hash" : "8d2e2dd83f51d03e890d424a51d3e0e84e904a43",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1031",
       "triggerID" : "8d2e2dd83f51d03e890d424a51d3e0e84e904a43",
       "triggerType" : "PUSH"
     }, {
       "hash" : "688381914087ff11acf2f8c9372dbf39afa0bd40",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1035",
       "triggerID" : "688381914087ff11acf2f8c9372dbf39afa0bd40",
       "triggerType" : "PUSH"
     }, {
       "hash" : "fc2c138901e82ed5f3fbf363c2a0ce8140ade050",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "fc2c138901e82ed5f3fbf363c2a0ce8140ade050",
       "triggerType" : "PUSH"
     }, {
       "hash" : "6ed9e1cb1e5a191752d6d39c7a9d5d41840f0cdf",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1039",
       "triggerID" : "6ed9e1cb1e5a191752d6d39c7a9d5d41840f0cdf",
       "triggerType" : "PUSH"
     }, {
       "hash" : "a510a7de1458bb996df5bbb341e8d010e4c91381",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "a510a7de1458bb996df5bbb341e8d010e4c91381",
       "triggerType" : "PUSH"
     }, {
       "hash" : "49677397cd117228742205944606f7a789090376",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1042",
       "triggerID" : "49677397cd117228742205944606f7a789090376",
       "triggerType" : "PUSH"
     }, {
       "hash" : "1630db0a37422aed942ff280560a9d6329df5015",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1044",
       "triggerID" : "1630db0a37422aed942ff280560a9d6329df5015",
       "triggerType" : "PUSH"
     }, {
       "hash" : "f11c76301d4ab524db0af57aa831be3a898ea0fb",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1048",
       "triggerID" : "f11c76301d4ab524db0af57aa831be3a898ea0fb",
       "triggerType" : "PUSH"
     }, {
       "hash" : "d1a0dc4c2017853a7ba0c3302f886043a63f29da",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1050",
       "triggerID" : "d1a0dc4c2017853a7ba0c3302f886043a63f29da",
       "triggerType" : "PUSH"
     }, {
       "hash" : "6a4c15ec39ded1f90b113b272554217698a98e1c",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1087",
       "triggerID" : "6a4c15ec39ded1f90b113b272554217698a98e1c",
       "triggerType" : "PUSH"
     }, {
       "hash" : "288e730d645a5e58e50a87ef5f44ba11a8fb0533",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1092",
       "triggerID" : "288e730d645a5e58e50a87ef5f44ba11a8fb0533",
       "triggerType" : "PUSH"
     }, {
       "hash" : "70a5c21321a0da3a0be8d85e206c24befa7b89f1",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1095",
       "triggerID" : "70a5c21321a0da3a0be8d85e206c24befa7b89f1",
       "triggerType" : "PUSH"
     }, {
       "hash" : "f072a986114949c5e426ce64a5f5568fc718e25d",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "f072a986114949c5e426ce64a5f5568fc718e25d",
       "triggerType" : "PUSH"
     }, {
       "hash" : "45b27475b1e8806ec8c566d6f151f4bc79c8549d",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1096",
       "triggerID" : "45b27475b1e8806ec8c566d6f151f4bc79c8549d",
       "triggerType" : "PUSH"
     }, {
       "hash" : "3c62ef22f6cde28110e5eb2fca8760a008d77723",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "3c62ef22f6cde28110e5eb2fca8760a008d77723",
       "triggerType" : "PUSH"
     }, {
       "hash" : "dc785fa04e542b2ccc45494e0292f5153c7ecea8",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1097",
       "triggerID" : "dc785fa04e542b2ccc45494e0292f5153c7ecea8",
       "triggerType" : "PUSH"
     }, {
       "hash" : "585ddb8049b9b9ccb057027e265b08a1b586df8c",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "585ddb8049b9b9ccb057027e265b08a1b586df8c",
       "triggerType" : "PUSH"
     }, {
       "hash" : "b62ff8675beed2c4487c788b9ca1a0ad4271b913",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "b62ff8675beed2c4487c788b9ca1a0ad4271b913",
       "triggerType" : "PUSH"
     }, {
       "hash" : "2d46048a7402907f348cc96da2839ba1b3e4932a",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1099",
       "triggerID" : "2d46048a7402907f348cc96da2839ba1b3e4932a",
       "triggerType" : "PUSH"
     }, {
       "hash" : "004fb6ebf23f796a4b77650e0530f0f1145e8b91",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1101",
       "triggerID" : "004fb6ebf23f796a4b77650e0530f0f1145e8b91",
       "triggerType" : "PUSH"
     }, {
       "hash" : "af0c33333b5e9ae6d59f20a6db1d7c016ea015d3",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1165",
       "triggerID" : "af0c33333b5e9ae6d59f20a6db1d7c016ea015d3",
       "triggerType" : "PUSH"
     }, {
       "hash" : "da8111118d71306500229619d3b7996fcd2526e8",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "da8111118d71306500229619d3b7996fcd2526e8",
       "triggerType" : "PUSH"
     }, {
       "hash" : "83257e2b9bd3c426cf90e11003a5f244a750d7fa",
       "status" : "FAILURE",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1167",
       "triggerID" : "83257e2b9bd3c426cf90e11003a5f244a750d7fa",
       "triggerType" : "PUSH"
     }, {
       "hash" : "e0de408f1b8153fdaaea4c87b27a7b69213b75b5",
       "status" : "PENDING",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1177",
       "triggerID" : "e0de408f1b8153fdaaea4c87b27a7b69213b75b5",
       "triggerType" : "PUSH"
     }, {
       "hash" : "3f7951f7ad7931f5dc0b977d7c33bc5429623535",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "3f7951f7ad7931f5dc0b977d7c33bc5429623535",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 9e668f65c6d4f73c76573236f8fa507dfb80edac UNKNOWN
   * 533f00e5546da0026bbe51e1c030ddd1c19fdd08 UNKNOWN
   * a2ee90c549332a7f17a7d9b2799849b77c08a8e6 UNKNOWN
   * fc2c138901e82ed5f3fbf363c2a0ce8140ade050 UNKNOWN
   * a510a7de1458bb996df5bbb341e8d010e4c91381 UNKNOWN
   * f072a986114949c5e426ce64a5f5568fc718e25d UNKNOWN
   * 3c62ef22f6cde28110e5eb2fca8760a008d77723 UNKNOWN
   * 585ddb8049b9b9ccb057027e265b08a1b586df8c UNKNOWN
   * b62ff8675beed2c4487c788b9ca1a0ad4271b913 UNKNOWN
   * da8111118d71306500229619d3b7996fcd2526e8 UNKNOWN
   * 83257e2b9bd3c426cf90e11003a5f244a750d7fa Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1167) 
   * e0de408f1b8153fdaaea4c87b27a7b69213b75b5 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1177) 
   * 3f7951f7ad7931f5dc0b977d7c33bc5429623535 UNKNOWN
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] [HUDI-8141] Incremental Query with Completion Time [hudi]

Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.

hudi-bot commented on PR #11947:
URL: https://github.com/apache/hudi/pull/11947#issuecomment-2417610469

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "090f256a315cbbf0c6a54d93cc2739dd693360e6",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=484",
       "triggerID" : "090f256a315cbbf0c6a54d93cc2739dd693360e6",
       "triggerType" : "PUSH"
     }, {
       "hash" : "0df5649013c1cf9e782fcbb67b8a2e7174f94512",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=985",
       "triggerID" : "0df5649013c1cf9e782fcbb67b8a2e7174f94512",
       "triggerType" : "PUSH"
     }, {
       "hash" : "d52e0128f409918644255abbe4376dc4a780bfc8",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=986",
       "triggerID" : "d52e0128f409918644255abbe4376dc4a780bfc8",
       "triggerType" : "PUSH"
     }, {
       "hash" : "9e668f65c6d4f73c76573236f8fa507dfb80edac",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "9e668f65c6d4f73c76573236f8fa507dfb80edac",
       "triggerType" : "PUSH"
     }, {
       "hash" : "f145131d524bf190986e5de600a326e32a27beca",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=987",
       "triggerID" : "f145131d524bf190986e5de600a326e32a27beca",
       "triggerType" : "PUSH"
     }, {
       "hash" : "0b008f807b42aa6fb2686922bb88d63e84a1f299",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=993",
       "triggerID" : "0b008f807b42aa6fb2686922bb88d63e84a1f299",
       "triggerType" : "PUSH"
     }, {
       "hash" : "65c01ad9f2b33d36d4890355feef91844438e732",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=995",
       "triggerID" : "65c01ad9f2b33d36d4890355feef91844438e732",
       "triggerType" : "PUSH"
     }, {
       "hash" : "955493fdc93842e8c99529cec35e39cecdbe9638",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=996",
       "triggerID" : "955493fdc93842e8c99529cec35e39cecdbe9638",
       "triggerType" : "PUSH"
     }, {
       "hash" : "533f00e5546da0026bbe51e1c030ddd1c19fdd08",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "533f00e5546da0026bbe51e1c030ddd1c19fdd08",
       "triggerType" : "PUSH"
     }, {
       "hash" : "05777be67e37f3a6a8eaa59e54d9e17e5fc6ba9b",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1016",
       "triggerID" : "05777be67e37f3a6a8eaa59e54d9e17e5fc6ba9b",
       "triggerType" : "PUSH"
     }, {
       "hash" : "a2ee90c549332a7f17a7d9b2799849b77c08a8e6",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "a2ee90c549332a7f17a7d9b2799849b77c08a8e6",
       "triggerType" : "PUSH"
     }, {
       "hash" : "05ce568b0b6ba14da9ca40d26b39ba478837d747",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1017",
       "triggerID" : "05ce568b0b6ba14da9ca40d26b39ba478837d747",
       "triggerType" : "PUSH"
     }, {
       "hash" : "840e3b33d39b1f86367bb6b0e65eadeb33de7765",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1025",
       "triggerID" : "840e3b33d39b1f86367bb6b0e65eadeb33de7765",
       "triggerType" : "PUSH"
     }, {
       "hash" : "8d2e2dd83f51d03e890d424a51d3e0e84e904a43",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1031",
       "triggerID" : "8d2e2dd83f51d03e890d424a51d3e0e84e904a43",
       "triggerType" : "PUSH"
     }, {
       "hash" : "688381914087ff11acf2f8c9372dbf39afa0bd40",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1035",
       "triggerID" : "688381914087ff11acf2f8c9372dbf39afa0bd40",
       "triggerType" : "PUSH"
     }, {
       "hash" : "fc2c138901e82ed5f3fbf363c2a0ce8140ade050",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "fc2c138901e82ed5f3fbf363c2a0ce8140ade050",
       "triggerType" : "PUSH"
     }, {
       "hash" : "6ed9e1cb1e5a191752d6d39c7a9d5d41840f0cdf",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1039",
       "triggerID" : "6ed9e1cb1e5a191752d6d39c7a9d5d41840f0cdf",
       "triggerType" : "PUSH"
     }, {
       "hash" : "a510a7de1458bb996df5bbb341e8d010e4c91381",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "a510a7de1458bb996df5bbb341e8d010e4c91381",
       "triggerType" : "PUSH"
     }, {
       "hash" : "49677397cd117228742205944606f7a789090376",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1042",
       "triggerID" : "49677397cd117228742205944606f7a789090376",
       "triggerType" : "PUSH"
     }, {
       "hash" : "1630db0a37422aed942ff280560a9d6329df5015",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1044",
       "triggerID" : "1630db0a37422aed942ff280560a9d6329df5015",
       "triggerType" : "PUSH"
     }, {
       "hash" : "f11c76301d4ab524db0af57aa831be3a898ea0fb",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1048",
       "triggerID" : "f11c76301d4ab524db0af57aa831be3a898ea0fb",
       "triggerType" : "PUSH"
     }, {
       "hash" : "d1a0dc4c2017853a7ba0c3302f886043a63f29da",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1050",
       "triggerID" : "d1a0dc4c2017853a7ba0c3302f886043a63f29da",
       "triggerType" : "PUSH"
     }, {
       "hash" : "6a4c15ec39ded1f90b113b272554217698a98e1c",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1087",
       "triggerID" : "6a4c15ec39ded1f90b113b272554217698a98e1c",
       "triggerType" : "PUSH"
     }, {
       "hash" : "288e730d645a5e58e50a87ef5f44ba11a8fb0533",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1092",
       "triggerID" : "288e730d645a5e58e50a87ef5f44ba11a8fb0533",
       "triggerType" : "PUSH"
     }, {
       "hash" : "70a5c21321a0da3a0be8d85e206c24befa7b89f1",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1095",
       "triggerID" : "70a5c21321a0da3a0be8d85e206c24befa7b89f1",
       "triggerType" : "PUSH"
     }, {
       "hash" : "f072a986114949c5e426ce64a5f5568fc718e25d",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "f072a986114949c5e426ce64a5f5568fc718e25d",
       "triggerType" : "PUSH"
     }, {
       "hash" : "45b27475b1e8806ec8c566d6f151f4bc79c8549d",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1096",
       "triggerID" : "45b27475b1e8806ec8c566d6f151f4bc79c8549d",
       "triggerType" : "PUSH"
     }, {
       "hash" : "3c62ef22f6cde28110e5eb2fca8760a008d77723",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "3c62ef22f6cde28110e5eb2fca8760a008d77723",
       "triggerType" : "PUSH"
     }, {
       "hash" : "dc785fa04e542b2ccc45494e0292f5153c7ecea8",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1097",
       "triggerID" : "dc785fa04e542b2ccc45494e0292f5153c7ecea8",
       "triggerType" : "PUSH"
     }, {
       "hash" : "585ddb8049b9b9ccb057027e265b08a1b586df8c",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "585ddb8049b9b9ccb057027e265b08a1b586df8c",
       "triggerType" : "PUSH"
     }, {
       "hash" : "b62ff8675beed2c4487c788b9ca1a0ad4271b913",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "b62ff8675beed2c4487c788b9ca1a0ad4271b913",
       "triggerType" : "PUSH"
     }, {
       "hash" : "2d46048a7402907f348cc96da2839ba1b3e4932a",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1099",
       "triggerID" : "2d46048a7402907f348cc96da2839ba1b3e4932a",
       "triggerType" : "PUSH"
     }, {
       "hash" : "004fb6ebf23f796a4b77650e0530f0f1145e8b91",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1101",
       "triggerID" : "004fb6ebf23f796a4b77650e0530f0f1145e8b91",
       "triggerType" : "PUSH"
     }, {
       "hash" : "af0c33333b5e9ae6d59f20a6db1d7c016ea015d3",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1165",
       "triggerID" : "af0c33333b5e9ae6d59f20a6db1d7c016ea015d3",
       "triggerType" : "PUSH"
     }, {
       "hash" : "da8111118d71306500229619d3b7996fcd2526e8",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "da8111118d71306500229619d3b7996fcd2526e8",
       "triggerType" : "PUSH"
     }, {
       "hash" : "83257e2b9bd3c426cf90e11003a5f244a750d7fa",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1167",
       "triggerID" : "83257e2b9bd3c426cf90e11003a5f244a750d7fa",
       "triggerType" : "PUSH"
     }, {
       "hash" : "e0de408f1b8153fdaaea4c87b27a7b69213b75b5",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1177",
       "triggerID" : "e0de408f1b8153fdaaea4c87b27a7b69213b75b5",
       "triggerType" : "PUSH"
     }, {
       "hash" : "3f7951f7ad7931f5dc0b977d7c33bc5429623535",
       "status" : "FAILURE",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1180",
       "triggerID" : "3f7951f7ad7931f5dc0b977d7c33bc5429623535",
       "triggerType" : "PUSH"
     }, {
       "hash" : "9cd6f06a753b63630e8245d51338d2bf9ab86b84",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "9cd6f06a753b63630e8245d51338d2bf9ab86b84",
       "triggerType" : "PUSH"
     }, {
       "hash" : "7380b902731a92e0645c02397e14591a05798620",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "7380b902731a92e0645c02397e14591a05798620",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 9e668f65c6d4f73c76573236f8fa507dfb80edac UNKNOWN
   * 533f00e5546da0026bbe51e1c030ddd1c19fdd08 UNKNOWN
   * a2ee90c549332a7f17a7d9b2799849b77c08a8e6 UNKNOWN
   * fc2c138901e82ed5f3fbf363c2a0ce8140ade050 UNKNOWN
   * a510a7de1458bb996df5bbb341e8d010e4c91381 UNKNOWN
   * f072a986114949c5e426ce64a5f5568fc718e25d UNKNOWN
   * 3c62ef22f6cde28110e5eb2fca8760a008d77723 UNKNOWN
   * 585ddb8049b9b9ccb057027e265b08a1b586df8c UNKNOWN
   * b62ff8675beed2c4487c788b9ca1a0ad4271b913 UNKNOWN
   * da8111118d71306500229619d3b7996fcd2526e8 UNKNOWN
   * 3f7951f7ad7931f5dc0b977d7c33bc5429623535 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1180) 
   * 9cd6f06a753b63630e8245d51338d2bf9ab86b84 UNKNOWN
   * 7380b902731a92e0645c02397e14591a05798620 UNKNOWN
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] [HUDI-8141] Incremental Query with Completion Time [hudi]

Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.

hudi-bot commented on PR #11947:
URL: https://github.com/apache/hudi/pull/11947#issuecomment-2418072211

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "090f256a315cbbf0c6a54d93cc2739dd693360e6",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=484",
       "triggerID" : "090f256a315cbbf0c6a54d93cc2739dd693360e6",
       "triggerType" : "PUSH"
     }, {
       "hash" : "0df5649013c1cf9e782fcbb67b8a2e7174f94512",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=985",
       "triggerID" : "0df5649013c1cf9e782fcbb67b8a2e7174f94512",
       "triggerType" : "PUSH"
     }, {
       "hash" : "d52e0128f409918644255abbe4376dc4a780bfc8",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=986",
       "triggerID" : "d52e0128f409918644255abbe4376dc4a780bfc8",
       "triggerType" : "PUSH"
     }, {
       "hash" : "9e668f65c6d4f73c76573236f8fa507dfb80edac",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "9e668f65c6d4f73c76573236f8fa507dfb80edac",
       "triggerType" : "PUSH"
     }, {
       "hash" : "f145131d524bf190986e5de600a326e32a27beca",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=987",
       "triggerID" : "f145131d524bf190986e5de600a326e32a27beca",
       "triggerType" : "PUSH"
     }, {
       "hash" : "0b008f807b42aa6fb2686922bb88d63e84a1f299",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=993",
       "triggerID" : "0b008f807b42aa6fb2686922bb88d63e84a1f299",
       "triggerType" : "PUSH"
     }, {
       "hash" : "65c01ad9f2b33d36d4890355feef91844438e732",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=995",
       "triggerID" : "65c01ad9f2b33d36d4890355feef91844438e732",
       "triggerType" : "PUSH"
     }, {
       "hash" : "955493fdc93842e8c99529cec35e39cecdbe9638",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=996",
       "triggerID" : "955493fdc93842e8c99529cec35e39cecdbe9638",
       "triggerType" : "PUSH"
     }, {
       "hash" : "533f00e5546da0026bbe51e1c030ddd1c19fdd08",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "533f00e5546da0026bbe51e1c030ddd1c19fdd08",
       "triggerType" : "PUSH"
     }, {
       "hash" : "05777be67e37f3a6a8eaa59e54d9e17e5fc6ba9b",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1016",
       "triggerID" : "05777be67e37f3a6a8eaa59e54d9e17e5fc6ba9b",
       "triggerType" : "PUSH"
     }, {
       "hash" : "a2ee90c549332a7f17a7d9b2799849b77c08a8e6",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "a2ee90c549332a7f17a7d9b2799849b77c08a8e6",
       "triggerType" : "PUSH"
     }, {
       "hash" : "05ce568b0b6ba14da9ca40d26b39ba478837d747",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1017",
       "triggerID" : "05ce568b0b6ba14da9ca40d26b39ba478837d747",
       "triggerType" : "PUSH"
     }, {
       "hash" : "840e3b33d39b1f86367bb6b0e65eadeb33de7765",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1025",
       "triggerID" : "840e3b33d39b1f86367bb6b0e65eadeb33de7765",
       "triggerType" : "PUSH"
     }, {
       "hash" : "8d2e2dd83f51d03e890d424a51d3e0e84e904a43",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1031",
       "triggerID" : "8d2e2dd83f51d03e890d424a51d3e0e84e904a43",
       "triggerType" : "PUSH"
     }, {
       "hash" : "688381914087ff11acf2f8c9372dbf39afa0bd40",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1035",
       "triggerID" : "688381914087ff11acf2f8c9372dbf39afa0bd40",
       "triggerType" : "PUSH"
     }, {
       "hash" : "fc2c138901e82ed5f3fbf363c2a0ce8140ade050",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "fc2c138901e82ed5f3fbf363c2a0ce8140ade050",
       "triggerType" : "PUSH"
     }, {
       "hash" : "6ed9e1cb1e5a191752d6d39c7a9d5d41840f0cdf",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1039",
       "triggerID" : "6ed9e1cb1e5a191752d6d39c7a9d5d41840f0cdf",
       "triggerType" : "PUSH"
     }, {
       "hash" : "a510a7de1458bb996df5bbb341e8d010e4c91381",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "a510a7de1458bb996df5bbb341e8d010e4c91381",
       "triggerType" : "PUSH"
     }, {
       "hash" : "49677397cd117228742205944606f7a789090376",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1042",
       "triggerID" : "49677397cd117228742205944606f7a789090376",
       "triggerType" : "PUSH"
     }, {
       "hash" : "1630db0a37422aed942ff280560a9d6329df5015",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1044",
       "triggerID" : "1630db0a37422aed942ff280560a9d6329df5015",
       "triggerType" : "PUSH"
     }, {
       "hash" : "f11c76301d4ab524db0af57aa831be3a898ea0fb",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1048",
       "triggerID" : "f11c76301d4ab524db0af57aa831be3a898ea0fb",
       "triggerType" : "PUSH"
     }, {
       "hash" : "d1a0dc4c2017853a7ba0c3302f886043a63f29da",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1050",
       "triggerID" : "d1a0dc4c2017853a7ba0c3302f886043a63f29da",
       "triggerType" : "PUSH"
     }, {
       "hash" : "6a4c15ec39ded1f90b113b272554217698a98e1c",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1087",
       "triggerID" : "6a4c15ec39ded1f90b113b272554217698a98e1c",
       "triggerType" : "PUSH"
     }, {
       "hash" : "288e730d645a5e58e50a87ef5f44ba11a8fb0533",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1092",
       "triggerID" : "288e730d645a5e58e50a87ef5f44ba11a8fb0533",
       "triggerType" : "PUSH"
     }, {
       "hash" : "70a5c21321a0da3a0be8d85e206c24befa7b89f1",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1095",
       "triggerID" : "70a5c21321a0da3a0be8d85e206c24befa7b89f1",
       "triggerType" : "PUSH"
     }, {
       "hash" : "f072a986114949c5e426ce64a5f5568fc718e25d",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "f072a986114949c5e426ce64a5f5568fc718e25d",
       "triggerType" : "PUSH"
     }, {
       "hash" : "45b27475b1e8806ec8c566d6f151f4bc79c8549d",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1096",
       "triggerID" : "45b27475b1e8806ec8c566d6f151f4bc79c8549d",
       "triggerType" : "PUSH"
     }, {
       "hash" : "3c62ef22f6cde28110e5eb2fca8760a008d77723",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "3c62ef22f6cde28110e5eb2fca8760a008d77723",
       "triggerType" : "PUSH"
     }, {
       "hash" : "dc785fa04e542b2ccc45494e0292f5153c7ecea8",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1097",
       "triggerID" : "dc785fa04e542b2ccc45494e0292f5153c7ecea8",
       "triggerType" : "PUSH"
     }, {
       "hash" : "585ddb8049b9b9ccb057027e265b08a1b586df8c",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "585ddb8049b9b9ccb057027e265b08a1b586df8c",
       "triggerType" : "PUSH"
     }, {
       "hash" : "b62ff8675beed2c4487c788b9ca1a0ad4271b913",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "b62ff8675beed2c4487c788b9ca1a0ad4271b913",
       "triggerType" : "PUSH"
     }, {
       "hash" : "2d46048a7402907f348cc96da2839ba1b3e4932a",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1099",
       "triggerID" : "2d46048a7402907f348cc96da2839ba1b3e4932a",
       "triggerType" : "PUSH"
     }, {
       "hash" : "004fb6ebf23f796a4b77650e0530f0f1145e8b91",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1101",
       "triggerID" : "004fb6ebf23f796a4b77650e0530f0f1145e8b91",
       "triggerType" : "PUSH"
     }, {
       "hash" : "af0c33333b5e9ae6d59f20a6db1d7c016ea015d3",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1165",
       "triggerID" : "af0c33333b5e9ae6d59f20a6db1d7c016ea015d3",
       "triggerType" : "PUSH"
     }, {
       "hash" : "da8111118d71306500229619d3b7996fcd2526e8",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "da8111118d71306500229619d3b7996fcd2526e8",
       "triggerType" : "PUSH"
     }, {
       "hash" : "83257e2b9bd3c426cf90e11003a5f244a750d7fa",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1167",
       "triggerID" : "83257e2b9bd3c426cf90e11003a5f244a750d7fa",
       "triggerType" : "PUSH"
     }, {
       "hash" : "e0de408f1b8153fdaaea4c87b27a7b69213b75b5",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1177",
       "triggerID" : "e0de408f1b8153fdaaea4c87b27a7b69213b75b5",
       "triggerType" : "PUSH"
     }, {
       "hash" : "3f7951f7ad7931f5dc0b977d7c33bc5429623535",
       "status" : "FAILURE",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1180",
       "triggerID" : "3f7951f7ad7931f5dc0b977d7c33bc5429623535",
       "triggerType" : "PUSH"
     }, {
       "hash" : "9cd6f06a753b63630e8245d51338d2bf9ab86b84",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "9cd6f06a753b63630e8245d51338d2bf9ab86b84",
       "triggerType" : "PUSH"
     }, {
       "hash" : "7380b902731a92e0645c02397e14591a05798620",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "7380b902731a92e0645c02397e14591a05798620",
       "triggerType" : "PUSH"
     }, {
       "hash" : "0bd5cca01a541845081d5430d48b021d4bdf0969",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "0bd5cca01a541845081d5430d48b021d4bdf0969",
       "triggerType" : "PUSH"
     }, {
       "hash" : "8f4e175ef554f015e34e0968b656b5de714ab430",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "8f4e175ef554f015e34e0968b656b5de714ab430",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 9e668f65c6d4f73c76573236f8fa507dfb80edac UNKNOWN
   * 533f00e5546da0026bbe51e1c030ddd1c19fdd08 UNKNOWN
   * a2ee90c549332a7f17a7d9b2799849b77c08a8e6 UNKNOWN
   * fc2c138901e82ed5f3fbf363c2a0ce8140ade050 UNKNOWN
   * a510a7de1458bb996df5bbb341e8d010e4c91381 UNKNOWN
   * f072a986114949c5e426ce64a5f5568fc718e25d UNKNOWN
   * 3c62ef22f6cde28110e5eb2fca8760a008d77723 UNKNOWN
   * 585ddb8049b9b9ccb057027e265b08a1b586df8c UNKNOWN
   * b62ff8675beed2c4487c788b9ca1a0ad4271b913 UNKNOWN
   * da8111118d71306500229619d3b7996fcd2526e8 UNKNOWN
   * 3f7951f7ad7931f5dc0b977d7c33bc5429623535 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1180) 
   * 9cd6f06a753b63630e8245d51338d2bf9ab86b84 UNKNOWN
   * 7380b902731a92e0645c02397e14591a05798620 UNKNOWN
   * 0bd5cca01a541845081d5430d48b021d4bdf0969 UNKNOWN
   * 8f4e175ef554f015e34e0968b656b5de714ab430 UNKNOWN
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] [HUDI-8141] Incremental Query with Completion Time [hudi]

Posted by "yihua (via GitHub)" <gi...@apache.org>.

yihua commented on code in PR #11947:
URL: https://github.com/apache/hudi/pull/11947#discussion_r1802135861


##########
hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/hudi/IncrementalRelation.scala:
##########
@@ -90,24 +78,24 @@ class IncrementalRelation(val sqlContext: SQLContext,
     throw new HoodieException("Incremental queries are not supported when meta fields are disabled")
   }
 
+  private val queryContext: IncrementalQueryAnalyzer.QueryContext =
+    IncrementalQueryAnalyzer.builder()
+      .metaClient(metaClient)
+      .startTime(optParams(DataSourceReadOptions.BEGIN_INSTANTTIME.key))
+      .endTime(optParams.getOrElse(DataSourceReadOptions.END_INSTANTTIME.key, null))
+      .rangeType(InstantRange.RangeType.OPEN_CLOSED)
+      .build()
+      .analyze()
+
+  private val commitsToReturn = List.concat(
+    queryContext.getArchivedInstants.asScala,
+    queryContext.getActiveInstants.asScala)

Review Comment:
   `queryContext.instants` contains only the instant times.  There is one place below that needs to get the replaced instants from `commitsToReturn` based on the action type, so `queryContext.getArchivedInstants` and `queryContext.getActiveInstants` are used.  Another way is to make `queryContext.instants` store `HoodieInstant` instead of `String` of instant times.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] [HUDI-8141] Incremental Query with Completion Time [hudi]

Posted by "danny0405 (via GitHub)" <gi...@apache.org>.

danny0405 commented on code in PR #11947:
URL: https://github.com/apache/hudi/pull/11947#discussion_r1805712611


##########
hudi-utilities/src/main/java/org/apache/hudi/utilities/sources/SnapshotLoadQuerySplitter.java:
##########
@@ -118,6 +124,32 @@ public QueryInfo getNextCheckpoint(Dataset<Row> df, QueryInfo queryInfo, Option<
         .orElse(queryInfo);
   }
 
+  public Option<CheckpointWithPredicates> getNextCheckpoint(Dataset<Row> df, QueryContext queryContext,
+                                                            Option<SourceProfileSupplier> sourceProfileSupplier) {
+    // the start instant would be included into the final query result. So we need to get
+    // a strictly lower timestamp to have query splitter include the start instant
+    Option<CheckpointWithPredicates> nextCheckpointWithPredicates =
+        getNextCheckpointWithPredicates(df, instantTimeMinusMillis(queryContext.getBeginInstant().get(), 1));
+    if (nextCheckpointWithPredicates.isPresent()) {
+      // getNextCheckpointWithPredicates is based on instant times,
+      // so we need to translate the instant time to the completion time
+      String endInstantTime = nextCheckpointWithPredicates.get().getEndInstant();
+      Option<String> endCompletionTime = Option.fromJavaOptional(queryContext.getInstants().stream()
+          .filter(instant -> endInstantTime.equals(instant.getTimestamp()))

Review Comment:
   Isn't it just the max completion time from query context?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] [HUDI-8141] Incremental Query with Completion Time [hudi]

Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.

hudi-bot commented on PR #11947:
URL: https://github.com/apache/hudi/pull/11947#issuecomment-2421539138

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "090f256a315cbbf0c6a54d93cc2739dd693360e6",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=484",
       "triggerID" : "090f256a315cbbf0c6a54d93cc2739dd693360e6",
       "triggerType" : "PUSH"
     }, {
       "hash" : "0df5649013c1cf9e782fcbb67b8a2e7174f94512",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=985",
       "triggerID" : "0df5649013c1cf9e782fcbb67b8a2e7174f94512",
       "triggerType" : "PUSH"
     }, {
       "hash" : "d52e0128f409918644255abbe4376dc4a780bfc8",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=986",
       "triggerID" : "d52e0128f409918644255abbe4376dc4a780bfc8",
       "triggerType" : "PUSH"
     }, {
       "hash" : "9e668f65c6d4f73c76573236f8fa507dfb80edac",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "9e668f65c6d4f73c76573236f8fa507dfb80edac",
       "triggerType" : "PUSH"
     }, {
       "hash" : "f145131d524bf190986e5de600a326e32a27beca",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=987",
       "triggerID" : "f145131d524bf190986e5de600a326e32a27beca",
       "triggerType" : "PUSH"
     }, {
       "hash" : "0b008f807b42aa6fb2686922bb88d63e84a1f299",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=993",
       "triggerID" : "0b008f807b42aa6fb2686922bb88d63e84a1f299",
       "triggerType" : "PUSH"
     }, {
       "hash" : "65c01ad9f2b33d36d4890355feef91844438e732",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=995",
       "triggerID" : "65c01ad9f2b33d36d4890355feef91844438e732",
       "triggerType" : "PUSH"
     }, {
       "hash" : "955493fdc93842e8c99529cec35e39cecdbe9638",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=996",
       "triggerID" : "955493fdc93842e8c99529cec35e39cecdbe9638",
       "triggerType" : "PUSH"
     }, {
       "hash" : "533f00e5546da0026bbe51e1c030ddd1c19fdd08",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "533f00e5546da0026bbe51e1c030ddd1c19fdd08",
       "triggerType" : "PUSH"
     }, {
       "hash" : "05777be67e37f3a6a8eaa59e54d9e17e5fc6ba9b",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1016",
       "triggerID" : "05777be67e37f3a6a8eaa59e54d9e17e5fc6ba9b",
       "triggerType" : "PUSH"
     }, {
       "hash" : "a2ee90c549332a7f17a7d9b2799849b77c08a8e6",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "a2ee90c549332a7f17a7d9b2799849b77c08a8e6",
       "triggerType" : "PUSH"
     }, {
       "hash" : "05ce568b0b6ba14da9ca40d26b39ba478837d747",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1017",
       "triggerID" : "05ce568b0b6ba14da9ca40d26b39ba478837d747",
       "triggerType" : "PUSH"
     }, {
       "hash" : "840e3b33d39b1f86367bb6b0e65eadeb33de7765",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1025",
       "triggerID" : "840e3b33d39b1f86367bb6b0e65eadeb33de7765",
       "triggerType" : "PUSH"
     }, {
       "hash" : "8d2e2dd83f51d03e890d424a51d3e0e84e904a43",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1031",
       "triggerID" : "8d2e2dd83f51d03e890d424a51d3e0e84e904a43",
       "triggerType" : "PUSH"
     }, {
       "hash" : "688381914087ff11acf2f8c9372dbf39afa0bd40",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1035",
       "triggerID" : "688381914087ff11acf2f8c9372dbf39afa0bd40",
       "triggerType" : "PUSH"
     }, {
       "hash" : "fc2c138901e82ed5f3fbf363c2a0ce8140ade050",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "fc2c138901e82ed5f3fbf363c2a0ce8140ade050",
       "triggerType" : "PUSH"
     }, {
       "hash" : "6ed9e1cb1e5a191752d6d39c7a9d5d41840f0cdf",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1039",
       "triggerID" : "6ed9e1cb1e5a191752d6d39c7a9d5d41840f0cdf",
       "triggerType" : "PUSH"
     }, {
       "hash" : "a510a7de1458bb996df5bbb341e8d010e4c91381",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "a510a7de1458bb996df5bbb341e8d010e4c91381",
       "triggerType" : "PUSH"
     }, {
       "hash" : "49677397cd117228742205944606f7a789090376",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1042",
       "triggerID" : "49677397cd117228742205944606f7a789090376",
       "triggerType" : "PUSH"
     }, {
       "hash" : "1630db0a37422aed942ff280560a9d6329df5015",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1044",
       "triggerID" : "1630db0a37422aed942ff280560a9d6329df5015",
       "triggerType" : "PUSH"
     }, {
       "hash" : "f11c76301d4ab524db0af57aa831be3a898ea0fb",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1048",
       "triggerID" : "f11c76301d4ab524db0af57aa831be3a898ea0fb",
       "triggerType" : "PUSH"
     }, {
       "hash" : "d1a0dc4c2017853a7ba0c3302f886043a63f29da",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1050",
       "triggerID" : "d1a0dc4c2017853a7ba0c3302f886043a63f29da",
       "triggerType" : "PUSH"
     }, {
       "hash" : "6a4c15ec39ded1f90b113b272554217698a98e1c",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1087",
       "triggerID" : "6a4c15ec39ded1f90b113b272554217698a98e1c",
       "triggerType" : "PUSH"
     }, {
       "hash" : "288e730d645a5e58e50a87ef5f44ba11a8fb0533",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1092",
       "triggerID" : "288e730d645a5e58e50a87ef5f44ba11a8fb0533",
       "triggerType" : "PUSH"
     }, {
       "hash" : "70a5c21321a0da3a0be8d85e206c24befa7b89f1",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1095",
       "triggerID" : "70a5c21321a0da3a0be8d85e206c24befa7b89f1",
       "triggerType" : "PUSH"
     }, {
       "hash" : "f072a986114949c5e426ce64a5f5568fc718e25d",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "f072a986114949c5e426ce64a5f5568fc718e25d",
       "triggerType" : "PUSH"
     }, {
       "hash" : "45b27475b1e8806ec8c566d6f151f4bc79c8549d",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1096",
       "triggerID" : "45b27475b1e8806ec8c566d6f151f4bc79c8549d",
       "triggerType" : "PUSH"
     }, {
       "hash" : "3c62ef22f6cde28110e5eb2fca8760a008d77723",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "3c62ef22f6cde28110e5eb2fca8760a008d77723",
       "triggerType" : "PUSH"
     }, {
       "hash" : "dc785fa04e542b2ccc45494e0292f5153c7ecea8",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1097",
       "triggerID" : "dc785fa04e542b2ccc45494e0292f5153c7ecea8",
       "triggerType" : "PUSH"
     }, {
       "hash" : "585ddb8049b9b9ccb057027e265b08a1b586df8c",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "585ddb8049b9b9ccb057027e265b08a1b586df8c",
       "triggerType" : "PUSH"
     }, {
       "hash" : "b62ff8675beed2c4487c788b9ca1a0ad4271b913",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "b62ff8675beed2c4487c788b9ca1a0ad4271b913",
       "triggerType" : "PUSH"
     }, {
       "hash" : "2d46048a7402907f348cc96da2839ba1b3e4932a",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1099",
       "triggerID" : "2d46048a7402907f348cc96da2839ba1b3e4932a",
       "triggerType" : "PUSH"
     }, {
       "hash" : "004fb6ebf23f796a4b77650e0530f0f1145e8b91",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1101",
       "triggerID" : "004fb6ebf23f796a4b77650e0530f0f1145e8b91",
       "triggerType" : "PUSH"
     }, {
       "hash" : "af0c33333b5e9ae6d59f20a6db1d7c016ea015d3",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1165",
       "triggerID" : "af0c33333b5e9ae6d59f20a6db1d7c016ea015d3",
       "triggerType" : "PUSH"
     }, {
       "hash" : "da8111118d71306500229619d3b7996fcd2526e8",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "da8111118d71306500229619d3b7996fcd2526e8",
       "triggerType" : "PUSH"
     }, {
       "hash" : "83257e2b9bd3c426cf90e11003a5f244a750d7fa",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1167",
       "triggerID" : "83257e2b9bd3c426cf90e11003a5f244a750d7fa",
       "triggerType" : "PUSH"
     }, {
       "hash" : "e0de408f1b8153fdaaea4c87b27a7b69213b75b5",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1177",
       "triggerID" : "e0de408f1b8153fdaaea4c87b27a7b69213b75b5",
       "triggerType" : "PUSH"
     }, {
       "hash" : "3f7951f7ad7931f5dc0b977d7c33bc5429623535",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1180",
       "triggerID" : "3f7951f7ad7931f5dc0b977d7c33bc5429623535",
       "triggerType" : "PUSH"
     }, {
       "hash" : "9cd6f06a753b63630e8245d51338d2bf9ab86b84",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "9cd6f06a753b63630e8245d51338d2bf9ab86b84",
       "triggerType" : "PUSH"
     }, {
       "hash" : "7380b902731a92e0645c02397e14591a05798620",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "7380b902731a92e0645c02397e14591a05798620",
       "triggerType" : "PUSH"
     }, {
       "hash" : "0bd5cca01a541845081d5430d48b021d4bdf0969",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "0bd5cca01a541845081d5430d48b021d4bdf0969",
       "triggerType" : "PUSH"
     }, {
       "hash" : "8f4e175ef554f015e34e0968b656b5de714ab430",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1182",
       "triggerID" : "8f4e175ef554f015e34e0968b656b5de714ab430",
       "triggerType" : "PUSH"
     }, {
       "hash" : "2023f3c882d0c2f5b96e28d2f21812dc0c64aaee",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "2023f3c882d0c2f5b96e28d2f21812dc0c64aaee",
       "triggerType" : "PUSH"
     }, {
       "hash" : "8394acf060d5c3953dbd67cf930b9b1d3b92d8b5",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1198",
       "triggerID" : "8394acf060d5c3953dbd67cf930b9b1d3b92d8b5",
       "triggerType" : "PUSH"
     }, {
       "hash" : "7ec42c105971bff463e359019405a7f4e7947d86",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "7ec42c105971bff463e359019405a7f4e7947d86",
       "triggerType" : "PUSH"
     }, {
       "hash" : "4b930786a7b2de09ac1653cd884ee76a7d1b2ca5",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1202",
       "triggerID" : "4b930786a7b2de09ac1653cd884ee76a7d1b2ca5",
       "triggerType" : "PUSH"
     }, {
       "hash" : "0e9a648c4e03b5a74d6a0085831805041a9a26a2",
       "status" : "SUCCESS",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1216",
       "triggerID" : "0e9a648c4e03b5a74d6a0085831805041a9a26a2",
       "triggerType" : "PUSH"
     }, {
       "hash" : "326a8729a5d1dd88bc474a5c273aec6ca219f298",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "326a8729a5d1dd88bc474a5c273aec6ca219f298",
       "triggerType" : "PUSH"
     }, {
       "hash" : "cad2f761fc50fba28e2fb3595e4b1270e39ac48a",
       "status" : "PENDING",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1221",
       "triggerID" : "cad2f761fc50fba28e2fb3595e4b1270e39ac48a",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 9e668f65c6d4f73c76573236f8fa507dfb80edac UNKNOWN
   * 533f00e5546da0026bbe51e1c030ddd1c19fdd08 UNKNOWN
   * a2ee90c549332a7f17a7d9b2799849b77c08a8e6 UNKNOWN
   * fc2c138901e82ed5f3fbf363c2a0ce8140ade050 UNKNOWN
   * a510a7de1458bb996df5bbb341e8d010e4c91381 UNKNOWN
   * f072a986114949c5e426ce64a5f5568fc718e25d UNKNOWN
   * 3c62ef22f6cde28110e5eb2fca8760a008d77723 UNKNOWN
   * 585ddb8049b9b9ccb057027e265b08a1b586df8c UNKNOWN
   * b62ff8675beed2c4487c788b9ca1a0ad4271b913 UNKNOWN
   * da8111118d71306500229619d3b7996fcd2526e8 UNKNOWN
   * 9cd6f06a753b63630e8245d51338d2bf9ab86b84 UNKNOWN
   * 7380b902731a92e0645c02397e14591a05798620 UNKNOWN
   * 0bd5cca01a541845081d5430d48b021d4bdf0969 UNKNOWN
   * 2023f3c882d0c2f5b96e28d2f21812dc0c64aaee UNKNOWN
   * 7ec42c105971bff463e359019405a7f4e7947d86 UNKNOWN
   * 0e9a648c4e03b5a74d6a0085831805041a9a26a2 Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1216) 
   * 326a8729a5d1dd88bc474a5c273aec6ca219f298 UNKNOWN
   * cad2f761fc50fba28e2fb3595e4b1270e39ac48a Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1221) 
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] [HUDI-8141] Incremental Query with Completion Time [hudi]

Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.

hudi-bot commented on PR #11947:
URL: https://github.com/apache/hudi/pull/11947#issuecomment-2423316036

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "090f256a315cbbf0c6a54d93cc2739dd693360e6",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=484",
       "triggerID" : "090f256a315cbbf0c6a54d93cc2739dd693360e6",
       "triggerType" : "PUSH"
     }, {
       "hash" : "0df5649013c1cf9e782fcbb67b8a2e7174f94512",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=985",
       "triggerID" : "0df5649013c1cf9e782fcbb67b8a2e7174f94512",
       "triggerType" : "PUSH"
     }, {
       "hash" : "d52e0128f409918644255abbe4376dc4a780bfc8",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=986",
       "triggerID" : "d52e0128f409918644255abbe4376dc4a780bfc8",
       "triggerType" : "PUSH"
     }, {
       "hash" : "9e668f65c6d4f73c76573236f8fa507dfb80edac",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "9e668f65c6d4f73c76573236f8fa507dfb80edac",
       "triggerType" : "PUSH"
     }, {
       "hash" : "f145131d524bf190986e5de600a326e32a27beca",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=987",
       "triggerID" : "f145131d524bf190986e5de600a326e32a27beca",
       "triggerType" : "PUSH"
     }, {
       "hash" : "0b008f807b42aa6fb2686922bb88d63e84a1f299",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=993",
       "triggerID" : "0b008f807b42aa6fb2686922bb88d63e84a1f299",
       "triggerType" : "PUSH"
     }, {
       "hash" : "65c01ad9f2b33d36d4890355feef91844438e732",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=995",
       "triggerID" : "65c01ad9f2b33d36d4890355feef91844438e732",
       "triggerType" : "PUSH"
     }, {
       "hash" : "955493fdc93842e8c99529cec35e39cecdbe9638",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=996",
       "triggerID" : "955493fdc93842e8c99529cec35e39cecdbe9638",
       "triggerType" : "PUSH"
     }, {
       "hash" : "533f00e5546da0026bbe51e1c030ddd1c19fdd08",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "533f00e5546da0026bbe51e1c030ddd1c19fdd08",
       "triggerType" : "PUSH"
     }, {
       "hash" : "05777be67e37f3a6a8eaa59e54d9e17e5fc6ba9b",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1016",
       "triggerID" : "05777be67e37f3a6a8eaa59e54d9e17e5fc6ba9b",
       "triggerType" : "PUSH"
     }, {
       "hash" : "a2ee90c549332a7f17a7d9b2799849b77c08a8e6",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "a2ee90c549332a7f17a7d9b2799849b77c08a8e6",
       "triggerType" : "PUSH"
     }, {
       "hash" : "05ce568b0b6ba14da9ca40d26b39ba478837d747",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1017",
       "triggerID" : "05ce568b0b6ba14da9ca40d26b39ba478837d747",
       "triggerType" : "PUSH"
     }, {
       "hash" : "840e3b33d39b1f86367bb6b0e65eadeb33de7765",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1025",
       "triggerID" : "840e3b33d39b1f86367bb6b0e65eadeb33de7765",
       "triggerType" : "PUSH"
     }, {
       "hash" : "8d2e2dd83f51d03e890d424a51d3e0e84e904a43",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1031",
       "triggerID" : "8d2e2dd83f51d03e890d424a51d3e0e84e904a43",
       "triggerType" : "PUSH"
     }, {
       "hash" : "688381914087ff11acf2f8c9372dbf39afa0bd40",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1035",
       "triggerID" : "688381914087ff11acf2f8c9372dbf39afa0bd40",
       "triggerType" : "PUSH"
     }, {
       "hash" : "fc2c138901e82ed5f3fbf363c2a0ce8140ade050",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "fc2c138901e82ed5f3fbf363c2a0ce8140ade050",
       "triggerType" : "PUSH"
     }, {
       "hash" : "6ed9e1cb1e5a191752d6d39c7a9d5d41840f0cdf",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1039",
       "triggerID" : "6ed9e1cb1e5a191752d6d39c7a9d5d41840f0cdf",
       "triggerType" : "PUSH"
     }, {
       "hash" : "a510a7de1458bb996df5bbb341e8d010e4c91381",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "a510a7de1458bb996df5bbb341e8d010e4c91381",
       "triggerType" : "PUSH"
     }, {
       "hash" : "49677397cd117228742205944606f7a789090376",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1042",
       "triggerID" : "49677397cd117228742205944606f7a789090376",
       "triggerType" : "PUSH"
     }, {
       "hash" : "1630db0a37422aed942ff280560a9d6329df5015",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1044",
       "triggerID" : "1630db0a37422aed942ff280560a9d6329df5015",
       "triggerType" : "PUSH"
     }, {
       "hash" : "f11c76301d4ab524db0af57aa831be3a898ea0fb",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1048",
       "triggerID" : "f11c76301d4ab524db0af57aa831be3a898ea0fb",
       "triggerType" : "PUSH"
     }, {
       "hash" : "d1a0dc4c2017853a7ba0c3302f886043a63f29da",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1050",
       "triggerID" : "d1a0dc4c2017853a7ba0c3302f886043a63f29da",
       "triggerType" : "PUSH"
     }, {
       "hash" : "6a4c15ec39ded1f90b113b272554217698a98e1c",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1087",
       "triggerID" : "6a4c15ec39ded1f90b113b272554217698a98e1c",
       "triggerType" : "PUSH"
     }, {
       "hash" : "288e730d645a5e58e50a87ef5f44ba11a8fb0533",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1092",
       "triggerID" : "288e730d645a5e58e50a87ef5f44ba11a8fb0533",
       "triggerType" : "PUSH"
     }, {
       "hash" : "70a5c21321a0da3a0be8d85e206c24befa7b89f1",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1095",
       "triggerID" : "70a5c21321a0da3a0be8d85e206c24befa7b89f1",
       "triggerType" : "PUSH"
     }, {
       "hash" : "f072a986114949c5e426ce64a5f5568fc718e25d",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "f072a986114949c5e426ce64a5f5568fc718e25d",
       "triggerType" : "PUSH"
     }, {
       "hash" : "45b27475b1e8806ec8c566d6f151f4bc79c8549d",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1096",
       "triggerID" : "45b27475b1e8806ec8c566d6f151f4bc79c8549d",
       "triggerType" : "PUSH"
     }, {
       "hash" : "3c62ef22f6cde28110e5eb2fca8760a008d77723",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "3c62ef22f6cde28110e5eb2fca8760a008d77723",
       "triggerType" : "PUSH"
     }, {
       "hash" : "dc785fa04e542b2ccc45494e0292f5153c7ecea8",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1097",
       "triggerID" : "dc785fa04e542b2ccc45494e0292f5153c7ecea8",
       "triggerType" : "PUSH"
     }, {
       "hash" : "585ddb8049b9b9ccb057027e265b08a1b586df8c",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "585ddb8049b9b9ccb057027e265b08a1b586df8c",
       "triggerType" : "PUSH"
     }, {
       "hash" : "b62ff8675beed2c4487c788b9ca1a0ad4271b913",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "b62ff8675beed2c4487c788b9ca1a0ad4271b913",
       "triggerType" : "PUSH"
     }, {
       "hash" : "2d46048a7402907f348cc96da2839ba1b3e4932a",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1099",
       "triggerID" : "2d46048a7402907f348cc96da2839ba1b3e4932a",
       "triggerType" : "PUSH"
     }, {
       "hash" : "004fb6ebf23f796a4b77650e0530f0f1145e8b91",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1101",
       "triggerID" : "004fb6ebf23f796a4b77650e0530f0f1145e8b91",
       "triggerType" : "PUSH"
     }, {
       "hash" : "af0c33333b5e9ae6d59f20a6db1d7c016ea015d3",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1165",
       "triggerID" : "af0c33333b5e9ae6d59f20a6db1d7c016ea015d3",
       "triggerType" : "PUSH"
     }, {
       "hash" : "da8111118d71306500229619d3b7996fcd2526e8",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "da8111118d71306500229619d3b7996fcd2526e8",
       "triggerType" : "PUSH"
     }, {
       "hash" : "83257e2b9bd3c426cf90e11003a5f244a750d7fa",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1167",
       "triggerID" : "83257e2b9bd3c426cf90e11003a5f244a750d7fa",
       "triggerType" : "PUSH"
     }, {
       "hash" : "e0de408f1b8153fdaaea4c87b27a7b69213b75b5",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1177",
       "triggerID" : "e0de408f1b8153fdaaea4c87b27a7b69213b75b5",
       "triggerType" : "PUSH"
     }, {
       "hash" : "3f7951f7ad7931f5dc0b977d7c33bc5429623535",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1180",
       "triggerID" : "3f7951f7ad7931f5dc0b977d7c33bc5429623535",
       "triggerType" : "PUSH"
     }, {
       "hash" : "9cd6f06a753b63630e8245d51338d2bf9ab86b84",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "9cd6f06a753b63630e8245d51338d2bf9ab86b84",
       "triggerType" : "PUSH"
     }, {
       "hash" : "7380b902731a92e0645c02397e14591a05798620",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "7380b902731a92e0645c02397e14591a05798620",
       "triggerType" : "PUSH"
     }, {
       "hash" : "0bd5cca01a541845081d5430d48b021d4bdf0969",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "0bd5cca01a541845081d5430d48b021d4bdf0969",
       "triggerType" : "PUSH"
     }, {
       "hash" : "8f4e175ef554f015e34e0968b656b5de714ab430",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1182",
       "triggerID" : "8f4e175ef554f015e34e0968b656b5de714ab430",
       "triggerType" : "PUSH"
     }, {
       "hash" : "2023f3c882d0c2f5b96e28d2f21812dc0c64aaee",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "2023f3c882d0c2f5b96e28d2f21812dc0c64aaee",
       "triggerType" : "PUSH"
     }, {
       "hash" : "8394acf060d5c3953dbd67cf930b9b1d3b92d8b5",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1198",
       "triggerID" : "8394acf060d5c3953dbd67cf930b9b1d3b92d8b5",
       "triggerType" : "PUSH"
     }, {
       "hash" : "7ec42c105971bff463e359019405a7f4e7947d86",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "7ec42c105971bff463e359019405a7f4e7947d86",
       "triggerType" : "PUSH"
     }, {
       "hash" : "4b930786a7b2de09ac1653cd884ee76a7d1b2ca5",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1202",
       "triggerID" : "4b930786a7b2de09ac1653cd884ee76a7d1b2ca5",
       "triggerType" : "PUSH"
     }, {
       "hash" : "0e9a648c4e03b5a74d6a0085831805041a9a26a2",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1216",
       "triggerID" : "0e9a648c4e03b5a74d6a0085831805041a9a26a2",
       "triggerType" : "PUSH"
     }, {
       "hash" : "326a8729a5d1dd88bc474a5c273aec6ca219f298",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "326a8729a5d1dd88bc474a5c273aec6ca219f298",
       "triggerType" : "PUSH"
     }, {
       "hash" : "cad2f761fc50fba28e2fb3595e4b1270e39ac48a",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1221",
       "triggerID" : "cad2f761fc50fba28e2fb3595e4b1270e39ac48a",
       "triggerType" : "PUSH"
     }, {
       "hash" : "d485a50aec34bc5707377afe96d696772832558e",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "d485a50aec34bc5707377afe96d696772832558e",
       "triggerType" : "PUSH"
     }, {
       "hash" : "38a98106f19fe4e51632a2c4289ec0f01805ba5e",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1224",
       "triggerID" : "38a98106f19fe4e51632a2c4289ec0f01805ba5e",
       "triggerType" : "PUSH"
     }, {
       "hash" : "7a6116cfbf4eba54e77194664225ed1c372d71be",
       "status" : "FAILURE",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1250",
       "triggerID" : "7a6116cfbf4eba54e77194664225ed1c372d71be",
       "triggerType" : "PUSH"
     }, {
       "hash" : "df721d8b5f610e156430f46b1a704310c70a4725",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "df721d8b5f610e156430f46b1a704310c70a4725",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 9e668f65c6d4f73c76573236f8fa507dfb80edac UNKNOWN
   * 533f00e5546da0026bbe51e1c030ddd1c19fdd08 UNKNOWN
   * a2ee90c549332a7f17a7d9b2799849b77c08a8e6 UNKNOWN
   * fc2c138901e82ed5f3fbf363c2a0ce8140ade050 UNKNOWN
   * a510a7de1458bb996df5bbb341e8d010e4c91381 UNKNOWN
   * f072a986114949c5e426ce64a5f5568fc718e25d UNKNOWN
   * 3c62ef22f6cde28110e5eb2fca8760a008d77723 UNKNOWN
   * 585ddb8049b9b9ccb057027e265b08a1b586df8c UNKNOWN
   * b62ff8675beed2c4487c788b9ca1a0ad4271b913 UNKNOWN
   * da8111118d71306500229619d3b7996fcd2526e8 UNKNOWN
   * 9cd6f06a753b63630e8245d51338d2bf9ab86b84 UNKNOWN
   * 7380b902731a92e0645c02397e14591a05798620 UNKNOWN
   * 0bd5cca01a541845081d5430d48b021d4bdf0969 UNKNOWN
   * 2023f3c882d0c2f5b96e28d2f21812dc0c64aaee UNKNOWN
   * 7ec42c105971bff463e359019405a7f4e7947d86 UNKNOWN
   * 326a8729a5d1dd88bc474a5c273aec6ca219f298 UNKNOWN
   * d485a50aec34bc5707377afe96d696772832558e UNKNOWN
   * 7a6116cfbf4eba54e77194664225ed1c372d71be Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1250) 
   * df721d8b5f610e156430f46b1a704310c70a4725 UNKNOWN
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] [HUDI-8141] Incremental Query with Completion Time [hudi]

Posted by "yihua (via GitHub)" <gi...@apache.org>.

yihua commented on code in PR #11947:
URL: https://github.com/apache/hudi/pull/11947#discussion_r1805921883


##########
hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/hudi/IncrementalRelation.scala:
##########
@@ -90,24 +78,24 @@ class IncrementalRelation(val sqlContext: SQLContext,
     throw new HoodieException("Incremental queries are not supported when meta fields are disabled")
   }
 
+  private val queryContext: IncrementalQueryAnalyzer.QueryContext =
+    IncrementalQueryAnalyzer.builder()
+      .metaClient(metaClient)
+      .startTime(optParams(DataSourceReadOptions.BEGIN_INSTANTTIME.key))
+      .endTime(optParams.getOrElse(DataSourceReadOptions.END_INSTANTTIME.key, null))
+      .rangeType(InstantRange.RangeType.OPEN_CLOSED)

Review Comment:
   This is fixed now by dynamically setting the range type for the incremental query.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] [HUDI-8141] Incremental Query with Completion Time [hudi]

Posted by "yihua (via GitHub)" <gi...@apache.org>.

yihua commented on code in PR #11947:
URL: https://github.com/apache/hudi/pull/11947#discussion_r1806021437


##########
hudi-common/src/main/java/org/apache/hudi/common/table/read/IncrementalQueryAnalyzer.java:
##########
@@ -413,9 +425,14 @@ public Option<String> getEndInstant() {
      * Returns the latest instant time which should be included physically in reading.
      */
     public String getLastInstant() {
+      ValidationUtils.checkArgument(!this.instants.isEmpty());

Review Comment:
   Fixed.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] [HUDI-8141] WIP:Always use completion time to find begin Instant for incremental source [hudi]

Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.

hudi-bot commented on PR #11947:
URL: https://github.com/apache/hudi/pull/11947#issuecomment-2351180927

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "090f256a315cbbf0c6a54d93cc2739dd693360e6",
       "status" : "FAILURE",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=484",
       "triggerID" : "090f256a315cbbf0c6a54d93cc2739dd693360e6",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 090f256a315cbbf0c6a54d93cc2739dd693360e6 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=484) 
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] [HUDI-8141] WIP:Always use completion time to find begin Instant for incremental source [hudi]

Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.

hudi-bot commented on PR #11947:
URL: https://github.com/apache/hudi/pull/11947#issuecomment-2401634233

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "090f256a315cbbf0c6a54d93cc2739dd693360e6",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=484",
       "triggerID" : "090f256a315cbbf0c6a54d93cc2739dd693360e6",
       "triggerType" : "PUSH"
     }, {
       "hash" : "0df5649013c1cf9e782fcbb67b8a2e7174f94512",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=985",
       "triggerID" : "0df5649013c1cf9e782fcbb67b8a2e7174f94512",
       "triggerType" : "PUSH"
     }, {
       "hash" : "d52e0128f409918644255abbe4376dc4a780bfc8",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=986",
       "triggerID" : "d52e0128f409918644255abbe4376dc4a780bfc8",
       "triggerType" : "PUSH"
     }, {
       "hash" : "9e668f65c6d4f73c76573236f8fa507dfb80edac",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "9e668f65c6d4f73c76573236f8fa507dfb80edac",
       "triggerType" : "PUSH"
     }, {
       "hash" : "f145131d524bf190986e5de600a326e32a27beca",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=987",
       "triggerID" : "f145131d524bf190986e5de600a326e32a27beca",
       "triggerType" : "PUSH"
     }, {
       "hash" : "0b008f807b42aa6fb2686922bb88d63e84a1f299",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=993",
       "triggerID" : "0b008f807b42aa6fb2686922bb88d63e84a1f299",
       "triggerType" : "PUSH"
     }, {
       "hash" : "65c01ad9f2b33d36d4890355feef91844438e732",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=995",
       "triggerID" : "65c01ad9f2b33d36d4890355feef91844438e732",
       "triggerType" : "PUSH"
     }, {
       "hash" : "955493fdc93842e8c99529cec35e39cecdbe9638",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=996",
       "triggerID" : "955493fdc93842e8c99529cec35e39cecdbe9638",
       "triggerType" : "PUSH"
     }, {
       "hash" : "533f00e5546da0026bbe51e1c030ddd1c19fdd08",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "533f00e5546da0026bbe51e1c030ddd1c19fdd08",
       "triggerType" : "PUSH"
     }, {
       "hash" : "05777be67e37f3a6a8eaa59e54d9e17e5fc6ba9b",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1016",
       "triggerID" : "05777be67e37f3a6a8eaa59e54d9e17e5fc6ba9b",
       "triggerType" : "PUSH"
     }, {
       "hash" : "a2ee90c549332a7f17a7d9b2799849b77c08a8e6",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "a2ee90c549332a7f17a7d9b2799849b77c08a8e6",
       "triggerType" : "PUSH"
     }, {
       "hash" : "05ce568b0b6ba14da9ca40d26b39ba478837d747",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1017",
       "triggerID" : "05ce568b0b6ba14da9ca40d26b39ba478837d747",
       "triggerType" : "PUSH"
     }, {
       "hash" : "840e3b33d39b1f86367bb6b0e65eadeb33de7765",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1025",
       "triggerID" : "840e3b33d39b1f86367bb6b0e65eadeb33de7765",
       "triggerType" : "PUSH"
     }, {
       "hash" : "8d2e2dd83f51d03e890d424a51d3e0e84e904a43",
       "status" : "FAILURE",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1031",
       "triggerID" : "8d2e2dd83f51d03e890d424a51d3e0e84e904a43",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 9e668f65c6d4f73c76573236f8fa507dfb80edac UNKNOWN
   * 533f00e5546da0026bbe51e1c030ddd1c19fdd08 UNKNOWN
   * a2ee90c549332a7f17a7d9b2799849b77c08a8e6 UNKNOWN
   * 8d2e2dd83f51d03e890d424a51d3e0e84e904a43 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1031) 
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] [HUDI-8141] Incremental Query with Completion Time [hudi]

Posted by "CTTY (via GitHub)" <gi...@apache.org>.

CTTY commented on code in PR #11947:
URL: https://github.com/apache/hudi/pull/11947#discussion_r1797918568


##########
hudi-spark-datasource/hudi-spark/src/test/scala/org/apache/spark/sql/hudi/procedure/TestCopyToTableProcedure.scala:
##########
@@ -188,10 +191,11 @@ class TestCopyToTableProcedure extends HoodieSparkProcedureTestBase {
       spark.sql(s"insert into $tableName select 2, 'a2', 20, 1500")
 
       // mark beginTime
-      val beginTime = spark.sql(s"select max(_hoodie_commit_time) from $tableName").collectAsList().get(0).get(0)
+      val fs = HadoopFSUtils.getFs(tablePath, spark.sessionState.newHadoopConf())
+      val beginCompletionTime = HoodieDataSourceHelpers.latestCommitCompletionTime(fs, tablePath)
       spark.sql(s"insert into $tableName select 3, 'a3', 30, 2000")
       spark.sql(s"insert into $tableName select 4, 'a4', 40, 2500")
-      val endTime = spark.sql(s"select max(_hoodie_commit_time) from $tableName").collectAsList().get(0).get(0)
+      val endCompletionTime = HoodieDataSourceHelpers.latestCommitCompletionTime(fs, tablePath)

Review Comment:
   There are few inserts between the two calls of `HoodieDataSourceHelpers.latestCommitCompletionTime(fs, tablePath)` so it won't be the same



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] [HUDI-8141] Incremental Query with Completion Time [hudi]

Posted by "CTTY (via GitHub)" <gi...@apache.org>.

CTTY commented on code in PR #11947:
URL: https://github.com/apache/hudi/pull/11947#discussion_r1797513909


##########
hudi-utilities/src/main/java/org/apache/hudi/utilities/sources/HoodieIncrSource.java:
##########
@@ -175,33 +181,72 @@ public Pair<Option<Dataset<Row>>, String> fetchNextBatch(Option<String> lastCkpt
     IncrSourceHelper.MissingCheckpointStrategy missingCheckpointStrategy =
         (containsConfigProperty(props, HoodieIncrSourceConfig.MISSING_CHECKPOINT_STRATEGY))
             ? IncrSourceHelper.MissingCheckpointStrategy.valueOf(
-            getStringWithAltKeys(props, HoodieIncrSourceConfig.MISSING_CHECKPOINT_STRATEGY))
+                getStringWithAltKeys(props, HoodieIncrSourceConfig.MISSING_CHECKPOINT_STRATEGY))
             : null;
     if (readLatestOnMissingCkpt) {
       missingCheckpointStrategy = IncrSourceHelper.MissingCheckpointStrategy.READ_LATEST;
     }
 
-    // Use begin Instant if set and non-empty
-    Option<String> beginInstant =
-        lastCkptStr.isPresent() ? lastCkptStr.get().isEmpty() ? Option.empty() : lastCkptStr : Option.empty();
+    HoodieTableMetaClient metaClient = HoodieTableMetaClient.builder()
+        .setConf(HadoopFSUtils.getStorageConfWithCopy(sparkContext.hadoopConfiguration()))
+        .setBasePath(srcPath)
+        .setLoadActiveTimelineOnLoad(true)
+        .build();
 
+    int numInstantsFromConfig = getIntWithAltKeys(props, HoodieIncrSourceConfig.NUM_INSTANTS_PER_FETCH);
+
+    String startTime;
+    if (lastCkptStr.isPresent() && !lastCkptStr.get().isEmpty()) {
+      startTime = lastCkptStr.get();
+    } else if (missingCheckpointStrategy != null) {
+      switch (missingCheckpointStrategy) {
+        case READ_UPTO_LATEST_COMMIT:
+          startTime = DEFAULT_BEGIN_TIMESTAMP;
+          // disrespect numInstantsFromConfig when reading up to latest
+          numInstantsFromConfig = -1;
+          break;
+        case READ_LATEST:
+          Option<HoodieInstant> lastInstant = metaClient
+              .getCommitsAndCompactionTimeline()
+              .filterCompletedInstants()
+              .lastInstant();
+          startTime = lastInstant
+              .map(hoodieInstant -> getStrictlyLowerTimestamp(hoodieInstant.getCompletionTime()))
+              .orElse(DEFAULT_BEGIN_TIMESTAMP);

Review Comment:
   We can not pass `null` or `Option.empty` directly to start time because it will fall into this case: https://github.com/apache/hudi/blob/master/hudi-common/src/main/java/org/apache/hudi/common/table/timeline/CompletionTimeQueryView.java#L278 
   if the `startTime` is `null` then it would disrespect the `missingCheckpointStrategy` and will always read the latest snapshot i



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] [HUDI-8141] Incremental Query with Completion Time [hudi]

Posted by "yihua (via GitHub)" <gi...@apache.org>.

yihua commented on code in PR #11947:
URL: https://github.com/apache/hudi/pull/11947#discussion_r1797786610


##########
hudi-spark-datasource/hudi-spark/src/main/java/org/apache/hudi/HoodieDataSourceHelpers.java:
##########
@@ -74,6 +74,15 @@ public static List<String> listCommitsSince(HoodieStorage storage, String basePa
         .map(HoodieInstant::getTimestamp).collect(Collectors.toList());
   }
 
+  public static List<String> listCompletionTimeSince(FileSystem fs, String basePath,

Review Comment:
   This method is not used.



##########
hudi-spark-datasource/hudi-spark/src/main/java/org/apache/hudi/HoodieDataSourceHelpers.java:
##########
@@ -88,6 +97,16 @@ public static String latestCommit(HoodieStorage storage, String basePath) {
     return timeline.lastInstant().get().getTimestamp();
   }
 
+  public static String latestCommitCompletionTime(FileSystem fs, String basePath) {
+    HoodieTimeline timeline = allCompletedCommitsCompactions(fs, basePath);
+    return timeline.lastInstant().get().getCompletionTime();
+  }
+
+  public static String latestCommitCompletionTime(HoodieStorage storage, String basePath) {
+    HoodieTimeline timeline = allCompletedCommitsCompactions(storage, basePath);
+    return timeline.lastInstant().get().getCompletionTime();
+  }

Review Comment:
   Similar here, this returns the completion time of the commit which has the largest instant time, which might not be the max completion time on the timeline.



##########
hudi-spark-datasource/hudi-spark/src/test/scala/org/apache/hudi/functional/TestCOWDataSource.scala:
##########
@@ -48,7 +48,6 @@ import org.apache.hudi.table.HoodieSparkTable
 import org.apache.hudi.testutils.HoodieSparkClientTestBase
 import org.apache.hudi.util.JFunction
 import org.apache.hudi.{AvroConversionUtils, DataSourceReadOptions, DataSourceWriteOptions, HoodieDataSourceHelpers, QuickstartUtils, ScalaAssertionSupport}
-

Review Comment:
   Let's keep the import grouping.



##########
hudi-spark-datasource/hudi-spark/src/main/java/org/apache/hudi/HoodieDataSourceHelpers.java:
##########
@@ -88,6 +97,16 @@ public static String latestCommit(HoodieStorage storage, String basePath) {
     return timeline.lastInstant().get().getTimestamp();
   }
 
+  public static String latestCommitCompletionTime(FileSystem fs, String basePath) {
+    HoodieTimeline timeline = allCompletedCommitsCompactions(fs, basePath);
+    return timeline.lastInstant().get().getCompletionTime();
+  }
+
+  public static String latestCommitCompletionTime(HoodieStorage storage, String basePath) {
+    HoodieTimeline timeline = allCompletedCommitsCompactions(storage, basePath);
+    return timeline.lastInstant().get().getCompletionTime();
+  }

Review Comment:
   Should these be moved to test utils if they are only used by the tests?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] [HUDI-8141] Incremental Query with Completion Time [hudi]

Posted by "CTTY (via GitHub)" <gi...@apache.org>.

CTTY commented on code in PR #11947:
URL: https://github.com/apache/hudi/pull/11947#discussion_r1797787473


##########
hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/hudi/IncrementalRelation.scala:
##########
@@ -220,15 +209,12 @@ class IncrementalRelation(val sqlContext: SQLContext,
 
       val sOpts = optParams.filter(p => !p._1.equalsIgnoreCase("path"))
 
-      val startInstantTime = optParams(DataSourceReadOptions.BEGIN_INSTANTTIME.key)
+      val startInstantTime = queryContext.getStartInstant.get()
       val startInstantArchived = commitTimeline.isBeforeTimelineStarts(startInstantTime)
-      val endInstantTime = optParams.getOrElse(DataSourceReadOptions.END_INSTANTTIME.key(), lastInstant.getTimestamp)
+      val endInstantTime = queryContext.getEndInstant.get()

Review Comment:
   This is a good point, I think it also makes no sense to check if `endInstantArchived` is true here because 
   1. end instant cannot come before start instant and 
   2. We should scan the full table if either start or end instant is archived.
   
   Will fix this



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] [HUDI-8141] Incremental Query with Completion Time [hudi]

Posted by "CTTY (via GitHub)" <gi...@apache.org>.

CTTY commented on code in PR #11947:
URL: https://github.com/apache/hudi/pull/11947#discussion_r1797784983


##########
hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/hudi/DataSourceOptions.scala:
##########
@@ -125,6 +125,15 @@ object DataSourceReadOptions {
       + HollowCommitHandling.USE_TRANSITION_TIME + "`, will use instant's "
       + "`stateTransitionTime` to perform comparison.")
 
+  /* TODO: this is a replacement of HoodieIncrSourceConfig.NUM_INSTANTS_PER_FETCH
+      to make incremental instant limit available in spark incremental relations
+      Need to link HoodieIncrSourceConfig.NUM_INSTANTS_PER_FETCH to this correctly and add doc*/
+  val INCREMENTAL_LIMIT: ConfigProperty[String] = ConfigProperty
+    .key("hoodie.streamer.source.hoodieincr.num_instants")

Review Comment:
   I think this makes sense, will remove this config and its usages in Hudi incremental relations



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] [HUDI-8141] Incremental Query with Completion Time [hudi]

Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.

hudi-bot commented on PR #11947:
URL: https://github.com/apache/hudi/pull/11947#issuecomment-2408834576

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "090f256a315cbbf0c6a54d93cc2739dd693360e6",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=484",
       "triggerID" : "090f256a315cbbf0c6a54d93cc2739dd693360e6",
       "triggerType" : "PUSH"
     }, {
       "hash" : "0df5649013c1cf9e782fcbb67b8a2e7174f94512",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=985",
       "triggerID" : "0df5649013c1cf9e782fcbb67b8a2e7174f94512",
       "triggerType" : "PUSH"
     }, {
       "hash" : "d52e0128f409918644255abbe4376dc4a780bfc8",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=986",
       "triggerID" : "d52e0128f409918644255abbe4376dc4a780bfc8",
       "triggerType" : "PUSH"
     }, {
       "hash" : "9e668f65c6d4f73c76573236f8fa507dfb80edac",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "9e668f65c6d4f73c76573236f8fa507dfb80edac",
       "triggerType" : "PUSH"
     }, {
       "hash" : "f145131d524bf190986e5de600a326e32a27beca",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=987",
       "triggerID" : "f145131d524bf190986e5de600a326e32a27beca",
       "triggerType" : "PUSH"
     }, {
       "hash" : "0b008f807b42aa6fb2686922bb88d63e84a1f299",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=993",
       "triggerID" : "0b008f807b42aa6fb2686922bb88d63e84a1f299",
       "triggerType" : "PUSH"
     }, {
       "hash" : "65c01ad9f2b33d36d4890355feef91844438e732",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=995",
       "triggerID" : "65c01ad9f2b33d36d4890355feef91844438e732",
       "triggerType" : "PUSH"
     }, {
       "hash" : "955493fdc93842e8c99529cec35e39cecdbe9638",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=996",
       "triggerID" : "955493fdc93842e8c99529cec35e39cecdbe9638",
       "triggerType" : "PUSH"
     }, {
       "hash" : "533f00e5546da0026bbe51e1c030ddd1c19fdd08",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "533f00e5546da0026bbe51e1c030ddd1c19fdd08",
       "triggerType" : "PUSH"
     }, {
       "hash" : "05777be67e37f3a6a8eaa59e54d9e17e5fc6ba9b",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1016",
       "triggerID" : "05777be67e37f3a6a8eaa59e54d9e17e5fc6ba9b",
       "triggerType" : "PUSH"
     }, {
       "hash" : "a2ee90c549332a7f17a7d9b2799849b77c08a8e6",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "a2ee90c549332a7f17a7d9b2799849b77c08a8e6",
       "triggerType" : "PUSH"
     }, {
       "hash" : "05ce568b0b6ba14da9ca40d26b39ba478837d747",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1017",
       "triggerID" : "05ce568b0b6ba14da9ca40d26b39ba478837d747",
       "triggerType" : "PUSH"
     }, {
       "hash" : "840e3b33d39b1f86367bb6b0e65eadeb33de7765",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1025",
       "triggerID" : "840e3b33d39b1f86367bb6b0e65eadeb33de7765",
       "triggerType" : "PUSH"
     }, {
       "hash" : "8d2e2dd83f51d03e890d424a51d3e0e84e904a43",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1031",
       "triggerID" : "8d2e2dd83f51d03e890d424a51d3e0e84e904a43",
       "triggerType" : "PUSH"
     }, {
       "hash" : "688381914087ff11acf2f8c9372dbf39afa0bd40",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1035",
       "triggerID" : "688381914087ff11acf2f8c9372dbf39afa0bd40",
       "triggerType" : "PUSH"
     }, {
       "hash" : "fc2c138901e82ed5f3fbf363c2a0ce8140ade050",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "fc2c138901e82ed5f3fbf363c2a0ce8140ade050",
       "triggerType" : "PUSH"
     }, {
       "hash" : "6ed9e1cb1e5a191752d6d39c7a9d5d41840f0cdf",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1039",
       "triggerID" : "6ed9e1cb1e5a191752d6d39c7a9d5d41840f0cdf",
       "triggerType" : "PUSH"
     }, {
       "hash" : "a510a7de1458bb996df5bbb341e8d010e4c91381",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "a510a7de1458bb996df5bbb341e8d010e4c91381",
       "triggerType" : "PUSH"
     }, {
       "hash" : "49677397cd117228742205944606f7a789090376",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1042",
       "triggerID" : "49677397cd117228742205944606f7a789090376",
       "triggerType" : "PUSH"
     }, {
       "hash" : "1630db0a37422aed942ff280560a9d6329df5015",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1044",
       "triggerID" : "1630db0a37422aed942ff280560a9d6329df5015",
       "triggerType" : "PUSH"
     }, {
       "hash" : "f11c76301d4ab524db0af57aa831be3a898ea0fb",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1048",
       "triggerID" : "f11c76301d4ab524db0af57aa831be3a898ea0fb",
       "triggerType" : "PUSH"
     }, {
       "hash" : "d1a0dc4c2017853a7ba0c3302f886043a63f29da",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1050",
       "triggerID" : "d1a0dc4c2017853a7ba0c3302f886043a63f29da",
       "triggerType" : "PUSH"
     }, {
       "hash" : "6a4c15ec39ded1f90b113b272554217698a98e1c",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1087",
       "triggerID" : "6a4c15ec39ded1f90b113b272554217698a98e1c",
       "triggerType" : "PUSH"
     }, {
       "hash" : "288e730d645a5e58e50a87ef5f44ba11a8fb0533",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1092",
       "triggerID" : "288e730d645a5e58e50a87ef5f44ba11a8fb0533",
       "triggerType" : "PUSH"
     }, {
       "hash" : "70a5c21321a0da3a0be8d85e206c24befa7b89f1",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1095",
       "triggerID" : "70a5c21321a0da3a0be8d85e206c24befa7b89f1",
       "triggerType" : "PUSH"
     }, {
       "hash" : "f072a986114949c5e426ce64a5f5568fc718e25d",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "f072a986114949c5e426ce64a5f5568fc718e25d",
       "triggerType" : "PUSH"
     }, {
       "hash" : "45b27475b1e8806ec8c566d6f151f4bc79c8549d",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1096",
       "triggerID" : "45b27475b1e8806ec8c566d6f151f4bc79c8549d",
       "triggerType" : "PUSH"
     }, {
       "hash" : "3c62ef22f6cde28110e5eb2fca8760a008d77723",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "3c62ef22f6cde28110e5eb2fca8760a008d77723",
       "triggerType" : "PUSH"
     }, {
       "hash" : "dc785fa04e542b2ccc45494e0292f5153c7ecea8",
       "status" : "FAILURE",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1097",
       "triggerID" : "dc785fa04e542b2ccc45494e0292f5153c7ecea8",
       "triggerType" : "PUSH"
     }, {
       "hash" : "585ddb8049b9b9ccb057027e265b08a1b586df8c",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "585ddb8049b9b9ccb057027e265b08a1b586df8c",
       "triggerType" : "PUSH"
     }, {
       "hash" : "b62ff8675beed2c4487c788b9ca1a0ad4271b913",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "b62ff8675beed2c4487c788b9ca1a0ad4271b913",
       "triggerType" : "PUSH"
     }, {
       "hash" : "2d46048a7402907f348cc96da2839ba1b3e4932a",
       "status" : "PENDING",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1099",
       "triggerID" : "2d46048a7402907f348cc96da2839ba1b3e4932a",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 9e668f65c6d4f73c76573236f8fa507dfb80edac UNKNOWN
   * 533f00e5546da0026bbe51e1c030ddd1c19fdd08 UNKNOWN
   * a2ee90c549332a7f17a7d9b2799849b77c08a8e6 UNKNOWN
   * fc2c138901e82ed5f3fbf363c2a0ce8140ade050 UNKNOWN
   * a510a7de1458bb996df5bbb341e8d010e4c91381 UNKNOWN
   * f072a986114949c5e426ce64a5f5568fc718e25d UNKNOWN
   * 3c62ef22f6cde28110e5eb2fca8760a008d77723 UNKNOWN
   * dc785fa04e542b2ccc45494e0292f5153c7ecea8 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1097) 
   * 585ddb8049b9b9ccb057027e265b08a1b586df8c UNKNOWN
   * b62ff8675beed2c4487c788b9ca1a0ad4271b913 UNKNOWN
   * 2d46048a7402907f348cc96da2839ba1b3e4932a Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1099) 
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] [HUDI-8141] Incremental Query with Completion Time [hudi]

Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.

hudi-bot commented on PR #11947:
URL: https://github.com/apache/hudi/pull/11947#issuecomment-2408788843

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "090f256a315cbbf0c6a54d93cc2739dd693360e6",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=484",
       "triggerID" : "090f256a315cbbf0c6a54d93cc2739dd693360e6",
       "triggerType" : "PUSH"
     }, {
       "hash" : "0df5649013c1cf9e782fcbb67b8a2e7174f94512",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=985",
       "triggerID" : "0df5649013c1cf9e782fcbb67b8a2e7174f94512",
       "triggerType" : "PUSH"
     }, {
       "hash" : "d52e0128f409918644255abbe4376dc4a780bfc8",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=986",
       "triggerID" : "d52e0128f409918644255abbe4376dc4a780bfc8",
       "triggerType" : "PUSH"
     }, {
       "hash" : "9e668f65c6d4f73c76573236f8fa507dfb80edac",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "9e668f65c6d4f73c76573236f8fa507dfb80edac",
       "triggerType" : "PUSH"
     }, {
       "hash" : "f145131d524bf190986e5de600a326e32a27beca",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=987",
       "triggerID" : "f145131d524bf190986e5de600a326e32a27beca",
       "triggerType" : "PUSH"
     }, {
       "hash" : "0b008f807b42aa6fb2686922bb88d63e84a1f299",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=993",
       "triggerID" : "0b008f807b42aa6fb2686922bb88d63e84a1f299",
       "triggerType" : "PUSH"
     }, {
       "hash" : "65c01ad9f2b33d36d4890355feef91844438e732",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=995",
       "triggerID" : "65c01ad9f2b33d36d4890355feef91844438e732",
       "triggerType" : "PUSH"
     }, {
       "hash" : "955493fdc93842e8c99529cec35e39cecdbe9638",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=996",
       "triggerID" : "955493fdc93842e8c99529cec35e39cecdbe9638",
       "triggerType" : "PUSH"
     }, {
       "hash" : "533f00e5546da0026bbe51e1c030ddd1c19fdd08",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "533f00e5546da0026bbe51e1c030ddd1c19fdd08",
       "triggerType" : "PUSH"
     }, {
       "hash" : "05777be67e37f3a6a8eaa59e54d9e17e5fc6ba9b",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1016",
       "triggerID" : "05777be67e37f3a6a8eaa59e54d9e17e5fc6ba9b",
       "triggerType" : "PUSH"
     }, {
       "hash" : "a2ee90c549332a7f17a7d9b2799849b77c08a8e6",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "a2ee90c549332a7f17a7d9b2799849b77c08a8e6",
       "triggerType" : "PUSH"
     }, {
       "hash" : "05ce568b0b6ba14da9ca40d26b39ba478837d747",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1017",
       "triggerID" : "05ce568b0b6ba14da9ca40d26b39ba478837d747",
       "triggerType" : "PUSH"
     }, {
       "hash" : "840e3b33d39b1f86367bb6b0e65eadeb33de7765",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1025",
       "triggerID" : "840e3b33d39b1f86367bb6b0e65eadeb33de7765",
       "triggerType" : "PUSH"
     }, {
       "hash" : "8d2e2dd83f51d03e890d424a51d3e0e84e904a43",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1031",
       "triggerID" : "8d2e2dd83f51d03e890d424a51d3e0e84e904a43",
       "triggerType" : "PUSH"
     }, {
       "hash" : "688381914087ff11acf2f8c9372dbf39afa0bd40",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1035",
       "triggerID" : "688381914087ff11acf2f8c9372dbf39afa0bd40",
       "triggerType" : "PUSH"
     }, {
       "hash" : "fc2c138901e82ed5f3fbf363c2a0ce8140ade050",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "fc2c138901e82ed5f3fbf363c2a0ce8140ade050",
       "triggerType" : "PUSH"
     }, {
       "hash" : "6ed9e1cb1e5a191752d6d39c7a9d5d41840f0cdf",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1039",
       "triggerID" : "6ed9e1cb1e5a191752d6d39c7a9d5d41840f0cdf",
       "triggerType" : "PUSH"
     }, {
       "hash" : "a510a7de1458bb996df5bbb341e8d010e4c91381",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "a510a7de1458bb996df5bbb341e8d010e4c91381",
       "triggerType" : "PUSH"
     }, {
       "hash" : "49677397cd117228742205944606f7a789090376",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1042",
       "triggerID" : "49677397cd117228742205944606f7a789090376",
       "triggerType" : "PUSH"
     }, {
       "hash" : "1630db0a37422aed942ff280560a9d6329df5015",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1044",
       "triggerID" : "1630db0a37422aed942ff280560a9d6329df5015",
       "triggerType" : "PUSH"
     }, {
       "hash" : "f11c76301d4ab524db0af57aa831be3a898ea0fb",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1048",
       "triggerID" : "f11c76301d4ab524db0af57aa831be3a898ea0fb",
       "triggerType" : "PUSH"
     }, {
       "hash" : "d1a0dc4c2017853a7ba0c3302f886043a63f29da",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1050",
       "triggerID" : "d1a0dc4c2017853a7ba0c3302f886043a63f29da",
       "triggerType" : "PUSH"
     }, {
       "hash" : "6a4c15ec39ded1f90b113b272554217698a98e1c",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1087",
       "triggerID" : "6a4c15ec39ded1f90b113b272554217698a98e1c",
       "triggerType" : "PUSH"
     }, {
       "hash" : "288e730d645a5e58e50a87ef5f44ba11a8fb0533",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1092",
       "triggerID" : "288e730d645a5e58e50a87ef5f44ba11a8fb0533",
       "triggerType" : "PUSH"
     }, {
       "hash" : "70a5c21321a0da3a0be8d85e206c24befa7b89f1",
       "status" : "CANCELED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1095",
       "triggerID" : "70a5c21321a0da3a0be8d85e206c24befa7b89f1",
       "triggerType" : "PUSH"
     }, {
       "hash" : "f072a986114949c5e426ce64a5f5568fc718e25d",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "f072a986114949c5e426ce64a5f5568fc718e25d",
       "triggerType" : "PUSH"
     }, {
       "hash" : "45b27475b1e8806ec8c566d6f151f4bc79c8549d",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "45b27475b1e8806ec8c566d6f151f4bc79c8549d",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 9e668f65c6d4f73c76573236f8fa507dfb80edac UNKNOWN
   * 533f00e5546da0026bbe51e1c030ddd1c19fdd08 UNKNOWN
   * a2ee90c549332a7f17a7d9b2799849b77c08a8e6 UNKNOWN
   * fc2c138901e82ed5f3fbf363c2a0ce8140ade050 UNKNOWN
   * a510a7de1458bb996df5bbb341e8d010e4c91381 UNKNOWN
   * 70a5c21321a0da3a0be8d85e206c24befa7b89f1 Azure: [CANCELED](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1095) 
   * f072a986114949c5e426ce64a5f5568fc718e25d UNKNOWN
   * 45b27475b1e8806ec8c566d6f151f4bc79c8549d UNKNOWN
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] [HUDI-8141] Always use completion time to find begin Instant for incremental source [hudi]

Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.

hudi-bot commented on PR #11947:
URL: https://github.com/apache/hudi/pull/11947#issuecomment-2403868148

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "090f256a315cbbf0c6a54d93cc2739dd693360e6",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=484",
       "triggerID" : "090f256a315cbbf0c6a54d93cc2739dd693360e6",
       "triggerType" : "PUSH"
     }, {
       "hash" : "0df5649013c1cf9e782fcbb67b8a2e7174f94512",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=985",
       "triggerID" : "0df5649013c1cf9e782fcbb67b8a2e7174f94512",
       "triggerType" : "PUSH"
     }, {
       "hash" : "d52e0128f409918644255abbe4376dc4a780bfc8",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=986",
       "triggerID" : "d52e0128f409918644255abbe4376dc4a780bfc8",
       "triggerType" : "PUSH"
     }, {
       "hash" : "9e668f65c6d4f73c76573236f8fa507dfb80edac",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "9e668f65c6d4f73c76573236f8fa507dfb80edac",
       "triggerType" : "PUSH"
     }, {
       "hash" : "f145131d524bf190986e5de600a326e32a27beca",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=987",
       "triggerID" : "f145131d524bf190986e5de600a326e32a27beca",
       "triggerType" : "PUSH"
     }, {
       "hash" : "0b008f807b42aa6fb2686922bb88d63e84a1f299",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=993",
       "triggerID" : "0b008f807b42aa6fb2686922bb88d63e84a1f299",
       "triggerType" : "PUSH"
     }, {
       "hash" : "65c01ad9f2b33d36d4890355feef91844438e732",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=995",
       "triggerID" : "65c01ad9f2b33d36d4890355feef91844438e732",
       "triggerType" : "PUSH"
     }, {
       "hash" : "955493fdc93842e8c99529cec35e39cecdbe9638",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=996",
       "triggerID" : "955493fdc93842e8c99529cec35e39cecdbe9638",
       "triggerType" : "PUSH"
     }, {
       "hash" : "533f00e5546da0026bbe51e1c030ddd1c19fdd08",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "533f00e5546da0026bbe51e1c030ddd1c19fdd08",
       "triggerType" : "PUSH"
     }, {
       "hash" : "05777be67e37f3a6a8eaa59e54d9e17e5fc6ba9b",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1016",
       "triggerID" : "05777be67e37f3a6a8eaa59e54d9e17e5fc6ba9b",
       "triggerType" : "PUSH"
     }, {
       "hash" : "a2ee90c549332a7f17a7d9b2799849b77c08a8e6",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "a2ee90c549332a7f17a7d9b2799849b77c08a8e6",
       "triggerType" : "PUSH"
     }, {
       "hash" : "05ce568b0b6ba14da9ca40d26b39ba478837d747",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1017",
       "triggerID" : "05ce568b0b6ba14da9ca40d26b39ba478837d747",
       "triggerType" : "PUSH"
     }, {
       "hash" : "840e3b33d39b1f86367bb6b0e65eadeb33de7765",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1025",
       "triggerID" : "840e3b33d39b1f86367bb6b0e65eadeb33de7765",
       "triggerType" : "PUSH"
     }, {
       "hash" : "8d2e2dd83f51d03e890d424a51d3e0e84e904a43",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1031",
       "triggerID" : "8d2e2dd83f51d03e890d424a51d3e0e84e904a43",
       "triggerType" : "PUSH"
     }, {
       "hash" : "688381914087ff11acf2f8c9372dbf39afa0bd40",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1035",
       "triggerID" : "688381914087ff11acf2f8c9372dbf39afa0bd40",
       "triggerType" : "PUSH"
     }, {
       "hash" : "fc2c138901e82ed5f3fbf363c2a0ce8140ade050",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "fc2c138901e82ed5f3fbf363c2a0ce8140ade050",
       "triggerType" : "PUSH"
     }, {
       "hash" : "6ed9e1cb1e5a191752d6d39c7a9d5d41840f0cdf",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1039",
       "triggerID" : "6ed9e1cb1e5a191752d6d39c7a9d5d41840f0cdf",
       "triggerType" : "PUSH"
     }, {
       "hash" : "a510a7de1458bb996df5bbb341e8d010e4c91381",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "a510a7de1458bb996df5bbb341e8d010e4c91381",
       "triggerType" : "PUSH"
     }, {
       "hash" : "49677397cd117228742205944606f7a789090376",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1042",
       "triggerID" : "49677397cd117228742205944606f7a789090376",
       "triggerType" : "PUSH"
     }, {
       "hash" : "1630db0a37422aed942ff280560a9d6329df5015",
       "status" : "CANCELED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1044",
       "triggerID" : "1630db0a37422aed942ff280560a9d6329df5015",
       "triggerType" : "PUSH"
     }, {
       "hash" : "f11c76301d4ab524db0af57aa831be3a898ea0fb",
       "status" : "PENDING",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1048",
       "triggerID" : "f11c76301d4ab524db0af57aa831be3a898ea0fb",
       "triggerType" : "PUSH"
     }, {
       "hash" : "d1a0dc4c2017853a7ba0c3302f886043a63f29da",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "d1a0dc4c2017853a7ba0c3302f886043a63f29da",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 9e668f65c6d4f73c76573236f8fa507dfb80edac UNKNOWN
   * 533f00e5546da0026bbe51e1c030ddd1c19fdd08 UNKNOWN
   * a2ee90c549332a7f17a7d9b2799849b77c08a8e6 UNKNOWN
   * fc2c138901e82ed5f3fbf363c2a0ce8140ade050 UNKNOWN
   * a510a7de1458bb996df5bbb341e8d010e4c91381 UNKNOWN
   * 1630db0a37422aed942ff280560a9d6329df5015 Azure: [CANCELED](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1044) 
   * f11c76301d4ab524db0af57aa831be3a898ea0fb Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1048) 
   * d1a0dc4c2017853a7ba0c3302f886043a63f29da UNKNOWN
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] [HUDI-8141] Incremental Query with Completion Time [hudi]

Posted by "CTTY (via GitHub)" <gi...@apache.org>.

CTTY commented on code in PR #11947:
URL: https://github.com/apache/hudi/pull/11947#discussion_r1798104588


##########
hudi-spark-datasource/hudi-spark/src/test/scala/org/apache/hudi/functional/TestIncrementalReadWithFullTableScan.scala:
##########
@@ -70,9 +70,11 @@ class TestIncrementalReadWithFullTableScan extends HoodieSparkClientTestBase {
       HoodieWriteConfig.TBL_NAME.key -> "hoodie_test",
       HoodieMetadataConfig.COMPACT_NUM_DELTA_COMMITS.key -> "1"
     )
+
+
     // Create 10 commits
     for (i <- 1 to 10) {
-      val records = recordsToStrings(dataGen.generateInserts("%05d".format(i), perBatchSize)).asScala.toList
+      val records = recordsToStrings(dataGen.generateInserts(System.currentTimeMillis(), perBatchSize)).asScala.toList

Review Comment:
   There are some tests in `TestHoodieIncrSource` that has instant times don't match completion time to test hollow commits handling that can be used to test this feature. 
   
   I've filed a jira to further improve the test coverage: https://issues.apache.org/jira/browse/HUDI-8356



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] [HUDI-8141] Incremental Query with Completion Time [hudi]

Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.

hudi-bot commented on PR #11947:
URL: https://github.com/apache/hudi/pull/11947#issuecomment-2408332504

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "090f256a315cbbf0c6a54d93cc2739dd693360e6",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=484",
       "triggerID" : "090f256a315cbbf0c6a54d93cc2739dd693360e6",
       "triggerType" : "PUSH"
     }, {
       "hash" : "0df5649013c1cf9e782fcbb67b8a2e7174f94512",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=985",
       "triggerID" : "0df5649013c1cf9e782fcbb67b8a2e7174f94512",
       "triggerType" : "PUSH"
     }, {
       "hash" : "d52e0128f409918644255abbe4376dc4a780bfc8",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=986",
       "triggerID" : "d52e0128f409918644255abbe4376dc4a780bfc8",
       "triggerType" : "PUSH"
     }, {
       "hash" : "9e668f65c6d4f73c76573236f8fa507dfb80edac",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "9e668f65c6d4f73c76573236f8fa507dfb80edac",
       "triggerType" : "PUSH"
     }, {
       "hash" : "f145131d524bf190986e5de600a326e32a27beca",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=987",
       "triggerID" : "f145131d524bf190986e5de600a326e32a27beca",
       "triggerType" : "PUSH"
     }, {
       "hash" : "0b008f807b42aa6fb2686922bb88d63e84a1f299",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=993",
       "triggerID" : "0b008f807b42aa6fb2686922bb88d63e84a1f299",
       "triggerType" : "PUSH"
     }, {
       "hash" : "65c01ad9f2b33d36d4890355feef91844438e732",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=995",
       "triggerID" : "65c01ad9f2b33d36d4890355feef91844438e732",
       "triggerType" : "PUSH"
     }, {
       "hash" : "955493fdc93842e8c99529cec35e39cecdbe9638",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=996",
       "triggerID" : "955493fdc93842e8c99529cec35e39cecdbe9638",
       "triggerType" : "PUSH"
     }, {
       "hash" : "533f00e5546da0026bbe51e1c030ddd1c19fdd08",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "533f00e5546da0026bbe51e1c030ddd1c19fdd08",
       "triggerType" : "PUSH"
     }, {
       "hash" : "05777be67e37f3a6a8eaa59e54d9e17e5fc6ba9b",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1016",
       "triggerID" : "05777be67e37f3a6a8eaa59e54d9e17e5fc6ba9b",
       "triggerType" : "PUSH"
     }, {
       "hash" : "a2ee90c549332a7f17a7d9b2799849b77c08a8e6",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "a2ee90c549332a7f17a7d9b2799849b77c08a8e6",
       "triggerType" : "PUSH"
     }, {
       "hash" : "05ce568b0b6ba14da9ca40d26b39ba478837d747",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1017",
       "triggerID" : "05ce568b0b6ba14da9ca40d26b39ba478837d747",
       "triggerType" : "PUSH"
     }, {
       "hash" : "840e3b33d39b1f86367bb6b0e65eadeb33de7765",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1025",
       "triggerID" : "840e3b33d39b1f86367bb6b0e65eadeb33de7765",
       "triggerType" : "PUSH"
     }, {
       "hash" : "8d2e2dd83f51d03e890d424a51d3e0e84e904a43",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1031",
       "triggerID" : "8d2e2dd83f51d03e890d424a51d3e0e84e904a43",
       "triggerType" : "PUSH"
     }, {
       "hash" : "688381914087ff11acf2f8c9372dbf39afa0bd40",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1035",
       "triggerID" : "688381914087ff11acf2f8c9372dbf39afa0bd40",
       "triggerType" : "PUSH"
     }, {
       "hash" : "fc2c138901e82ed5f3fbf363c2a0ce8140ade050",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "fc2c138901e82ed5f3fbf363c2a0ce8140ade050",
       "triggerType" : "PUSH"
     }, {
       "hash" : "6ed9e1cb1e5a191752d6d39c7a9d5d41840f0cdf",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1039",
       "triggerID" : "6ed9e1cb1e5a191752d6d39c7a9d5d41840f0cdf",
       "triggerType" : "PUSH"
     }, {
       "hash" : "a510a7de1458bb996df5bbb341e8d010e4c91381",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "a510a7de1458bb996df5bbb341e8d010e4c91381",
       "triggerType" : "PUSH"
     }, {
       "hash" : "49677397cd117228742205944606f7a789090376",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1042",
       "triggerID" : "49677397cd117228742205944606f7a789090376",
       "triggerType" : "PUSH"
     }, {
       "hash" : "1630db0a37422aed942ff280560a9d6329df5015",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1044",
       "triggerID" : "1630db0a37422aed942ff280560a9d6329df5015",
       "triggerType" : "PUSH"
     }, {
       "hash" : "f11c76301d4ab524db0af57aa831be3a898ea0fb",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1048",
       "triggerID" : "f11c76301d4ab524db0af57aa831be3a898ea0fb",
       "triggerType" : "PUSH"
     }, {
       "hash" : "d1a0dc4c2017853a7ba0c3302f886043a63f29da",
       "status" : "FAILURE",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1050",
       "triggerID" : "d1a0dc4c2017853a7ba0c3302f886043a63f29da",
       "triggerType" : "PUSH"
     }, {
       "hash" : "6a4c15ec39ded1f90b113b272554217698a98e1c",
       "status" : "PENDING",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1087",
       "triggerID" : "6a4c15ec39ded1f90b113b272554217698a98e1c",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 9e668f65c6d4f73c76573236f8fa507dfb80edac UNKNOWN
   * 533f00e5546da0026bbe51e1c030ddd1c19fdd08 UNKNOWN
   * a2ee90c549332a7f17a7d9b2799849b77c08a8e6 UNKNOWN
   * fc2c138901e82ed5f3fbf363c2a0ce8140ade050 UNKNOWN
   * a510a7de1458bb996df5bbb341e8d010e4c91381 UNKNOWN
   * d1a0dc4c2017853a7ba0c3302f886043a63f29da Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1050) 
   * 6a4c15ec39ded1f90b113b272554217698a98e1c Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1087) 
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] [HUDI-8141] Incremental Query with Completion Time [hudi]

Posted by "danny0405 (via GitHub)" <gi...@apache.org>.

danny0405 commented on code in PR #11947:
URL: https://github.com/apache/hudi/pull/11947#discussion_r1798659033


##########
hudi-utilities/src/main/java/org/apache/hudi/utilities/sources/helpers/IncrSourceHelper.java:
##########
@@ -174,6 +170,64 @@ public static QueryInfo generateQueryInfo(JavaSparkContext jssc, String srcBaseP
     }
   }
 
+  public static IncrementalQueryAnalyzer getIncrementalQueryAnalyzer(
+      JavaSparkContext jssc,
+      String srcPath,
+      Option<String> lastCkptStr,
+      MissingCheckpointStrategy missingCheckpointStrategy,
+      int numInstantsFromConfig,
+      Option<SourceProfile<Integer>> latestSourceProfile) {
+    HoodieTableMetaClient metaClient = HoodieTableMetaClient.builder()
+        .setConf(HadoopFSUtils.getStorageConfWithCopy(jssc.hadoopConfiguration()))
+        .setBasePath(srcPath)
+        .setLoadActiveTimelineOnLoad(true)
+        .build();
+
+    String startTime;
+    if (lastCkptStr.isPresent() && !lastCkptStr.get().isEmpty()) {
+      startTime = lastCkptStr.get();
+    } else if (missingCheckpointStrategy != null) {
+      switch (missingCheckpointStrategy) {
+        case READ_UPTO_LATEST_COMMIT:
+          startTime = DEFAULT_BEGIN_TIMESTAMP;
+          // disrespect numInstantsFromConfig when reading up to latest
+          numInstantsFromConfig = -1;
+          break;
+        case READ_LATEST:
+          Option<HoodieInstant> lastInstant = metaClient
+              .getCommitsAndCompactionTimeline()
+              .filterCompletedInstants()
+              .lastInstant();
+          startTime = lastInstant
+              .map(hoodieInstant -> instantTimeMinusMillis(hoodieInstant.getCompletionTime(), 1))

Review Comment:
   Use the `closed_open` range type instead.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] [HUDI-8141] Incremental Query with Completion Time [hudi]

Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.

hudi-bot commented on PR #11947:
URL: https://github.com/apache/hudi/pull/11947#issuecomment-2423197284

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "090f256a315cbbf0c6a54d93cc2739dd693360e6",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=484",
       "triggerID" : "090f256a315cbbf0c6a54d93cc2739dd693360e6",
       "triggerType" : "PUSH"
     }, {
       "hash" : "0df5649013c1cf9e782fcbb67b8a2e7174f94512",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=985",
       "triggerID" : "0df5649013c1cf9e782fcbb67b8a2e7174f94512",
       "triggerType" : "PUSH"
     }, {
       "hash" : "d52e0128f409918644255abbe4376dc4a780bfc8",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=986",
       "triggerID" : "d52e0128f409918644255abbe4376dc4a780bfc8",
       "triggerType" : "PUSH"
     }, {
       "hash" : "9e668f65c6d4f73c76573236f8fa507dfb80edac",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "9e668f65c6d4f73c76573236f8fa507dfb80edac",
       "triggerType" : "PUSH"
     }, {
       "hash" : "f145131d524bf190986e5de600a326e32a27beca",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=987",
       "triggerID" : "f145131d524bf190986e5de600a326e32a27beca",
       "triggerType" : "PUSH"
     }, {
       "hash" : "0b008f807b42aa6fb2686922bb88d63e84a1f299",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=993",
       "triggerID" : "0b008f807b42aa6fb2686922bb88d63e84a1f299",
       "triggerType" : "PUSH"
     }, {
       "hash" : "65c01ad9f2b33d36d4890355feef91844438e732",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=995",
       "triggerID" : "65c01ad9f2b33d36d4890355feef91844438e732",
       "triggerType" : "PUSH"
     }, {
       "hash" : "955493fdc93842e8c99529cec35e39cecdbe9638",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=996",
       "triggerID" : "955493fdc93842e8c99529cec35e39cecdbe9638",
       "triggerType" : "PUSH"
     }, {
       "hash" : "533f00e5546da0026bbe51e1c030ddd1c19fdd08",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "533f00e5546da0026bbe51e1c030ddd1c19fdd08",
       "triggerType" : "PUSH"
     }, {
       "hash" : "05777be67e37f3a6a8eaa59e54d9e17e5fc6ba9b",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1016",
       "triggerID" : "05777be67e37f3a6a8eaa59e54d9e17e5fc6ba9b",
       "triggerType" : "PUSH"
     }, {
       "hash" : "a2ee90c549332a7f17a7d9b2799849b77c08a8e6",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "a2ee90c549332a7f17a7d9b2799849b77c08a8e6",
       "triggerType" : "PUSH"
     }, {
       "hash" : "05ce568b0b6ba14da9ca40d26b39ba478837d747",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1017",
       "triggerID" : "05ce568b0b6ba14da9ca40d26b39ba478837d747",
       "triggerType" : "PUSH"
     }, {
       "hash" : "840e3b33d39b1f86367bb6b0e65eadeb33de7765",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1025",
       "triggerID" : "840e3b33d39b1f86367bb6b0e65eadeb33de7765",
       "triggerType" : "PUSH"
     }, {
       "hash" : "8d2e2dd83f51d03e890d424a51d3e0e84e904a43",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1031",
       "triggerID" : "8d2e2dd83f51d03e890d424a51d3e0e84e904a43",
       "triggerType" : "PUSH"
     }, {
       "hash" : "688381914087ff11acf2f8c9372dbf39afa0bd40",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1035",
       "triggerID" : "688381914087ff11acf2f8c9372dbf39afa0bd40",
       "triggerType" : "PUSH"
     }, {
       "hash" : "fc2c138901e82ed5f3fbf363c2a0ce8140ade050",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "fc2c138901e82ed5f3fbf363c2a0ce8140ade050",
       "triggerType" : "PUSH"
     }, {
       "hash" : "6ed9e1cb1e5a191752d6d39c7a9d5d41840f0cdf",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1039",
       "triggerID" : "6ed9e1cb1e5a191752d6d39c7a9d5d41840f0cdf",
       "triggerType" : "PUSH"
     }, {
       "hash" : "a510a7de1458bb996df5bbb341e8d010e4c91381",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "a510a7de1458bb996df5bbb341e8d010e4c91381",
       "triggerType" : "PUSH"
     }, {
       "hash" : "49677397cd117228742205944606f7a789090376",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1042",
       "triggerID" : "49677397cd117228742205944606f7a789090376",
       "triggerType" : "PUSH"
     }, {
       "hash" : "1630db0a37422aed942ff280560a9d6329df5015",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1044",
       "triggerID" : "1630db0a37422aed942ff280560a9d6329df5015",
       "triggerType" : "PUSH"
     }, {
       "hash" : "f11c76301d4ab524db0af57aa831be3a898ea0fb",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1048",
       "triggerID" : "f11c76301d4ab524db0af57aa831be3a898ea0fb",
       "triggerType" : "PUSH"
     }, {
       "hash" : "d1a0dc4c2017853a7ba0c3302f886043a63f29da",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1050",
       "triggerID" : "d1a0dc4c2017853a7ba0c3302f886043a63f29da",
       "triggerType" : "PUSH"
     }, {
       "hash" : "6a4c15ec39ded1f90b113b272554217698a98e1c",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1087",
       "triggerID" : "6a4c15ec39ded1f90b113b272554217698a98e1c",
       "triggerType" : "PUSH"
     }, {
       "hash" : "288e730d645a5e58e50a87ef5f44ba11a8fb0533",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1092",
       "triggerID" : "288e730d645a5e58e50a87ef5f44ba11a8fb0533",
       "triggerType" : "PUSH"
     }, {
       "hash" : "70a5c21321a0da3a0be8d85e206c24befa7b89f1",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1095",
       "triggerID" : "70a5c21321a0da3a0be8d85e206c24befa7b89f1",
       "triggerType" : "PUSH"
     }, {
       "hash" : "f072a986114949c5e426ce64a5f5568fc718e25d",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "f072a986114949c5e426ce64a5f5568fc718e25d",
       "triggerType" : "PUSH"
     }, {
       "hash" : "45b27475b1e8806ec8c566d6f151f4bc79c8549d",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1096",
       "triggerID" : "45b27475b1e8806ec8c566d6f151f4bc79c8549d",
       "triggerType" : "PUSH"
     }, {
       "hash" : "3c62ef22f6cde28110e5eb2fca8760a008d77723",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "3c62ef22f6cde28110e5eb2fca8760a008d77723",
       "triggerType" : "PUSH"
     }, {
       "hash" : "dc785fa04e542b2ccc45494e0292f5153c7ecea8",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1097",
       "triggerID" : "dc785fa04e542b2ccc45494e0292f5153c7ecea8",
       "triggerType" : "PUSH"
     }, {
       "hash" : "585ddb8049b9b9ccb057027e265b08a1b586df8c",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "585ddb8049b9b9ccb057027e265b08a1b586df8c",
       "triggerType" : "PUSH"
     }, {
       "hash" : "b62ff8675beed2c4487c788b9ca1a0ad4271b913",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "b62ff8675beed2c4487c788b9ca1a0ad4271b913",
       "triggerType" : "PUSH"
     }, {
       "hash" : "2d46048a7402907f348cc96da2839ba1b3e4932a",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1099",
       "triggerID" : "2d46048a7402907f348cc96da2839ba1b3e4932a",
       "triggerType" : "PUSH"
     }, {
       "hash" : "004fb6ebf23f796a4b77650e0530f0f1145e8b91",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1101",
       "triggerID" : "004fb6ebf23f796a4b77650e0530f0f1145e8b91",
       "triggerType" : "PUSH"
     }, {
       "hash" : "af0c33333b5e9ae6d59f20a6db1d7c016ea015d3",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1165",
       "triggerID" : "af0c33333b5e9ae6d59f20a6db1d7c016ea015d3",
       "triggerType" : "PUSH"
     }, {
       "hash" : "da8111118d71306500229619d3b7996fcd2526e8",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "da8111118d71306500229619d3b7996fcd2526e8",
       "triggerType" : "PUSH"
     }, {
       "hash" : "83257e2b9bd3c426cf90e11003a5f244a750d7fa",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1167",
       "triggerID" : "83257e2b9bd3c426cf90e11003a5f244a750d7fa",
       "triggerType" : "PUSH"
     }, {
       "hash" : "e0de408f1b8153fdaaea4c87b27a7b69213b75b5",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1177",
       "triggerID" : "e0de408f1b8153fdaaea4c87b27a7b69213b75b5",
       "triggerType" : "PUSH"
     }, {
       "hash" : "3f7951f7ad7931f5dc0b977d7c33bc5429623535",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1180",
       "triggerID" : "3f7951f7ad7931f5dc0b977d7c33bc5429623535",
       "triggerType" : "PUSH"
     }, {
       "hash" : "9cd6f06a753b63630e8245d51338d2bf9ab86b84",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "9cd6f06a753b63630e8245d51338d2bf9ab86b84",
       "triggerType" : "PUSH"
     }, {
       "hash" : "7380b902731a92e0645c02397e14591a05798620",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "7380b902731a92e0645c02397e14591a05798620",
       "triggerType" : "PUSH"
     }, {
       "hash" : "0bd5cca01a541845081d5430d48b021d4bdf0969",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "0bd5cca01a541845081d5430d48b021d4bdf0969",
       "triggerType" : "PUSH"
     }, {
       "hash" : "8f4e175ef554f015e34e0968b656b5de714ab430",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1182",
       "triggerID" : "8f4e175ef554f015e34e0968b656b5de714ab430",
       "triggerType" : "PUSH"
     }, {
       "hash" : "2023f3c882d0c2f5b96e28d2f21812dc0c64aaee",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "2023f3c882d0c2f5b96e28d2f21812dc0c64aaee",
       "triggerType" : "PUSH"
     }, {
       "hash" : "8394acf060d5c3953dbd67cf930b9b1d3b92d8b5",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1198",
       "triggerID" : "8394acf060d5c3953dbd67cf930b9b1d3b92d8b5",
       "triggerType" : "PUSH"
     }, {
       "hash" : "7ec42c105971bff463e359019405a7f4e7947d86",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "7ec42c105971bff463e359019405a7f4e7947d86",
       "triggerType" : "PUSH"
     }, {
       "hash" : "4b930786a7b2de09ac1653cd884ee76a7d1b2ca5",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1202",
       "triggerID" : "4b930786a7b2de09ac1653cd884ee76a7d1b2ca5",
       "triggerType" : "PUSH"
     }, {
       "hash" : "0e9a648c4e03b5a74d6a0085831805041a9a26a2",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1216",
       "triggerID" : "0e9a648c4e03b5a74d6a0085831805041a9a26a2",
       "triggerType" : "PUSH"
     }, {
       "hash" : "326a8729a5d1dd88bc474a5c273aec6ca219f298",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "326a8729a5d1dd88bc474a5c273aec6ca219f298",
       "triggerType" : "PUSH"
     }, {
       "hash" : "cad2f761fc50fba28e2fb3595e4b1270e39ac48a",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1221",
       "triggerID" : "cad2f761fc50fba28e2fb3595e4b1270e39ac48a",
       "triggerType" : "PUSH"
     }, {
       "hash" : "d485a50aec34bc5707377afe96d696772832558e",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "d485a50aec34bc5707377afe96d696772832558e",
       "triggerType" : "PUSH"
     }, {
       "hash" : "38a98106f19fe4e51632a2c4289ec0f01805ba5e",
       "status" : "SUCCESS",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1224",
       "triggerID" : "38a98106f19fe4e51632a2c4289ec0f01805ba5e",
       "triggerType" : "PUSH"
     }, {
       "hash" : "7a6116cfbf4eba54e77194664225ed1c372d71be",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "7a6116cfbf4eba54e77194664225ed1c372d71be",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 9e668f65c6d4f73c76573236f8fa507dfb80edac UNKNOWN
   * 533f00e5546da0026bbe51e1c030ddd1c19fdd08 UNKNOWN
   * a2ee90c549332a7f17a7d9b2799849b77c08a8e6 UNKNOWN
   * fc2c138901e82ed5f3fbf363c2a0ce8140ade050 UNKNOWN
   * a510a7de1458bb996df5bbb341e8d010e4c91381 UNKNOWN
   * f072a986114949c5e426ce64a5f5568fc718e25d UNKNOWN
   * 3c62ef22f6cde28110e5eb2fca8760a008d77723 UNKNOWN
   * 585ddb8049b9b9ccb057027e265b08a1b586df8c UNKNOWN
   * b62ff8675beed2c4487c788b9ca1a0ad4271b913 UNKNOWN
   * da8111118d71306500229619d3b7996fcd2526e8 UNKNOWN
   * 9cd6f06a753b63630e8245d51338d2bf9ab86b84 UNKNOWN
   * 7380b902731a92e0645c02397e14591a05798620 UNKNOWN
   * 0bd5cca01a541845081d5430d48b021d4bdf0969 UNKNOWN
   * 2023f3c882d0c2f5b96e28d2f21812dc0c64aaee UNKNOWN
   * 7ec42c105971bff463e359019405a7f4e7947d86 UNKNOWN
   * 326a8729a5d1dd88bc474a5c273aec6ca219f298 UNKNOWN
   * d485a50aec34bc5707377afe96d696772832558e UNKNOWN
   * 38a98106f19fe4e51632a2c4289ec0f01805ba5e Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1224) 
   * 7a6116cfbf4eba54e77194664225ed1c372d71be UNKNOWN
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] [HUDI-8141] Incremental Query with Completion Time [hudi]

Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.

hudi-bot commented on PR #11947:
URL: https://github.com/apache/hudi/pull/11947#issuecomment-2408378449

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "090f256a315cbbf0c6a54d93cc2739dd693360e6",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=484",
       "triggerID" : "090f256a315cbbf0c6a54d93cc2739dd693360e6",
       "triggerType" : "PUSH"
     }, {
       "hash" : "0df5649013c1cf9e782fcbb67b8a2e7174f94512",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=985",
       "triggerID" : "0df5649013c1cf9e782fcbb67b8a2e7174f94512",
       "triggerType" : "PUSH"
     }, {
       "hash" : "d52e0128f409918644255abbe4376dc4a780bfc8",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=986",
       "triggerID" : "d52e0128f409918644255abbe4376dc4a780bfc8",
       "triggerType" : "PUSH"
     }, {
       "hash" : "9e668f65c6d4f73c76573236f8fa507dfb80edac",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "9e668f65c6d4f73c76573236f8fa507dfb80edac",
       "triggerType" : "PUSH"
     }, {
       "hash" : "f145131d524bf190986e5de600a326e32a27beca",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=987",
       "triggerID" : "f145131d524bf190986e5de600a326e32a27beca",
       "triggerType" : "PUSH"
     }, {
       "hash" : "0b008f807b42aa6fb2686922bb88d63e84a1f299",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=993",
       "triggerID" : "0b008f807b42aa6fb2686922bb88d63e84a1f299",
       "triggerType" : "PUSH"
     }, {
       "hash" : "65c01ad9f2b33d36d4890355feef91844438e732",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=995",
       "triggerID" : "65c01ad9f2b33d36d4890355feef91844438e732",
       "triggerType" : "PUSH"
     }, {
       "hash" : "955493fdc93842e8c99529cec35e39cecdbe9638",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=996",
       "triggerID" : "955493fdc93842e8c99529cec35e39cecdbe9638",
       "triggerType" : "PUSH"
     }, {
       "hash" : "533f00e5546da0026bbe51e1c030ddd1c19fdd08",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "533f00e5546da0026bbe51e1c030ddd1c19fdd08",
       "triggerType" : "PUSH"
     }, {
       "hash" : "05777be67e37f3a6a8eaa59e54d9e17e5fc6ba9b",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1016",
       "triggerID" : "05777be67e37f3a6a8eaa59e54d9e17e5fc6ba9b",
       "triggerType" : "PUSH"
     }, {
       "hash" : "a2ee90c549332a7f17a7d9b2799849b77c08a8e6",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "a2ee90c549332a7f17a7d9b2799849b77c08a8e6",
       "triggerType" : "PUSH"
     }, {
       "hash" : "05ce568b0b6ba14da9ca40d26b39ba478837d747",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1017",
       "triggerID" : "05ce568b0b6ba14da9ca40d26b39ba478837d747",
       "triggerType" : "PUSH"
     }, {
       "hash" : "840e3b33d39b1f86367bb6b0e65eadeb33de7765",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1025",
       "triggerID" : "840e3b33d39b1f86367bb6b0e65eadeb33de7765",
       "triggerType" : "PUSH"
     }, {
       "hash" : "8d2e2dd83f51d03e890d424a51d3e0e84e904a43",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1031",
       "triggerID" : "8d2e2dd83f51d03e890d424a51d3e0e84e904a43",
       "triggerType" : "PUSH"
     }, {
       "hash" : "688381914087ff11acf2f8c9372dbf39afa0bd40",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1035",
       "triggerID" : "688381914087ff11acf2f8c9372dbf39afa0bd40",
       "triggerType" : "PUSH"
     }, {
       "hash" : "fc2c138901e82ed5f3fbf363c2a0ce8140ade050",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "fc2c138901e82ed5f3fbf363c2a0ce8140ade050",
       "triggerType" : "PUSH"
     }, {
       "hash" : "6ed9e1cb1e5a191752d6d39c7a9d5d41840f0cdf",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1039",
       "triggerID" : "6ed9e1cb1e5a191752d6d39c7a9d5d41840f0cdf",
       "triggerType" : "PUSH"
     }, {
       "hash" : "a510a7de1458bb996df5bbb341e8d010e4c91381",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "a510a7de1458bb996df5bbb341e8d010e4c91381",
       "triggerType" : "PUSH"
     }, {
       "hash" : "49677397cd117228742205944606f7a789090376",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1042",
       "triggerID" : "49677397cd117228742205944606f7a789090376",
       "triggerType" : "PUSH"
     }, {
       "hash" : "1630db0a37422aed942ff280560a9d6329df5015",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1044",
       "triggerID" : "1630db0a37422aed942ff280560a9d6329df5015",
       "triggerType" : "PUSH"
     }, {
       "hash" : "f11c76301d4ab524db0af57aa831be3a898ea0fb",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1048",
       "triggerID" : "f11c76301d4ab524db0af57aa831be3a898ea0fb",
       "triggerType" : "PUSH"
     }, {
       "hash" : "d1a0dc4c2017853a7ba0c3302f886043a63f29da",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1050",
       "triggerID" : "d1a0dc4c2017853a7ba0c3302f886043a63f29da",
       "triggerType" : "PUSH"
     }, {
       "hash" : "6a4c15ec39ded1f90b113b272554217698a98e1c",
       "status" : "SUCCESS",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1087",
       "triggerID" : "6a4c15ec39ded1f90b113b272554217698a98e1c",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 9e668f65c6d4f73c76573236f8fa507dfb80edac UNKNOWN
   * 533f00e5546da0026bbe51e1c030ddd1c19fdd08 UNKNOWN
   * a2ee90c549332a7f17a7d9b2799849b77c08a8e6 UNKNOWN
   * fc2c138901e82ed5f3fbf363c2a0ce8140ade050 UNKNOWN
   * a510a7de1458bb996df5bbb341e8d010e4c91381 UNKNOWN
   * 6a4c15ec39ded1f90b113b272554217698a98e1c Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1087) 
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] [HUDI-8141] Incremental Query with Completion Time [hudi]

Posted by "CTTY (via GitHub)" <gi...@apache.org>.

CTTY commented on code in PR #11947:
URL: https://github.com/apache/hudi/pull/11947#discussion_r1797780048


##########
hudi-common/src/main/java/org/apache/hudi/common/table/timeline/CompletionTimeQueryView.java:
##########
@@ -223,6 +224,25 @@ public List<String> getStartTimes(
         InstantRange.RangeType.CLOSED_CLOSED, earliestInstantTimeFunc);
   }
 
+  /**
+   * Queries the instant start time with given completion time range.
+   *
+   * @param rangeStart              The query range start completion time.
+   * @param rangeEnd                The query range end completion time.
+   * @param rangeType               The range type.
+   *                                with the minimum completion time.
+   *
+   * @return The sorted instant time list.
+   */
+  @VisibleForTesting
+  public List<String> getStartTimes(
+      String rangeStart,
+      String rangeEnd,
+      RangeType rangeType) {
+    return getStartTimes(metaClient.getCommitsTimeline().filterCompletedInstants(), Option.ofNullable(rangeStart), Option.ofNullable(rangeEnd),
+        rangeType, s -> HoodieInstantTimeGenerator.instantTimeMinusMillis(s, MILLI_SECONDS_IN_ONE_DAY));

Review Comment:
   Good idea, I'll convert this to a static `Function`



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] [HUDI-8141] Incremental Query with Completion Time [hudi]

Posted by "danny0405 (via GitHub)" <gi...@apache.org>.

danny0405 commented on code in PR #11947:
URL: https://github.com/apache/hudi/pull/11947#discussion_r1805697591


##########
hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/spark/sql/hudi/streaming/HoodieStreamSource.scala:
##########
@@ -186,11 +160,11 @@ class HoodieStreamSource(
     }
   }
 
-  private def startCommitTime(startOffset: HoodieSourceOffset): String = {
+  private def startCompletionTime(startOffset: HoodieSourceOffset): String = {
     startOffset match {
-      case INIT_OFFSET => startOffset.commitTime
-      case HoodieSourceOffset(commitTime) =>
-        commitTime
+      case INIT_OFFSET => startOffset.completionTime
+      case HoodieSourceOffset(completionTime) =>
+        HoodieInstantTimeGenerator.instantTimePlusMillis(completionTime, 1)

Review Comment:
   Isn't the begin completion time of current read just the end completion time of last read?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] [HUDI-8141] Incremental Query with Completion Time [hudi]

Posted by "yihua (via GitHub)" <gi...@apache.org>.

yihua commented on code in PR #11947:
URL: https://github.com/apache/hudi/pull/11947#discussion_r1805923225


##########
hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/hudi/MergeOnReadIncrementalRelation.scala:
##########
@@ -201,55 +199,42 @@ trait HoodieIncrementalRelationTrait extends HoodieBaseRelation {
     val fallbackToFullTableScan = optParams.getOrElse(DataSourceReadOptions.INCREMENTAL_FALLBACK_TO_FULL_TABLE_SCAN.key,
       DataSourceReadOptions.INCREMENTAL_FALLBACK_TO_FULL_TABLE_SCAN.defaultValue).toBoolean
 
-    fallbackToFullTableScan && (startInstantArchived || endInstantArchived
+    fallbackToFullTableScan && (startInstantArchived
       || affectedFilesInCommits.asScala.exists(fileStatus => !metaClient.getStorage.exists(fileStatus.getPath)))
   }
 
-  protected lazy val includedCommits: immutable.Seq[HoodieInstant] = {
-    if (!startInstantArchived || !endInstantArchived) {
-      // If endTimestamp commit is not archived, will filter instants
-      // before endTimestamp.
-      if (hollowCommitHandling == USE_TRANSITION_TIME) {
-        super.timeline.findInstantsInRangeByCompletionTime(startTimestamp, endTimestamp).getInstants.asScala.toList
-      } else {
-        super.timeline.findInstantsInRange(startTimestamp, endTimestamp).getInstants.asScala.toList
-      }
-    } else {
-      super.timeline.getInstants.asScala.toList
-    }
-  }
+  protected lazy val queryContext: IncrementalQueryAnalyzer.QueryContext =
+    IncrementalQueryAnalyzer.builder()
+      .metaClient(metaClient)
+      .startTime(optParams(DataSourceReadOptions.BEGIN_INSTANTTIME.key))
+      .endTime(optParams.getOrElse(DataSourceReadOptions.END_INSTANTTIME.key, null))
+      .rangeType(InstantRange.RangeType.OPEN_CLOSED)

Review Comment:
   After the discussion, I see your point now.  I've changed the `IncrementalRelation`, `MergeOnReadIncremetnalRelation`, and `CDCRelation` to take `rangeType` as an argument so it can be dynamically changed based on different scenarios.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] [HUDI-8141] Incremental Query with Completion Time [hudi]

Posted by "yihua (via GitHub)" <gi...@apache.org>.

yihua commented on code in PR #11947:
URL: https://github.com/apache/hudi/pull/11947#discussion_r1801779609


##########
hudi-common/src/main/java/org/apache/hudi/common/table/timeline/CompletionTimeQueryView.java:
##########
@@ -189,46 +193,78 @@ public Option<String> getCompletionTime(String beginTime) {
    * <p>By default, assumes there is at most 1 day time of duration for an instant to accelerate the queries.
    *
    * @param timeline The timeline.
-   * @param rangeStart   The query range start completion time.
-   * @param rangeEnd     The query range end completion time.
+   * @param rangeStartCompletionTime   The query range start completion time.
+   * @param rangeEndCompletionTime     The query range end completion time.
    * @param rangeType    The range type.
    *
    * @return The sorted instant time list.
    */
   public List<String> getStartTimes(
       HoodieTimeline timeline,
-      Option<String> rangeStart,
-      Option<String> rangeEnd,
+      Option<String> rangeStartCompletionTime,
+      Option<String> rangeEndCompletionTime,
       InstantRange.RangeType rangeType) {
     // assumes any instant/transaction lasts at most 1 day to optimize the query efficiency.
-    return getStartTimes(timeline, rangeStart, rangeEnd, rangeType, s -> HoodieInstantTimeGenerator.instantTimeMinusMillis(s, MILLI_SECONDS_IN_ONE_DAY));
+    return getStartTimes(
+        timeline,
+        rangeStartCompletionTime,
+        rangeEndCompletionTime,
+        rangeType,
+        GET_INSTANT_ONE_DAY_BEFORE);
   }
 
   /**
    * Queries the instant start time with given completion time range.
    *
-   * @param rangeStart              The query range start completion time.
-   * @param rangeEnd                The query range end completion time.
+   * @param rangeStartCompletionTime              The query range start completion time.
+   * @param rangeEndCompletionTime                The query range end completion time.
    * @param earliestInstantTimeFunc The function to generate the earliest start time boundary
    *                                with the minimum completion time.
    *
    * @return The sorted instant time list.
    */
   @VisibleForTesting
   public List<String> getStartTimes(
-      String rangeStart,
-      String rangeEnd,
+      String rangeStartCompletionTime,
+      String rangeEndCompletionTime,
       Function<String, String> earliestInstantTimeFunc) {
-    return getStartTimes(metaClient.getCommitsTimeline().filterCompletedInstants(), Option.ofNullable(rangeStart), Option.ofNullable(rangeEnd),
-        InstantRange.RangeType.CLOSED_CLOSED, earliestInstantTimeFunc);
+    return getStartTimes(
+        metaClient.getCommitsTimeline().filterCompletedInstants(),
+        Option.ofNullable(rangeStartCompletionTime),
+        Option.ofNullable(rangeEndCompletionTime),
+        InstantRange.RangeType.CLOSED_CLOSED,
+        earliestInstantTimeFunc);
+  }
+
+  /**
+   * Queries the instant start time with given completion time range.
+   *
+   * @param rangeStartCompletionTime              The query range start completion time.
+   * @param rangeEndCompletionTime                The query range end completion time.
+   * @param rangeType               The range type.
+   *                                with the minimum completion time.
+   *
+   * @return The sorted instant time list.
+   */
+  @VisibleForTesting
+  public List<String> getStartTimes(

Review Comment:
   Looks like this is not used any more.



##########
hudi-common/src/main/java/org/apache/hudi/common/table/timeline/HoodieDefaultTimeline.java:
##########
@@ -629,16 +649,23 @@ private Set<String> getOrCreatePendingClusteringInstantSet() {
   /**
    * Returns the first non savepoint commit on the timeline.
    */
-  private static Option<HoodieInstant> findFirstNonSavepointCommit(List<HoodieInstant> instants) {
+  private static Option<HoodieInstant> findFirstNonSavepointCommit(
+      List<HoodieInstant> instants,
+      boolean byCompletionTime) {
     Set<String> savepointTimestamps = instants.stream()
         .filter(entry -> entry.getAction().equals(HoodieTimeline.SAVEPOINT_ACTION))
-        .map(HoodieInstant::getTimestamp)
+        .map(byCompletionTime
+            ? HoodieInstant::getCompletionTime
+            : HoodieInstant::getTimestamp)
         .collect(Collectors.toSet());
     if (!savepointTimestamps.isEmpty()) {
       // There are chances that there could be holes in the timeline due to archival and savepoint interplay.
       // So, the first non-savepoint commit is considered as beginning of the active timeline.
       return Option.fromJavaOptional(instants.stream()
-          .filter(entry -> !savepointTimestamps.contains(entry.getTimestamp()))
+          .filter(entry -> !savepointTimestamps.contains(

Review Comment:
   This still assumes that the `instants` is sorted by completion time to give the correct results, but the passed-in `instants` is always sorted by instant time.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] [HUDI-8141] Incremental Query with Completion Time [hudi]

Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.

hudi-bot commented on PR #11947:
URL: https://github.com/apache/hudi/pull/11947#issuecomment-2415346095

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "090f256a315cbbf0c6a54d93cc2739dd693360e6",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=484",
       "triggerID" : "090f256a315cbbf0c6a54d93cc2739dd693360e6",
       "triggerType" : "PUSH"
     }, {
       "hash" : "0df5649013c1cf9e782fcbb67b8a2e7174f94512",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=985",
       "triggerID" : "0df5649013c1cf9e782fcbb67b8a2e7174f94512",
       "triggerType" : "PUSH"
     }, {
       "hash" : "d52e0128f409918644255abbe4376dc4a780bfc8",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=986",
       "triggerID" : "d52e0128f409918644255abbe4376dc4a780bfc8",
       "triggerType" : "PUSH"
     }, {
       "hash" : "9e668f65c6d4f73c76573236f8fa507dfb80edac",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "9e668f65c6d4f73c76573236f8fa507dfb80edac",
       "triggerType" : "PUSH"
     }, {
       "hash" : "f145131d524bf190986e5de600a326e32a27beca",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=987",
       "triggerID" : "f145131d524bf190986e5de600a326e32a27beca",
       "triggerType" : "PUSH"
     }, {
       "hash" : "0b008f807b42aa6fb2686922bb88d63e84a1f299",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=993",
       "triggerID" : "0b008f807b42aa6fb2686922bb88d63e84a1f299",
       "triggerType" : "PUSH"
     }, {
       "hash" : "65c01ad9f2b33d36d4890355feef91844438e732",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=995",
       "triggerID" : "65c01ad9f2b33d36d4890355feef91844438e732",
       "triggerType" : "PUSH"
     }, {
       "hash" : "955493fdc93842e8c99529cec35e39cecdbe9638",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=996",
       "triggerID" : "955493fdc93842e8c99529cec35e39cecdbe9638",
       "triggerType" : "PUSH"
     }, {
       "hash" : "533f00e5546da0026bbe51e1c030ddd1c19fdd08",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "533f00e5546da0026bbe51e1c030ddd1c19fdd08",
       "triggerType" : "PUSH"
     }, {
       "hash" : "05777be67e37f3a6a8eaa59e54d9e17e5fc6ba9b",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1016",
       "triggerID" : "05777be67e37f3a6a8eaa59e54d9e17e5fc6ba9b",
       "triggerType" : "PUSH"
     }, {
       "hash" : "a2ee90c549332a7f17a7d9b2799849b77c08a8e6",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "a2ee90c549332a7f17a7d9b2799849b77c08a8e6",
       "triggerType" : "PUSH"
     }, {
       "hash" : "05ce568b0b6ba14da9ca40d26b39ba478837d747",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1017",
       "triggerID" : "05ce568b0b6ba14da9ca40d26b39ba478837d747",
       "triggerType" : "PUSH"
     }, {
       "hash" : "840e3b33d39b1f86367bb6b0e65eadeb33de7765",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1025",
       "triggerID" : "840e3b33d39b1f86367bb6b0e65eadeb33de7765",
       "triggerType" : "PUSH"
     }, {
       "hash" : "8d2e2dd83f51d03e890d424a51d3e0e84e904a43",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1031",
       "triggerID" : "8d2e2dd83f51d03e890d424a51d3e0e84e904a43",
       "triggerType" : "PUSH"
     }, {
       "hash" : "688381914087ff11acf2f8c9372dbf39afa0bd40",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1035",
       "triggerID" : "688381914087ff11acf2f8c9372dbf39afa0bd40",
       "triggerType" : "PUSH"
     }, {
       "hash" : "fc2c138901e82ed5f3fbf363c2a0ce8140ade050",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "fc2c138901e82ed5f3fbf363c2a0ce8140ade050",
       "triggerType" : "PUSH"
     }, {
       "hash" : "6ed9e1cb1e5a191752d6d39c7a9d5d41840f0cdf",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1039",
       "triggerID" : "6ed9e1cb1e5a191752d6d39c7a9d5d41840f0cdf",
       "triggerType" : "PUSH"
     }, {
       "hash" : "a510a7de1458bb996df5bbb341e8d010e4c91381",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "a510a7de1458bb996df5bbb341e8d010e4c91381",
       "triggerType" : "PUSH"
     }, {
       "hash" : "49677397cd117228742205944606f7a789090376",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1042",
       "triggerID" : "49677397cd117228742205944606f7a789090376",
       "triggerType" : "PUSH"
     }, {
       "hash" : "1630db0a37422aed942ff280560a9d6329df5015",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1044",
       "triggerID" : "1630db0a37422aed942ff280560a9d6329df5015",
       "triggerType" : "PUSH"
     }, {
       "hash" : "f11c76301d4ab524db0af57aa831be3a898ea0fb",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1048",
       "triggerID" : "f11c76301d4ab524db0af57aa831be3a898ea0fb",
       "triggerType" : "PUSH"
     }, {
       "hash" : "d1a0dc4c2017853a7ba0c3302f886043a63f29da",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1050",
       "triggerID" : "d1a0dc4c2017853a7ba0c3302f886043a63f29da",
       "triggerType" : "PUSH"
     }, {
       "hash" : "6a4c15ec39ded1f90b113b272554217698a98e1c",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1087",
       "triggerID" : "6a4c15ec39ded1f90b113b272554217698a98e1c",
       "triggerType" : "PUSH"
     }, {
       "hash" : "288e730d645a5e58e50a87ef5f44ba11a8fb0533",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1092",
       "triggerID" : "288e730d645a5e58e50a87ef5f44ba11a8fb0533",
       "triggerType" : "PUSH"
     }, {
       "hash" : "70a5c21321a0da3a0be8d85e206c24befa7b89f1",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1095",
       "triggerID" : "70a5c21321a0da3a0be8d85e206c24befa7b89f1",
       "triggerType" : "PUSH"
     }, {
       "hash" : "f072a986114949c5e426ce64a5f5568fc718e25d",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "f072a986114949c5e426ce64a5f5568fc718e25d",
       "triggerType" : "PUSH"
     }, {
       "hash" : "45b27475b1e8806ec8c566d6f151f4bc79c8549d",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1096",
       "triggerID" : "45b27475b1e8806ec8c566d6f151f4bc79c8549d",
       "triggerType" : "PUSH"
     }, {
       "hash" : "3c62ef22f6cde28110e5eb2fca8760a008d77723",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "3c62ef22f6cde28110e5eb2fca8760a008d77723",
       "triggerType" : "PUSH"
     }, {
       "hash" : "dc785fa04e542b2ccc45494e0292f5153c7ecea8",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1097",
       "triggerID" : "dc785fa04e542b2ccc45494e0292f5153c7ecea8",
       "triggerType" : "PUSH"
     }, {
       "hash" : "585ddb8049b9b9ccb057027e265b08a1b586df8c",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "585ddb8049b9b9ccb057027e265b08a1b586df8c",
       "triggerType" : "PUSH"
     }, {
       "hash" : "b62ff8675beed2c4487c788b9ca1a0ad4271b913",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "b62ff8675beed2c4487c788b9ca1a0ad4271b913",
       "triggerType" : "PUSH"
     }, {
       "hash" : "2d46048a7402907f348cc96da2839ba1b3e4932a",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1099",
       "triggerID" : "2d46048a7402907f348cc96da2839ba1b3e4932a",
       "triggerType" : "PUSH"
     }, {
       "hash" : "004fb6ebf23f796a4b77650e0530f0f1145e8b91",
       "status" : "SUCCESS",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1101",
       "triggerID" : "004fb6ebf23f796a4b77650e0530f0f1145e8b91",
       "triggerType" : "PUSH"
     }, {
       "hash" : "af0c33333b5e9ae6d59f20a6db1d7c016ea015d3",
       "status" : "PENDING",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1165",
       "triggerID" : "af0c33333b5e9ae6d59f20a6db1d7c016ea015d3",
       "triggerType" : "PUSH"
     }, {
       "hash" : "da8111118d71306500229619d3b7996fcd2526e8",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "da8111118d71306500229619d3b7996fcd2526e8",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 9e668f65c6d4f73c76573236f8fa507dfb80edac UNKNOWN
   * 533f00e5546da0026bbe51e1c030ddd1c19fdd08 UNKNOWN
   * a2ee90c549332a7f17a7d9b2799849b77c08a8e6 UNKNOWN
   * fc2c138901e82ed5f3fbf363c2a0ce8140ade050 UNKNOWN
   * a510a7de1458bb996df5bbb341e8d010e4c91381 UNKNOWN
   * f072a986114949c5e426ce64a5f5568fc718e25d UNKNOWN
   * 3c62ef22f6cde28110e5eb2fca8760a008d77723 UNKNOWN
   * 585ddb8049b9b9ccb057027e265b08a1b586df8c UNKNOWN
   * b62ff8675beed2c4487c788b9ca1a0ad4271b913 UNKNOWN
   * 004fb6ebf23f796a4b77650e0530f0f1145e8b91 Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1101) 
   * af0c33333b5e9ae6d59f20a6db1d7c016ea015d3 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1165) 
   * da8111118d71306500229619d3b7996fcd2526e8 UNKNOWN
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] [HUDI-8141] Incremental Query with Completion Time [hudi]

Posted by "yihua (via GitHub)" <gi...@apache.org>.

yihua commented on code in PR #11947:
URL: https://github.com/apache/hudi/pull/11947#discussion_r1801755856


##########
hudi-utilities/src/main/java/org/apache/hudi/utilities/sources/HoodieIncrSource.java:
##########
@@ -175,33 +174,25 @@ public Pair<Option<Dataset<Row>>, String> fetchNextBatch(Option<String> lastCkpt
     IncrSourceHelper.MissingCheckpointStrategy missingCheckpointStrategy =
         (containsConfigProperty(props, HoodieIncrSourceConfig.MISSING_CHECKPOINT_STRATEGY))
             ? IncrSourceHelper.MissingCheckpointStrategy.valueOf(
-            getStringWithAltKeys(props, HoodieIncrSourceConfig.MISSING_CHECKPOINT_STRATEGY))
+                getStringWithAltKeys(props, HoodieIncrSourceConfig.MISSING_CHECKPOINT_STRATEGY))
             : null;
     if (readLatestOnMissingCkpt) {
       missingCheckpointStrategy = IncrSourceHelper.MissingCheckpointStrategy.READ_LATEST;
     }
 
-    // Use begin Instant if set and non-empty
-    Option<String> beginInstant =
-        lastCkptStr.isPresent() ? lastCkptStr.get().isEmpty() ? Option.empty() : lastCkptStr : Option.empty();
+    IncrementalQueryAnalyzer analyzer = IncrSourceHelper.getIncrementalQueryAnalyzer(
+        sparkContext, srcPath, lastCkptStr, missingCheckpointStrategy,
+        getIntWithAltKeys(props, HoodieIncrSourceConfig.NUM_INSTANTS_PER_FETCH),
+        getLatestSourceProfile());

Review Comment:
   The `IncrementalQueryAnalyzer` scans the archived timeline if the lastCkptStr falls into the archived timeline range, especially the first incremental pull without the last checkpoint, and the number of instants included in the predicate can be huge.  So we should see if this part can be improved.



##########
hudi-utilities/src/main/java/org/apache/hudi/utilities/sources/helpers/IncrSourceHelper.java:
##########
@@ -277,6 +331,23 @@ public static Dataset<Row> coalesceOrRepartition(Dataset dataset, int numPartiti
     return dataset;
   }
 
+  /**
+   * Copy IncrementalQueryAnalyzer.QueryContext with info from CheckpointWithPredicates updated
+   * */
+  public static QueryContext withUpdatedCheckpoint(

Review Comment:
   ```suggestion
     public static QueryContext withUpdatedCheckpointAndPredicate(
   ```



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] [HUDI-8141] Incremental Query with Completion Time [hudi]

Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.

hudi-bot commented on PR #11947:
URL: https://github.com/apache/hudi/pull/11947#issuecomment-2416053804

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "090f256a315cbbf0c6a54d93cc2739dd693360e6",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=484",
       "triggerID" : "090f256a315cbbf0c6a54d93cc2739dd693360e6",
       "triggerType" : "PUSH"
     }, {
       "hash" : "0df5649013c1cf9e782fcbb67b8a2e7174f94512",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=985",
       "triggerID" : "0df5649013c1cf9e782fcbb67b8a2e7174f94512",
       "triggerType" : "PUSH"
     }, {
       "hash" : "d52e0128f409918644255abbe4376dc4a780bfc8",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=986",
       "triggerID" : "d52e0128f409918644255abbe4376dc4a780bfc8",
       "triggerType" : "PUSH"
     }, {
       "hash" : "9e668f65c6d4f73c76573236f8fa507dfb80edac",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "9e668f65c6d4f73c76573236f8fa507dfb80edac",
       "triggerType" : "PUSH"
     }, {
       "hash" : "f145131d524bf190986e5de600a326e32a27beca",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=987",
       "triggerID" : "f145131d524bf190986e5de600a326e32a27beca",
       "triggerType" : "PUSH"
     }, {
       "hash" : "0b008f807b42aa6fb2686922bb88d63e84a1f299",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=993",
       "triggerID" : "0b008f807b42aa6fb2686922bb88d63e84a1f299",
       "triggerType" : "PUSH"
     }, {
       "hash" : "65c01ad9f2b33d36d4890355feef91844438e732",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=995",
       "triggerID" : "65c01ad9f2b33d36d4890355feef91844438e732",
       "triggerType" : "PUSH"
     }, {
       "hash" : "955493fdc93842e8c99529cec35e39cecdbe9638",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=996",
       "triggerID" : "955493fdc93842e8c99529cec35e39cecdbe9638",
       "triggerType" : "PUSH"
     }, {
       "hash" : "533f00e5546da0026bbe51e1c030ddd1c19fdd08",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "533f00e5546da0026bbe51e1c030ddd1c19fdd08",
       "triggerType" : "PUSH"
     }, {
       "hash" : "05777be67e37f3a6a8eaa59e54d9e17e5fc6ba9b",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1016",
       "triggerID" : "05777be67e37f3a6a8eaa59e54d9e17e5fc6ba9b",
       "triggerType" : "PUSH"
     }, {
       "hash" : "a2ee90c549332a7f17a7d9b2799849b77c08a8e6",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "a2ee90c549332a7f17a7d9b2799849b77c08a8e6",
       "triggerType" : "PUSH"
     }, {
       "hash" : "05ce568b0b6ba14da9ca40d26b39ba478837d747",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1017",
       "triggerID" : "05ce568b0b6ba14da9ca40d26b39ba478837d747",
       "triggerType" : "PUSH"
     }, {
       "hash" : "840e3b33d39b1f86367bb6b0e65eadeb33de7765",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1025",
       "triggerID" : "840e3b33d39b1f86367bb6b0e65eadeb33de7765",
       "triggerType" : "PUSH"
     }, {
       "hash" : "8d2e2dd83f51d03e890d424a51d3e0e84e904a43",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1031",
       "triggerID" : "8d2e2dd83f51d03e890d424a51d3e0e84e904a43",
       "triggerType" : "PUSH"
     }, {
       "hash" : "688381914087ff11acf2f8c9372dbf39afa0bd40",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1035",
       "triggerID" : "688381914087ff11acf2f8c9372dbf39afa0bd40",
       "triggerType" : "PUSH"
     }, {
       "hash" : "fc2c138901e82ed5f3fbf363c2a0ce8140ade050",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "fc2c138901e82ed5f3fbf363c2a0ce8140ade050",
       "triggerType" : "PUSH"
     }, {
       "hash" : "6ed9e1cb1e5a191752d6d39c7a9d5d41840f0cdf",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1039",
       "triggerID" : "6ed9e1cb1e5a191752d6d39c7a9d5d41840f0cdf",
       "triggerType" : "PUSH"
     }, {
       "hash" : "a510a7de1458bb996df5bbb341e8d010e4c91381",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "a510a7de1458bb996df5bbb341e8d010e4c91381",
       "triggerType" : "PUSH"
     }, {
       "hash" : "49677397cd117228742205944606f7a789090376",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1042",
       "triggerID" : "49677397cd117228742205944606f7a789090376",
       "triggerType" : "PUSH"
     }, {
       "hash" : "1630db0a37422aed942ff280560a9d6329df5015",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1044",
       "triggerID" : "1630db0a37422aed942ff280560a9d6329df5015",
       "triggerType" : "PUSH"
     }, {
       "hash" : "f11c76301d4ab524db0af57aa831be3a898ea0fb",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1048",
       "triggerID" : "f11c76301d4ab524db0af57aa831be3a898ea0fb",
       "triggerType" : "PUSH"
     }, {
       "hash" : "d1a0dc4c2017853a7ba0c3302f886043a63f29da",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1050",
       "triggerID" : "d1a0dc4c2017853a7ba0c3302f886043a63f29da",
       "triggerType" : "PUSH"
     }, {
       "hash" : "6a4c15ec39ded1f90b113b272554217698a98e1c",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1087",
       "triggerID" : "6a4c15ec39ded1f90b113b272554217698a98e1c",
       "triggerType" : "PUSH"
     }, {
       "hash" : "288e730d645a5e58e50a87ef5f44ba11a8fb0533",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1092",
       "triggerID" : "288e730d645a5e58e50a87ef5f44ba11a8fb0533",
       "triggerType" : "PUSH"
     }, {
       "hash" : "70a5c21321a0da3a0be8d85e206c24befa7b89f1",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1095",
       "triggerID" : "70a5c21321a0da3a0be8d85e206c24befa7b89f1",
       "triggerType" : "PUSH"
     }, {
       "hash" : "f072a986114949c5e426ce64a5f5568fc718e25d",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "f072a986114949c5e426ce64a5f5568fc718e25d",
       "triggerType" : "PUSH"
     }, {
       "hash" : "45b27475b1e8806ec8c566d6f151f4bc79c8549d",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1096",
       "triggerID" : "45b27475b1e8806ec8c566d6f151f4bc79c8549d",
       "triggerType" : "PUSH"
     }, {
       "hash" : "3c62ef22f6cde28110e5eb2fca8760a008d77723",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "3c62ef22f6cde28110e5eb2fca8760a008d77723",
       "triggerType" : "PUSH"
     }, {
       "hash" : "dc785fa04e542b2ccc45494e0292f5153c7ecea8",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1097",
       "triggerID" : "dc785fa04e542b2ccc45494e0292f5153c7ecea8",
       "triggerType" : "PUSH"
     }, {
       "hash" : "585ddb8049b9b9ccb057027e265b08a1b586df8c",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "585ddb8049b9b9ccb057027e265b08a1b586df8c",
       "triggerType" : "PUSH"
     }, {
       "hash" : "b62ff8675beed2c4487c788b9ca1a0ad4271b913",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "b62ff8675beed2c4487c788b9ca1a0ad4271b913",
       "triggerType" : "PUSH"
     }, {
       "hash" : "2d46048a7402907f348cc96da2839ba1b3e4932a",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1099",
       "triggerID" : "2d46048a7402907f348cc96da2839ba1b3e4932a",
       "triggerType" : "PUSH"
     }, {
       "hash" : "004fb6ebf23f796a4b77650e0530f0f1145e8b91",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1101",
       "triggerID" : "004fb6ebf23f796a4b77650e0530f0f1145e8b91",
       "triggerType" : "PUSH"
     }, {
       "hash" : "af0c33333b5e9ae6d59f20a6db1d7c016ea015d3",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1165",
       "triggerID" : "af0c33333b5e9ae6d59f20a6db1d7c016ea015d3",
       "triggerType" : "PUSH"
     }, {
       "hash" : "da8111118d71306500229619d3b7996fcd2526e8",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "da8111118d71306500229619d3b7996fcd2526e8",
       "triggerType" : "PUSH"
     }, {
       "hash" : "83257e2b9bd3c426cf90e11003a5f244a750d7fa",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1167",
       "triggerID" : "83257e2b9bd3c426cf90e11003a5f244a750d7fa",
       "triggerType" : "PUSH"
     }, {
       "hash" : "e0de408f1b8153fdaaea4c87b27a7b69213b75b5",
       "status" : "CANCELED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1177",
       "triggerID" : "e0de408f1b8153fdaaea4c87b27a7b69213b75b5",
       "triggerType" : "PUSH"
     }, {
       "hash" : "3f7951f7ad7931f5dc0b977d7c33bc5429623535",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "3f7951f7ad7931f5dc0b977d7c33bc5429623535",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 9e668f65c6d4f73c76573236f8fa507dfb80edac UNKNOWN
   * 533f00e5546da0026bbe51e1c030ddd1c19fdd08 UNKNOWN
   * a2ee90c549332a7f17a7d9b2799849b77c08a8e6 UNKNOWN
   * fc2c138901e82ed5f3fbf363c2a0ce8140ade050 UNKNOWN
   * a510a7de1458bb996df5bbb341e8d010e4c91381 UNKNOWN
   * f072a986114949c5e426ce64a5f5568fc718e25d UNKNOWN
   * 3c62ef22f6cde28110e5eb2fca8760a008d77723 UNKNOWN
   * 585ddb8049b9b9ccb057027e265b08a1b586df8c UNKNOWN
   * b62ff8675beed2c4487c788b9ca1a0ad4271b913 UNKNOWN
   * da8111118d71306500229619d3b7996fcd2526e8 UNKNOWN
   * e0de408f1b8153fdaaea4c87b27a7b69213b75b5 Azure: [CANCELED](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1177) 
   * 3f7951f7ad7931f5dc0b977d7c33bc5429623535 UNKNOWN
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] [HUDI-8141] Incremental Query with Completion Time [hudi]

Posted by "danny0405 (via GitHub)" <gi...@apache.org>.

danny0405 commented on code in PR #11947:
URL: https://github.com/apache/hudi/pull/11947#discussion_r1798653269


##########
hudi-common/src/main/java/org/apache/hudi/common/table/timeline/CompletionTimeQueryView.java:
##########
@@ -189,46 +193,78 @@ public Option<String> getCompletionTime(String beginTime) {
    * <p>By default, assumes there is at most 1 day time of duration for an instant to accelerate the queries.
    *
    * @param timeline The timeline.
-   * @param rangeStart   The query range start completion time.
-   * @param rangeEnd     The query range end completion time.
+   * @param rangeStartCompletionTime   The query range start completion time.
+   * @param rangeEndCompletionTime     The query range end completion time.
    * @param rangeType    The range type.
    *
    * @return The sorted instant time list.
    */
   public List<String> getStartTimes(
       HoodieTimeline timeline,
-      Option<String> rangeStart,
-      Option<String> rangeEnd,
+      Option<String> rangeStartCompletionTime,
+      Option<String> rangeEndCompletionTime,
       InstantRange.RangeType rangeType) {
     // assumes any instant/transaction lasts at most 1 day to optimize the query efficiency.
-    return getStartTimes(timeline, rangeStart, rangeEnd, rangeType, s -> HoodieInstantTimeGenerator.instantTimeMinusMillis(s, MILLI_SECONDS_IN_ONE_DAY));
+    return getStartTimes(
+        timeline,
+        rangeStartCompletionTime,
+        rangeEndCompletionTime,
+        rangeType,
+        GET_INSTANT_ONE_DAY_BEFORE);
   }
 
   /**
    * Queries the instant start time with given completion time range.
    *
-   * @param rangeStart              The query range start completion time.
-   * @param rangeEnd                The query range end completion time.
+   * @param rangeStartCompletionTime              The query range start completion time.
+   * @param rangeEndCompletionTime                The query range end completion time.
    * @param earliestInstantTimeFunc The function to generate the earliest start time boundary
    *                                with the minimum completion time.
    *
    * @return The sorted instant time list.
    */
   @VisibleForTesting
   public List<String> getStartTimes(
-      String rangeStart,
-      String rangeEnd,
+      String rangeStartCompletionTime,
+      String rangeEndCompletionTime,
       Function<String, String> earliestInstantTimeFunc) {
-    return getStartTimes(metaClient.getCommitsTimeline().filterCompletedInstants(), Option.ofNullable(rangeStart), Option.ofNullable(rangeEnd),
-        InstantRange.RangeType.CLOSED_CLOSED, earliestInstantTimeFunc);
+    return getStartTimes(
+        metaClient.getCommitsTimeline().filterCompletedInstants(),
+        Option.ofNullable(rangeStartCompletionTime),
+        Option.ofNullable(rangeEndCompletionTime),
+        InstantRange.RangeType.CLOSED_CLOSED,
+        earliestInstantTimeFunc);
+  }
+
+  /**
+   * Queries the instant start time with given completion time range.
+   *
+   * @param rangeStartCompletionTime              The query range start completion time.
+   * @param rangeEndCompletionTime                The query range end completion time.
+   * @param rangeType               The range type.
+   *                                with the minimum completion time.
+   *
+   * @return The sorted instant time list.
+   */
+  @VisibleForTesting
+  public List<String> getStartTimes(
+      String rangeStartCompletionTime,
+      String rangeEndCompletionTime,

Review Comment:
   ditto



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] [HUDI-8141] Incremental Query with Completion Time [hudi]

Posted by "yihua (via GitHub)" <gi...@apache.org>.

yihua commented on code in PR #11947:
URL: https://github.com/apache/hudi/pull/11947#discussion_r1804270406


##########
hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/hudi/IncrementalRelation.scala:
##########
@@ -90,24 +78,24 @@ class IncrementalRelation(val sqlContext: SQLContext,
     throw new HoodieException("Incremental queries are not supported when meta fields are disabled")
   }
 
+  private val queryContext: IncrementalQueryAnalyzer.QueryContext =
+    IncrementalQueryAnalyzer.builder()
+      .metaClient(metaClient)
+      .startTime(optParams(DataSourceReadOptions.BEGIN_INSTANTTIME.key))
+      .endTime(optParams.getOrElse(DataSourceReadOptions.END_INSTANTTIME.key, null))
+      .rangeType(InstantRange.RangeType.OPEN_CLOSED)

Review Comment:
   I've fixed that to be `CLOSED_CLOSED` in my latter git commits.  The Hudi incremental source or stream source first determines the range of instants though the incremental query analyzer with either `OPEN_CLOSED` range for resuming from the last checkpoint, or `CLOSED_CLOSED` range for the first batch without checkpoint.  Then the begin and end completion is passed down to the relation with both as inclusive.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] [HUDI-8141] Incremental Query with Completion Time [hudi]

Posted by "yihua (via GitHub)" <gi...@apache.org>.

yihua commented on code in PR #11947:
URL: https://github.com/apache/hudi/pull/11947#discussion_r1797491098


##########
hudi-utilities/src/main/java/org/apache/hudi/utilities/sources/HoodieIncrSource.java:
##########
@@ -212,35 +257,43 @@ public Pair<Option<Dataset<Row>>, String> fetchNextBatch(Option<String> lastCkpt
           .collect(Collectors.toMap(Pair::getLeft, Pair::getRight));
       reader = reader.options(optionsMap);
     }
+    boolean shouldFullScan =
+        queryContext.getActiveTimeline().isBeforeTimelineStartsByCompletionTime(startTime)
+        && missingCheckpointStrategy == MissingCheckpointStrategy.READ_UPTO_LATEST_COMMIT;
     Dataset<Row> source;
-    // Do Incr pull. Set end instant if available
-    if (queryInfo.isIncremental()) {
-      source = reader
-          .options(readOpts)
-          .option(QUERY_TYPE().key(), QUERY_TYPE_INCREMENTAL_OPT_VAL())
-          .option(BEGIN_INSTANTTIME().key(), queryInfo.getStartInstant())
-          .option(END_INSTANTTIME().key(), queryInfo.getEndInstant())
-          .option(INCREMENTAL_FALLBACK_TO_FULL_TABLE_SCAN().key(),
-              props.getString(INCREMENTAL_FALLBACK_TO_FULL_TABLE_SCAN().key(),
-                  INCREMENTAL_FALLBACK_TO_FULL_TABLE_SCAN().defaultValue()))
-          .option(INCREMENTAL_READ_HANDLE_HOLLOW_COMMIT().key(), handlingMode.name())
-          .load(srcPath);
-    } else {
-      // if checkpoint is missing from source table, and if strategy is set to READ_UPTO_LATEST_COMMIT, we have to issue snapshot query
+    if (instantRange.isEmpty() || shouldFullScan) {
+      // snapshot query
       Dataset<Row> snapshot = reader
           .options(readOpts)
           .option(DataSourceReadOptions.QUERY_TYPE().key(), DataSourceReadOptions.QUERY_TYPE_SNAPSHOT_OPT_VAL())
           .load(srcPath);
       if (snapshotLoadQuerySplitter.isPresent()) {
-        queryInfo = snapshotLoadQuerySplitter.get().getNextCheckpoint(snapshot, queryInfo, sourceProfileSupplier);
+        queryContext = snapshotLoadQuerySplitter.get().getNextCheckpoint(snapshot, queryContext, sourceProfileSupplier);
+        // update endTime/next checkpoint
+        endTime = queryContext.getEndInstant().orElse(queryContext.getLastInstant());
       }
+      Set<String> validInstants = new HashSet<>(queryContext.getInstants());
+      snapshot = queryContext.getPredicateFilter().map(snapshot::filter).orElse(snapshot);
       source = snapshot
           // add filtering so that only interested records are returned.
-          .filter(String.format("%s > '%s'", HoodieRecord.COMMIT_TIME_METADATA_FIELD,
-              queryInfo.getStartInstant()))
+          // completion time comparison uses ( , ], but when comparing start time we need to use [, ]
+          .filter(String.format("%s >= '%s'", HoodieRecord.COMMIT_TIME_METADATA_FIELD,
+              queryContext.getStartInstant().get()))
           .filter(String.format("%s <= '%s'", HoodieRecord.COMMIT_TIME_METADATA_FIELD,
-              queryInfo.getEndInstant()));
-      source = queryInfo.getPredicateFilter().map(source::filter).orElse(source);
+              queryContext.getEndInstant().orElse(queryContext.getLastInstant())))
+          .filter((Row row) -> validInstants.contains(row.getAs(HoodieRecord.COMMIT_TIME_METADATA_FIELD)));

Review Comment:
   Does this translate into `IN` predicate (`new Column(HoodieRecord.COMMIT_TIME_METADATA_FIELD).isin(validInstants)`) which can be pushed down?  We should avoid scanning all rows for evaluating this predicate.



##########
hudi-utilities/src/main/java/org/apache/hudi/utilities/sources/HoodieIncrSource.java:
##########
@@ -212,35 +257,43 @@ public Pair<Option<Dataset<Row>>, String> fetchNextBatch(Option<String> lastCkpt
           .collect(Collectors.toMap(Pair::getLeft, Pair::getRight));
       reader = reader.options(optionsMap);
     }
+    boolean shouldFullScan =
+        queryContext.getActiveTimeline().isBeforeTimelineStartsByCompletionTime(startTime)
+        && missingCheckpointStrategy == MissingCheckpointStrategy.READ_UPTO_LATEST_COMMIT;
     Dataset<Row> source;
-    // Do Incr pull. Set end instant if available
-    if (queryInfo.isIncremental()) {
-      source = reader
-          .options(readOpts)
-          .option(QUERY_TYPE().key(), QUERY_TYPE_INCREMENTAL_OPT_VAL())
-          .option(BEGIN_INSTANTTIME().key(), queryInfo.getStartInstant())
-          .option(END_INSTANTTIME().key(), queryInfo.getEndInstant())
-          .option(INCREMENTAL_FALLBACK_TO_FULL_TABLE_SCAN().key(),
-              props.getString(INCREMENTAL_FALLBACK_TO_FULL_TABLE_SCAN().key(),
-                  INCREMENTAL_FALLBACK_TO_FULL_TABLE_SCAN().defaultValue()))
-          .option(INCREMENTAL_READ_HANDLE_HOLLOW_COMMIT().key(), handlingMode.name())
-          .load(srcPath);
-    } else {
-      // if checkpoint is missing from source table, and if strategy is set to READ_UPTO_LATEST_COMMIT, we have to issue snapshot query
+    if (instantRange.isEmpty() || shouldFullScan) {
+      // snapshot query
       Dataset<Row> snapshot = reader
           .options(readOpts)
           .option(DataSourceReadOptions.QUERY_TYPE().key(), DataSourceReadOptions.QUERY_TYPE_SNAPSHOT_OPT_VAL())
           .load(srcPath);
       if (snapshotLoadQuerySplitter.isPresent()) {
-        queryInfo = snapshotLoadQuerySplitter.get().getNextCheckpoint(snapshot, queryInfo, sourceProfileSupplier);
+        queryContext = snapshotLoadQuerySplitter.get().getNextCheckpoint(snapshot, queryContext, sourceProfileSupplier);
+        // update endTime/next checkpoint
+        endTime = queryContext.getEndInstant().orElse(queryContext.getLastInstant());
       }
+      Set<String> validInstants = new HashSet<>(queryContext.getInstants());
+      snapshot = queryContext.getPredicateFilter().map(snapshot::filter).orElse(snapshot);

Review Comment:
   It looks like the `predicateFilter` is never set?



##########
hudi-utilities/src/main/java/org/apache/hudi/utilities/sources/HoodieIncrSource.java:
##########
@@ -212,35 +257,43 @@ public Pair<Option<Dataset<Row>>, String> fetchNextBatch(Option<String> lastCkpt
           .collect(Collectors.toMap(Pair::getLeft, Pair::getRight));
       reader = reader.options(optionsMap);
     }
+    boolean shouldFullScan =
+        queryContext.getActiveTimeline().isBeforeTimelineStartsByCompletionTime(startTime)
+        && missingCheckpointStrategy == MissingCheckpointStrategy.READ_UPTO_LATEST_COMMIT;
     Dataset<Row> source;
-    // Do Incr pull. Set end instant if available
-    if (queryInfo.isIncremental()) {
-      source = reader
-          .options(readOpts)
-          .option(QUERY_TYPE().key(), QUERY_TYPE_INCREMENTAL_OPT_VAL())
-          .option(BEGIN_INSTANTTIME().key(), queryInfo.getStartInstant())
-          .option(END_INSTANTTIME().key(), queryInfo.getEndInstant())
-          .option(INCREMENTAL_FALLBACK_TO_FULL_TABLE_SCAN().key(),
-              props.getString(INCREMENTAL_FALLBACK_TO_FULL_TABLE_SCAN().key(),
-                  INCREMENTAL_FALLBACK_TO_FULL_TABLE_SCAN().defaultValue()))
-          .option(INCREMENTAL_READ_HANDLE_HOLLOW_COMMIT().key(), handlingMode.name())
-          .load(srcPath);
-    } else {
-      // if checkpoint is missing from source table, and if strategy is set to READ_UPTO_LATEST_COMMIT, we have to issue snapshot query
+    if (instantRange.isEmpty() || shouldFullScan) {
+      // snapshot query
       Dataset<Row> snapshot = reader
           .options(readOpts)
           .option(DataSourceReadOptions.QUERY_TYPE().key(), DataSourceReadOptions.QUERY_TYPE_SNAPSHOT_OPT_VAL())
           .load(srcPath);
       if (snapshotLoadQuerySplitter.isPresent()) {
-        queryInfo = snapshotLoadQuerySplitter.get().getNextCheckpoint(snapshot, queryInfo, sourceProfileSupplier);
+        queryContext = snapshotLoadQuerySplitter.get().getNextCheckpoint(snapshot, queryContext, sourceProfileSupplier);
+        // update endTime/next checkpoint
+        endTime = queryContext.getEndInstant().orElse(queryContext.getLastInstant());
       }
+      Set<String> validInstants = new HashSet<>(queryContext.getInstants());
+      snapshot = queryContext.getPredicateFilter().map(snapshot::filter).orElse(snapshot);
       source = snapshot
           // add filtering so that only interested records are returned.
-          .filter(String.format("%s > '%s'", HoodieRecord.COMMIT_TIME_METADATA_FIELD,
-              queryInfo.getStartInstant()))
+          // completion time comparison uses ( , ], but when comparing start time we need to use [, ]
+          .filter(String.format("%s >= '%s'", HoodieRecord.COMMIT_TIME_METADATA_FIELD,
+              queryContext.getStartInstant().get()))
           .filter(String.format("%s <= '%s'", HoodieRecord.COMMIT_TIME_METADATA_FIELD,
-              queryInfo.getEndInstant()));
-      source = queryInfo.getPredicateFilter().map(source::filter).orElse(source);
+              queryContext.getEndInstant().orElse(queryContext.getLastInstant())))

Review Comment:
   Could `queryContext.getStartInstant()` or `queryContext.getEndInstant()` be null?



##########
hudi-utilities/src/main/java/org/apache/hudi/utilities/sources/HoodieIncrSource.java:
##########
@@ -212,35 +257,43 @@ public Pair<Option<Dataset<Row>>, String> fetchNextBatch(Option<String> lastCkpt
           .collect(Collectors.toMap(Pair::getLeft, Pair::getRight));
       reader = reader.options(optionsMap);
     }
+    boolean shouldFullScan =
+        queryContext.getActiveTimeline().isBeforeTimelineStartsByCompletionTime(startTime)
+        && missingCheckpointStrategy == MissingCheckpointStrategy.READ_UPTO_LATEST_COMMIT;
     Dataset<Row> source;
-    // Do Incr pull. Set end instant if available
-    if (queryInfo.isIncremental()) {
-      source = reader
-          .options(readOpts)
-          .option(QUERY_TYPE().key(), QUERY_TYPE_INCREMENTAL_OPT_VAL())
-          .option(BEGIN_INSTANTTIME().key(), queryInfo.getStartInstant())
-          .option(END_INSTANTTIME().key(), queryInfo.getEndInstant())
-          .option(INCREMENTAL_FALLBACK_TO_FULL_TABLE_SCAN().key(),
-              props.getString(INCREMENTAL_FALLBACK_TO_FULL_TABLE_SCAN().key(),
-                  INCREMENTAL_FALLBACK_TO_FULL_TABLE_SCAN().defaultValue()))
-          .option(INCREMENTAL_READ_HANDLE_HOLLOW_COMMIT().key(), handlingMode.name())
-          .load(srcPath);
-    } else {
-      // if checkpoint is missing from source table, and if strategy is set to READ_UPTO_LATEST_COMMIT, we have to issue snapshot query
+    if (instantRange.isEmpty() || shouldFullScan) {
+      // snapshot query
       Dataset<Row> snapshot = reader
           .options(readOpts)
           .option(DataSourceReadOptions.QUERY_TYPE().key(), DataSourceReadOptions.QUERY_TYPE_SNAPSHOT_OPT_VAL())
           .load(srcPath);
       if (snapshotLoadQuerySplitter.isPresent()) {
-        queryInfo = snapshotLoadQuerySplitter.get().getNextCheckpoint(snapshot, queryInfo, sourceProfileSupplier);
+        queryContext = snapshotLoadQuerySplitter.get().getNextCheckpoint(snapshot, queryContext, sourceProfileSupplier);
+        // update endTime/next checkpoint
+        endTime = queryContext.getEndInstant().orElse(queryContext.getLastInstant());

Review Comment:
   I think we need to consider that the checkpoint can be more than the completion time itself, like `completionTime#key`.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] [HUDI-8141] Incremental Query with Completion Time [hudi]

Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.

hudi-bot commented on PR #11947:
URL: https://github.com/apache/hudi/pull/11947#issuecomment-2419013968

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "090f256a315cbbf0c6a54d93cc2739dd693360e6",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=484",
       "triggerID" : "090f256a315cbbf0c6a54d93cc2739dd693360e6",
       "triggerType" : "PUSH"
     }, {
       "hash" : "0df5649013c1cf9e782fcbb67b8a2e7174f94512",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=985",
       "triggerID" : "0df5649013c1cf9e782fcbb67b8a2e7174f94512",
       "triggerType" : "PUSH"
     }, {
       "hash" : "d52e0128f409918644255abbe4376dc4a780bfc8",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=986",
       "triggerID" : "d52e0128f409918644255abbe4376dc4a780bfc8",
       "triggerType" : "PUSH"
     }, {
       "hash" : "9e668f65c6d4f73c76573236f8fa507dfb80edac",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "9e668f65c6d4f73c76573236f8fa507dfb80edac",
       "triggerType" : "PUSH"
     }, {
       "hash" : "f145131d524bf190986e5de600a326e32a27beca",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=987",
       "triggerID" : "f145131d524bf190986e5de600a326e32a27beca",
       "triggerType" : "PUSH"
     }, {
       "hash" : "0b008f807b42aa6fb2686922bb88d63e84a1f299",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=993",
       "triggerID" : "0b008f807b42aa6fb2686922bb88d63e84a1f299",
       "triggerType" : "PUSH"
     }, {
       "hash" : "65c01ad9f2b33d36d4890355feef91844438e732",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=995",
       "triggerID" : "65c01ad9f2b33d36d4890355feef91844438e732",
       "triggerType" : "PUSH"
     }, {
       "hash" : "955493fdc93842e8c99529cec35e39cecdbe9638",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=996",
       "triggerID" : "955493fdc93842e8c99529cec35e39cecdbe9638",
       "triggerType" : "PUSH"
     }, {
       "hash" : "533f00e5546da0026bbe51e1c030ddd1c19fdd08",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "533f00e5546da0026bbe51e1c030ddd1c19fdd08",
       "triggerType" : "PUSH"
     }, {
       "hash" : "05777be67e37f3a6a8eaa59e54d9e17e5fc6ba9b",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1016",
       "triggerID" : "05777be67e37f3a6a8eaa59e54d9e17e5fc6ba9b",
       "triggerType" : "PUSH"
     }, {
       "hash" : "a2ee90c549332a7f17a7d9b2799849b77c08a8e6",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "a2ee90c549332a7f17a7d9b2799849b77c08a8e6",
       "triggerType" : "PUSH"
     }, {
       "hash" : "05ce568b0b6ba14da9ca40d26b39ba478837d747",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1017",
       "triggerID" : "05ce568b0b6ba14da9ca40d26b39ba478837d747",
       "triggerType" : "PUSH"
     }, {
       "hash" : "840e3b33d39b1f86367bb6b0e65eadeb33de7765",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1025",
       "triggerID" : "840e3b33d39b1f86367bb6b0e65eadeb33de7765",
       "triggerType" : "PUSH"
     }, {
       "hash" : "8d2e2dd83f51d03e890d424a51d3e0e84e904a43",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1031",
       "triggerID" : "8d2e2dd83f51d03e890d424a51d3e0e84e904a43",
       "triggerType" : "PUSH"
     }, {
       "hash" : "688381914087ff11acf2f8c9372dbf39afa0bd40",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1035",
       "triggerID" : "688381914087ff11acf2f8c9372dbf39afa0bd40",
       "triggerType" : "PUSH"
     }, {
       "hash" : "fc2c138901e82ed5f3fbf363c2a0ce8140ade050",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "fc2c138901e82ed5f3fbf363c2a0ce8140ade050",
       "triggerType" : "PUSH"
     }, {
       "hash" : "6ed9e1cb1e5a191752d6d39c7a9d5d41840f0cdf",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1039",
       "triggerID" : "6ed9e1cb1e5a191752d6d39c7a9d5d41840f0cdf",
       "triggerType" : "PUSH"
     }, {
       "hash" : "a510a7de1458bb996df5bbb341e8d010e4c91381",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "a510a7de1458bb996df5bbb341e8d010e4c91381",
       "triggerType" : "PUSH"
     }, {
       "hash" : "49677397cd117228742205944606f7a789090376",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1042",
       "triggerID" : "49677397cd117228742205944606f7a789090376",
       "triggerType" : "PUSH"
     }, {
       "hash" : "1630db0a37422aed942ff280560a9d6329df5015",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1044",
       "triggerID" : "1630db0a37422aed942ff280560a9d6329df5015",
       "triggerType" : "PUSH"
     }, {
       "hash" : "f11c76301d4ab524db0af57aa831be3a898ea0fb",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1048",
       "triggerID" : "f11c76301d4ab524db0af57aa831be3a898ea0fb",
       "triggerType" : "PUSH"
     }, {
       "hash" : "d1a0dc4c2017853a7ba0c3302f886043a63f29da",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1050",
       "triggerID" : "d1a0dc4c2017853a7ba0c3302f886043a63f29da",
       "triggerType" : "PUSH"
     }, {
       "hash" : "6a4c15ec39ded1f90b113b272554217698a98e1c",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1087",
       "triggerID" : "6a4c15ec39ded1f90b113b272554217698a98e1c",
       "triggerType" : "PUSH"
     }, {
       "hash" : "288e730d645a5e58e50a87ef5f44ba11a8fb0533",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1092",
       "triggerID" : "288e730d645a5e58e50a87ef5f44ba11a8fb0533",
       "triggerType" : "PUSH"
     }, {
       "hash" : "70a5c21321a0da3a0be8d85e206c24befa7b89f1",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1095",
       "triggerID" : "70a5c21321a0da3a0be8d85e206c24befa7b89f1",
       "triggerType" : "PUSH"
     }, {
       "hash" : "f072a986114949c5e426ce64a5f5568fc718e25d",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "f072a986114949c5e426ce64a5f5568fc718e25d",
       "triggerType" : "PUSH"
     }, {
       "hash" : "45b27475b1e8806ec8c566d6f151f4bc79c8549d",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1096",
       "triggerID" : "45b27475b1e8806ec8c566d6f151f4bc79c8549d",
       "triggerType" : "PUSH"
     }, {
       "hash" : "3c62ef22f6cde28110e5eb2fca8760a008d77723",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "3c62ef22f6cde28110e5eb2fca8760a008d77723",
       "triggerType" : "PUSH"
     }, {
       "hash" : "dc785fa04e542b2ccc45494e0292f5153c7ecea8",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1097",
       "triggerID" : "dc785fa04e542b2ccc45494e0292f5153c7ecea8",
       "triggerType" : "PUSH"
     }, {
       "hash" : "585ddb8049b9b9ccb057027e265b08a1b586df8c",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "585ddb8049b9b9ccb057027e265b08a1b586df8c",
       "triggerType" : "PUSH"
     }, {
       "hash" : "b62ff8675beed2c4487c788b9ca1a0ad4271b913",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "b62ff8675beed2c4487c788b9ca1a0ad4271b913",
       "triggerType" : "PUSH"
     }, {
       "hash" : "2d46048a7402907f348cc96da2839ba1b3e4932a",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1099",
       "triggerID" : "2d46048a7402907f348cc96da2839ba1b3e4932a",
       "triggerType" : "PUSH"
     }, {
       "hash" : "004fb6ebf23f796a4b77650e0530f0f1145e8b91",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1101",
       "triggerID" : "004fb6ebf23f796a4b77650e0530f0f1145e8b91",
       "triggerType" : "PUSH"
     }, {
       "hash" : "af0c33333b5e9ae6d59f20a6db1d7c016ea015d3",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1165",
       "triggerID" : "af0c33333b5e9ae6d59f20a6db1d7c016ea015d3",
       "triggerType" : "PUSH"
     }, {
       "hash" : "da8111118d71306500229619d3b7996fcd2526e8",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "da8111118d71306500229619d3b7996fcd2526e8",
       "triggerType" : "PUSH"
     }, {
       "hash" : "83257e2b9bd3c426cf90e11003a5f244a750d7fa",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1167",
       "triggerID" : "83257e2b9bd3c426cf90e11003a5f244a750d7fa",
       "triggerType" : "PUSH"
     }, {
       "hash" : "e0de408f1b8153fdaaea4c87b27a7b69213b75b5",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1177",
       "triggerID" : "e0de408f1b8153fdaaea4c87b27a7b69213b75b5",
       "triggerType" : "PUSH"
     }, {
       "hash" : "3f7951f7ad7931f5dc0b977d7c33bc5429623535",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1180",
       "triggerID" : "3f7951f7ad7931f5dc0b977d7c33bc5429623535",
       "triggerType" : "PUSH"
     }, {
       "hash" : "9cd6f06a753b63630e8245d51338d2bf9ab86b84",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "9cd6f06a753b63630e8245d51338d2bf9ab86b84",
       "triggerType" : "PUSH"
     }, {
       "hash" : "7380b902731a92e0645c02397e14591a05798620",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "7380b902731a92e0645c02397e14591a05798620",
       "triggerType" : "PUSH"
     }, {
       "hash" : "0bd5cca01a541845081d5430d48b021d4bdf0969",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "0bd5cca01a541845081d5430d48b021d4bdf0969",
       "triggerType" : "PUSH"
     }, {
       "hash" : "8f4e175ef554f015e34e0968b656b5de714ab430",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1182",
       "triggerID" : "8f4e175ef554f015e34e0968b656b5de714ab430",
       "triggerType" : "PUSH"
     }, {
       "hash" : "2023f3c882d0c2f5b96e28d2f21812dc0c64aaee",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "2023f3c882d0c2f5b96e28d2f21812dc0c64aaee",
       "triggerType" : "PUSH"
     }, {
       "hash" : "8394acf060d5c3953dbd67cf930b9b1d3b92d8b5",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1198",
       "triggerID" : "8394acf060d5c3953dbd67cf930b9b1d3b92d8b5",
       "triggerType" : "PUSH"
     }, {
       "hash" : "7ec42c105971bff463e359019405a7f4e7947d86",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "7ec42c105971bff463e359019405a7f4e7947d86",
       "triggerType" : "PUSH"
     }, {
       "hash" : "4b930786a7b2de09ac1653cd884ee76a7d1b2ca5",
       "status" : "SUCCESS",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1202",
       "triggerID" : "4b930786a7b2de09ac1653cd884ee76a7d1b2ca5",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 9e668f65c6d4f73c76573236f8fa507dfb80edac UNKNOWN
   * 533f00e5546da0026bbe51e1c030ddd1c19fdd08 UNKNOWN
   * a2ee90c549332a7f17a7d9b2799849b77c08a8e6 UNKNOWN
   * fc2c138901e82ed5f3fbf363c2a0ce8140ade050 UNKNOWN
   * a510a7de1458bb996df5bbb341e8d010e4c91381 UNKNOWN
   * f072a986114949c5e426ce64a5f5568fc718e25d UNKNOWN
   * 3c62ef22f6cde28110e5eb2fca8760a008d77723 UNKNOWN
   * 585ddb8049b9b9ccb057027e265b08a1b586df8c UNKNOWN
   * b62ff8675beed2c4487c788b9ca1a0ad4271b913 UNKNOWN
   * da8111118d71306500229619d3b7996fcd2526e8 UNKNOWN
   * 9cd6f06a753b63630e8245d51338d2bf9ab86b84 UNKNOWN
   * 7380b902731a92e0645c02397e14591a05798620 UNKNOWN
   * 0bd5cca01a541845081d5430d48b021d4bdf0969 UNKNOWN
   * 2023f3c882d0c2f5b96e28d2f21812dc0c64aaee UNKNOWN
   * 7ec42c105971bff463e359019405a7f4e7947d86 UNKNOWN
   * 4b930786a7b2de09ac1653cd884ee76a7d1b2ca5 Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1202) 
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] [HUDI-8141] Incremental Query with Completion Time [hudi]

Posted by "yihua (via GitHub)" <gi...@apache.org>.

yihua commented on code in PR #11947:
URL: https://github.com/apache/hudi/pull/11947#discussion_r1803999310


##########
hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/hudi/IncrementalRelation.scala:
##########
@@ -90,24 +78,24 @@ class IncrementalRelation(val sqlContext: SQLContext,
     throw new HoodieException("Incremental queries are not supported when meta fields are disabled")
   }
 
+  private val queryContext: IncrementalQueryAnalyzer.QueryContext =
+    IncrementalQueryAnalyzer.builder()
+      .metaClient(metaClient)
+      .startTime(optParams(DataSourceReadOptions.BEGIN_INSTANTTIME.key))
+      .endTime(optParams.getOrElse(DataSourceReadOptions.END_INSTANTTIME.key, null))
+      .rangeType(InstantRange.RangeType.OPEN_CLOSED)
+      .build()
+      .analyze()
+
+  private val commitsToReturn = List.concat(
+    queryContext.getArchivedInstants.asScala,
+    queryContext.getActiveInstants.asScala)
+  // TODO: investigate failure: TestHoodieIncrSource#testHoodieIncrSourceInflightCommitBeforeCompletedCommit
+  //    .sortBy(_.getCompletionTime)  // why would sorting by completion time fail this test?

Review Comment:
   There is no issue in production code.  The test logic is not realistic causing the test to fail.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] [HUDI-8141] Incremental Query with Completion Time [hudi]

Posted by "danny0405 (via GitHub)" <gi...@apache.org>.

danny0405 commented on code in PR #11947:
URL: https://github.com/apache/hudi/pull/11947#discussion_r1806007346


##########
hudi-utilities/src/main/java/org/apache/hudi/utilities/sources/HoodieIncrSource.java:
##########
@@ -175,33 +177,27 @@ public Pair<Option<Dataset<Row>>, String> fetchNextBatch(Option<String> lastCkpt
     IncrSourceHelper.MissingCheckpointStrategy missingCheckpointStrategy =
         (containsConfigProperty(props, HoodieIncrSourceConfig.MISSING_CHECKPOINT_STRATEGY))
             ? IncrSourceHelper.MissingCheckpointStrategy.valueOf(
-            getStringWithAltKeys(props, HoodieIncrSourceConfig.MISSING_CHECKPOINT_STRATEGY))
+                getStringWithAltKeys(props, HoodieIncrSourceConfig.MISSING_CHECKPOINT_STRATEGY))
             : null;
     if (readLatestOnMissingCkpt) {
       missingCheckpointStrategy = IncrSourceHelper.MissingCheckpointStrategy.READ_LATEST;
     }
 
-    // Use begin Instant if set and non-empty
-    Option<String> beginInstant =
-        lastCkptStr.isPresent() ? lastCkptStr.get().isEmpty() ? Option.empty() : lastCkptStr : Option.empty();
+    IncrementalQueryAnalyzer analyzer = IncrSourceHelper.getIncrementalQueryAnalyzer(
+        sparkContext, srcPath, lastCkptStr, missingCheckpointStrategy,
+        getIntWithAltKeys(props, HoodieIncrSourceConfig.NUM_INSTANTS_PER_FETCH),
+        getLatestSourceProfile());
+    QueryContext queryContext = analyzer.analyze();
+    Option<InstantRange> instantRange = queryContext.getInstantRange();
 
-    // If source profile exists, use the numInstants from source profile.
-    final int numInstantsFromConfig = getIntWithAltKeys(props, HoodieIncrSourceConfig.NUM_INSTANTS_PER_FETCH);
-    int numInstantsPerFetch = getLatestSourceProfile().map(sourceProfile -> {
-      int numInstantsFromSourceProfile = sourceProfile.getSourceSpecificContext();
-      LOG.info("Overriding numInstantsPerFetch from source profile numInstantsFromSourceProfile {} , numInstantsFromConfig {}", numInstantsFromSourceProfile, numInstantsFromConfig);
-      return numInstantsFromSourceProfile;
-    }).orElse(numInstantsFromConfig);
-
-    HollowCommitHandling handlingMode = getHollowCommitHandleMode(props);
-    QueryInfo queryInfo = generateQueryInfo(sparkContext, srcPath,
-        numInstantsPerFetch, beginInstant, missingCheckpointStrategy, handlingMode,
-        HoodieRecord.COMMIT_TIME_METADATA_FIELD, HoodieRecord.RECORD_KEY_METADATA_FIELD,
-        null, false, Option.empty());
-
-    if (queryInfo.areStartAndEndInstantsEqual()) {
+    String endCompletionTime;
+    // analyzer.getBeginCompletionTime() is empty only when reading the latest instant
+    // in the first batch
+    if (queryContext.isEmpty()
+        || (endCompletionTime = queryContext.getMaxCompletionTime())
+        .equals(analyzer.getStartCompletionTime().orElseGet(() -> null))) {
       LOG.info("Already caught up. No new data to process");
-      return Pair.of(Option.empty(), queryInfo.getEndInstant());
+      return Pair.of(Option.empty(), analyzer.getStartCompletionTime().get());

Review Comment:
   start or end?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] [HUDI-8141] Support incremental query with completion time [hudi]

Posted by "yihua (via GitHub)" <gi...@apache.org>.

yihua merged PR #11947:
URL: https://github.com/apache/hudi/pull/11947


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] [HUDI-8141] Incremental Query with Completion Time [hudi]

Posted by "danny0405 (via GitHub)" <gi...@apache.org>.

danny0405 commented on code in PR #11947:
URL: https://github.com/apache/hudi/pull/11947#discussion_r1805711690


##########
hudi-utilities/src/main/java/org/apache/hudi/utilities/sources/HoodieIncrSource.java:
##########
@@ -212,35 +209,58 @@ public Pair<Option<Dataset<Row>>, String> fetchNextBatch(Option<String> lastCkpt
           .collect(Collectors.toMap(Pair::getLeft, Pair::getRight));
       reader = reader.options(optionsMap);
     }
+
+    boolean shouldFullScan =
+        missingCheckpointStrategy == MissingCheckpointStrategy.READ_UPTO_LATEST_COMMIT
+            && queryContext.getActiveTimeline()
+            .isBeforeTimelineStartsByCompletionTime(analyzer.getBeginCompletionTime().get());
     Dataset<Row> source;
-    // Do Incr pull. Set end instant if available
-    if (queryInfo.isIncremental()) {
-      source = reader
-          .options(readOpts)
-          .option(QUERY_TYPE().key(), QUERY_TYPE_INCREMENTAL_OPT_VAL())
-          .option(BEGIN_INSTANTTIME().key(), queryInfo.getStartInstant())
-          .option(END_INSTANTTIME().key(), queryInfo.getEndInstant())
-          .option(INCREMENTAL_FALLBACK_TO_FULL_TABLE_SCAN().key(),
-              props.getString(INCREMENTAL_FALLBACK_TO_FULL_TABLE_SCAN().key(),
-                  INCREMENTAL_FALLBACK_TO_FULL_TABLE_SCAN().defaultValue()))
-          .option(INCREMENTAL_READ_HANDLE_HOLLOW_COMMIT().key(), handlingMode.name())
-          .load(srcPath);
-    } else {
-      // if checkpoint is missing from source table, and if strategy is set to READ_UPTO_LATEST_COMMIT, we have to issue snapshot query
+    if (instantRange.isEmpty() || shouldFullScan) {
+      // snapshot query
       Dataset<Row> snapshot = reader
           .options(readOpts)
           .option(DataSourceReadOptions.QUERY_TYPE().key(), DataSourceReadOptions.QUERY_TYPE_SNAPSHOT_OPT_VAL())
           .load(srcPath);
+
+      Option<String> predicate = Option.empty();
+      List<String> instantTimeList = queryContext.getInstantTimeList();
       if (snapshotLoadQuerySplitter.isPresent()) {
-        queryInfo = snapshotLoadQuerySplitter.get().getNextCheckpoint(snapshot, queryInfo, sourceProfileSupplier);
+        Option<SnapshotLoadQuerySplitter.CheckpointWithPredicates> newCheckpointAndPredicate =
+            snapshotLoadQuerySplitter.get().getNextCheckpoint(snapshot, queryContext, sourceProfileSupplier);
+        if (newCheckpointAndPredicate.isPresent()) {
+          endCompletionTime = newCheckpointAndPredicate.get().endInstant;
+          predicate = Option.of(newCheckpointAndPredicate.get().predicateFilter);
+          instantTimeList = queryContext.getInstants().stream()
+              .filter(instant -> HoodieTimeline.compareTimestamps(
+                  instant.getCompletionTime(), HoodieTimeline.LESSER_THAN_OR_EQUALS, newCheckpointAndPredicate.get().endInstant))
+              .map(HoodieInstant::getTimestamp)
+              .collect(Collectors.toList());
+        } else {
+          endCompletionTime = queryContext.getEndInstant().orElse(queryContext.getLastInstant());

Review Comment:
   Seems incorrect to use the instant time here.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] [HUDI-8141] Incremental Query with Completion Time [hudi]

Posted by "danny0405 (via GitHub)" <gi...@apache.org>.

danny0405 commented on code in PR #11947:
URL: https://github.com/apache/hudi/pull/11947#discussion_r1805712611


##########
hudi-utilities/src/main/java/org/apache/hudi/utilities/sources/SnapshotLoadQuerySplitter.java:
##########
@@ -118,6 +124,32 @@ public QueryInfo getNextCheckpoint(Dataset<Row> df, QueryInfo queryInfo, Option<
         .orElse(queryInfo);
   }
 
+  public Option<CheckpointWithPredicates> getNextCheckpoint(Dataset<Row> df, QueryContext queryContext,
+                                                            Option<SourceProfileSupplier> sourceProfileSupplier) {
+    // the start instant would be included into the final query result. So we need to get
+    // a strictly lower timestamp to have query splitter include the start instant
+    Option<CheckpointWithPredicates> nextCheckpointWithPredicates =
+        getNextCheckpointWithPredicates(df, instantTimeMinusMillis(queryContext.getBeginInstant().get(), 1));
+    if (nextCheckpointWithPredicates.isPresent()) {
+      // getNextCheckpointWithPredicates is based on instant times,
+      // so we need to translate the instant time to the completion time
+      String endInstantTime = nextCheckpointWithPredicates.get().getEndInstant();
+      Option<String> endCompletionTime = Option.fromJavaOptional(queryContext.getInstants().stream()
+          .filter(instant -> endInstantTime.equals(instant.getTimestamp()))

Review Comment:
   Isn't is just the max completion time from query context?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] [HUDI-8141] Incremental Query with Completion Time [hudi]

Posted by "danny0405 (via GitHub)" <gi...@apache.org>.

danny0405 commented on code in PR #11947:
URL: https://github.com/apache/hudi/pull/11947#discussion_r1806001721


##########
hudi-utilities/src/main/java/org/apache/hudi/utilities/sources/HoodieIncrSource.java:
##########
@@ -212,35 +208,58 @@ public Pair<Option<Dataset<Row>>, String> fetchNextBatch(Option<String> lastCkpt
           .collect(Collectors.toMap(Pair::getLeft, Pair::getRight));
       reader = reader.options(optionsMap);
     }
+
+    boolean shouldFullScan =
+        missingCheckpointStrategy == MissingCheckpointStrategy.READ_UPTO_LATEST_COMMIT
+            && queryContext.getActiveTimeline()
+            .isBeforeTimelineStartsByCompletionTime(analyzer.getStartCompletionTime().get());
     Dataset<Row> source;
-    // Do Incr pull. Set end instant if available
-    if (queryInfo.isIncremental()) {
-      source = reader
-          .options(readOpts)
-          .option(QUERY_TYPE().key(), QUERY_TYPE_INCREMENTAL_OPT_VAL())
-          .option(BEGIN_INSTANTTIME().key(), queryInfo.getStartInstant())
-          .option(END_INSTANTTIME().key(), queryInfo.getEndInstant())
-          .option(INCREMENTAL_FALLBACK_TO_FULL_TABLE_SCAN().key(),
-              props.getString(INCREMENTAL_FALLBACK_TO_FULL_TABLE_SCAN().key(),
-                  INCREMENTAL_FALLBACK_TO_FULL_TABLE_SCAN().defaultValue()))
-          .option(INCREMENTAL_READ_HANDLE_HOLLOW_COMMIT().key(), handlingMode.name())
-          .load(srcPath);
-    } else {
-      // if checkpoint is missing from source table, and if strategy is set to READ_UPTO_LATEST_COMMIT, we have to issue snapshot query
+    if (instantRange.isEmpty() || shouldFullScan) {
+      // snapshot query
       Dataset<Row> snapshot = reader
           .options(readOpts)
           .option(DataSourceReadOptions.QUERY_TYPE().key(), DataSourceReadOptions.QUERY_TYPE_SNAPSHOT_OPT_VAL())
           .load(srcPath);
+
+      Option<String> predicate = Option.empty();
+      List<String> instantTimeList = queryContext.getInstantTimeList();
       if (snapshotLoadQuerySplitter.isPresent()) {
-        queryInfo = snapshotLoadQuerySplitter.get().getNextCheckpoint(snapshot, queryInfo, sourceProfileSupplier);
+        Option<SnapshotLoadQuerySplitter.CheckpointWithPredicates> newCheckpointAndPredicate =
+            snapshotLoadQuerySplitter.get().getNextCheckpointWithPredicates(snapshot, queryContext);
+        if (newCheckpointAndPredicate.isPresent()) {
+          endCompletionTime = newCheckpointAndPredicate.get().endInstant;
+          predicate = Option.of(newCheckpointAndPredicate.get().predicateFilter);
+          instantTimeList = queryContext.getInstants().stream()
+              .filter(instant -> HoodieTimeline.compareTimestamps(
+                  instant.getCompletionTime(), HoodieTimeline.LESSER_THAN_OR_EQUALS, newCheckpointAndPredicate.get().endInstant))

Review Comment:
   The analyzer already ensure all the instants returned are included in the completion time range, so is this filtering redundant?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] [HUDI-8141] Incremental Query with Completion Time [hudi]

Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.

hudi-bot commented on PR #11947:
URL: https://github.com/apache/hudi/pull/11947#issuecomment-2408656619

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "090f256a315cbbf0c6a54d93cc2739dd693360e6",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=484",
       "triggerID" : "090f256a315cbbf0c6a54d93cc2739dd693360e6",
       "triggerType" : "PUSH"
     }, {
       "hash" : "0df5649013c1cf9e782fcbb67b8a2e7174f94512",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=985",
       "triggerID" : "0df5649013c1cf9e782fcbb67b8a2e7174f94512",
       "triggerType" : "PUSH"
     }, {
       "hash" : "d52e0128f409918644255abbe4376dc4a780bfc8",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=986",
       "triggerID" : "d52e0128f409918644255abbe4376dc4a780bfc8",
       "triggerType" : "PUSH"
     }, {
       "hash" : "9e668f65c6d4f73c76573236f8fa507dfb80edac",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "9e668f65c6d4f73c76573236f8fa507dfb80edac",
       "triggerType" : "PUSH"
     }, {
       "hash" : "f145131d524bf190986e5de600a326e32a27beca",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=987",
       "triggerID" : "f145131d524bf190986e5de600a326e32a27beca",
       "triggerType" : "PUSH"
     }, {
       "hash" : "0b008f807b42aa6fb2686922bb88d63e84a1f299",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=993",
       "triggerID" : "0b008f807b42aa6fb2686922bb88d63e84a1f299",
       "triggerType" : "PUSH"
     }, {
       "hash" : "65c01ad9f2b33d36d4890355feef91844438e732",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=995",
       "triggerID" : "65c01ad9f2b33d36d4890355feef91844438e732",
       "triggerType" : "PUSH"
     }, {
       "hash" : "955493fdc93842e8c99529cec35e39cecdbe9638",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=996",
       "triggerID" : "955493fdc93842e8c99529cec35e39cecdbe9638",
       "triggerType" : "PUSH"
     }, {
       "hash" : "533f00e5546da0026bbe51e1c030ddd1c19fdd08",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "533f00e5546da0026bbe51e1c030ddd1c19fdd08",
       "triggerType" : "PUSH"
     }, {
       "hash" : "05777be67e37f3a6a8eaa59e54d9e17e5fc6ba9b",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1016",
       "triggerID" : "05777be67e37f3a6a8eaa59e54d9e17e5fc6ba9b",
       "triggerType" : "PUSH"
     }, {
       "hash" : "a2ee90c549332a7f17a7d9b2799849b77c08a8e6",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "a2ee90c549332a7f17a7d9b2799849b77c08a8e6",
       "triggerType" : "PUSH"
     }, {
       "hash" : "05ce568b0b6ba14da9ca40d26b39ba478837d747",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1017",
       "triggerID" : "05ce568b0b6ba14da9ca40d26b39ba478837d747",
       "triggerType" : "PUSH"
     }, {
       "hash" : "840e3b33d39b1f86367bb6b0e65eadeb33de7765",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1025",
       "triggerID" : "840e3b33d39b1f86367bb6b0e65eadeb33de7765",
       "triggerType" : "PUSH"
     }, {
       "hash" : "8d2e2dd83f51d03e890d424a51d3e0e84e904a43",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1031",
       "triggerID" : "8d2e2dd83f51d03e890d424a51d3e0e84e904a43",
       "triggerType" : "PUSH"
     }, {
       "hash" : "688381914087ff11acf2f8c9372dbf39afa0bd40",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1035",
       "triggerID" : "688381914087ff11acf2f8c9372dbf39afa0bd40",
       "triggerType" : "PUSH"
     }, {
       "hash" : "fc2c138901e82ed5f3fbf363c2a0ce8140ade050",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "fc2c138901e82ed5f3fbf363c2a0ce8140ade050",
       "triggerType" : "PUSH"
     }, {
       "hash" : "6ed9e1cb1e5a191752d6d39c7a9d5d41840f0cdf",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1039",
       "triggerID" : "6ed9e1cb1e5a191752d6d39c7a9d5d41840f0cdf",
       "triggerType" : "PUSH"
     }, {
       "hash" : "a510a7de1458bb996df5bbb341e8d010e4c91381",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "a510a7de1458bb996df5bbb341e8d010e4c91381",
       "triggerType" : "PUSH"
     }, {
       "hash" : "49677397cd117228742205944606f7a789090376",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1042",
       "triggerID" : "49677397cd117228742205944606f7a789090376",
       "triggerType" : "PUSH"
     }, {
       "hash" : "1630db0a37422aed942ff280560a9d6329df5015",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1044",
       "triggerID" : "1630db0a37422aed942ff280560a9d6329df5015",
       "triggerType" : "PUSH"
     }, {
       "hash" : "f11c76301d4ab524db0af57aa831be3a898ea0fb",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1048",
       "triggerID" : "f11c76301d4ab524db0af57aa831be3a898ea0fb",
       "triggerType" : "PUSH"
     }, {
       "hash" : "d1a0dc4c2017853a7ba0c3302f886043a63f29da",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1050",
       "triggerID" : "d1a0dc4c2017853a7ba0c3302f886043a63f29da",
       "triggerType" : "PUSH"
     }, {
       "hash" : "6a4c15ec39ded1f90b113b272554217698a98e1c",
       "status" : "SUCCESS",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1087",
       "triggerID" : "6a4c15ec39ded1f90b113b272554217698a98e1c",
       "triggerType" : "PUSH"
     }, {
       "hash" : "288e730d645a5e58e50a87ef5f44ba11a8fb0533",
       "status" : "PENDING",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1092",
       "triggerID" : "288e730d645a5e58e50a87ef5f44ba11a8fb0533",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 9e668f65c6d4f73c76573236f8fa507dfb80edac UNKNOWN
   * 533f00e5546da0026bbe51e1c030ddd1c19fdd08 UNKNOWN
   * a2ee90c549332a7f17a7d9b2799849b77c08a8e6 UNKNOWN
   * fc2c138901e82ed5f3fbf363c2a0ce8140ade050 UNKNOWN
   * a510a7de1458bb996df5bbb341e8d010e4c91381 UNKNOWN
   * 6a4c15ec39ded1f90b113b272554217698a98e1c Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1087) 
   * 288e730d645a5e58e50a87ef5f44ba11a8fb0533 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1092) 
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] [HUDI-8141] Incremental Query with Completion Time [hudi]

Posted by "CTTY (via GitHub)" <gi...@apache.org>.

CTTY commented on code in PR #11947:
URL: https://github.com/apache/hudi/pull/11947#discussion_r1797748764


##########
hudi-utilities/src/main/java/org/apache/hudi/utilities/sources/HoodieIncrSource.java:
##########
@@ -212,35 +257,43 @@ public Pair<Option<Dataset<Row>>, String> fetchNextBatch(Option<String> lastCkpt
           .collect(Collectors.toMap(Pair::getLeft, Pair::getRight));
       reader = reader.options(optionsMap);
     }
+    boolean shouldFullScan =
+        queryContext.getActiveTimeline().isBeforeTimelineStartsByCompletionTime(startTime)
+        && missingCheckpointStrategy == MissingCheckpointStrategy.READ_UPTO_LATEST_COMMIT;
     Dataset<Row> source;
-    // Do Incr pull. Set end instant if available
-    if (queryInfo.isIncremental()) {
-      source = reader
-          .options(readOpts)
-          .option(QUERY_TYPE().key(), QUERY_TYPE_INCREMENTAL_OPT_VAL())
-          .option(BEGIN_INSTANTTIME().key(), queryInfo.getStartInstant())
-          .option(END_INSTANTTIME().key(), queryInfo.getEndInstant())
-          .option(INCREMENTAL_FALLBACK_TO_FULL_TABLE_SCAN().key(),
-              props.getString(INCREMENTAL_FALLBACK_TO_FULL_TABLE_SCAN().key(),
-                  INCREMENTAL_FALLBACK_TO_FULL_TABLE_SCAN().defaultValue()))
-          .option(INCREMENTAL_READ_HANDLE_HOLLOW_COMMIT().key(), handlingMode.name())
-          .load(srcPath);
-    } else {
-      // if checkpoint is missing from source table, and if strategy is set to READ_UPTO_LATEST_COMMIT, we have to issue snapshot query
+    if (instantRange.isEmpty() || shouldFullScan) {
+      // snapshot query
       Dataset<Row> snapshot = reader
           .options(readOpts)
           .option(DataSourceReadOptions.QUERY_TYPE().key(), DataSourceReadOptions.QUERY_TYPE_SNAPSHOT_OPT_VAL())
           .load(srcPath);
       if (snapshotLoadQuerySplitter.isPresent()) {
-        queryInfo = snapshotLoadQuerySplitter.get().getNextCheckpoint(snapshot, queryInfo, sourceProfileSupplier);
+        queryContext = snapshotLoadQuerySplitter.get().getNextCheckpoint(snapshot, queryContext, sourceProfileSupplier);
+        // update endTime/next checkpoint
+        endTime = queryContext.getEndInstant().orElse(queryContext.getLastInstant());
       }
+      Set<String> validInstants = new HashSet<>(queryContext.getInstants());
+      snapshot = queryContext.getPredicateFilter().map(snapshot::filter).orElse(snapshot);
       source = snapshot
           // add filtering so that only interested records are returned.
-          .filter(String.format("%s > '%s'", HoodieRecord.COMMIT_TIME_METADATA_FIELD,
-              queryInfo.getStartInstant()))
+          // completion time comparison uses ( , ], but when comparing start time we need to use [, ]
+          .filter(String.format("%s >= '%s'", HoodieRecord.COMMIT_TIME_METADATA_FIELD,
+              queryContext.getStartInstant().get()))
           .filter(String.format("%s <= '%s'", HoodieRecord.COMMIT_TIME_METADATA_FIELD,
-              queryInfo.getEndInstant()));
-      source = queryInfo.getPredicateFilter().map(source::filter).orElse(source);
+              queryContext.getEndInstant().orElse(queryContext.getLastInstant())))

Review Comment:
   There are only two cases for `queryContext.getStartInstant()` and `queryContext.getEndInstant()`:
   - Both of them are null
   - Both of them have values
   
   If both of them are null, then we would return early when checking `queryContext.isEmpty()` above



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] [HUDI-8141] Incremental Query with Completion Time [hudi]

Posted by "danny0405 (via GitHub)" <gi...@apache.org>.

danny0405 commented on code in PR #11947:
URL: https://github.com/apache/hudi/pull/11947#discussion_r1798652506


##########
hudi-common/src/main/java/org/apache/hudi/common/table/read/IncrementalQueryAnalyzer.java:
##########
@@ -376,14 +382,16 @@ private QueryContext(
         List<HoodieInstant> archivedInstants,
         List<HoodieInstant> activeInstants,
         HoodieTimeline activeTimeline,
-        @Nullable HoodieTimeline archivedTimeline) {
+        @Nullable HoodieTimeline archivedTimeline,
+        @Nullable String predicateFilter) {
       this.startInstant = Option.ofNullable(startInstant);
       this.endInstant = Option.ofNullable(endInstant);
       this.archivedInstants = archivedInstants;
       this.activeInstants = activeInstants;
       this.activeTimeline = activeTimeline;
       this.archivedTimeline = archivedTimeline;
       this.instants = instants;
+      this.predicateFilter = Option.ofNullable(predicateFilter);

Review Comment:
   I didn't see any code in `IncrementalQueryAnalyzer` to utilitize the `predicateFilter`, so why to hold it here?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] [HUDI-8141] Incremental Query with Completion Time [hudi]

Posted by "CTTY (via GitHub)" <gi...@apache.org>.

CTTY commented on code in PR #11947:
URL: https://github.com/apache/hudi/pull/11947#discussion_r1797905527


##########
hudi-spark-datasource/hudi-spark/src/test/scala/org/apache/hudi/functional/TestCOWDataSourceStorage.scala:
##########
@@ -185,7 +188,7 @@ class TestCOWDataSourceStorage extends SparkClientFunctionalTestHarness {
     val hoodieIncViewDF1 = spark.read.format("org.apache.hudi")
       .option(DataSourceReadOptions.QUERY_TYPE.key, DataSourceReadOptions.QUERY_TYPE_INCREMENTAL_OPT_VAL)
       .option(DataSourceReadOptions.BEGIN_INSTANTTIME.key, "000")
-      .option(DataSourceReadOptions.END_INSTANTTIME.key, firstCommit)
+      .option(DataSourceReadOptions.END_INSTANTTIME.key, completionTime1)

Review Comment:
   this query is to pull the first commit. so the start time should be a completion time that's before the 1st commit's completion time. And the end time should be completionTime1. 
   
   Basically to fetch commit in this range: `(completionTime1, completionTime1]`



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] [HUDI-8141] Incremental Query with Completion Time [hudi]

Posted by "CTTY (via GitHub)" <gi...@apache.org>.

CTTY commented on code in PR #11947:
URL: https://github.com/apache/hudi/pull/11947#discussion_r1798105220


##########
hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/hudi/IncrementalRelation.scala:
##########
@@ -90,24 +78,25 @@ class IncrementalRelation(val sqlContext: SQLContext,
     throw new HoodieException("Incremental queries are not supported when meta fields are disabled")
   }
 
+  private val queryContext: IncrementalQueryAnalyzer.QueryContext =
+    IncrementalQueryAnalyzer.builder()
+      .metaClient(metaClient)
+      .startTime(optParams(DataSourceReadOptions.BEGIN_INSTANTTIME.key))
+      .endTime(optParams.getOrElse(DataSourceReadOptions.END_INSTANTTIME.key, null))
+      .rangeType(InstantRange.RangeType.OPEN_CLOSED)
+      .limit(optParams.getOrElse(
+        DataSourceReadOptions.INCREMENTAL_LIMIT.key,
+        DataSourceReadOptions.INCREMENTAL_LIMIT.defaultValue).toInt)
+      .build()
+      .analyze()
+
+  private val commitsToReturn = List.concat(
+    queryContext.getArchivedInstants.asScala,
+    queryContext.getActiveInstants.asScala)
+
   private val useEndInstantSchema = optParams.getOrElse(INCREMENTAL_READ_SCHEMA_USE_END_INSTANTTIME.key,
     INCREMENTAL_READ_SCHEMA_USE_END_INSTANTTIME.defaultValue).toBoolean
 
-  private val lastInstant = commitTimeline.lastInstant().get()
-
-  private val commitsTimelineToReturn = {
-    if (hollowCommitHandling == USE_TRANSITION_TIME) {
-      commitTimeline.findInstantsInRangeByCompletionTime(
-        optParams(DataSourceReadOptions.BEGIN_INSTANTTIME.key),
-        optParams.getOrElse(DataSourceReadOptions.END_INSTANTTIME.key(), lastInstant.getCompletionTime))
-    } else {
-      commitTimeline.findInstantsInRange(
-        optParams(DataSourceReadOptions.BEGIN_INSTANTTIME.key),
-        optParams.getOrElse(DataSourceReadOptions.END_INSTANTTIME.key(), lastInstant.getTimestamp))
-    }
-  }
-  private val commitsToReturn = commitsTimelineToReturn.getInstantsAsStream.iterator().asScala.toList

Review Comment:
   JIRA: https://issues.apache.org/jira/browse/HUDI-8359



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] [HUDI-8141] Incremental Query with Completion Time [hudi]

Posted by "danny0405 (via GitHub)" <gi...@apache.org>.

danny0405 commented on code in PR #11947:
URL: https://github.com/apache/hudi/pull/11947#discussion_r1798653149


##########
hudi-common/src/main/java/org/apache/hudi/common/table/timeline/CompletionTimeQueryView.java:
##########
@@ -189,46 +193,78 @@ public Option<String> getCompletionTime(String beginTime) {
    * <p>By default, assumes there is at most 1 day time of duration for an instant to accelerate the queries.
    *
    * @param timeline The timeline.
-   * @param rangeStart   The query range start completion time.
-   * @param rangeEnd     The query range end completion time.
+   * @param rangeStartCompletionTime   The query range start completion time.
+   * @param rangeEndCompletionTime     The query range end completion time.
    * @param rangeType    The range type.
    *
    * @return The sorted instant time list.
    */
   public List<String> getStartTimes(
       HoodieTimeline timeline,
-      Option<String> rangeStart,
-      Option<String> rangeEnd,
+      Option<String> rangeStartCompletionTime,

Review Comment:
   `beginCompletionTime` and `endCompletionTime`, can you also fix the doc indentation to be left justifying.



##########
hudi-common/src/main/java/org/apache/hudi/common/table/timeline/CompletionTimeQueryView.java:
##########
@@ -189,46 +193,78 @@ public Option<String> getCompletionTime(String beginTime) {
    * <p>By default, assumes there is at most 1 day time of duration for an instant to accelerate the queries.
    *
    * @param timeline The timeline.
-   * @param rangeStart   The query range start completion time.
-   * @param rangeEnd     The query range end completion time.
+   * @param rangeStartCompletionTime   The query range start completion time.
+   * @param rangeEndCompletionTime     The query range end completion time.
    * @param rangeType    The range type.
    *
    * @return The sorted instant time list.
    */
   public List<String> getStartTimes(
       HoodieTimeline timeline,
-      Option<String> rangeStart,
-      Option<String> rangeEnd,
+      Option<String> rangeStartCompletionTime,
+      Option<String> rangeEndCompletionTime,
       InstantRange.RangeType rangeType) {
     // assumes any instant/transaction lasts at most 1 day to optimize the query efficiency.
-    return getStartTimes(timeline, rangeStart, rangeEnd, rangeType, s -> HoodieInstantTimeGenerator.instantTimeMinusMillis(s, MILLI_SECONDS_IN_ONE_DAY));
+    return getStartTimes(
+        timeline,
+        rangeStartCompletionTime,
+        rangeEndCompletionTime,
+        rangeType,
+        GET_INSTANT_ONE_DAY_BEFORE);
   }
 
   /**
    * Queries the instant start time with given completion time range.
    *
-   * @param rangeStart              The query range start completion time.
-   * @param rangeEnd                The query range end completion time.
+   * @param rangeStartCompletionTime              The query range start completion time.
+   * @param rangeEndCompletionTime                The query range end completion time.

Review Comment:
   ditto.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] [HUDI-8141] Incremental Query with Completion Time [hudi]

Posted by "danny0405 (via GitHub)" <gi...@apache.org>.

danny0405 commented on code in PR #11947:
URL: https://github.com/apache/hudi/pull/11947#discussion_r1798656034


##########
hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/hudi/MergeOnReadIncrementalRelation.scala:
##########
@@ -201,55 +199,42 @@ trait HoodieIncrementalRelationTrait extends HoodieBaseRelation {
     val fallbackToFullTableScan = optParams.getOrElse(DataSourceReadOptions.INCREMENTAL_FALLBACK_TO_FULL_TABLE_SCAN.key,
       DataSourceReadOptions.INCREMENTAL_FALLBACK_TO_FULL_TABLE_SCAN.defaultValue).toBoolean
 
-    fallbackToFullTableScan && (startInstantArchived || endInstantArchived
+    fallbackToFullTableScan && (startInstantArchived
       || affectedFilesInCommits.asScala.exists(fileStatus => !metaClient.getStorage.exists(fileStatus.getPath)))
   }
 
-  protected lazy val includedCommits: immutable.Seq[HoodieInstant] = {
-    if (!startInstantArchived || !endInstantArchived) {
-      // If endTimestamp commit is not archived, will filter instants
-      // before endTimestamp.
-      if (hollowCommitHandling == USE_TRANSITION_TIME) {
-        super.timeline.findInstantsInRangeByCompletionTime(startTimestamp, endTimestamp).getInstants.asScala.toList
-      } else {
-        super.timeline.findInstantsInRange(startTimestamp, endTimestamp).getInstants.asScala.toList
-      }
-    } else {
-      super.timeline.getInstants.asScala.toList
-    }
-  }
+  protected lazy val queryContext: IncrementalQueryAnalyzer.QueryContext =
+    IncrementalQueryAnalyzer.builder()
+      .metaClient(metaClient)
+      .startTime(optParams(DataSourceReadOptions.BEGIN_INSTANTTIME.key))
+      .endTime(optParams.getOrElse(DataSourceReadOptions.END_INSTANTTIME.key, null))
+      .rangeType(InstantRange.RangeType.OPEN_CLOSED)
+      .build()
+      .analyze()
+
+  protected lazy val includedCommits: immutable.Seq[HoodieInstant] = List.concat(
+    queryContext.getArchivedInstants.asScala,

Review Comment:
   use instants in query context instead.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] [HUDI-8141] Incremental Query with Completion Time [hudi]

Posted by "yihua (via GitHub)" <gi...@apache.org>.

yihua commented on code in PR #11947:
URL: https://github.com/apache/hudi/pull/11947#discussion_r1802456538


##########
hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/hudi/IncrementalRelation.scala:
##########
@@ -90,24 +78,24 @@ class IncrementalRelation(val sqlContext: SQLContext,
     throw new HoodieException("Incremental queries are not supported when meta fields are disabled")
   }
 
+  private val queryContext: IncrementalQueryAnalyzer.QueryContext =
+    IncrementalQueryAnalyzer.builder()
+      .metaClient(metaClient)
+      .startTime(optParams(DataSourceReadOptions.BEGIN_INSTANTTIME.key))
+      .endTime(optParams.getOrElse(DataSourceReadOptions.END_INSTANTTIME.key, null))
+      .rangeType(InstantRange.RangeType.OPEN_CLOSED)

Review Comment:
   I've changed `IncrSourceHelper#getIncrementalQueryAnalyzer` to use different `RangeType` based on whether there is last checkpoint or not, so `CLOSED_CLOSED` is used for the first batch, while `OPEN_CLOSED` is used for subsequent batches.  I also got rid of the usage of `instantTimeMinusMillis(hoodieInstant.getCompletionTime(), 1)`.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] [HUDI-8141] WIP:Always use completion time to find begin Instant for incremental source [hudi]

Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.

hudi-bot commented on PR #11947:
URL: https://github.com/apache/hudi/pull/11947#issuecomment-2400504035

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "090f256a315cbbf0c6a54d93cc2739dd693360e6",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=484",
       "triggerID" : "090f256a315cbbf0c6a54d93cc2739dd693360e6",
       "triggerType" : "PUSH"
     }, {
       "hash" : "0df5649013c1cf9e782fcbb67b8a2e7174f94512",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=985",
       "triggerID" : "0df5649013c1cf9e782fcbb67b8a2e7174f94512",
       "triggerType" : "PUSH"
     }, {
       "hash" : "d52e0128f409918644255abbe4376dc4a780bfc8",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=986",
       "triggerID" : "d52e0128f409918644255abbe4376dc4a780bfc8",
       "triggerType" : "PUSH"
     }, {
       "hash" : "9e668f65c6d4f73c76573236f8fa507dfb80edac",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "9e668f65c6d4f73c76573236f8fa507dfb80edac",
       "triggerType" : "PUSH"
     }, {
       "hash" : "f145131d524bf190986e5de600a326e32a27beca",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=987",
       "triggerID" : "f145131d524bf190986e5de600a326e32a27beca",
       "triggerType" : "PUSH"
     }, {
       "hash" : "0b008f807b42aa6fb2686922bb88d63e84a1f299",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=993",
       "triggerID" : "0b008f807b42aa6fb2686922bb88d63e84a1f299",
       "triggerType" : "PUSH"
     }, {
       "hash" : "65c01ad9f2b33d36d4890355feef91844438e732",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=995",
       "triggerID" : "65c01ad9f2b33d36d4890355feef91844438e732",
       "triggerType" : "PUSH"
     }, {
       "hash" : "955493fdc93842e8c99529cec35e39cecdbe9638",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=996",
       "triggerID" : "955493fdc93842e8c99529cec35e39cecdbe9638",
       "triggerType" : "PUSH"
     }, {
       "hash" : "533f00e5546da0026bbe51e1c030ddd1c19fdd08",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "533f00e5546da0026bbe51e1c030ddd1c19fdd08",
       "triggerType" : "PUSH"
     }, {
       "hash" : "05777be67e37f3a6a8eaa59e54d9e17e5fc6ba9b",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1016",
       "triggerID" : "05777be67e37f3a6a8eaa59e54d9e17e5fc6ba9b",
       "triggerType" : "PUSH"
     }, {
       "hash" : "a2ee90c549332a7f17a7d9b2799849b77c08a8e6",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "a2ee90c549332a7f17a7d9b2799849b77c08a8e6",
       "triggerType" : "PUSH"
     }, {
       "hash" : "05ce568b0b6ba14da9ca40d26b39ba478837d747",
       "status" : "FAILURE",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1017",
       "triggerID" : "05ce568b0b6ba14da9ca40d26b39ba478837d747",
       "triggerType" : "PUSH"
     }, {
       "hash" : "840e3b33d39b1f86367bb6b0e65eadeb33de7765",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "840e3b33d39b1f86367bb6b0e65eadeb33de7765",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 9e668f65c6d4f73c76573236f8fa507dfb80edac UNKNOWN
   * 533f00e5546da0026bbe51e1c030ddd1c19fdd08 UNKNOWN
   * a2ee90c549332a7f17a7d9b2799849b77c08a8e6 UNKNOWN
   * 05ce568b0b6ba14da9ca40d26b39ba478837d747 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1017) 
   * 840e3b33d39b1f86367bb6b0e65eadeb33de7765 UNKNOWN
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] [HUDI-8141] WIP:Always use completion time to find begin Instant for incremental source [hudi]

Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.

hudi-bot commented on PR #11947:
URL: https://github.com/apache/hudi/pull/11947#issuecomment-2400733433

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "090f256a315cbbf0c6a54d93cc2739dd693360e6",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=484",
       "triggerID" : "090f256a315cbbf0c6a54d93cc2739dd693360e6",
       "triggerType" : "PUSH"
     }, {
       "hash" : "0df5649013c1cf9e782fcbb67b8a2e7174f94512",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=985",
       "triggerID" : "0df5649013c1cf9e782fcbb67b8a2e7174f94512",
       "triggerType" : "PUSH"
     }, {
       "hash" : "d52e0128f409918644255abbe4376dc4a780bfc8",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=986",
       "triggerID" : "d52e0128f409918644255abbe4376dc4a780bfc8",
       "triggerType" : "PUSH"
     }, {
       "hash" : "9e668f65c6d4f73c76573236f8fa507dfb80edac",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "9e668f65c6d4f73c76573236f8fa507dfb80edac",
       "triggerType" : "PUSH"
     }, {
       "hash" : "f145131d524bf190986e5de600a326e32a27beca",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=987",
       "triggerID" : "f145131d524bf190986e5de600a326e32a27beca",
       "triggerType" : "PUSH"
     }, {
       "hash" : "0b008f807b42aa6fb2686922bb88d63e84a1f299",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=993",
       "triggerID" : "0b008f807b42aa6fb2686922bb88d63e84a1f299",
       "triggerType" : "PUSH"
     }, {
       "hash" : "65c01ad9f2b33d36d4890355feef91844438e732",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=995",
       "triggerID" : "65c01ad9f2b33d36d4890355feef91844438e732",
       "triggerType" : "PUSH"
     }, {
       "hash" : "955493fdc93842e8c99529cec35e39cecdbe9638",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=996",
       "triggerID" : "955493fdc93842e8c99529cec35e39cecdbe9638",
       "triggerType" : "PUSH"
     }, {
       "hash" : "533f00e5546da0026bbe51e1c030ddd1c19fdd08",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "533f00e5546da0026bbe51e1c030ddd1c19fdd08",
       "triggerType" : "PUSH"
     }, {
       "hash" : "05777be67e37f3a6a8eaa59e54d9e17e5fc6ba9b",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1016",
       "triggerID" : "05777be67e37f3a6a8eaa59e54d9e17e5fc6ba9b",
       "triggerType" : "PUSH"
     }, {
       "hash" : "a2ee90c549332a7f17a7d9b2799849b77c08a8e6",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "a2ee90c549332a7f17a7d9b2799849b77c08a8e6",
       "triggerType" : "PUSH"
     }, {
       "hash" : "05ce568b0b6ba14da9ca40d26b39ba478837d747",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1017",
       "triggerID" : "05ce568b0b6ba14da9ca40d26b39ba478837d747",
       "triggerType" : "PUSH"
     }, {
       "hash" : "840e3b33d39b1f86367bb6b0e65eadeb33de7765",
       "status" : "FAILURE",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1025",
       "triggerID" : "840e3b33d39b1f86367bb6b0e65eadeb33de7765",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 9e668f65c6d4f73c76573236f8fa507dfb80edac UNKNOWN
   * 533f00e5546da0026bbe51e1c030ddd1c19fdd08 UNKNOWN
   * a2ee90c549332a7f17a7d9b2799849b77c08a8e6 UNKNOWN
   * 840e3b33d39b1f86367bb6b0e65eadeb33de7765 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1025) 
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] [HUDI-8141] Always use completion time to find begin Instant for incremental source [hudi]

Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.

hudi-bot commented on PR #11947:
URL: https://github.com/apache/hudi/pull/11947#issuecomment-2403869060

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "090f256a315cbbf0c6a54d93cc2739dd693360e6",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=484",
       "triggerID" : "090f256a315cbbf0c6a54d93cc2739dd693360e6",
       "triggerType" : "PUSH"
     }, {
       "hash" : "0df5649013c1cf9e782fcbb67b8a2e7174f94512",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=985",
       "triggerID" : "0df5649013c1cf9e782fcbb67b8a2e7174f94512",
       "triggerType" : "PUSH"
     }, {
       "hash" : "d52e0128f409918644255abbe4376dc4a780bfc8",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=986",
       "triggerID" : "d52e0128f409918644255abbe4376dc4a780bfc8",
       "triggerType" : "PUSH"
     }, {
       "hash" : "9e668f65c6d4f73c76573236f8fa507dfb80edac",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "9e668f65c6d4f73c76573236f8fa507dfb80edac",
       "triggerType" : "PUSH"
     }, {
       "hash" : "f145131d524bf190986e5de600a326e32a27beca",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=987",
       "triggerID" : "f145131d524bf190986e5de600a326e32a27beca",
       "triggerType" : "PUSH"
     }, {
       "hash" : "0b008f807b42aa6fb2686922bb88d63e84a1f299",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=993",
       "triggerID" : "0b008f807b42aa6fb2686922bb88d63e84a1f299",
       "triggerType" : "PUSH"
     }, {
       "hash" : "65c01ad9f2b33d36d4890355feef91844438e732",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=995",
       "triggerID" : "65c01ad9f2b33d36d4890355feef91844438e732",
       "triggerType" : "PUSH"
     }, {
       "hash" : "955493fdc93842e8c99529cec35e39cecdbe9638",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=996",
       "triggerID" : "955493fdc93842e8c99529cec35e39cecdbe9638",
       "triggerType" : "PUSH"
     }, {
       "hash" : "533f00e5546da0026bbe51e1c030ddd1c19fdd08",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "533f00e5546da0026bbe51e1c030ddd1c19fdd08",
       "triggerType" : "PUSH"
     }, {
       "hash" : "05777be67e37f3a6a8eaa59e54d9e17e5fc6ba9b",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1016",
       "triggerID" : "05777be67e37f3a6a8eaa59e54d9e17e5fc6ba9b",
       "triggerType" : "PUSH"
     }, {
       "hash" : "a2ee90c549332a7f17a7d9b2799849b77c08a8e6",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "a2ee90c549332a7f17a7d9b2799849b77c08a8e6",
       "triggerType" : "PUSH"
     }, {
       "hash" : "05ce568b0b6ba14da9ca40d26b39ba478837d747",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1017",
       "triggerID" : "05ce568b0b6ba14da9ca40d26b39ba478837d747",
       "triggerType" : "PUSH"
     }, {
       "hash" : "840e3b33d39b1f86367bb6b0e65eadeb33de7765",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1025",
       "triggerID" : "840e3b33d39b1f86367bb6b0e65eadeb33de7765",
       "triggerType" : "PUSH"
     }, {
       "hash" : "8d2e2dd83f51d03e890d424a51d3e0e84e904a43",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1031",
       "triggerID" : "8d2e2dd83f51d03e890d424a51d3e0e84e904a43",
       "triggerType" : "PUSH"
     }, {
       "hash" : "688381914087ff11acf2f8c9372dbf39afa0bd40",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1035",
       "triggerID" : "688381914087ff11acf2f8c9372dbf39afa0bd40",
       "triggerType" : "PUSH"
     }, {
       "hash" : "fc2c138901e82ed5f3fbf363c2a0ce8140ade050",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "fc2c138901e82ed5f3fbf363c2a0ce8140ade050",
       "triggerType" : "PUSH"
     }, {
       "hash" : "6ed9e1cb1e5a191752d6d39c7a9d5d41840f0cdf",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1039",
       "triggerID" : "6ed9e1cb1e5a191752d6d39c7a9d5d41840f0cdf",
       "triggerType" : "PUSH"
     }, {
       "hash" : "a510a7de1458bb996df5bbb341e8d010e4c91381",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "a510a7de1458bb996df5bbb341e8d010e4c91381",
       "triggerType" : "PUSH"
     }, {
       "hash" : "49677397cd117228742205944606f7a789090376",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1042",
       "triggerID" : "49677397cd117228742205944606f7a789090376",
       "triggerType" : "PUSH"
     }, {
       "hash" : "1630db0a37422aed942ff280560a9d6329df5015",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1044",
       "triggerID" : "1630db0a37422aed942ff280560a9d6329df5015",
       "triggerType" : "PUSH"
     }, {
       "hash" : "f11c76301d4ab524db0af57aa831be3a898ea0fb",
       "status" : "CANCELED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1048",
       "triggerID" : "f11c76301d4ab524db0af57aa831be3a898ea0fb",
       "triggerType" : "PUSH"
     }, {
       "hash" : "d1a0dc4c2017853a7ba0c3302f886043a63f29da",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "d1a0dc4c2017853a7ba0c3302f886043a63f29da",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 9e668f65c6d4f73c76573236f8fa507dfb80edac UNKNOWN
   * 533f00e5546da0026bbe51e1c030ddd1c19fdd08 UNKNOWN
   * a2ee90c549332a7f17a7d9b2799849b77c08a8e6 UNKNOWN
   * fc2c138901e82ed5f3fbf363c2a0ce8140ade050 UNKNOWN
   * a510a7de1458bb996df5bbb341e8d010e4c91381 UNKNOWN
   * f11c76301d4ab524db0af57aa831be3a898ea0fb Azure: [CANCELED](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1048) 
   * d1a0dc4c2017853a7ba0c3302f886043a63f29da UNKNOWN
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] [HUDI-8141] Always use completion time to find begin Instant for incremental source [hudi]

Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.

hudi-bot commented on PR #11947:
URL: https://github.com/apache/hudi/pull/11947#issuecomment-2403839507

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "090f256a315cbbf0c6a54d93cc2739dd693360e6",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=484",
       "triggerID" : "090f256a315cbbf0c6a54d93cc2739dd693360e6",
       "triggerType" : "PUSH"
     }, {
       "hash" : "0df5649013c1cf9e782fcbb67b8a2e7174f94512",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=985",
       "triggerID" : "0df5649013c1cf9e782fcbb67b8a2e7174f94512",
       "triggerType" : "PUSH"
     }, {
       "hash" : "d52e0128f409918644255abbe4376dc4a780bfc8",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=986",
       "triggerID" : "d52e0128f409918644255abbe4376dc4a780bfc8",
       "triggerType" : "PUSH"
     }, {
       "hash" : "9e668f65c6d4f73c76573236f8fa507dfb80edac",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "9e668f65c6d4f73c76573236f8fa507dfb80edac",
       "triggerType" : "PUSH"
     }, {
       "hash" : "f145131d524bf190986e5de600a326e32a27beca",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=987",
       "triggerID" : "f145131d524bf190986e5de600a326e32a27beca",
       "triggerType" : "PUSH"
     }, {
       "hash" : "0b008f807b42aa6fb2686922bb88d63e84a1f299",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=993",
       "triggerID" : "0b008f807b42aa6fb2686922bb88d63e84a1f299",
       "triggerType" : "PUSH"
     }, {
       "hash" : "65c01ad9f2b33d36d4890355feef91844438e732",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=995",
       "triggerID" : "65c01ad9f2b33d36d4890355feef91844438e732",
       "triggerType" : "PUSH"
     }, {
       "hash" : "955493fdc93842e8c99529cec35e39cecdbe9638",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=996",
       "triggerID" : "955493fdc93842e8c99529cec35e39cecdbe9638",
       "triggerType" : "PUSH"
     }, {
       "hash" : "533f00e5546da0026bbe51e1c030ddd1c19fdd08",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "533f00e5546da0026bbe51e1c030ddd1c19fdd08",
       "triggerType" : "PUSH"
     }, {
       "hash" : "05777be67e37f3a6a8eaa59e54d9e17e5fc6ba9b",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1016",
       "triggerID" : "05777be67e37f3a6a8eaa59e54d9e17e5fc6ba9b",
       "triggerType" : "PUSH"
     }, {
       "hash" : "a2ee90c549332a7f17a7d9b2799849b77c08a8e6",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "a2ee90c549332a7f17a7d9b2799849b77c08a8e6",
       "triggerType" : "PUSH"
     }, {
       "hash" : "05ce568b0b6ba14da9ca40d26b39ba478837d747",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1017",
       "triggerID" : "05ce568b0b6ba14da9ca40d26b39ba478837d747",
       "triggerType" : "PUSH"
     }, {
       "hash" : "840e3b33d39b1f86367bb6b0e65eadeb33de7765",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1025",
       "triggerID" : "840e3b33d39b1f86367bb6b0e65eadeb33de7765",
       "triggerType" : "PUSH"
     }, {
       "hash" : "8d2e2dd83f51d03e890d424a51d3e0e84e904a43",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1031",
       "triggerID" : "8d2e2dd83f51d03e890d424a51d3e0e84e904a43",
       "triggerType" : "PUSH"
     }, {
       "hash" : "688381914087ff11acf2f8c9372dbf39afa0bd40",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1035",
       "triggerID" : "688381914087ff11acf2f8c9372dbf39afa0bd40",
       "triggerType" : "PUSH"
     }, {
       "hash" : "fc2c138901e82ed5f3fbf363c2a0ce8140ade050",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "fc2c138901e82ed5f3fbf363c2a0ce8140ade050",
       "triggerType" : "PUSH"
     }, {
       "hash" : "6ed9e1cb1e5a191752d6d39c7a9d5d41840f0cdf",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1039",
       "triggerID" : "6ed9e1cb1e5a191752d6d39c7a9d5d41840f0cdf",
       "triggerType" : "PUSH"
     }, {
       "hash" : "a510a7de1458bb996df5bbb341e8d010e4c91381",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "a510a7de1458bb996df5bbb341e8d010e4c91381",
       "triggerType" : "PUSH"
     }, {
       "hash" : "49677397cd117228742205944606f7a789090376",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1042",
       "triggerID" : "49677397cd117228742205944606f7a789090376",
       "triggerType" : "PUSH"
     }, {
       "hash" : "1630db0a37422aed942ff280560a9d6329df5015",
       "status" : "CANCELED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1044",
       "triggerID" : "1630db0a37422aed942ff280560a9d6329df5015",
       "triggerType" : "PUSH"
     }, {
       "hash" : "f11c76301d4ab524db0af57aa831be3a898ea0fb",
       "status" : "PENDING",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1048",
       "triggerID" : "f11c76301d4ab524db0af57aa831be3a898ea0fb",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 9e668f65c6d4f73c76573236f8fa507dfb80edac UNKNOWN
   * 533f00e5546da0026bbe51e1c030ddd1c19fdd08 UNKNOWN
   * a2ee90c549332a7f17a7d9b2799849b77c08a8e6 UNKNOWN
   * fc2c138901e82ed5f3fbf363c2a0ce8140ade050 UNKNOWN
   * a510a7de1458bb996df5bbb341e8d010e4c91381 UNKNOWN
   * 1630db0a37422aed942ff280560a9d6329df5015 Azure: [CANCELED](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1044) 
   * f11c76301d4ab524db0af57aa831be3a898ea0fb Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1048) 
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] [HUDI-8141] Incremental Query with Completion Time [hudi]

Posted by "danny0405 (via GitHub)" <gi...@apache.org>.

danny0405 commented on code in PR #11947:
URL: https://github.com/apache/hudi/pull/11947#discussion_r1803972453


##########
hudi-utilities/src/main/java/org/apache/hudi/utilities/sources/SnapshotLoadQuerySplitter.java:
##########
@@ -118,6 +124,32 @@ public QueryInfo getNextCheckpoint(Dataset<Row> df, QueryInfo queryInfo, Option<
         .orElse(queryInfo);
   }
 
+  public Option<CheckpointWithPredicates> getNextCheckpoint(Dataset<Row> df, QueryContext queryContext,
+                                                            Option<SourceProfileSupplier> sourceProfileSupplier) {
+    // the start instant would be included into the final query result. So we need to get
+    // a strictly lower timestamp to have query splitter include the start instant
+    Option<CheckpointWithPredicates> nextCheckpointWithPredicates =
+        getNextCheckpointWithPredicates(df, instantTimeMinusMillis(queryContext.getBeginInstant().get(), 1));

Review Comment:
   Can we define the behavior through range type?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] [HUDI-8141] Incremental Query with Completion Time [hudi]

Posted by "danny0405 (via GitHub)" <gi...@apache.org>.

danny0405 commented on code in PR #11947:
URL: https://github.com/apache/hudi/pull/11947#discussion_r1803961786


##########
hudi-common/src/test/java/org/apache/hudi/common/testutils/HoodieTestDataGenerator.java:
##########
@@ -680,7 +680,7 @@ public List<HoodieRecord> generateInsertsAsPerSchema(String commitTime, Integer
   public List<HoodieRecord> generateInserts(String instantTime, Integer n) {
     return generateInserts(instantTime, n, false);
   }
-
+  

Review Comment:
   unnecessary change.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] [HUDI-8141] Incremental Query with Completion Time [hudi]

Posted by "yihua (via GitHub)" <gi...@apache.org>.

yihua commented on code in PR #11947:
URL: https://github.com/apache/hudi/pull/11947#discussion_r1804227698


##########
hudi-common/src/test/java/org/apache/hudi/common/testutils/HoodieTestDataGenerator.java:
##########
@@ -680,7 +680,7 @@ public List<HoodieRecord> generateInsertsAsPerSchema(String commitTime, Integer
   public List<HoodieRecord> generateInserts(String instantTime, Integer n) {
     return generateInserts(instantTime, n, false);
   }
-
+  

Review Comment:
   Fixed.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] [HUDI-8141] Incremental Query with Completion Time [hudi]

Posted by "yihua (via GitHub)" <gi...@apache.org>.

yihua commented on code in PR #11947:
URL: https://github.com/apache/hudi/pull/11947#discussion_r1802115138


##########
hudi-common/src/main/java/org/apache/hudi/common/table/timeline/CompletionTimeQueryView.java:
##########
@@ -189,46 +193,78 @@ public Option<String> getCompletionTime(String beginTime) {
    * <p>By default, assumes there is at most 1 day time of duration for an instant to accelerate the queries.
    *
    * @param timeline The timeline.
-   * @param rangeStart   The query range start completion time.
-   * @param rangeEnd     The query range end completion time.
+   * @param rangeStartCompletionTime   The query range start completion time.
+   * @param rangeEndCompletionTime     The query range end completion time.
    * @param rangeType    The range type.
    *
    * @return The sorted instant time list.
    */
   public List<String> getStartTimes(
       HoodieTimeline timeline,
-      Option<String> rangeStart,
-      Option<String> rangeEnd,
+      Option<String> rangeStartCompletionTime,
+      Option<String> rangeEndCompletionTime,
       InstantRange.RangeType rangeType) {
     // assumes any instant/transaction lasts at most 1 day to optimize the query efficiency.
-    return getStartTimes(timeline, rangeStart, rangeEnd, rangeType, s -> HoodieInstantTimeGenerator.instantTimeMinusMillis(s, MILLI_SECONDS_IN_ONE_DAY));
+    return getStartTimes(
+        timeline,
+        rangeStartCompletionTime,
+        rangeEndCompletionTime,
+        rangeType,
+        GET_INSTANT_ONE_DAY_BEFORE);
   }
 
   /**
    * Queries the instant start time with given completion time range.
    *
-   * @param rangeStart              The query range start completion time.
-   * @param rangeEnd                The query range end completion time.
+   * @param rangeStartCompletionTime              The query range start completion time.
+   * @param rangeEndCompletionTime                The query range end completion time.
    * @param earliestInstantTimeFunc The function to generate the earliest start time boundary
    *                                with the minimum completion time.
    *
    * @return The sorted instant time list.
    */
   @VisibleForTesting
   public List<String> getStartTimes(
-      String rangeStart,
-      String rangeEnd,
+      String rangeStartCompletionTime,
+      String rangeEndCompletionTime,
       Function<String, String> earliestInstantTimeFunc) {
-    return getStartTimes(metaClient.getCommitsTimeline().filterCompletedInstants(), Option.ofNullable(rangeStart), Option.ofNullable(rangeEnd),
-        InstantRange.RangeType.CLOSED_CLOSED, earliestInstantTimeFunc);
+    return getStartTimes(
+        metaClient.getCommitsTimeline().filterCompletedInstants(),
+        Option.ofNullable(rangeStartCompletionTime),
+        Option.ofNullable(rangeEndCompletionTime),
+        InstantRange.RangeType.CLOSED_CLOSED,
+        earliestInstantTimeFunc);
+  }
+
+  /**
+   * Queries the instant start time with given completion time range.
+   *
+   * @param rangeStartCompletionTime              The query range start completion time.
+   * @param rangeEndCompletionTime                The query range end completion time.
+   * @param rangeType               The range type.
+   *                                with the minimum completion time.
+   *
+   * @return The sorted instant time list.
+   */
+  @VisibleForTesting
+  public List<String> getStartTimes(
+      String rangeStartCompletionTime,
+      String rangeEndCompletionTime,
+      RangeType rangeType) {
+    return getStartTimes(
+        metaClient.getCommitsTimeline().filterCompletedInstants(),
+        Option.ofNullable(rangeStartCompletionTime),
+        Option.ofNullable(rangeEndCompletionTime),
+        rangeType,
+        GET_INSTANT_ONE_DAY_BEFORE);
   }
 
   /**
    * Queries the instant start time with given completion time range.
    *
    * @param timeline            The timeline.
-   * @param rangeStart              The query range start completion time.
-   * @param rangeEnd                The query range end completion time.
+   * @param rangeStartCompletionTime              The query range start completion time.
+   * @param rangeEndCompletionTime                The query range end completion time.

Review Comment:
   Done.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] [HUDI-8141] Incremental Query with Completion Time [hudi]

Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.

hudi-bot commented on PR #11947:
URL: https://github.com/apache/hudi/pull/11947#issuecomment-2415372196

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "090f256a315cbbf0c6a54d93cc2739dd693360e6",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=484",
       "triggerID" : "090f256a315cbbf0c6a54d93cc2739dd693360e6",
       "triggerType" : "PUSH"
     }, {
       "hash" : "0df5649013c1cf9e782fcbb67b8a2e7174f94512",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=985",
       "triggerID" : "0df5649013c1cf9e782fcbb67b8a2e7174f94512",
       "triggerType" : "PUSH"
     }, {
       "hash" : "d52e0128f409918644255abbe4376dc4a780bfc8",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=986",
       "triggerID" : "d52e0128f409918644255abbe4376dc4a780bfc8",
       "triggerType" : "PUSH"
     }, {
       "hash" : "9e668f65c6d4f73c76573236f8fa507dfb80edac",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "9e668f65c6d4f73c76573236f8fa507dfb80edac",
       "triggerType" : "PUSH"
     }, {
       "hash" : "f145131d524bf190986e5de600a326e32a27beca",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=987",
       "triggerID" : "f145131d524bf190986e5de600a326e32a27beca",
       "triggerType" : "PUSH"
     }, {
       "hash" : "0b008f807b42aa6fb2686922bb88d63e84a1f299",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=993",
       "triggerID" : "0b008f807b42aa6fb2686922bb88d63e84a1f299",
       "triggerType" : "PUSH"
     }, {
       "hash" : "65c01ad9f2b33d36d4890355feef91844438e732",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=995",
       "triggerID" : "65c01ad9f2b33d36d4890355feef91844438e732",
       "triggerType" : "PUSH"
     }, {
       "hash" : "955493fdc93842e8c99529cec35e39cecdbe9638",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=996",
       "triggerID" : "955493fdc93842e8c99529cec35e39cecdbe9638",
       "triggerType" : "PUSH"
     }, {
       "hash" : "533f00e5546da0026bbe51e1c030ddd1c19fdd08",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "533f00e5546da0026bbe51e1c030ddd1c19fdd08",
       "triggerType" : "PUSH"
     }, {
       "hash" : "05777be67e37f3a6a8eaa59e54d9e17e5fc6ba9b",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1016",
       "triggerID" : "05777be67e37f3a6a8eaa59e54d9e17e5fc6ba9b",
       "triggerType" : "PUSH"
     }, {
       "hash" : "a2ee90c549332a7f17a7d9b2799849b77c08a8e6",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "a2ee90c549332a7f17a7d9b2799849b77c08a8e6",
       "triggerType" : "PUSH"
     }, {
       "hash" : "05ce568b0b6ba14da9ca40d26b39ba478837d747",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1017",
       "triggerID" : "05ce568b0b6ba14da9ca40d26b39ba478837d747",
       "triggerType" : "PUSH"
     }, {
       "hash" : "840e3b33d39b1f86367bb6b0e65eadeb33de7765",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1025",
       "triggerID" : "840e3b33d39b1f86367bb6b0e65eadeb33de7765",
       "triggerType" : "PUSH"
     }, {
       "hash" : "8d2e2dd83f51d03e890d424a51d3e0e84e904a43",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1031",
       "triggerID" : "8d2e2dd83f51d03e890d424a51d3e0e84e904a43",
       "triggerType" : "PUSH"
     }, {
       "hash" : "688381914087ff11acf2f8c9372dbf39afa0bd40",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1035",
       "triggerID" : "688381914087ff11acf2f8c9372dbf39afa0bd40",
       "triggerType" : "PUSH"
     }, {
       "hash" : "fc2c138901e82ed5f3fbf363c2a0ce8140ade050",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "fc2c138901e82ed5f3fbf363c2a0ce8140ade050",
       "triggerType" : "PUSH"
     }, {
       "hash" : "6ed9e1cb1e5a191752d6d39c7a9d5d41840f0cdf",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1039",
       "triggerID" : "6ed9e1cb1e5a191752d6d39c7a9d5d41840f0cdf",
       "triggerType" : "PUSH"
     }, {
       "hash" : "a510a7de1458bb996df5bbb341e8d010e4c91381",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "a510a7de1458bb996df5bbb341e8d010e4c91381",
       "triggerType" : "PUSH"
     }, {
       "hash" : "49677397cd117228742205944606f7a789090376",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1042",
       "triggerID" : "49677397cd117228742205944606f7a789090376",
       "triggerType" : "PUSH"
     }, {
       "hash" : "1630db0a37422aed942ff280560a9d6329df5015",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1044",
       "triggerID" : "1630db0a37422aed942ff280560a9d6329df5015",
       "triggerType" : "PUSH"
     }, {
       "hash" : "f11c76301d4ab524db0af57aa831be3a898ea0fb",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1048",
       "triggerID" : "f11c76301d4ab524db0af57aa831be3a898ea0fb",
       "triggerType" : "PUSH"
     }, {
       "hash" : "d1a0dc4c2017853a7ba0c3302f886043a63f29da",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1050",
       "triggerID" : "d1a0dc4c2017853a7ba0c3302f886043a63f29da",
       "triggerType" : "PUSH"
     }, {
       "hash" : "6a4c15ec39ded1f90b113b272554217698a98e1c",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1087",
       "triggerID" : "6a4c15ec39ded1f90b113b272554217698a98e1c",
       "triggerType" : "PUSH"
     }, {
       "hash" : "288e730d645a5e58e50a87ef5f44ba11a8fb0533",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1092",
       "triggerID" : "288e730d645a5e58e50a87ef5f44ba11a8fb0533",
       "triggerType" : "PUSH"
     }, {
       "hash" : "70a5c21321a0da3a0be8d85e206c24befa7b89f1",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1095",
       "triggerID" : "70a5c21321a0da3a0be8d85e206c24befa7b89f1",
       "triggerType" : "PUSH"
     }, {
       "hash" : "f072a986114949c5e426ce64a5f5568fc718e25d",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "f072a986114949c5e426ce64a5f5568fc718e25d",
       "triggerType" : "PUSH"
     }, {
       "hash" : "45b27475b1e8806ec8c566d6f151f4bc79c8549d",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1096",
       "triggerID" : "45b27475b1e8806ec8c566d6f151f4bc79c8549d",
       "triggerType" : "PUSH"
     }, {
       "hash" : "3c62ef22f6cde28110e5eb2fca8760a008d77723",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "3c62ef22f6cde28110e5eb2fca8760a008d77723",
       "triggerType" : "PUSH"
     }, {
       "hash" : "dc785fa04e542b2ccc45494e0292f5153c7ecea8",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1097",
       "triggerID" : "dc785fa04e542b2ccc45494e0292f5153c7ecea8",
       "triggerType" : "PUSH"
     }, {
       "hash" : "585ddb8049b9b9ccb057027e265b08a1b586df8c",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "585ddb8049b9b9ccb057027e265b08a1b586df8c",
       "triggerType" : "PUSH"
     }, {
       "hash" : "b62ff8675beed2c4487c788b9ca1a0ad4271b913",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "b62ff8675beed2c4487c788b9ca1a0ad4271b913",
       "triggerType" : "PUSH"
     }, {
       "hash" : "2d46048a7402907f348cc96da2839ba1b3e4932a",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1099",
       "triggerID" : "2d46048a7402907f348cc96da2839ba1b3e4932a",
       "triggerType" : "PUSH"
     }, {
       "hash" : "004fb6ebf23f796a4b77650e0530f0f1145e8b91",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1101",
       "triggerID" : "004fb6ebf23f796a4b77650e0530f0f1145e8b91",
       "triggerType" : "PUSH"
     }, {
       "hash" : "af0c33333b5e9ae6d59f20a6db1d7c016ea015d3",
       "status" : "CANCELED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1165",
       "triggerID" : "af0c33333b5e9ae6d59f20a6db1d7c016ea015d3",
       "triggerType" : "PUSH"
     }, {
       "hash" : "da8111118d71306500229619d3b7996fcd2526e8",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "da8111118d71306500229619d3b7996fcd2526e8",
       "triggerType" : "PUSH"
     }, {
       "hash" : "83257e2b9bd3c426cf90e11003a5f244a750d7fa",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "83257e2b9bd3c426cf90e11003a5f244a750d7fa",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 9e668f65c6d4f73c76573236f8fa507dfb80edac UNKNOWN
   * 533f00e5546da0026bbe51e1c030ddd1c19fdd08 UNKNOWN
   * a2ee90c549332a7f17a7d9b2799849b77c08a8e6 UNKNOWN
   * fc2c138901e82ed5f3fbf363c2a0ce8140ade050 UNKNOWN
   * a510a7de1458bb996df5bbb341e8d010e4c91381 UNKNOWN
   * f072a986114949c5e426ce64a5f5568fc718e25d UNKNOWN
   * 3c62ef22f6cde28110e5eb2fca8760a008d77723 UNKNOWN
   * 585ddb8049b9b9ccb057027e265b08a1b586df8c UNKNOWN
   * b62ff8675beed2c4487c788b9ca1a0ad4271b913 UNKNOWN
   * af0c33333b5e9ae6d59f20a6db1d7c016ea015d3 Azure: [CANCELED](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1165) 
   * da8111118d71306500229619d3b7996fcd2526e8 UNKNOWN
   * 83257e2b9bd3c426cf90e11003a5f244a750d7fa UNKNOWN
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] [HUDI-8141] Incremental Query with Completion Time [hudi]

Posted by "yihua (via GitHub)" <gi...@apache.org>.

yihua commented on code in PR #11947:
URL: https://github.com/apache/hudi/pull/11947#discussion_r1802141492


##########
hudi-utilities/src/main/java/org/apache/hudi/utilities/sources/HoodieIncrSource.java:
##########
@@ -212,35 +203,43 @@ public Pair<Option<Dataset<Row>>, String> fetchNextBatch(Option<String> lastCkpt
           .collect(Collectors.toMap(Pair::getLeft, Pair::getRight));
       reader = reader.options(optionsMap);
     }
+    boolean shouldFullScan =
+        queryContext.getActiveTimeline().isBeforeTimelineStartsByCompletionTime(completionTimeToResumeFrom)
+        && missingCheckpointStrategy == MissingCheckpointStrategy.READ_UPTO_LATEST_COMMIT;
     Dataset<Row> source;
-    // Do Incr pull. Set end instant if available
-    if (queryInfo.isIncremental()) {
-      source = reader
-          .options(readOpts)
-          .option(QUERY_TYPE().key(), QUERY_TYPE_INCREMENTAL_OPT_VAL())
-          .option(BEGIN_INSTANTTIME().key(), queryInfo.getStartInstant())
-          .option(END_INSTANTTIME().key(), queryInfo.getEndInstant())
-          .option(INCREMENTAL_FALLBACK_TO_FULL_TABLE_SCAN().key(),
-              props.getString(INCREMENTAL_FALLBACK_TO_FULL_TABLE_SCAN().key(),
-                  INCREMENTAL_FALLBACK_TO_FULL_TABLE_SCAN().defaultValue()))
-          .option(INCREMENTAL_READ_HANDLE_HOLLOW_COMMIT().key(), handlingMode.name())
-          .load(srcPath);
-    } else {
-      // if checkpoint is missing from source table, and if strategy is set to READ_UPTO_LATEST_COMMIT, we have to issue snapshot query
+    if (instantRange.isEmpty() || shouldFullScan) {
+      // snapshot query
       Dataset<Row> snapshot = reader
           .options(readOpts)
           .option(DataSourceReadOptions.QUERY_TYPE().key(), DataSourceReadOptions.QUERY_TYPE_SNAPSHOT_OPT_VAL())
           .load(srcPath);
       if (snapshotLoadQuerySplitter.isPresent()) {
-        queryInfo = snapshotLoadQuerySplitter.get().getNextCheckpoint(snapshot, queryInfo, sourceProfileSupplier);
+        queryContext = snapshotLoadQuerySplitter.get().getNextCheckpoint(snapshot, queryContext, sourceProfileSupplier);
+        // update endTime/next checkpoint
+        completionTimeToStopAt = queryContext.getEndInstant().orElse(queryContext.getLastInstant());
       }
+
+      snapshot = queryContext.getPredicateFilter().map(snapshot::filter).orElse(snapshot);
       source = snapshot
           // add filtering so that only interested records are returned.
-          .filter(String.format("%s > '%s'", HoodieRecord.COMMIT_TIME_METADATA_FIELD,
-              queryInfo.getStartInstant()))
+          // completion time comparison uses ( , ], but when comparing start time we need to use [, ]

Review Comment:
   Removed.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] [HUDI-8141] Incremental Query with Completion Time [hudi]

Posted by "danny0405 (via GitHub)" <gi...@apache.org>.

danny0405 commented on code in PR #11947:
URL: https://github.com/apache/hudi/pull/11947#discussion_r1802196899


##########
hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/hudi/IncrementalRelation.scala:
##########
@@ -90,24 +78,24 @@ class IncrementalRelation(val sqlContext: SQLContext,
     throw new HoodieException("Incremental queries are not supported when meta fields are disabled")
   }
 
+  private val queryContext: IncrementalQueryAnalyzer.QueryContext =
+    IncrementalQueryAnalyzer.builder()
+      .metaClient(metaClient)
+      .startTime(optParams(DataSourceReadOptions.BEGIN_INSTANTTIME.key))
+      .endTime(optParams.getOrElse(DataSourceReadOptions.END_INSTANTTIME.key, null))
+      .rangeType(InstantRange.RangeType.OPEN_CLOSED)

Review Comment:
   > In Spark the hoodie.datasource.read.begin.instanttime is exclusive
   
   Not sure it is correct, at least in commen sense of incremental read, the start offset should be inclusive by default.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] [HUDI-8141] Incremental Query with Completion Time [hudi]

Posted by "danny0405 (via GitHub)" <gi...@apache.org>.

danny0405 commented on code in PR #11947:
URL: https://github.com/apache/hudi/pull/11947#discussion_r1802198656


##########
hudi-common/src/main/java/org/apache/hudi/common/table/read/IncrementalQueryAnalyzer.java:
##########
@@ -376,14 +382,16 @@ private QueryContext(
         List<HoodieInstant> archivedInstants,
         List<HoodieInstant> activeInstants,
         HoodieTimeline activeTimeline,
-        @Nullable HoodieTimeline archivedTimeline) {
+        @Nullable HoodieTimeline archivedTimeline,
+        @Nullable String predicateFilter) {
       this.startInstant = Option.ofNullable(startInstant);
       this.endInstant = Option.ofNullable(endInstant);
       this.archivedInstants = archivedInstants;
       this.activeInstants = activeInstants;
       this.activeTimeline = activeTimeline;
       this.archivedTimeline = archivedTimeline;
       this.instants = instants;
+      this.predicateFilter = Option.ofNullable(predicateFilter);

Review Comment:
   The `QueryContext` is designed to be immutable, let's mantain the unrelated `predicateFilter` outside of it.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] [HUDI-8141] WIP:Always use completion time to find begin Instant for incremental source [hudi]

Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.

hudi-bot commented on PR #11947:
URL: https://github.com/apache/hudi/pull/11947#issuecomment-2403246895

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "090f256a315cbbf0c6a54d93cc2739dd693360e6",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=484",
       "triggerID" : "090f256a315cbbf0c6a54d93cc2739dd693360e6",
       "triggerType" : "PUSH"
     }, {
       "hash" : "0df5649013c1cf9e782fcbb67b8a2e7174f94512",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=985",
       "triggerID" : "0df5649013c1cf9e782fcbb67b8a2e7174f94512",
       "triggerType" : "PUSH"
     }, {
       "hash" : "d52e0128f409918644255abbe4376dc4a780bfc8",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=986",
       "triggerID" : "d52e0128f409918644255abbe4376dc4a780bfc8",
       "triggerType" : "PUSH"
     }, {
       "hash" : "9e668f65c6d4f73c76573236f8fa507dfb80edac",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "9e668f65c6d4f73c76573236f8fa507dfb80edac",
       "triggerType" : "PUSH"
     }, {
       "hash" : "f145131d524bf190986e5de600a326e32a27beca",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=987",
       "triggerID" : "f145131d524bf190986e5de600a326e32a27beca",
       "triggerType" : "PUSH"
     }, {
       "hash" : "0b008f807b42aa6fb2686922bb88d63e84a1f299",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=993",
       "triggerID" : "0b008f807b42aa6fb2686922bb88d63e84a1f299",
       "triggerType" : "PUSH"
     }, {
       "hash" : "65c01ad9f2b33d36d4890355feef91844438e732",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=995",
       "triggerID" : "65c01ad9f2b33d36d4890355feef91844438e732",
       "triggerType" : "PUSH"
     }, {
       "hash" : "955493fdc93842e8c99529cec35e39cecdbe9638",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=996",
       "triggerID" : "955493fdc93842e8c99529cec35e39cecdbe9638",
       "triggerType" : "PUSH"
     }, {
       "hash" : "533f00e5546da0026bbe51e1c030ddd1c19fdd08",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "533f00e5546da0026bbe51e1c030ddd1c19fdd08",
       "triggerType" : "PUSH"
     }, {
       "hash" : "05777be67e37f3a6a8eaa59e54d9e17e5fc6ba9b",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1016",
       "triggerID" : "05777be67e37f3a6a8eaa59e54d9e17e5fc6ba9b",
       "triggerType" : "PUSH"
     }, {
       "hash" : "a2ee90c549332a7f17a7d9b2799849b77c08a8e6",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "a2ee90c549332a7f17a7d9b2799849b77c08a8e6",
       "triggerType" : "PUSH"
     }, {
       "hash" : "05ce568b0b6ba14da9ca40d26b39ba478837d747",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1017",
       "triggerID" : "05ce568b0b6ba14da9ca40d26b39ba478837d747",
       "triggerType" : "PUSH"
     }, {
       "hash" : "840e3b33d39b1f86367bb6b0e65eadeb33de7765",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1025",
       "triggerID" : "840e3b33d39b1f86367bb6b0e65eadeb33de7765",
       "triggerType" : "PUSH"
     }, {
       "hash" : "8d2e2dd83f51d03e890d424a51d3e0e84e904a43",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1031",
       "triggerID" : "8d2e2dd83f51d03e890d424a51d3e0e84e904a43",
       "triggerType" : "PUSH"
     }, {
       "hash" : "688381914087ff11acf2f8c9372dbf39afa0bd40",
       "status" : "SUCCESS",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1035",
       "triggerID" : "688381914087ff11acf2f8c9372dbf39afa0bd40",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 9e668f65c6d4f73c76573236f8fa507dfb80edac UNKNOWN
   * 533f00e5546da0026bbe51e1c030ddd1c19fdd08 UNKNOWN
   * a2ee90c549332a7f17a7d9b2799849b77c08a8e6 UNKNOWN
   * 688381914087ff11acf2f8c9372dbf39afa0bd40 Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1035) 
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] [HUDI-8141] WIP:Always use completion time to find begin Instant for incremental source [hudi]

Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.

hudi-bot commented on PR #11947:
URL: https://github.com/apache/hudi/pull/11947#issuecomment-2403608616

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "090f256a315cbbf0c6a54d93cc2739dd693360e6",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=484",
       "triggerID" : "090f256a315cbbf0c6a54d93cc2739dd693360e6",
       "triggerType" : "PUSH"
     }, {
       "hash" : "0df5649013c1cf9e782fcbb67b8a2e7174f94512",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=985",
       "triggerID" : "0df5649013c1cf9e782fcbb67b8a2e7174f94512",
       "triggerType" : "PUSH"
     }, {
       "hash" : "d52e0128f409918644255abbe4376dc4a780bfc8",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=986",
       "triggerID" : "d52e0128f409918644255abbe4376dc4a780bfc8",
       "triggerType" : "PUSH"
     }, {
       "hash" : "9e668f65c6d4f73c76573236f8fa507dfb80edac",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "9e668f65c6d4f73c76573236f8fa507dfb80edac",
       "triggerType" : "PUSH"
     }, {
       "hash" : "f145131d524bf190986e5de600a326e32a27beca",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=987",
       "triggerID" : "f145131d524bf190986e5de600a326e32a27beca",
       "triggerType" : "PUSH"
     }, {
       "hash" : "0b008f807b42aa6fb2686922bb88d63e84a1f299",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=993",
       "triggerID" : "0b008f807b42aa6fb2686922bb88d63e84a1f299",
       "triggerType" : "PUSH"
     }, {
       "hash" : "65c01ad9f2b33d36d4890355feef91844438e732",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=995",
       "triggerID" : "65c01ad9f2b33d36d4890355feef91844438e732",
       "triggerType" : "PUSH"
     }, {
       "hash" : "955493fdc93842e8c99529cec35e39cecdbe9638",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=996",
       "triggerID" : "955493fdc93842e8c99529cec35e39cecdbe9638",
       "triggerType" : "PUSH"
     }, {
       "hash" : "533f00e5546da0026bbe51e1c030ddd1c19fdd08",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "533f00e5546da0026bbe51e1c030ddd1c19fdd08",
       "triggerType" : "PUSH"
     }, {
       "hash" : "05777be67e37f3a6a8eaa59e54d9e17e5fc6ba9b",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1016",
       "triggerID" : "05777be67e37f3a6a8eaa59e54d9e17e5fc6ba9b",
       "triggerType" : "PUSH"
     }, {
       "hash" : "a2ee90c549332a7f17a7d9b2799849b77c08a8e6",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "a2ee90c549332a7f17a7d9b2799849b77c08a8e6",
       "triggerType" : "PUSH"
     }, {
       "hash" : "05ce568b0b6ba14da9ca40d26b39ba478837d747",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1017",
       "triggerID" : "05ce568b0b6ba14da9ca40d26b39ba478837d747",
       "triggerType" : "PUSH"
     }, {
       "hash" : "840e3b33d39b1f86367bb6b0e65eadeb33de7765",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1025",
       "triggerID" : "840e3b33d39b1f86367bb6b0e65eadeb33de7765",
       "triggerType" : "PUSH"
     }, {
       "hash" : "8d2e2dd83f51d03e890d424a51d3e0e84e904a43",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1031",
       "triggerID" : "8d2e2dd83f51d03e890d424a51d3e0e84e904a43",
       "triggerType" : "PUSH"
     }, {
       "hash" : "688381914087ff11acf2f8c9372dbf39afa0bd40",
       "status" : "SUCCESS",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1035",
       "triggerID" : "688381914087ff11acf2f8c9372dbf39afa0bd40",
       "triggerType" : "PUSH"
     }, {
       "hash" : "fc2c138901e82ed5f3fbf363c2a0ce8140ade050",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "fc2c138901e82ed5f3fbf363c2a0ce8140ade050",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 9e668f65c6d4f73c76573236f8fa507dfb80edac UNKNOWN
   * 533f00e5546da0026bbe51e1c030ddd1c19fdd08 UNKNOWN
   * a2ee90c549332a7f17a7d9b2799849b77c08a8e6 UNKNOWN
   * 688381914087ff11acf2f8c9372dbf39afa0bd40 Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1035) 
   * fc2c138901e82ed5f3fbf363c2a0ce8140ade050 UNKNOWN
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] [HUDI-8141] WIP:Always use completion time to find begin Instant for incremental source [hudi]

Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.

hudi-bot commented on PR #11947:
URL: https://github.com/apache/hudi/pull/11947#issuecomment-2403742210

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "090f256a315cbbf0c6a54d93cc2739dd693360e6",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=484",
       "triggerID" : "090f256a315cbbf0c6a54d93cc2739dd693360e6",
       "triggerType" : "PUSH"
     }, {
       "hash" : "0df5649013c1cf9e782fcbb67b8a2e7174f94512",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=985",
       "triggerID" : "0df5649013c1cf9e782fcbb67b8a2e7174f94512",
       "triggerType" : "PUSH"
     }, {
       "hash" : "d52e0128f409918644255abbe4376dc4a780bfc8",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=986",
       "triggerID" : "d52e0128f409918644255abbe4376dc4a780bfc8",
       "triggerType" : "PUSH"
     }, {
       "hash" : "9e668f65c6d4f73c76573236f8fa507dfb80edac",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "9e668f65c6d4f73c76573236f8fa507dfb80edac",
       "triggerType" : "PUSH"
     }, {
       "hash" : "f145131d524bf190986e5de600a326e32a27beca",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=987",
       "triggerID" : "f145131d524bf190986e5de600a326e32a27beca",
       "triggerType" : "PUSH"
     }, {
       "hash" : "0b008f807b42aa6fb2686922bb88d63e84a1f299",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=993",
       "triggerID" : "0b008f807b42aa6fb2686922bb88d63e84a1f299",
       "triggerType" : "PUSH"
     }, {
       "hash" : "65c01ad9f2b33d36d4890355feef91844438e732",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=995",
       "triggerID" : "65c01ad9f2b33d36d4890355feef91844438e732",
       "triggerType" : "PUSH"
     }, {
       "hash" : "955493fdc93842e8c99529cec35e39cecdbe9638",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=996",
       "triggerID" : "955493fdc93842e8c99529cec35e39cecdbe9638",
       "triggerType" : "PUSH"
     }, {
       "hash" : "533f00e5546da0026bbe51e1c030ddd1c19fdd08",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "533f00e5546da0026bbe51e1c030ddd1c19fdd08",
       "triggerType" : "PUSH"
     }, {
       "hash" : "05777be67e37f3a6a8eaa59e54d9e17e5fc6ba9b",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1016",
       "triggerID" : "05777be67e37f3a6a8eaa59e54d9e17e5fc6ba9b",
       "triggerType" : "PUSH"
     }, {
       "hash" : "a2ee90c549332a7f17a7d9b2799849b77c08a8e6",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "a2ee90c549332a7f17a7d9b2799849b77c08a8e6",
       "triggerType" : "PUSH"
     }, {
       "hash" : "05ce568b0b6ba14da9ca40d26b39ba478837d747",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1017",
       "triggerID" : "05ce568b0b6ba14da9ca40d26b39ba478837d747",
       "triggerType" : "PUSH"
     }, {
       "hash" : "840e3b33d39b1f86367bb6b0e65eadeb33de7765",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1025",
       "triggerID" : "840e3b33d39b1f86367bb6b0e65eadeb33de7765",
       "triggerType" : "PUSH"
     }, {
       "hash" : "8d2e2dd83f51d03e890d424a51d3e0e84e904a43",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1031",
       "triggerID" : "8d2e2dd83f51d03e890d424a51d3e0e84e904a43",
       "triggerType" : "PUSH"
     }, {
       "hash" : "688381914087ff11acf2f8c9372dbf39afa0bd40",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1035",
       "triggerID" : "688381914087ff11acf2f8c9372dbf39afa0bd40",
       "triggerType" : "PUSH"
     }, {
       "hash" : "fc2c138901e82ed5f3fbf363c2a0ce8140ade050",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "fc2c138901e82ed5f3fbf363c2a0ce8140ade050",
       "triggerType" : "PUSH"
     }, {
       "hash" : "6ed9e1cb1e5a191752d6d39c7a9d5d41840f0cdf",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1039",
       "triggerID" : "6ed9e1cb1e5a191752d6d39c7a9d5d41840f0cdf",
       "triggerType" : "PUSH"
     }, {
       "hash" : "a510a7de1458bb996df5bbb341e8d010e4c91381",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "a510a7de1458bb996df5bbb341e8d010e4c91381",
       "triggerType" : "PUSH"
     }, {
       "hash" : "49677397cd117228742205944606f7a789090376",
       "status" : "CANCELED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1042",
       "triggerID" : "49677397cd117228742205944606f7a789090376",
       "triggerType" : "PUSH"
     }, {
       "hash" : "1630db0a37422aed942ff280560a9d6329df5015",
       "status" : "PENDING",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1044",
       "triggerID" : "1630db0a37422aed942ff280560a9d6329df5015",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 9e668f65c6d4f73c76573236f8fa507dfb80edac UNKNOWN
   * 533f00e5546da0026bbe51e1c030ddd1c19fdd08 UNKNOWN
   * a2ee90c549332a7f17a7d9b2799849b77c08a8e6 UNKNOWN
   * fc2c138901e82ed5f3fbf363c2a0ce8140ade050 UNKNOWN
   * a510a7de1458bb996df5bbb341e8d010e4c91381 UNKNOWN
   * 49677397cd117228742205944606f7a789090376 Azure: [CANCELED](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1042) 
   * 1630db0a37422aed942ff280560a9d6329df5015 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1044) 
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] [HUDI-8141] Always use completion time to find begin Instant for incremental source [hudi]

Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.

hudi-bot commented on PR #11947:
URL: https://github.com/apache/hudi/pull/11947#issuecomment-2404044702

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "090f256a315cbbf0c6a54d93cc2739dd693360e6",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=484",
       "triggerID" : "090f256a315cbbf0c6a54d93cc2739dd693360e6",
       "triggerType" : "PUSH"
     }, {
       "hash" : "0df5649013c1cf9e782fcbb67b8a2e7174f94512",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=985",
       "triggerID" : "0df5649013c1cf9e782fcbb67b8a2e7174f94512",
       "triggerType" : "PUSH"
     }, {
       "hash" : "d52e0128f409918644255abbe4376dc4a780bfc8",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=986",
       "triggerID" : "d52e0128f409918644255abbe4376dc4a780bfc8",
       "triggerType" : "PUSH"
     }, {
       "hash" : "9e668f65c6d4f73c76573236f8fa507dfb80edac",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "9e668f65c6d4f73c76573236f8fa507dfb80edac",
       "triggerType" : "PUSH"
     }, {
       "hash" : "f145131d524bf190986e5de600a326e32a27beca",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=987",
       "triggerID" : "f145131d524bf190986e5de600a326e32a27beca",
       "triggerType" : "PUSH"
     }, {
       "hash" : "0b008f807b42aa6fb2686922bb88d63e84a1f299",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=993",
       "triggerID" : "0b008f807b42aa6fb2686922bb88d63e84a1f299",
       "triggerType" : "PUSH"
     }, {
       "hash" : "65c01ad9f2b33d36d4890355feef91844438e732",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=995",
       "triggerID" : "65c01ad9f2b33d36d4890355feef91844438e732",
       "triggerType" : "PUSH"
     }, {
       "hash" : "955493fdc93842e8c99529cec35e39cecdbe9638",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=996",
       "triggerID" : "955493fdc93842e8c99529cec35e39cecdbe9638",
       "triggerType" : "PUSH"
     }, {
       "hash" : "533f00e5546da0026bbe51e1c030ddd1c19fdd08",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "533f00e5546da0026bbe51e1c030ddd1c19fdd08",
       "triggerType" : "PUSH"
     }, {
       "hash" : "05777be67e37f3a6a8eaa59e54d9e17e5fc6ba9b",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1016",
       "triggerID" : "05777be67e37f3a6a8eaa59e54d9e17e5fc6ba9b",
       "triggerType" : "PUSH"
     }, {
       "hash" : "a2ee90c549332a7f17a7d9b2799849b77c08a8e6",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "a2ee90c549332a7f17a7d9b2799849b77c08a8e6",
       "triggerType" : "PUSH"
     }, {
       "hash" : "05ce568b0b6ba14da9ca40d26b39ba478837d747",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1017",
       "triggerID" : "05ce568b0b6ba14da9ca40d26b39ba478837d747",
       "triggerType" : "PUSH"
     }, {
       "hash" : "840e3b33d39b1f86367bb6b0e65eadeb33de7765",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1025",
       "triggerID" : "840e3b33d39b1f86367bb6b0e65eadeb33de7765",
       "triggerType" : "PUSH"
     }, {
       "hash" : "8d2e2dd83f51d03e890d424a51d3e0e84e904a43",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1031",
       "triggerID" : "8d2e2dd83f51d03e890d424a51d3e0e84e904a43",
       "triggerType" : "PUSH"
     }, {
       "hash" : "688381914087ff11acf2f8c9372dbf39afa0bd40",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1035",
       "triggerID" : "688381914087ff11acf2f8c9372dbf39afa0bd40",
       "triggerType" : "PUSH"
     }, {
       "hash" : "fc2c138901e82ed5f3fbf363c2a0ce8140ade050",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "fc2c138901e82ed5f3fbf363c2a0ce8140ade050",
       "triggerType" : "PUSH"
     }, {
       "hash" : "6ed9e1cb1e5a191752d6d39c7a9d5d41840f0cdf",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1039",
       "triggerID" : "6ed9e1cb1e5a191752d6d39c7a9d5d41840f0cdf",
       "triggerType" : "PUSH"
     }, {
       "hash" : "a510a7de1458bb996df5bbb341e8d010e4c91381",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "a510a7de1458bb996df5bbb341e8d010e4c91381",
       "triggerType" : "PUSH"
     }, {
       "hash" : "49677397cd117228742205944606f7a789090376",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1042",
       "triggerID" : "49677397cd117228742205944606f7a789090376",
       "triggerType" : "PUSH"
     }, {
       "hash" : "1630db0a37422aed942ff280560a9d6329df5015",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1044",
       "triggerID" : "1630db0a37422aed942ff280560a9d6329df5015",
       "triggerType" : "PUSH"
     }, {
       "hash" : "f11c76301d4ab524db0af57aa831be3a898ea0fb",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1048",
       "triggerID" : "f11c76301d4ab524db0af57aa831be3a898ea0fb",
       "triggerType" : "PUSH"
     }, {
       "hash" : "d1a0dc4c2017853a7ba0c3302f886043a63f29da",
       "status" : "FAILURE",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1050",
       "triggerID" : "d1a0dc4c2017853a7ba0c3302f886043a63f29da",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 9e668f65c6d4f73c76573236f8fa507dfb80edac UNKNOWN
   * 533f00e5546da0026bbe51e1c030ddd1c19fdd08 UNKNOWN
   * a2ee90c549332a7f17a7d9b2799849b77c08a8e6 UNKNOWN
   * fc2c138901e82ed5f3fbf363c2a0ce8140ade050 UNKNOWN
   * a510a7de1458bb996df5bbb341e8d010e4c91381 UNKNOWN
   * d1a0dc4c2017853a7ba0c3302f886043a63f29da Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1050) 
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] [HUDI-8141] Incremental Query with Completion Time [hudi]

Posted by "CTTY (via GitHub)" <gi...@apache.org>.

CTTY commented on code in PR #11947:
URL: https://github.com/apache/hudi/pull/11947#discussion_r1798095067


##########
hudi-utilities/src/main/java/org/apache/hudi/utilities/sources/HoodieIncrSource.java:
##########
@@ -212,35 +257,43 @@ public Pair<Option<Dataset<Row>>, String> fetchNextBatch(Option<String> lastCkpt
           .collect(Collectors.toMap(Pair::getLeft, Pair::getRight));
       reader = reader.options(optionsMap);
     }
+    boolean shouldFullScan =
+        queryContext.getActiveTimeline().isBeforeTimelineStartsByCompletionTime(startTime)
+        && missingCheckpointStrategy == MissingCheckpointStrategy.READ_UPTO_LATEST_COMMIT;
     Dataset<Row> source;
-    // Do Incr pull. Set end instant if available
-    if (queryInfo.isIncremental()) {
-      source = reader
-          .options(readOpts)
-          .option(QUERY_TYPE().key(), QUERY_TYPE_INCREMENTAL_OPT_VAL())
-          .option(BEGIN_INSTANTTIME().key(), queryInfo.getStartInstant())
-          .option(END_INSTANTTIME().key(), queryInfo.getEndInstant())
-          .option(INCREMENTAL_FALLBACK_TO_FULL_TABLE_SCAN().key(),
-              props.getString(INCREMENTAL_FALLBACK_TO_FULL_TABLE_SCAN().key(),
-                  INCREMENTAL_FALLBACK_TO_FULL_TABLE_SCAN().defaultValue()))
-          .option(INCREMENTAL_READ_HANDLE_HOLLOW_COMMIT().key(), handlingMode.name())
-          .load(srcPath);
-    } else {
-      // if checkpoint is missing from source table, and if strategy is set to READ_UPTO_LATEST_COMMIT, we have to issue snapshot query
+    if (instantRange.isEmpty() || shouldFullScan) {
+      // snapshot query
       Dataset<Row> snapshot = reader
           .options(readOpts)
           .option(DataSourceReadOptions.QUERY_TYPE().key(), DataSourceReadOptions.QUERY_TYPE_SNAPSHOT_OPT_VAL())
           .load(srcPath);
       if (snapshotLoadQuerySplitter.isPresent()) {
-        queryInfo = snapshotLoadQuerySplitter.get().getNextCheckpoint(snapshot, queryInfo, sourceProfileSupplier);
+        queryContext = snapshotLoadQuerySplitter.get().getNextCheckpoint(snapshot, queryContext, sourceProfileSupplier);
+        // update endTime/next checkpoint
+        endTime = queryContext.getEndInstant().orElse(queryContext.getLastInstant());
       }
+      Set<String> validInstants = new HashSet<>(queryContext.getInstants());
+      snapshot = queryContext.getPredicateFilter().map(snapshot::filter).orElse(snapshot);
       source = snapshot
           // add filtering so that only interested records are returned.
-          .filter(String.format("%s > '%s'", HoodieRecord.COMMIT_TIME_METADATA_FIELD,
-              queryInfo.getStartInstant()))
+          // completion time comparison uses ( , ], but when comparing start time we need to use [, ]
+          .filter(String.format("%s >= '%s'", HoodieRecord.COMMIT_TIME_METADATA_FIELD,
+              queryContext.getStartInstant().get()))
           .filter(String.format("%s <= '%s'", HoodieRecord.COMMIT_TIME_METADATA_FIELD,
-              queryInfo.getEndInstant()));
-      source = queryInfo.getPredicateFilter().map(source::filter).orElse(source);
+              queryContext.getEndInstant().orElse(queryContext.getLastInstant())))
+          .filter((Row row) -> validInstants.contains(row.getAs(HoodieRecord.COMMIT_TIME_METADATA_FIELD)));

Review Comment:
   Fixed by switching to use pure string expression



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] [HUDI-8141] Incremental Query with Completion Time [hudi]

Posted by "yihua (via GitHub)" <gi...@apache.org>.

yihua commented on code in PR #11947:
URL: https://github.com/apache/hudi/pull/11947#discussion_r1797777212


##########
hudi-common/src/main/java/org/apache/hudi/common/table/timeline/TimelineUtils.java:
##########
@@ -308,6 +309,38 @@ public static HoodieTimeline getCommitsTimelineAfter(
     return timelineSinceLastSync;
   }
 
+  public static HoodieTimeline getCommitsTimeLineAfterByCompletionTimeRange(
+      HoodieTableMetaClient metaClient,
+      String exclusiveStartCompletionTime,
+      String inclusiveEndCompletionTime) {
+    HoodieDefaultTimeline writeTimeline = metaClient.getActiveTimeline().getWriteTimeline();
+
+    HoodieDefaultTimeline timeline;
+    if (writeTimeline.isBeforeTimelineStartsByCompletionTime(exclusiveStartCompletionTime)) {
+      // need to load archived timeline as well
+      try (CompletionTimeQueryView view = new CompletionTimeQueryView(metaClient)) {
+        List<String> instants = view.getStartTimes(
+            exclusiveStartCompletionTime,
+            inclusiveEndCompletionTime,
+            RangeType.OPEN_CLOSED);
+        if (instants.isEmpty()) {
+          LOG.warn("Did not find any instant when getting timeline, using write timeline for incremental read");
+          timeline = writeTimeline;
+        } else {
+          timeline = metaClient.getArchivedTimeline(instants.get(0))
+              .mergeTimeline(writeTimeline)
+              .mergeTimeline(writeTimeline);

Review Comment:
   If this is still needed, only need to do one merge here.



##########
hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/hudi/HoodieMergeOnReadRDD.scala:
##########
@@ -82,7 +82,8 @@ class HoodieMergeOnReadRDD(@transient sc: SparkContext,
                            @transient fileSplits: Seq[HoodieMergeOnReadFileSplit],
                            includeStartTime: Boolean = false,
                            startTimestamp: String = null,
-                           endTimestamp: String = null)
+                           endTimestamp: String = null,
+                           includedTimestamps: Set[String] = null)

Review Comment:
   Are all these timestamps still representing instant times, not completion time?  The scaladocs should also be updated.



##########
hudi-common/src/test/java/org/apache/hudi/common/testutils/HoodieTestDataGenerator.java:
##########
@@ -681,6 +681,10 @@ public List<HoodieRecord> generateInserts(String instantTime, Integer n) {
     return generateInserts(instantTime, n, false);
   }
 
+  public List<HoodieRecord> generateInserts(Long instantTime, Integer n) {
+    return generateInserts(String.valueOf(instantTime), n, false);
+  }

Review Comment:
   Could caller just use String type instead of adding redundant test methods here?



##########
hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/hudi/DataSourceOptions.scala:
##########
@@ -125,6 +125,15 @@ object DataSourceReadOptions {
       + HollowCommitHandling.USE_TRANSITION_TIME + "`, will use instant's "
       + "`stateTransitionTime` to perform comparison.")
 
+  /* TODO: this is a replacement of HoodieIncrSourceConfig.NUM_INSTANTS_PER_FETCH
+      to make incremental instant limit available in spark incremental relations
+      Need to link HoodieIncrSourceConfig.NUM_INSTANTS_PER_FETCH to this correctly and add doc*/
+  val INCREMENTAL_LIMIT: ConfigProperty[String] = ConfigProperty
+    .key("hoodie.streamer.source.hoodieincr.num_instants")

Review Comment:
   Spark should not be aware of this config.  We can pass down the correct instant range without using the `num_instants` config.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] [HUDI-8141] Incremental Query with Completion Time [hudi]

Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.

hudi-bot commented on PR #11947:
URL: https://github.com/apache/hudi/pull/11947#issuecomment-2408807740

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "090f256a315cbbf0c6a54d93cc2739dd693360e6",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=484",
       "triggerID" : "090f256a315cbbf0c6a54d93cc2739dd693360e6",
       "triggerType" : "PUSH"
     }, {
       "hash" : "0df5649013c1cf9e782fcbb67b8a2e7174f94512",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=985",
       "triggerID" : "0df5649013c1cf9e782fcbb67b8a2e7174f94512",
       "triggerType" : "PUSH"
     }, {
       "hash" : "d52e0128f409918644255abbe4376dc4a780bfc8",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=986",
       "triggerID" : "d52e0128f409918644255abbe4376dc4a780bfc8",
       "triggerType" : "PUSH"
     }, {
       "hash" : "9e668f65c6d4f73c76573236f8fa507dfb80edac",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "9e668f65c6d4f73c76573236f8fa507dfb80edac",
       "triggerType" : "PUSH"
     }, {
       "hash" : "f145131d524bf190986e5de600a326e32a27beca",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=987",
       "triggerID" : "f145131d524bf190986e5de600a326e32a27beca",
       "triggerType" : "PUSH"
     }, {
       "hash" : "0b008f807b42aa6fb2686922bb88d63e84a1f299",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=993",
       "triggerID" : "0b008f807b42aa6fb2686922bb88d63e84a1f299",
       "triggerType" : "PUSH"
     }, {
       "hash" : "65c01ad9f2b33d36d4890355feef91844438e732",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=995",
       "triggerID" : "65c01ad9f2b33d36d4890355feef91844438e732",
       "triggerType" : "PUSH"
     }, {
       "hash" : "955493fdc93842e8c99529cec35e39cecdbe9638",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=996",
       "triggerID" : "955493fdc93842e8c99529cec35e39cecdbe9638",
       "triggerType" : "PUSH"
     }, {
       "hash" : "533f00e5546da0026bbe51e1c030ddd1c19fdd08",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "533f00e5546da0026bbe51e1c030ddd1c19fdd08",
       "triggerType" : "PUSH"
     }, {
       "hash" : "05777be67e37f3a6a8eaa59e54d9e17e5fc6ba9b",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1016",
       "triggerID" : "05777be67e37f3a6a8eaa59e54d9e17e5fc6ba9b",
       "triggerType" : "PUSH"
     }, {
       "hash" : "a2ee90c549332a7f17a7d9b2799849b77c08a8e6",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "a2ee90c549332a7f17a7d9b2799849b77c08a8e6",
       "triggerType" : "PUSH"
     }, {
       "hash" : "05ce568b0b6ba14da9ca40d26b39ba478837d747",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1017",
       "triggerID" : "05ce568b0b6ba14da9ca40d26b39ba478837d747",
       "triggerType" : "PUSH"
     }, {
       "hash" : "840e3b33d39b1f86367bb6b0e65eadeb33de7765",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1025",
       "triggerID" : "840e3b33d39b1f86367bb6b0e65eadeb33de7765",
       "triggerType" : "PUSH"
     }, {
       "hash" : "8d2e2dd83f51d03e890d424a51d3e0e84e904a43",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1031",
       "triggerID" : "8d2e2dd83f51d03e890d424a51d3e0e84e904a43",
       "triggerType" : "PUSH"
     }, {
       "hash" : "688381914087ff11acf2f8c9372dbf39afa0bd40",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1035",
       "triggerID" : "688381914087ff11acf2f8c9372dbf39afa0bd40",
       "triggerType" : "PUSH"
     }, {
       "hash" : "fc2c138901e82ed5f3fbf363c2a0ce8140ade050",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "fc2c138901e82ed5f3fbf363c2a0ce8140ade050",
       "triggerType" : "PUSH"
     }, {
       "hash" : "6ed9e1cb1e5a191752d6d39c7a9d5d41840f0cdf",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1039",
       "triggerID" : "6ed9e1cb1e5a191752d6d39c7a9d5d41840f0cdf",
       "triggerType" : "PUSH"
     }, {
       "hash" : "a510a7de1458bb996df5bbb341e8d010e4c91381",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "a510a7de1458bb996df5bbb341e8d010e4c91381",
       "triggerType" : "PUSH"
     }, {
       "hash" : "49677397cd117228742205944606f7a789090376",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1042",
       "triggerID" : "49677397cd117228742205944606f7a789090376",
       "triggerType" : "PUSH"
     }, {
       "hash" : "1630db0a37422aed942ff280560a9d6329df5015",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1044",
       "triggerID" : "1630db0a37422aed942ff280560a9d6329df5015",
       "triggerType" : "PUSH"
     }, {
       "hash" : "f11c76301d4ab524db0af57aa831be3a898ea0fb",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1048",
       "triggerID" : "f11c76301d4ab524db0af57aa831be3a898ea0fb",
       "triggerType" : "PUSH"
     }, {
       "hash" : "d1a0dc4c2017853a7ba0c3302f886043a63f29da",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1050",
       "triggerID" : "d1a0dc4c2017853a7ba0c3302f886043a63f29da",
       "triggerType" : "PUSH"
     }, {
       "hash" : "6a4c15ec39ded1f90b113b272554217698a98e1c",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1087",
       "triggerID" : "6a4c15ec39ded1f90b113b272554217698a98e1c",
       "triggerType" : "PUSH"
     }, {
       "hash" : "288e730d645a5e58e50a87ef5f44ba11a8fb0533",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1092",
       "triggerID" : "288e730d645a5e58e50a87ef5f44ba11a8fb0533",
       "triggerType" : "PUSH"
     }, {
       "hash" : "70a5c21321a0da3a0be8d85e206c24befa7b89f1",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1095",
       "triggerID" : "70a5c21321a0da3a0be8d85e206c24befa7b89f1",
       "triggerType" : "PUSH"
     }, {
       "hash" : "f072a986114949c5e426ce64a5f5568fc718e25d",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "f072a986114949c5e426ce64a5f5568fc718e25d",
       "triggerType" : "PUSH"
     }, {
       "hash" : "45b27475b1e8806ec8c566d6f151f4bc79c8549d",
       "status" : "CANCELED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1096",
       "triggerID" : "45b27475b1e8806ec8c566d6f151f4bc79c8549d",
       "triggerType" : "PUSH"
     }, {
       "hash" : "3c62ef22f6cde28110e5eb2fca8760a008d77723",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "3c62ef22f6cde28110e5eb2fca8760a008d77723",
       "triggerType" : "PUSH"
     }, {
       "hash" : "dc785fa04e542b2ccc45494e0292f5153c7ecea8",
       "status" : "PENDING",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1097",
       "triggerID" : "dc785fa04e542b2ccc45494e0292f5153c7ecea8",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 9e668f65c6d4f73c76573236f8fa507dfb80edac UNKNOWN
   * 533f00e5546da0026bbe51e1c030ddd1c19fdd08 UNKNOWN
   * a2ee90c549332a7f17a7d9b2799849b77c08a8e6 UNKNOWN
   * fc2c138901e82ed5f3fbf363c2a0ce8140ade050 UNKNOWN
   * a510a7de1458bb996df5bbb341e8d010e4c91381 UNKNOWN
   * f072a986114949c5e426ce64a5f5568fc718e25d UNKNOWN
   * 45b27475b1e8806ec8c566d6f151f4bc79c8549d Azure: [CANCELED](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1096) 
   * 3c62ef22f6cde28110e5eb2fca8760a008d77723 UNKNOWN
   * dc785fa04e542b2ccc45494e0292f5153c7ecea8 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1097) 
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] [HUDI-8141] Incremental Query with Completion Time [hudi]

Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.

hudi-bot commented on PR #11947:
URL: https://github.com/apache/hudi/pull/11947#issuecomment-2415396814

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "090f256a315cbbf0c6a54d93cc2739dd693360e6",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=484",
       "triggerID" : "090f256a315cbbf0c6a54d93cc2739dd693360e6",
       "triggerType" : "PUSH"
     }, {
       "hash" : "0df5649013c1cf9e782fcbb67b8a2e7174f94512",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=985",
       "triggerID" : "0df5649013c1cf9e782fcbb67b8a2e7174f94512",
       "triggerType" : "PUSH"
     }, {
       "hash" : "d52e0128f409918644255abbe4376dc4a780bfc8",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=986",
       "triggerID" : "d52e0128f409918644255abbe4376dc4a780bfc8",
       "triggerType" : "PUSH"
     }, {
       "hash" : "9e668f65c6d4f73c76573236f8fa507dfb80edac",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "9e668f65c6d4f73c76573236f8fa507dfb80edac",
       "triggerType" : "PUSH"
     }, {
       "hash" : "f145131d524bf190986e5de600a326e32a27beca",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=987",
       "triggerID" : "f145131d524bf190986e5de600a326e32a27beca",
       "triggerType" : "PUSH"
     }, {
       "hash" : "0b008f807b42aa6fb2686922bb88d63e84a1f299",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=993",
       "triggerID" : "0b008f807b42aa6fb2686922bb88d63e84a1f299",
       "triggerType" : "PUSH"
     }, {
       "hash" : "65c01ad9f2b33d36d4890355feef91844438e732",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=995",
       "triggerID" : "65c01ad9f2b33d36d4890355feef91844438e732",
       "triggerType" : "PUSH"
     }, {
       "hash" : "955493fdc93842e8c99529cec35e39cecdbe9638",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=996",
       "triggerID" : "955493fdc93842e8c99529cec35e39cecdbe9638",
       "triggerType" : "PUSH"
     }, {
       "hash" : "533f00e5546da0026bbe51e1c030ddd1c19fdd08",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "533f00e5546da0026bbe51e1c030ddd1c19fdd08",
       "triggerType" : "PUSH"
     }, {
       "hash" : "05777be67e37f3a6a8eaa59e54d9e17e5fc6ba9b",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1016",
       "triggerID" : "05777be67e37f3a6a8eaa59e54d9e17e5fc6ba9b",
       "triggerType" : "PUSH"
     }, {
       "hash" : "a2ee90c549332a7f17a7d9b2799849b77c08a8e6",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "a2ee90c549332a7f17a7d9b2799849b77c08a8e6",
       "triggerType" : "PUSH"
     }, {
       "hash" : "05ce568b0b6ba14da9ca40d26b39ba478837d747",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1017",
       "triggerID" : "05ce568b0b6ba14da9ca40d26b39ba478837d747",
       "triggerType" : "PUSH"
     }, {
       "hash" : "840e3b33d39b1f86367bb6b0e65eadeb33de7765",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1025",
       "triggerID" : "840e3b33d39b1f86367bb6b0e65eadeb33de7765",
       "triggerType" : "PUSH"
     }, {
       "hash" : "8d2e2dd83f51d03e890d424a51d3e0e84e904a43",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1031",
       "triggerID" : "8d2e2dd83f51d03e890d424a51d3e0e84e904a43",
       "triggerType" : "PUSH"
     }, {
       "hash" : "688381914087ff11acf2f8c9372dbf39afa0bd40",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1035",
       "triggerID" : "688381914087ff11acf2f8c9372dbf39afa0bd40",
       "triggerType" : "PUSH"
     }, {
       "hash" : "fc2c138901e82ed5f3fbf363c2a0ce8140ade050",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "fc2c138901e82ed5f3fbf363c2a0ce8140ade050",
       "triggerType" : "PUSH"
     }, {
       "hash" : "6ed9e1cb1e5a191752d6d39c7a9d5d41840f0cdf",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1039",
       "triggerID" : "6ed9e1cb1e5a191752d6d39c7a9d5d41840f0cdf",
       "triggerType" : "PUSH"
     }, {
       "hash" : "a510a7de1458bb996df5bbb341e8d010e4c91381",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "a510a7de1458bb996df5bbb341e8d010e4c91381",
       "triggerType" : "PUSH"
     }, {
       "hash" : "49677397cd117228742205944606f7a789090376",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1042",
       "triggerID" : "49677397cd117228742205944606f7a789090376",
       "triggerType" : "PUSH"
     }, {
       "hash" : "1630db0a37422aed942ff280560a9d6329df5015",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1044",
       "triggerID" : "1630db0a37422aed942ff280560a9d6329df5015",
       "triggerType" : "PUSH"
     }, {
       "hash" : "f11c76301d4ab524db0af57aa831be3a898ea0fb",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1048",
       "triggerID" : "f11c76301d4ab524db0af57aa831be3a898ea0fb",
       "triggerType" : "PUSH"
     }, {
       "hash" : "d1a0dc4c2017853a7ba0c3302f886043a63f29da",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1050",
       "triggerID" : "d1a0dc4c2017853a7ba0c3302f886043a63f29da",
       "triggerType" : "PUSH"
     }, {
       "hash" : "6a4c15ec39ded1f90b113b272554217698a98e1c",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1087",
       "triggerID" : "6a4c15ec39ded1f90b113b272554217698a98e1c",
       "triggerType" : "PUSH"
     }, {
       "hash" : "288e730d645a5e58e50a87ef5f44ba11a8fb0533",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1092",
       "triggerID" : "288e730d645a5e58e50a87ef5f44ba11a8fb0533",
       "triggerType" : "PUSH"
     }, {
       "hash" : "70a5c21321a0da3a0be8d85e206c24befa7b89f1",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1095",
       "triggerID" : "70a5c21321a0da3a0be8d85e206c24befa7b89f1",
       "triggerType" : "PUSH"
     }, {
       "hash" : "f072a986114949c5e426ce64a5f5568fc718e25d",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "f072a986114949c5e426ce64a5f5568fc718e25d",
       "triggerType" : "PUSH"
     }, {
       "hash" : "45b27475b1e8806ec8c566d6f151f4bc79c8549d",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1096",
       "triggerID" : "45b27475b1e8806ec8c566d6f151f4bc79c8549d",
       "triggerType" : "PUSH"
     }, {
       "hash" : "3c62ef22f6cde28110e5eb2fca8760a008d77723",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "3c62ef22f6cde28110e5eb2fca8760a008d77723",
       "triggerType" : "PUSH"
     }, {
       "hash" : "dc785fa04e542b2ccc45494e0292f5153c7ecea8",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1097",
       "triggerID" : "dc785fa04e542b2ccc45494e0292f5153c7ecea8",
       "triggerType" : "PUSH"
     }, {
       "hash" : "585ddb8049b9b9ccb057027e265b08a1b586df8c",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "585ddb8049b9b9ccb057027e265b08a1b586df8c",
       "triggerType" : "PUSH"
     }, {
       "hash" : "b62ff8675beed2c4487c788b9ca1a0ad4271b913",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "b62ff8675beed2c4487c788b9ca1a0ad4271b913",
       "triggerType" : "PUSH"
     }, {
       "hash" : "2d46048a7402907f348cc96da2839ba1b3e4932a",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1099",
       "triggerID" : "2d46048a7402907f348cc96da2839ba1b3e4932a",
       "triggerType" : "PUSH"
     }, {
       "hash" : "004fb6ebf23f796a4b77650e0530f0f1145e8b91",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1101",
       "triggerID" : "004fb6ebf23f796a4b77650e0530f0f1145e8b91",
       "triggerType" : "PUSH"
     }, {
       "hash" : "af0c33333b5e9ae6d59f20a6db1d7c016ea015d3",
       "status" : "CANCELED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1165",
       "triggerID" : "af0c33333b5e9ae6d59f20a6db1d7c016ea015d3",
       "triggerType" : "PUSH"
     }, {
       "hash" : "da8111118d71306500229619d3b7996fcd2526e8",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "da8111118d71306500229619d3b7996fcd2526e8",
       "triggerType" : "PUSH"
     }, {
       "hash" : "83257e2b9bd3c426cf90e11003a5f244a750d7fa",
       "status" : "PENDING",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1167",
       "triggerID" : "83257e2b9bd3c426cf90e11003a5f244a750d7fa",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 9e668f65c6d4f73c76573236f8fa507dfb80edac UNKNOWN
   * 533f00e5546da0026bbe51e1c030ddd1c19fdd08 UNKNOWN
   * a2ee90c549332a7f17a7d9b2799849b77c08a8e6 UNKNOWN
   * fc2c138901e82ed5f3fbf363c2a0ce8140ade050 UNKNOWN
   * a510a7de1458bb996df5bbb341e8d010e4c91381 UNKNOWN
   * f072a986114949c5e426ce64a5f5568fc718e25d UNKNOWN
   * 3c62ef22f6cde28110e5eb2fca8760a008d77723 UNKNOWN
   * 585ddb8049b9b9ccb057027e265b08a1b586df8c UNKNOWN
   * b62ff8675beed2c4487c788b9ca1a0ad4271b913 UNKNOWN
   * af0c33333b5e9ae6d59f20a6db1d7c016ea015d3 Azure: [CANCELED](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1165) 
   * da8111118d71306500229619d3b7996fcd2526e8 UNKNOWN
   * 83257e2b9bd3c426cf90e11003a5f244a750d7fa Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1167) 
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] [HUDI-8141] Incremental Query with Completion Time [hudi]

Posted by "danny0405 (via GitHub)" <gi...@apache.org>.

danny0405 commented on code in PR #11947:
URL: https://github.com/apache/hudi/pull/11947#discussion_r1802209333


##########
hudi-utilities/src/main/java/org/apache/hudi/utilities/sources/helpers/IncrSourceHelper.java:
##########
@@ -174,6 +170,64 @@ public static QueryInfo generateQueryInfo(JavaSparkContext jssc, String srcBaseP
     }
   }
 
+  public static IncrementalQueryAnalyzer getIncrementalQueryAnalyzer(
+      JavaSparkContext jssc,
+      String srcPath,
+      Option<String> lastCkptStr,
+      MissingCheckpointStrategy missingCheckpointStrategy,
+      int numInstantsFromConfig,
+      Option<SourceProfile<Integer>> latestSourceProfile) {
+    HoodieTableMetaClient metaClient = HoodieTableMetaClient.builder()
+        .setConf(HadoopFSUtils.getStorageConfWithCopy(jssc.hadoopConfiguration()))
+        .setBasePath(srcPath)
+        .setLoadActiveTimelineOnLoad(true)
+        .build();
+
+    String startTime;
+    if (lastCkptStr.isPresent() && !lastCkptStr.get().isEmpty()) {
+      startTime = lastCkptStr.get();
+    } else if (missingCheckpointStrategy != null) {
+      switch (missingCheckpointStrategy) {
+        case READ_UPTO_LATEST_COMMIT:
+          startTime = DEFAULT_BEGIN_TIMESTAMP;
+          // disrespect numInstantsFromConfig when reading up to latest
+          numInstantsFromConfig = -1;
+          break;
+        case READ_LATEST:
+          Option<HoodieInstant> lastInstant = metaClient
+              .getCommitsAndCompactionTimeline()
+              .filterCompletedInstants()
+              .lastInstant();
+          startTime = lastInstant
+              .map(hoodieInstant -> instantTimeMinusMillis(hoodieInstant.getCompletionTime(), 1))

Review Comment:
   Still I kind of think the exclusion of start instant does not make sense, for e.g for the very first commit to consume, it should be inclusive, it should only be exclusive when we are sure the read is a following up concinuous read after the preceeding reads.
   
   Also for batch inc read, I kind of think the start commit should be inclusive.
   
   So from code insights, let's make the inclusiveness/exclusiveness adajustable instead of hard code, and only make the start instant exclusive when we are sure the read is a continuous read.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] [HUDI-8141] Incremental Query with Completion Time [hudi]

Posted by "yihua (via GitHub)" <gi...@apache.org>.

yihua commented on code in PR #11947:
URL: https://github.com/apache/hudi/pull/11947#discussion_r1802138152


##########
hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/hudi/MergeOnReadIncrementalRelation.scala:
##########
@@ -201,55 +199,42 @@ trait HoodieIncrementalRelationTrait extends HoodieBaseRelation {
     val fallbackToFullTableScan = optParams.getOrElse(DataSourceReadOptions.INCREMENTAL_FALLBACK_TO_FULL_TABLE_SCAN.key,
       DataSourceReadOptions.INCREMENTAL_FALLBACK_TO_FULL_TABLE_SCAN.defaultValue).toBoolean
 
-    fallbackToFullTableScan && (startInstantArchived || endInstantArchived
+    fallbackToFullTableScan && (startInstantArchived
       || affectedFilesInCommits.asScala.exists(fileStatus => !metaClient.getStorage.exists(fileStatus.getPath)))
   }
 
-  protected lazy val includedCommits: immutable.Seq[HoodieInstant] = {
-    if (!startInstantArchived || !endInstantArchived) {
-      // If endTimestamp commit is not archived, will filter instants
-      // before endTimestamp.
-      if (hollowCommitHandling == USE_TRANSITION_TIME) {
-        super.timeline.findInstantsInRangeByCompletionTime(startTimestamp, endTimestamp).getInstants.asScala.toList
-      } else {
-        super.timeline.findInstantsInRange(startTimestamp, endTimestamp).getInstants.asScala.toList
-      }
-    } else {
-      super.timeline.getInstants.asScala.toList
-    }
-  }
+  protected lazy val queryContext: IncrementalQueryAnalyzer.QueryContext =
+    IncrementalQueryAnalyzer.builder()
+      .metaClient(metaClient)
+      .startTime(optParams(DataSourceReadOptions.BEGIN_INSTANTTIME.key))
+      .endTime(optParams.getOrElse(DataSourceReadOptions.END_INSTANTTIME.key, null))
+      .rangeType(InstantRange.RangeType.OPEN_CLOSED)

Review Comment:
   The same as https://github.com/apache/hudi/pull/11947#discussion_r1802132570.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] [HUDI-8141] Incremental Query with Completion Time [hudi]

Posted by "yihua (via GitHub)" <gi...@apache.org>.

yihua commented on code in PR #11947:
URL: https://github.com/apache/hudi/pull/11947#discussion_r1802136616


##########
hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/hudi/IncrementalRelation.scala:
##########
@@ -305,14 +286,17 @@ class IncrementalRelation(val sqlContext: SQLContext,
   }
 
   private def fullTableScanDataFrame(startInstantTime: String, endInstantTime: String): DataFrame = {
+    val commitTimesToReturn = commitsToReturn.map(_.getTimestamp)
     val hudiDF = sqlContext.read
       .format("hudi_v1")
       .schema(usedSchema)
       .load(basePath.toString)
-      .filter(String.format("%s > '%s'", HoodieRecord.COMMIT_TIME_METADATA_FIELD, //Notice the > in place of >= because we are working with optParam instead of first commit > optParam
+      // using closed_closed range here since we are using start instant from QueryContext
+      .filter(String.format("%s >= '%s'", HoodieRecord.COMMIT_TIME_METADATA_FIELD,
         startInstantTime))
       .filter(String.format("%s <= '%s'", HoodieRecord.COMMIT_TIME_METADATA_FIELD,

Review Comment:
   I think we can remove it as the `IN` predicate should also enable predicate pushdown and efficient filtering with column ranges in parquet.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] [HUDI-8141] Incremental Query with Completion Time [hudi]

Posted by "danny0405 (via GitHub)" <gi...@apache.org>.

danny0405 commented on code in PR #11947:
URL: https://github.com/apache/hudi/pull/11947#discussion_r1802196899


##########
hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/hudi/IncrementalRelation.scala:
##########
@@ -90,24 +78,24 @@ class IncrementalRelation(val sqlContext: SQLContext,
     throw new HoodieException("Incremental queries are not supported when meta fields are disabled")
   }
 
+  private val queryContext: IncrementalQueryAnalyzer.QueryContext =
+    IncrementalQueryAnalyzer.builder()
+      .metaClient(metaClient)
+      .startTime(optParams(DataSourceReadOptions.BEGIN_INSTANTTIME.key))
+      .endTime(optParams.getOrElse(DataSourceReadOptions.END_INSTANTTIME.key, null))
+      .rangeType(InstantRange.RangeType.OPEN_CLOSED)

Review Comment:
   > In Spark the hoodie.datasource.read.begin.instanttime is exclusive
   
   Not sure it is correct, at least in commen sense of incremental read, the start offset should be inclusive by default.
   
   And this is also not consistent with the semantics definition in `IncrementalQueryAnalyzer`.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] [HUDI-8141] Incremental Query with Completion Time [hudi]

Posted by "yihua (via GitHub)" <gi...@apache.org>.

yihua commented on code in PR #11947:
URL: https://github.com/apache/hudi/pull/11947#discussion_r1802429850


##########
hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/hudi/MergeOnReadIncrementalRelation.scala:
##########
@@ -186,12 +184,12 @@ trait HoodieIncrementalRelationTrait extends HoodieBaseRelation {
 
   protected def endTimestamp: String = optParams.getOrElse(
     DataSourceReadOptions.END_INSTANTTIME.key,
-    if (hollowCommitHandling == USE_TRANSITION_TIME) super.timeline.lastInstant().get.getCompletionTime
-    else super.timeline.lastInstant().get.getTimestamp)
-
-  protected def startInstantArchived: Boolean = super.timeline.isBeforeTimelineStarts(startTimestamp)
+    super.timeline.getInstantsAsStream.iterator().asScala.toList
+      .map(_.getCompletionTime)

Review Comment:
   Fixed.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] [HUDI-8141] Incremental Query with Completion Time [hudi]

Posted by "CTTY (via GitHub)" <gi...@apache.org>.

CTTY commented on code in PR #11947:
URL: https://github.com/apache/hudi/pull/11947#discussion_r1797947572


##########
hudi-spark-datasource/hudi-spark/src/main/java/org/apache/hudi/HoodieDataSourceHelpers.java:
##########
@@ -88,6 +97,16 @@ public static String latestCommit(HoodieStorage storage, String basePath) {
     return timeline.lastInstant().get().getTimestamp();
   }
 
+  public static String latestCommitCompletionTime(FileSystem fs, String basePath) {
+    HoodieTimeline timeline = allCompletedCommitsCompactions(fs, basePath);
+    return timeline.lastInstant().get().getCompletionTime();
+  }
+
+  public static String latestCommitCompletionTime(HoodieStorage storage, String basePath) {
+    HoodieTimeline timeline = allCompletedCommitsCompactions(storage, basePath);
+    return timeline.lastInstant().get().getCompletionTime();
+  }

Review Comment:
   I moved these to `org.apache.hudi.testutils.DataSourceTestUtils`



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] [HUDI-8141] Incremental Query with Completion Time [hudi]

Posted by "CTTY (via GitHub)" <gi...@apache.org>.

CTTY commented on code in PR #11947:
URL: https://github.com/apache/hudi/pull/11947#discussion_r1797941540


##########
hudi-spark-datasource/hudi-spark/src/test/scala/org/apache/hudi/functional/TestStreamingSource.scala:
##########
@@ -46,7 +46,7 @@ class TestStreamingSource extends StreamTest {
   )
   private val columns = Seq("id", "name", "price", "ts")
 
-  val handlingMode: HollowCommitHandling = BLOCK
+  val handlingMode: HollowCommitHandling = USE_TRANSITION_TIME

Review Comment:
   This should be removed



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] [HUDI-8141] Incremental Query with Completion Time [hudi]

Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.

hudi-bot commented on PR #11947:
URL: https://github.com/apache/hudi/pull/11947#issuecomment-2408833671

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "090f256a315cbbf0c6a54d93cc2739dd693360e6",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=484",
       "triggerID" : "090f256a315cbbf0c6a54d93cc2739dd693360e6",
       "triggerType" : "PUSH"
     }, {
       "hash" : "0df5649013c1cf9e782fcbb67b8a2e7174f94512",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=985",
       "triggerID" : "0df5649013c1cf9e782fcbb67b8a2e7174f94512",
       "triggerType" : "PUSH"
     }, {
       "hash" : "d52e0128f409918644255abbe4376dc4a780bfc8",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=986",
       "triggerID" : "d52e0128f409918644255abbe4376dc4a780bfc8",
       "triggerType" : "PUSH"
     }, {
       "hash" : "9e668f65c6d4f73c76573236f8fa507dfb80edac",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "9e668f65c6d4f73c76573236f8fa507dfb80edac",
       "triggerType" : "PUSH"
     }, {
       "hash" : "f145131d524bf190986e5de600a326e32a27beca",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=987",
       "triggerID" : "f145131d524bf190986e5de600a326e32a27beca",
       "triggerType" : "PUSH"
     }, {
       "hash" : "0b008f807b42aa6fb2686922bb88d63e84a1f299",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=993",
       "triggerID" : "0b008f807b42aa6fb2686922bb88d63e84a1f299",
       "triggerType" : "PUSH"
     }, {
       "hash" : "65c01ad9f2b33d36d4890355feef91844438e732",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=995",
       "triggerID" : "65c01ad9f2b33d36d4890355feef91844438e732",
       "triggerType" : "PUSH"
     }, {
       "hash" : "955493fdc93842e8c99529cec35e39cecdbe9638",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=996",
       "triggerID" : "955493fdc93842e8c99529cec35e39cecdbe9638",
       "triggerType" : "PUSH"
     }, {
       "hash" : "533f00e5546da0026bbe51e1c030ddd1c19fdd08",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "533f00e5546da0026bbe51e1c030ddd1c19fdd08",
       "triggerType" : "PUSH"
     }, {
       "hash" : "05777be67e37f3a6a8eaa59e54d9e17e5fc6ba9b",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1016",
       "triggerID" : "05777be67e37f3a6a8eaa59e54d9e17e5fc6ba9b",
       "triggerType" : "PUSH"
     }, {
       "hash" : "a2ee90c549332a7f17a7d9b2799849b77c08a8e6",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "a2ee90c549332a7f17a7d9b2799849b77c08a8e6",
       "triggerType" : "PUSH"
     }, {
       "hash" : "05ce568b0b6ba14da9ca40d26b39ba478837d747",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1017",
       "triggerID" : "05ce568b0b6ba14da9ca40d26b39ba478837d747",
       "triggerType" : "PUSH"
     }, {
       "hash" : "840e3b33d39b1f86367bb6b0e65eadeb33de7765",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1025",
       "triggerID" : "840e3b33d39b1f86367bb6b0e65eadeb33de7765",
       "triggerType" : "PUSH"
     }, {
       "hash" : "8d2e2dd83f51d03e890d424a51d3e0e84e904a43",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1031",
       "triggerID" : "8d2e2dd83f51d03e890d424a51d3e0e84e904a43",
       "triggerType" : "PUSH"
     }, {
       "hash" : "688381914087ff11acf2f8c9372dbf39afa0bd40",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1035",
       "triggerID" : "688381914087ff11acf2f8c9372dbf39afa0bd40",
       "triggerType" : "PUSH"
     }, {
       "hash" : "fc2c138901e82ed5f3fbf363c2a0ce8140ade050",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "fc2c138901e82ed5f3fbf363c2a0ce8140ade050",
       "triggerType" : "PUSH"
     }, {
       "hash" : "6ed9e1cb1e5a191752d6d39c7a9d5d41840f0cdf",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1039",
       "triggerID" : "6ed9e1cb1e5a191752d6d39c7a9d5d41840f0cdf",
       "triggerType" : "PUSH"
     }, {
       "hash" : "a510a7de1458bb996df5bbb341e8d010e4c91381",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "a510a7de1458bb996df5bbb341e8d010e4c91381",
       "triggerType" : "PUSH"
     }, {
       "hash" : "49677397cd117228742205944606f7a789090376",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1042",
       "triggerID" : "49677397cd117228742205944606f7a789090376",
       "triggerType" : "PUSH"
     }, {
       "hash" : "1630db0a37422aed942ff280560a9d6329df5015",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1044",
       "triggerID" : "1630db0a37422aed942ff280560a9d6329df5015",
       "triggerType" : "PUSH"
     }, {
       "hash" : "f11c76301d4ab524db0af57aa831be3a898ea0fb",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1048",
       "triggerID" : "f11c76301d4ab524db0af57aa831be3a898ea0fb",
       "triggerType" : "PUSH"
     }, {
       "hash" : "d1a0dc4c2017853a7ba0c3302f886043a63f29da",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1050",
       "triggerID" : "d1a0dc4c2017853a7ba0c3302f886043a63f29da",
       "triggerType" : "PUSH"
     }, {
       "hash" : "6a4c15ec39ded1f90b113b272554217698a98e1c",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1087",
       "triggerID" : "6a4c15ec39ded1f90b113b272554217698a98e1c",
       "triggerType" : "PUSH"
     }, {
       "hash" : "288e730d645a5e58e50a87ef5f44ba11a8fb0533",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1092",
       "triggerID" : "288e730d645a5e58e50a87ef5f44ba11a8fb0533",
       "triggerType" : "PUSH"
     }, {
       "hash" : "70a5c21321a0da3a0be8d85e206c24befa7b89f1",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1095",
       "triggerID" : "70a5c21321a0da3a0be8d85e206c24befa7b89f1",
       "triggerType" : "PUSH"
     }, {
       "hash" : "f072a986114949c5e426ce64a5f5568fc718e25d",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "f072a986114949c5e426ce64a5f5568fc718e25d",
       "triggerType" : "PUSH"
     }, {
       "hash" : "45b27475b1e8806ec8c566d6f151f4bc79c8549d",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1096",
       "triggerID" : "45b27475b1e8806ec8c566d6f151f4bc79c8549d",
       "triggerType" : "PUSH"
     }, {
       "hash" : "3c62ef22f6cde28110e5eb2fca8760a008d77723",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "3c62ef22f6cde28110e5eb2fca8760a008d77723",
       "triggerType" : "PUSH"
     }, {
       "hash" : "dc785fa04e542b2ccc45494e0292f5153c7ecea8",
       "status" : "FAILURE",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1097",
       "triggerID" : "dc785fa04e542b2ccc45494e0292f5153c7ecea8",
       "triggerType" : "PUSH"
     }, {
       "hash" : "585ddb8049b9b9ccb057027e265b08a1b586df8c",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "585ddb8049b9b9ccb057027e265b08a1b586df8c",
       "triggerType" : "PUSH"
     }, {
       "hash" : "b62ff8675beed2c4487c788b9ca1a0ad4271b913",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "b62ff8675beed2c4487c788b9ca1a0ad4271b913",
       "triggerType" : "PUSH"
     }, {
       "hash" : "2d46048a7402907f348cc96da2839ba1b3e4932a",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "2d46048a7402907f348cc96da2839ba1b3e4932a",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 9e668f65c6d4f73c76573236f8fa507dfb80edac UNKNOWN
   * 533f00e5546da0026bbe51e1c030ddd1c19fdd08 UNKNOWN
   * a2ee90c549332a7f17a7d9b2799849b77c08a8e6 UNKNOWN
   * fc2c138901e82ed5f3fbf363c2a0ce8140ade050 UNKNOWN
   * a510a7de1458bb996df5bbb341e8d010e4c91381 UNKNOWN
   * f072a986114949c5e426ce64a5f5568fc718e25d UNKNOWN
   * 3c62ef22f6cde28110e5eb2fca8760a008d77723 UNKNOWN
   * dc785fa04e542b2ccc45494e0292f5153c7ecea8 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1097) 
   * 585ddb8049b9b9ccb057027e265b08a1b586df8c UNKNOWN
   * b62ff8675beed2c4487c788b9ca1a0ad4271b913 UNKNOWN
   * 2d46048a7402907f348cc96da2839ba1b3e4932a UNKNOWN
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] [HUDI-8141] Incremental Query with Completion Time [hudi]

Posted by "danny0405 (via GitHub)" <gi...@apache.org>.

danny0405 commented on code in PR #11947:
URL: https://github.com/apache/hudi/pull/11947#discussion_r1802204446


##########
hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/spark/sql/hudi/streaming/HoodieStreamSource.scala:
##########
@@ -71,19 +69,6 @@ class HoodieStreamSource(
     parameters.get(DataSourceReadOptions.QUERY_TYPE.key).contains(DataSourceReadOptions.QUERY_TYPE_INCREMENTAL_OPT_VAL) &&
     parameters.get(DataSourceReadOptions.INCREMENTAL_FORMAT.key).contains(DataSourceReadOptions.INCREMENTAL_FORMAT_CDC_VAL)
 
-  /**
-   * When hollow commits are found while doing streaming read , unlike batch incremental query,
-   * we do not use [[HollowCommitHandling.FAIL]] by default, instead we use [[HollowCommitHandling.BLOCK]]
-   * to block processing data from going beyond the hollow commits to avoid unintentional skip.
-   *
-   * Users can set [[DataSourceReadOptions.INCREMENTAL_READ_HANDLE_HOLLOW_COMMIT]] to
-   * [[HollowCommitHandling.USE_TRANSITION_TIME]] to avoid the blocking behavior.
-   */
-  private val hollowCommitHandling: HollowCommitHandling =

Review Comment:
   > We still need to handle hollow commits if the source table is version 6 or below
   
   Are you saying we fallback to to instant time incremental queries? Maybe just telling user the semantics is migrated for 1.0 release reader.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] [HUDI-8141] Incremental Query with Completion Time [hudi]

Posted by "danny0405 (via GitHub)" <gi...@apache.org>.

danny0405 commented on code in PR #11947:
URL: https://github.com/apache/hudi/pull/11947#discussion_r1798653394


##########
hudi-common/src/main/java/org/apache/hudi/common/table/timeline/CompletionTimeQueryView.java:
##########
@@ -189,46 +193,78 @@ public Option<String> getCompletionTime(String beginTime) {
    * <p>By default, assumes there is at most 1 day time of duration for an instant to accelerate the queries.
    *
    * @param timeline The timeline.
-   * @param rangeStart   The query range start completion time.
-   * @param rangeEnd     The query range end completion time.
+   * @param rangeStartCompletionTime   The query range start completion time.
+   * @param rangeEndCompletionTime     The query range end completion time.
    * @param rangeType    The range type.
    *
    * @return The sorted instant time list.
    */
   public List<String> getStartTimes(
       HoodieTimeline timeline,
-      Option<String> rangeStart,
-      Option<String> rangeEnd,
+      Option<String> rangeStartCompletionTime,
+      Option<String> rangeEndCompletionTime,
       InstantRange.RangeType rangeType) {
     // assumes any instant/transaction lasts at most 1 day to optimize the query efficiency.
-    return getStartTimes(timeline, rangeStart, rangeEnd, rangeType, s -> HoodieInstantTimeGenerator.instantTimeMinusMillis(s, MILLI_SECONDS_IN_ONE_DAY));
+    return getStartTimes(
+        timeline,
+        rangeStartCompletionTime,
+        rangeEndCompletionTime,
+        rangeType,
+        GET_INSTANT_ONE_DAY_BEFORE);
   }
 
   /**
    * Queries the instant start time with given completion time range.
    *
-   * @param rangeStart              The query range start completion time.
-   * @param rangeEnd                The query range end completion time.
+   * @param rangeStartCompletionTime              The query range start completion time.
+   * @param rangeEndCompletionTime                The query range end completion time.
    * @param earliestInstantTimeFunc The function to generate the earliest start time boundary
    *                                with the minimum completion time.
    *
    * @return The sorted instant time list.
    */
   @VisibleForTesting
   public List<String> getStartTimes(
-      String rangeStart,
-      String rangeEnd,
+      String rangeStartCompletionTime,
+      String rangeEndCompletionTime,
       Function<String, String> earliestInstantTimeFunc) {
-    return getStartTimes(metaClient.getCommitsTimeline().filterCompletedInstants(), Option.ofNullable(rangeStart), Option.ofNullable(rangeEnd),
-        InstantRange.RangeType.CLOSED_CLOSED, earliestInstantTimeFunc);
+    return getStartTimes(
+        metaClient.getCommitsTimeline().filterCompletedInstants(),
+        Option.ofNullable(rangeStartCompletionTime),
+        Option.ofNullable(rangeEndCompletionTime),
+        InstantRange.RangeType.CLOSED_CLOSED,
+        earliestInstantTimeFunc);
+  }
+
+  /**
+   * Queries the instant start time with given completion time range.
+   *
+   * @param rangeStartCompletionTime              The query range start completion time.
+   * @param rangeEndCompletionTime                The query range end completion time.
+   * @param rangeType               The range type.
+   *                                with the minimum completion time.
+   *
+   * @return The sorted instant time list.
+   */
+  @VisibleForTesting
+  public List<String> getStartTimes(
+      String rangeStartCompletionTime,
+      String rangeEndCompletionTime,
+      RangeType rangeType) {
+    return getStartTimes(
+        metaClient.getCommitsTimeline().filterCompletedInstants(),
+        Option.ofNullable(rangeStartCompletionTime),
+        Option.ofNullable(rangeEndCompletionTime),
+        rangeType,
+        GET_INSTANT_ONE_DAY_BEFORE);
   }
 
   /**
    * Queries the instant start time with given completion time range.
    *
    * @param timeline            The timeline.
-   * @param rangeStart              The query range start completion time.
-   * @param rangeEnd                The query range end completion time.
+   * @param rangeStartCompletionTime              The query range start completion time.
+   * @param rangeEndCompletionTime                The query range end completion time.

Review Comment:
   `beginCompletionTime` and `endCompletionTime`, can you also fix the doc indentation to be left justifying.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] [HUDI-8141] Incremental Query with Completion Time [hudi]

Posted by "danny0405 (via GitHub)" <gi...@apache.org>.

danny0405 commented on code in PR #11947:
URL: https://github.com/apache/hudi/pull/11947#discussion_r1798658319


##########
hudi-utilities/src/main/java/org/apache/hudi/utilities/sources/HoodieIncrSource.java:
##########
@@ -212,35 +203,43 @@ public Pair<Option<Dataset<Row>>, String> fetchNextBatch(Option<String> lastCkpt
           .collect(Collectors.toMap(Pair::getLeft, Pair::getRight));
       reader = reader.options(optionsMap);
     }
+    boolean shouldFullScan =
+        queryContext.getActiveTimeline().isBeforeTimelineStartsByCompletionTime(completionTimeToResumeFrom)
+        && missingCheckpointStrategy == MissingCheckpointStrategy.READ_UPTO_LATEST_COMMIT;
     Dataset<Row> source;
-    // Do Incr pull. Set end instant if available
-    if (queryInfo.isIncremental()) {
-      source = reader
-          .options(readOpts)
-          .option(QUERY_TYPE().key(), QUERY_TYPE_INCREMENTAL_OPT_VAL())
-          .option(BEGIN_INSTANTTIME().key(), queryInfo.getStartInstant())
-          .option(END_INSTANTTIME().key(), queryInfo.getEndInstant())
-          .option(INCREMENTAL_FALLBACK_TO_FULL_TABLE_SCAN().key(),
-              props.getString(INCREMENTAL_FALLBACK_TO_FULL_TABLE_SCAN().key(),
-                  INCREMENTAL_FALLBACK_TO_FULL_TABLE_SCAN().defaultValue()))
-          .option(INCREMENTAL_READ_HANDLE_HOLLOW_COMMIT().key(), handlingMode.name())
-          .load(srcPath);
-    } else {
-      // if checkpoint is missing from source table, and if strategy is set to READ_UPTO_LATEST_COMMIT, we have to issue snapshot query
+    if (instantRange.isEmpty() || shouldFullScan) {
+      // snapshot query
       Dataset<Row> snapshot = reader
           .options(readOpts)
           .option(DataSourceReadOptions.QUERY_TYPE().key(), DataSourceReadOptions.QUERY_TYPE_SNAPSHOT_OPT_VAL())
           .load(srcPath);
       if (snapshotLoadQuerySplitter.isPresent()) {
-        queryInfo = snapshotLoadQuerySplitter.get().getNextCheckpoint(snapshot, queryInfo, sourceProfileSupplier);
+        queryContext = snapshotLoadQuerySplitter.get().getNextCheckpoint(snapshot, queryContext, sourceProfileSupplier);
+        // update endTime/next checkpoint
+        completionTimeToStopAt = queryContext.getEndInstant().orElse(queryContext.getLastInstant());
       }
+
+      snapshot = queryContext.getPredicateFilter().map(snapshot::filter).orElse(snapshot);
       source = snapshot
           // add filtering so that only interested records are returned.
-          .filter(String.format("%s > '%s'", HoodieRecord.COMMIT_TIME_METADATA_FIELD,
-              queryInfo.getStartInstant()))
+          // completion time comparison uses ( , ], but when comparing start time we need to use [, ]

Review Comment:
   Do we still need the `<=` and `>=` predicates given the predicate `in`?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] [HUDI-8141] Incremental Query with Completion Time [hudi]

Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.

hudi-bot commented on PR #11947:
URL: https://github.com/apache/hudi/pull/11947#issuecomment-2415337915

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "090f256a315cbbf0c6a54d93cc2739dd693360e6",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=484",
       "triggerID" : "090f256a315cbbf0c6a54d93cc2739dd693360e6",
       "triggerType" : "PUSH"
     }, {
       "hash" : "0df5649013c1cf9e782fcbb67b8a2e7174f94512",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=985",
       "triggerID" : "0df5649013c1cf9e782fcbb67b8a2e7174f94512",
       "triggerType" : "PUSH"
     }, {
       "hash" : "d52e0128f409918644255abbe4376dc4a780bfc8",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=986",
       "triggerID" : "d52e0128f409918644255abbe4376dc4a780bfc8",
       "triggerType" : "PUSH"
     }, {
       "hash" : "9e668f65c6d4f73c76573236f8fa507dfb80edac",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "9e668f65c6d4f73c76573236f8fa507dfb80edac",
       "triggerType" : "PUSH"
     }, {
       "hash" : "f145131d524bf190986e5de600a326e32a27beca",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=987",
       "triggerID" : "f145131d524bf190986e5de600a326e32a27beca",
       "triggerType" : "PUSH"
     }, {
       "hash" : "0b008f807b42aa6fb2686922bb88d63e84a1f299",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=993",
       "triggerID" : "0b008f807b42aa6fb2686922bb88d63e84a1f299",
       "triggerType" : "PUSH"
     }, {
       "hash" : "65c01ad9f2b33d36d4890355feef91844438e732",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=995",
       "triggerID" : "65c01ad9f2b33d36d4890355feef91844438e732",
       "triggerType" : "PUSH"
     }, {
       "hash" : "955493fdc93842e8c99529cec35e39cecdbe9638",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=996",
       "triggerID" : "955493fdc93842e8c99529cec35e39cecdbe9638",
       "triggerType" : "PUSH"
     }, {
       "hash" : "533f00e5546da0026bbe51e1c030ddd1c19fdd08",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "533f00e5546da0026bbe51e1c030ddd1c19fdd08",
       "triggerType" : "PUSH"
     }, {
       "hash" : "05777be67e37f3a6a8eaa59e54d9e17e5fc6ba9b",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1016",
       "triggerID" : "05777be67e37f3a6a8eaa59e54d9e17e5fc6ba9b",
       "triggerType" : "PUSH"
     }, {
       "hash" : "a2ee90c549332a7f17a7d9b2799849b77c08a8e6",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "a2ee90c549332a7f17a7d9b2799849b77c08a8e6",
       "triggerType" : "PUSH"
     }, {
       "hash" : "05ce568b0b6ba14da9ca40d26b39ba478837d747",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1017",
       "triggerID" : "05ce568b0b6ba14da9ca40d26b39ba478837d747",
       "triggerType" : "PUSH"
     }, {
       "hash" : "840e3b33d39b1f86367bb6b0e65eadeb33de7765",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1025",
       "triggerID" : "840e3b33d39b1f86367bb6b0e65eadeb33de7765",
       "triggerType" : "PUSH"
     }, {
       "hash" : "8d2e2dd83f51d03e890d424a51d3e0e84e904a43",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1031",
       "triggerID" : "8d2e2dd83f51d03e890d424a51d3e0e84e904a43",
       "triggerType" : "PUSH"
     }, {
       "hash" : "688381914087ff11acf2f8c9372dbf39afa0bd40",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1035",
       "triggerID" : "688381914087ff11acf2f8c9372dbf39afa0bd40",
       "triggerType" : "PUSH"
     }, {
       "hash" : "fc2c138901e82ed5f3fbf363c2a0ce8140ade050",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "fc2c138901e82ed5f3fbf363c2a0ce8140ade050",
       "triggerType" : "PUSH"
     }, {
       "hash" : "6ed9e1cb1e5a191752d6d39c7a9d5d41840f0cdf",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1039",
       "triggerID" : "6ed9e1cb1e5a191752d6d39c7a9d5d41840f0cdf",
       "triggerType" : "PUSH"
     }, {
       "hash" : "a510a7de1458bb996df5bbb341e8d010e4c91381",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "a510a7de1458bb996df5bbb341e8d010e4c91381",
       "triggerType" : "PUSH"
     }, {
       "hash" : "49677397cd117228742205944606f7a789090376",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1042",
       "triggerID" : "49677397cd117228742205944606f7a789090376",
       "triggerType" : "PUSH"
     }, {
       "hash" : "1630db0a37422aed942ff280560a9d6329df5015",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1044",
       "triggerID" : "1630db0a37422aed942ff280560a9d6329df5015",
       "triggerType" : "PUSH"
     }, {
       "hash" : "f11c76301d4ab524db0af57aa831be3a898ea0fb",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1048",
       "triggerID" : "f11c76301d4ab524db0af57aa831be3a898ea0fb",
       "triggerType" : "PUSH"
     }, {
       "hash" : "d1a0dc4c2017853a7ba0c3302f886043a63f29da",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1050",
       "triggerID" : "d1a0dc4c2017853a7ba0c3302f886043a63f29da",
       "triggerType" : "PUSH"
     }, {
       "hash" : "6a4c15ec39ded1f90b113b272554217698a98e1c",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1087",
       "triggerID" : "6a4c15ec39ded1f90b113b272554217698a98e1c",
       "triggerType" : "PUSH"
     }, {
       "hash" : "288e730d645a5e58e50a87ef5f44ba11a8fb0533",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1092",
       "triggerID" : "288e730d645a5e58e50a87ef5f44ba11a8fb0533",
       "triggerType" : "PUSH"
     }, {
       "hash" : "70a5c21321a0da3a0be8d85e206c24befa7b89f1",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1095",
       "triggerID" : "70a5c21321a0da3a0be8d85e206c24befa7b89f1",
       "triggerType" : "PUSH"
     }, {
       "hash" : "f072a986114949c5e426ce64a5f5568fc718e25d",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "f072a986114949c5e426ce64a5f5568fc718e25d",
       "triggerType" : "PUSH"
     }, {
       "hash" : "45b27475b1e8806ec8c566d6f151f4bc79c8549d",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1096",
       "triggerID" : "45b27475b1e8806ec8c566d6f151f4bc79c8549d",
       "triggerType" : "PUSH"
     }, {
       "hash" : "3c62ef22f6cde28110e5eb2fca8760a008d77723",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "3c62ef22f6cde28110e5eb2fca8760a008d77723",
       "triggerType" : "PUSH"
     }, {
       "hash" : "dc785fa04e542b2ccc45494e0292f5153c7ecea8",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1097",
       "triggerID" : "dc785fa04e542b2ccc45494e0292f5153c7ecea8",
       "triggerType" : "PUSH"
     }, {
       "hash" : "585ddb8049b9b9ccb057027e265b08a1b586df8c",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "585ddb8049b9b9ccb057027e265b08a1b586df8c",
       "triggerType" : "PUSH"
     }, {
       "hash" : "b62ff8675beed2c4487c788b9ca1a0ad4271b913",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "b62ff8675beed2c4487c788b9ca1a0ad4271b913",
       "triggerType" : "PUSH"
     }, {
       "hash" : "2d46048a7402907f348cc96da2839ba1b3e4932a",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1099",
       "triggerID" : "2d46048a7402907f348cc96da2839ba1b3e4932a",
       "triggerType" : "PUSH"
     }, {
       "hash" : "004fb6ebf23f796a4b77650e0530f0f1145e8b91",
       "status" : "SUCCESS",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1101",
       "triggerID" : "004fb6ebf23f796a4b77650e0530f0f1145e8b91",
       "triggerType" : "PUSH"
     }, {
       "hash" : "af0c33333b5e9ae6d59f20a6db1d7c016ea015d3",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "af0c33333b5e9ae6d59f20a6db1d7c016ea015d3",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 9e668f65c6d4f73c76573236f8fa507dfb80edac UNKNOWN
   * 533f00e5546da0026bbe51e1c030ddd1c19fdd08 UNKNOWN
   * a2ee90c549332a7f17a7d9b2799849b77c08a8e6 UNKNOWN
   * fc2c138901e82ed5f3fbf363c2a0ce8140ade050 UNKNOWN
   * a510a7de1458bb996df5bbb341e8d010e4c91381 UNKNOWN
   * f072a986114949c5e426ce64a5f5568fc718e25d UNKNOWN
   * 3c62ef22f6cde28110e5eb2fca8760a008d77723 UNKNOWN
   * 585ddb8049b9b9ccb057027e265b08a1b586df8c UNKNOWN
   * b62ff8675beed2c4487c788b9ca1a0ad4271b913 UNKNOWN
   * 004fb6ebf23f796a4b77650e0530f0f1145e8b91 Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1101) 
   * af0c33333b5e9ae6d59f20a6db1d7c016ea015d3 UNKNOWN
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] [HUDI-8141] Incremental Query with Completion Time [hudi]

Posted by "CTTY (via GitHub)" <gi...@apache.org>.

CTTY commented on code in PR #11947:
URL: https://github.com/apache/hudi/pull/11947#discussion_r1797786645


##########
hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/hudi/IncrementalRelation.scala:
##########
@@ -90,24 +78,25 @@ class IncrementalRelation(val sqlContext: SQLContext,
     throw new HoodieException("Incremental queries are not supported when meta fields are disabled")
   }
 
+  private val queryContext: IncrementalQueryAnalyzer.QueryContext =
+    IncrementalQueryAnalyzer.builder()
+      .metaClient(metaClient)
+      .startTime(optParams(DataSourceReadOptions.BEGIN_INSTANTTIME.key))
+      .endTime(optParams.getOrElse(DataSourceReadOptions.END_INSTANTTIME.key, null))

Review Comment:
   Yes, I think we will need to rename these configs. I'll file a jira to track this



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] [HUDI-8141] Incremental Query with Completion Time [hudi]

Posted by "yihua (via GitHub)" <gi...@apache.org>.

yihua commented on code in PR #11947:
URL: https://github.com/apache/hudi/pull/11947#discussion_r1805754603


##########
hudi-utilities/src/main/java/org/apache/hudi/utilities/sources/SnapshotLoadQuerySplitter.java:
##########
@@ -118,6 +124,32 @@ public QueryInfo getNextCheckpoint(Dataset<Row> df, QueryInfo queryInfo, Option<
         .orElse(queryInfo);
   }
 
+  public Option<CheckpointWithPredicates> getNextCheckpoint(Dataset<Row> df, QueryContext queryContext,
+                                                            Option<SourceProfileSupplier> sourceProfileSupplier) {
+    // the start instant would be included into the final query result. So we need to get
+    // a strictly lower timestamp to have query splitter include the start instant
+    Option<CheckpointWithPredicates> nextCheckpointWithPredicates =
+        getNextCheckpointWithPredicates(df, instantTimeMinusMillis(queryContext.getBeginInstant().get(), 1));
+    if (nextCheckpointWithPredicates.isPresent()) {
+      // getNextCheckpointWithPredicates is based on instant times,
+      // so we need to translate the instant time to the completion time
+      String endInstantTime = nextCheckpointWithPredicates.get().getEndInstant();
+      Option<String> endCompletionTime = Option.fromJavaOptional(queryContext.getInstants().stream()
+          .filter(instant -> endInstantTime.equals(instant.getTimestamp()))

Review Comment:
   The end completion time or next checkpoint returned by the custom `SnapshotLoadQuerySplitter` implementation can be earlier than the max completion time from query context.  For example, the query context returns these instants with instant and completion time, `[(t10, t20), (t30, t40), (t50, t60)]` with the max completion time of `t60`, the custom `SnapshotLoadQuerySplitter` implementation may take and process this as input and only return `t30` as the next instant to stop (thus `t40` as the `endCompletionTime`) after custom logic of further limiting the instants to read.  So we need to account for such a logic.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] [HUDI-8141] Incremental Query with Completion Time [hudi]

Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.

hudi-bot commented on PR #11947:
URL: https://github.com/apache/hudi/pull/11947#issuecomment-2418486498

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "090f256a315cbbf0c6a54d93cc2739dd693360e6",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=484",
       "triggerID" : "090f256a315cbbf0c6a54d93cc2739dd693360e6",
       "triggerType" : "PUSH"
     }, {
       "hash" : "0df5649013c1cf9e782fcbb67b8a2e7174f94512",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=985",
       "triggerID" : "0df5649013c1cf9e782fcbb67b8a2e7174f94512",
       "triggerType" : "PUSH"
     }, {
       "hash" : "d52e0128f409918644255abbe4376dc4a780bfc8",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=986",
       "triggerID" : "d52e0128f409918644255abbe4376dc4a780bfc8",
       "triggerType" : "PUSH"
     }, {
       "hash" : "9e668f65c6d4f73c76573236f8fa507dfb80edac",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "9e668f65c6d4f73c76573236f8fa507dfb80edac",
       "triggerType" : "PUSH"
     }, {
       "hash" : "f145131d524bf190986e5de600a326e32a27beca",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=987",
       "triggerID" : "f145131d524bf190986e5de600a326e32a27beca",
       "triggerType" : "PUSH"
     }, {
       "hash" : "0b008f807b42aa6fb2686922bb88d63e84a1f299",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=993",
       "triggerID" : "0b008f807b42aa6fb2686922bb88d63e84a1f299",
       "triggerType" : "PUSH"
     }, {
       "hash" : "65c01ad9f2b33d36d4890355feef91844438e732",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=995",
       "triggerID" : "65c01ad9f2b33d36d4890355feef91844438e732",
       "triggerType" : "PUSH"
     }, {
       "hash" : "955493fdc93842e8c99529cec35e39cecdbe9638",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=996",
       "triggerID" : "955493fdc93842e8c99529cec35e39cecdbe9638",
       "triggerType" : "PUSH"
     }, {
       "hash" : "533f00e5546da0026bbe51e1c030ddd1c19fdd08",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "533f00e5546da0026bbe51e1c030ddd1c19fdd08",
       "triggerType" : "PUSH"
     }, {
       "hash" : "05777be67e37f3a6a8eaa59e54d9e17e5fc6ba9b",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1016",
       "triggerID" : "05777be67e37f3a6a8eaa59e54d9e17e5fc6ba9b",
       "triggerType" : "PUSH"
     }, {
       "hash" : "a2ee90c549332a7f17a7d9b2799849b77c08a8e6",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "a2ee90c549332a7f17a7d9b2799849b77c08a8e6",
       "triggerType" : "PUSH"
     }, {
       "hash" : "05ce568b0b6ba14da9ca40d26b39ba478837d747",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1017",
       "triggerID" : "05ce568b0b6ba14da9ca40d26b39ba478837d747",
       "triggerType" : "PUSH"
     }, {
       "hash" : "840e3b33d39b1f86367bb6b0e65eadeb33de7765",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1025",
       "triggerID" : "840e3b33d39b1f86367bb6b0e65eadeb33de7765",
       "triggerType" : "PUSH"
     }, {
       "hash" : "8d2e2dd83f51d03e890d424a51d3e0e84e904a43",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1031",
       "triggerID" : "8d2e2dd83f51d03e890d424a51d3e0e84e904a43",
       "triggerType" : "PUSH"
     }, {
       "hash" : "688381914087ff11acf2f8c9372dbf39afa0bd40",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1035",
       "triggerID" : "688381914087ff11acf2f8c9372dbf39afa0bd40",
       "triggerType" : "PUSH"
     }, {
       "hash" : "fc2c138901e82ed5f3fbf363c2a0ce8140ade050",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "fc2c138901e82ed5f3fbf363c2a0ce8140ade050",
       "triggerType" : "PUSH"
     }, {
       "hash" : "6ed9e1cb1e5a191752d6d39c7a9d5d41840f0cdf",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1039",
       "triggerID" : "6ed9e1cb1e5a191752d6d39c7a9d5d41840f0cdf",
       "triggerType" : "PUSH"
     }, {
       "hash" : "a510a7de1458bb996df5bbb341e8d010e4c91381",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "a510a7de1458bb996df5bbb341e8d010e4c91381",
       "triggerType" : "PUSH"
     }, {
       "hash" : "49677397cd117228742205944606f7a789090376",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1042",
       "triggerID" : "49677397cd117228742205944606f7a789090376",
       "triggerType" : "PUSH"
     }, {
       "hash" : "1630db0a37422aed942ff280560a9d6329df5015",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1044",
       "triggerID" : "1630db0a37422aed942ff280560a9d6329df5015",
       "triggerType" : "PUSH"
     }, {
       "hash" : "f11c76301d4ab524db0af57aa831be3a898ea0fb",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1048",
       "triggerID" : "f11c76301d4ab524db0af57aa831be3a898ea0fb",
       "triggerType" : "PUSH"
     }, {
       "hash" : "d1a0dc4c2017853a7ba0c3302f886043a63f29da",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1050",
       "triggerID" : "d1a0dc4c2017853a7ba0c3302f886043a63f29da",
       "triggerType" : "PUSH"
     }, {
       "hash" : "6a4c15ec39ded1f90b113b272554217698a98e1c",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1087",
       "triggerID" : "6a4c15ec39ded1f90b113b272554217698a98e1c",
       "triggerType" : "PUSH"
     }, {
       "hash" : "288e730d645a5e58e50a87ef5f44ba11a8fb0533",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1092",
       "triggerID" : "288e730d645a5e58e50a87ef5f44ba11a8fb0533",
       "triggerType" : "PUSH"
     }, {
       "hash" : "70a5c21321a0da3a0be8d85e206c24befa7b89f1",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1095",
       "triggerID" : "70a5c21321a0da3a0be8d85e206c24befa7b89f1",
       "triggerType" : "PUSH"
     }, {
       "hash" : "f072a986114949c5e426ce64a5f5568fc718e25d",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "f072a986114949c5e426ce64a5f5568fc718e25d",
       "triggerType" : "PUSH"
     }, {
       "hash" : "45b27475b1e8806ec8c566d6f151f4bc79c8549d",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1096",
       "triggerID" : "45b27475b1e8806ec8c566d6f151f4bc79c8549d",
       "triggerType" : "PUSH"
     }, {
       "hash" : "3c62ef22f6cde28110e5eb2fca8760a008d77723",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "3c62ef22f6cde28110e5eb2fca8760a008d77723",
       "triggerType" : "PUSH"
     }, {
       "hash" : "dc785fa04e542b2ccc45494e0292f5153c7ecea8",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1097",
       "triggerID" : "dc785fa04e542b2ccc45494e0292f5153c7ecea8",
       "triggerType" : "PUSH"
     }, {
       "hash" : "585ddb8049b9b9ccb057027e265b08a1b586df8c",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "585ddb8049b9b9ccb057027e265b08a1b586df8c",
       "triggerType" : "PUSH"
     }, {
       "hash" : "b62ff8675beed2c4487c788b9ca1a0ad4271b913",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "b62ff8675beed2c4487c788b9ca1a0ad4271b913",
       "triggerType" : "PUSH"
     }, {
       "hash" : "2d46048a7402907f348cc96da2839ba1b3e4932a",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1099",
       "triggerID" : "2d46048a7402907f348cc96da2839ba1b3e4932a",
       "triggerType" : "PUSH"
     }, {
       "hash" : "004fb6ebf23f796a4b77650e0530f0f1145e8b91",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1101",
       "triggerID" : "004fb6ebf23f796a4b77650e0530f0f1145e8b91",
       "triggerType" : "PUSH"
     }, {
       "hash" : "af0c33333b5e9ae6d59f20a6db1d7c016ea015d3",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1165",
       "triggerID" : "af0c33333b5e9ae6d59f20a6db1d7c016ea015d3",
       "triggerType" : "PUSH"
     }, {
       "hash" : "da8111118d71306500229619d3b7996fcd2526e8",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "da8111118d71306500229619d3b7996fcd2526e8",
       "triggerType" : "PUSH"
     }, {
       "hash" : "83257e2b9bd3c426cf90e11003a5f244a750d7fa",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1167",
       "triggerID" : "83257e2b9bd3c426cf90e11003a5f244a750d7fa",
       "triggerType" : "PUSH"
     }, {
       "hash" : "e0de408f1b8153fdaaea4c87b27a7b69213b75b5",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1177",
       "triggerID" : "e0de408f1b8153fdaaea4c87b27a7b69213b75b5",
       "triggerType" : "PUSH"
     }, {
       "hash" : "3f7951f7ad7931f5dc0b977d7c33bc5429623535",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1180",
       "triggerID" : "3f7951f7ad7931f5dc0b977d7c33bc5429623535",
       "triggerType" : "PUSH"
     }, {
       "hash" : "9cd6f06a753b63630e8245d51338d2bf9ab86b84",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "9cd6f06a753b63630e8245d51338d2bf9ab86b84",
       "triggerType" : "PUSH"
     }, {
       "hash" : "7380b902731a92e0645c02397e14591a05798620",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "7380b902731a92e0645c02397e14591a05798620",
       "triggerType" : "PUSH"
     }, {
       "hash" : "0bd5cca01a541845081d5430d48b021d4bdf0969",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "0bd5cca01a541845081d5430d48b021d4bdf0969",
       "triggerType" : "PUSH"
     }, {
       "hash" : "8f4e175ef554f015e34e0968b656b5de714ab430",
       "status" : "SUCCESS",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1182",
       "triggerID" : "8f4e175ef554f015e34e0968b656b5de714ab430",
       "triggerType" : "PUSH"
     }, {
       "hash" : "2023f3c882d0c2f5b96e28d2f21812dc0c64aaee",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "2023f3c882d0c2f5b96e28d2f21812dc0c64aaee",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 9e668f65c6d4f73c76573236f8fa507dfb80edac UNKNOWN
   * 533f00e5546da0026bbe51e1c030ddd1c19fdd08 UNKNOWN
   * a2ee90c549332a7f17a7d9b2799849b77c08a8e6 UNKNOWN
   * fc2c138901e82ed5f3fbf363c2a0ce8140ade050 UNKNOWN
   * a510a7de1458bb996df5bbb341e8d010e4c91381 UNKNOWN
   * f072a986114949c5e426ce64a5f5568fc718e25d UNKNOWN
   * 3c62ef22f6cde28110e5eb2fca8760a008d77723 UNKNOWN
   * 585ddb8049b9b9ccb057027e265b08a1b586df8c UNKNOWN
   * b62ff8675beed2c4487c788b9ca1a0ad4271b913 UNKNOWN
   * da8111118d71306500229619d3b7996fcd2526e8 UNKNOWN
   * 9cd6f06a753b63630e8245d51338d2bf9ab86b84 UNKNOWN
   * 7380b902731a92e0645c02397e14591a05798620 UNKNOWN
   * 0bd5cca01a541845081d5430d48b021d4bdf0969 UNKNOWN
   * 8f4e175ef554f015e34e0968b656b5de714ab430 Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1182) 
   * 2023f3c882d0c2f5b96e28d2f21812dc0c64aaee UNKNOWN
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] [HUDI-8141] Incremental Query with Completion Time [hudi]

Posted by "CTTY (via GitHub)" <gi...@apache.org>.

CTTY commented on code in PR #11947:
URL: https://github.com/apache/hudi/pull/11947#discussion_r1797911286


##########
hudi-spark-datasource/hudi-spark/src/test/scala/org/apache/hudi/functional/TestSparkDataSource.scala:
##########
@@ -138,7 +140,7 @@ class TestSparkDataSource extends SparkClientFunctionalTestHarness {
     val hoodieIncViewDf1 = spark.read.format("org.apache.hudi")
       .option(DataSourceReadOptions.QUERY_TYPE.key, DataSourceReadOptions.QUERY_TYPE_INCREMENTAL_OPT_VAL)
       .option(DataSourceReadOptions.BEGIN_INSTANTTIME.key, "000")
-      .option(DataSourceReadOptions.END_INSTANTTIME.key, firstCommit)
+      .option(DataSourceReadOptions.END_INSTANTTIME.key, commitCompletionTime1)

Review Comment:
   Same here, I'll add `val beforeCommitCompletionTime1 = HoodieInstantTimeGenerator.instantTimeMinusMillis(commitCompletionTime1, 1)`



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] [HUDI-8141] Incremental Query with Completion Time [hudi]

Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.

hudi-bot commented on PR #11947:
URL: https://github.com/apache/hudi/pull/11947#issuecomment-2418869760

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "090f256a315cbbf0c6a54d93cc2739dd693360e6",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=484",
       "triggerID" : "090f256a315cbbf0c6a54d93cc2739dd693360e6",
       "triggerType" : "PUSH"
     }, {
       "hash" : "0df5649013c1cf9e782fcbb67b8a2e7174f94512",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=985",
       "triggerID" : "0df5649013c1cf9e782fcbb67b8a2e7174f94512",
       "triggerType" : "PUSH"
     }, {
       "hash" : "d52e0128f409918644255abbe4376dc4a780bfc8",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=986",
       "triggerID" : "d52e0128f409918644255abbe4376dc4a780bfc8",
       "triggerType" : "PUSH"
     }, {
       "hash" : "9e668f65c6d4f73c76573236f8fa507dfb80edac",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "9e668f65c6d4f73c76573236f8fa507dfb80edac",
       "triggerType" : "PUSH"
     }, {
       "hash" : "f145131d524bf190986e5de600a326e32a27beca",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=987",
       "triggerID" : "f145131d524bf190986e5de600a326e32a27beca",
       "triggerType" : "PUSH"
     }, {
       "hash" : "0b008f807b42aa6fb2686922bb88d63e84a1f299",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=993",
       "triggerID" : "0b008f807b42aa6fb2686922bb88d63e84a1f299",
       "triggerType" : "PUSH"
     }, {
       "hash" : "65c01ad9f2b33d36d4890355feef91844438e732",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=995",
       "triggerID" : "65c01ad9f2b33d36d4890355feef91844438e732",
       "triggerType" : "PUSH"
     }, {
       "hash" : "955493fdc93842e8c99529cec35e39cecdbe9638",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=996",
       "triggerID" : "955493fdc93842e8c99529cec35e39cecdbe9638",
       "triggerType" : "PUSH"
     }, {
       "hash" : "533f00e5546da0026bbe51e1c030ddd1c19fdd08",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "533f00e5546da0026bbe51e1c030ddd1c19fdd08",
       "triggerType" : "PUSH"
     }, {
       "hash" : "05777be67e37f3a6a8eaa59e54d9e17e5fc6ba9b",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1016",
       "triggerID" : "05777be67e37f3a6a8eaa59e54d9e17e5fc6ba9b",
       "triggerType" : "PUSH"
     }, {
       "hash" : "a2ee90c549332a7f17a7d9b2799849b77c08a8e6",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "a2ee90c549332a7f17a7d9b2799849b77c08a8e6",
       "triggerType" : "PUSH"
     }, {
       "hash" : "05ce568b0b6ba14da9ca40d26b39ba478837d747",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1017",
       "triggerID" : "05ce568b0b6ba14da9ca40d26b39ba478837d747",
       "triggerType" : "PUSH"
     }, {
       "hash" : "840e3b33d39b1f86367bb6b0e65eadeb33de7765",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1025",
       "triggerID" : "840e3b33d39b1f86367bb6b0e65eadeb33de7765",
       "triggerType" : "PUSH"
     }, {
       "hash" : "8d2e2dd83f51d03e890d424a51d3e0e84e904a43",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1031",
       "triggerID" : "8d2e2dd83f51d03e890d424a51d3e0e84e904a43",
       "triggerType" : "PUSH"
     }, {
       "hash" : "688381914087ff11acf2f8c9372dbf39afa0bd40",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1035",
       "triggerID" : "688381914087ff11acf2f8c9372dbf39afa0bd40",
       "triggerType" : "PUSH"
     }, {
       "hash" : "fc2c138901e82ed5f3fbf363c2a0ce8140ade050",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "fc2c138901e82ed5f3fbf363c2a0ce8140ade050",
       "triggerType" : "PUSH"
     }, {
       "hash" : "6ed9e1cb1e5a191752d6d39c7a9d5d41840f0cdf",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1039",
       "triggerID" : "6ed9e1cb1e5a191752d6d39c7a9d5d41840f0cdf",
       "triggerType" : "PUSH"
     }, {
       "hash" : "a510a7de1458bb996df5bbb341e8d010e4c91381",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "a510a7de1458bb996df5bbb341e8d010e4c91381",
       "triggerType" : "PUSH"
     }, {
       "hash" : "49677397cd117228742205944606f7a789090376",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1042",
       "triggerID" : "49677397cd117228742205944606f7a789090376",
       "triggerType" : "PUSH"
     }, {
       "hash" : "1630db0a37422aed942ff280560a9d6329df5015",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1044",
       "triggerID" : "1630db0a37422aed942ff280560a9d6329df5015",
       "triggerType" : "PUSH"
     }, {
       "hash" : "f11c76301d4ab524db0af57aa831be3a898ea0fb",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1048",
       "triggerID" : "f11c76301d4ab524db0af57aa831be3a898ea0fb",
       "triggerType" : "PUSH"
     }, {
       "hash" : "d1a0dc4c2017853a7ba0c3302f886043a63f29da",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1050",
       "triggerID" : "d1a0dc4c2017853a7ba0c3302f886043a63f29da",
       "triggerType" : "PUSH"
     }, {
       "hash" : "6a4c15ec39ded1f90b113b272554217698a98e1c",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1087",
       "triggerID" : "6a4c15ec39ded1f90b113b272554217698a98e1c",
       "triggerType" : "PUSH"
     }, {
       "hash" : "288e730d645a5e58e50a87ef5f44ba11a8fb0533",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1092",
       "triggerID" : "288e730d645a5e58e50a87ef5f44ba11a8fb0533",
       "triggerType" : "PUSH"
     }, {
       "hash" : "70a5c21321a0da3a0be8d85e206c24befa7b89f1",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1095",
       "triggerID" : "70a5c21321a0da3a0be8d85e206c24befa7b89f1",
       "triggerType" : "PUSH"
     }, {
       "hash" : "f072a986114949c5e426ce64a5f5568fc718e25d",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "f072a986114949c5e426ce64a5f5568fc718e25d",
       "triggerType" : "PUSH"
     }, {
       "hash" : "45b27475b1e8806ec8c566d6f151f4bc79c8549d",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1096",
       "triggerID" : "45b27475b1e8806ec8c566d6f151f4bc79c8549d",
       "triggerType" : "PUSH"
     }, {
       "hash" : "3c62ef22f6cde28110e5eb2fca8760a008d77723",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "3c62ef22f6cde28110e5eb2fca8760a008d77723",
       "triggerType" : "PUSH"
     }, {
       "hash" : "dc785fa04e542b2ccc45494e0292f5153c7ecea8",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1097",
       "triggerID" : "dc785fa04e542b2ccc45494e0292f5153c7ecea8",
       "triggerType" : "PUSH"
     }, {
       "hash" : "585ddb8049b9b9ccb057027e265b08a1b586df8c",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "585ddb8049b9b9ccb057027e265b08a1b586df8c",
       "triggerType" : "PUSH"
     }, {
       "hash" : "b62ff8675beed2c4487c788b9ca1a0ad4271b913",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "b62ff8675beed2c4487c788b9ca1a0ad4271b913",
       "triggerType" : "PUSH"
     }, {
       "hash" : "2d46048a7402907f348cc96da2839ba1b3e4932a",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1099",
       "triggerID" : "2d46048a7402907f348cc96da2839ba1b3e4932a",
       "triggerType" : "PUSH"
     }, {
       "hash" : "004fb6ebf23f796a4b77650e0530f0f1145e8b91",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1101",
       "triggerID" : "004fb6ebf23f796a4b77650e0530f0f1145e8b91",
       "triggerType" : "PUSH"
     }, {
       "hash" : "af0c33333b5e9ae6d59f20a6db1d7c016ea015d3",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1165",
       "triggerID" : "af0c33333b5e9ae6d59f20a6db1d7c016ea015d3",
       "triggerType" : "PUSH"
     }, {
       "hash" : "da8111118d71306500229619d3b7996fcd2526e8",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "da8111118d71306500229619d3b7996fcd2526e8",
       "triggerType" : "PUSH"
     }, {
       "hash" : "83257e2b9bd3c426cf90e11003a5f244a750d7fa",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1167",
       "triggerID" : "83257e2b9bd3c426cf90e11003a5f244a750d7fa",
       "triggerType" : "PUSH"
     }, {
       "hash" : "e0de408f1b8153fdaaea4c87b27a7b69213b75b5",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1177",
       "triggerID" : "e0de408f1b8153fdaaea4c87b27a7b69213b75b5",
       "triggerType" : "PUSH"
     }, {
       "hash" : "3f7951f7ad7931f5dc0b977d7c33bc5429623535",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1180",
       "triggerID" : "3f7951f7ad7931f5dc0b977d7c33bc5429623535",
       "triggerType" : "PUSH"
     }, {
       "hash" : "9cd6f06a753b63630e8245d51338d2bf9ab86b84",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "9cd6f06a753b63630e8245d51338d2bf9ab86b84",
       "triggerType" : "PUSH"
     }, {
       "hash" : "7380b902731a92e0645c02397e14591a05798620",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "7380b902731a92e0645c02397e14591a05798620",
       "triggerType" : "PUSH"
     }, {
       "hash" : "0bd5cca01a541845081d5430d48b021d4bdf0969",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "0bd5cca01a541845081d5430d48b021d4bdf0969",
       "triggerType" : "PUSH"
     }, {
       "hash" : "8f4e175ef554f015e34e0968b656b5de714ab430",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1182",
       "triggerID" : "8f4e175ef554f015e34e0968b656b5de714ab430",
       "triggerType" : "PUSH"
     }, {
       "hash" : "2023f3c882d0c2f5b96e28d2f21812dc0c64aaee",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "2023f3c882d0c2f5b96e28d2f21812dc0c64aaee",
       "triggerType" : "PUSH"
     }, {
       "hash" : "8394acf060d5c3953dbd67cf930b9b1d3b92d8b5",
       "status" : "SUCCESS",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1198",
       "triggerID" : "8394acf060d5c3953dbd67cf930b9b1d3b92d8b5",
       "triggerType" : "PUSH"
     }, {
       "hash" : "7ec42c105971bff463e359019405a7f4e7947d86",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "7ec42c105971bff463e359019405a7f4e7947d86",
       "triggerType" : "PUSH"
     }, {
       "hash" : "4b930786a7b2de09ac1653cd884ee76a7d1b2ca5",
       "status" : "PENDING",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1202",
       "triggerID" : "4b930786a7b2de09ac1653cd884ee76a7d1b2ca5",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 9e668f65c6d4f73c76573236f8fa507dfb80edac UNKNOWN
   * 533f00e5546da0026bbe51e1c030ddd1c19fdd08 UNKNOWN
   * a2ee90c549332a7f17a7d9b2799849b77c08a8e6 UNKNOWN
   * fc2c138901e82ed5f3fbf363c2a0ce8140ade050 UNKNOWN
   * a510a7de1458bb996df5bbb341e8d010e4c91381 UNKNOWN
   * f072a986114949c5e426ce64a5f5568fc718e25d UNKNOWN
   * 3c62ef22f6cde28110e5eb2fca8760a008d77723 UNKNOWN
   * 585ddb8049b9b9ccb057027e265b08a1b586df8c UNKNOWN
   * b62ff8675beed2c4487c788b9ca1a0ad4271b913 UNKNOWN
   * da8111118d71306500229619d3b7996fcd2526e8 UNKNOWN
   * 9cd6f06a753b63630e8245d51338d2bf9ab86b84 UNKNOWN
   * 7380b902731a92e0645c02397e14591a05798620 UNKNOWN
   * 0bd5cca01a541845081d5430d48b021d4bdf0969 UNKNOWN
   * 2023f3c882d0c2f5b96e28d2f21812dc0c64aaee UNKNOWN
   * 8394acf060d5c3953dbd67cf930b9b1d3b92d8b5 Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1198) 
   * 7ec42c105971bff463e359019405a7f4e7947d86 UNKNOWN
   * 4b930786a7b2de09ac1653cd884ee76a7d1b2ca5 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1202) 
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] [HUDI-8141] Incremental Query with Completion Time [hudi]

Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.

hudi-bot commented on PR #11947:
URL: https://github.com/apache/hudi/pull/11947#issuecomment-2423303493

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "090f256a315cbbf0c6a54d93cc2739dd693360e6",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=484",
       "triggerID" : "090f256a315cbbf0c6a54d93cc2739dd693360e6",
       "triggerType" : "PUSH"
     }, {
       "hash" : "0df5649013c1cf9e782fcbb67b8a2e7174f94512",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=985",
       "triggerID" : "0df5649013c1cf9e782fcbb67b8a2e7174f94512",
       "triggerType" : "PUSH"
     }, {
       "hash" : "d52e0128f409918644255abbe4376dc4a780bfc8",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=986",
       "triggerID" : "d52e0128f409918644255abbe4376dc4a780bfc8",
       "triggerType" : "PUSH"
     }, {
       "hash" : "9e668f65c6d4f73c76573236f8fa507dfb80edac",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "9e668f65c6d4f73c76573236f8fa507dfb80edac",
       "triggerType" : "PUSH"
     }, {
       "hash" : "f145131d524bf190986e5de600a326e32a27beca",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=987",
       "triggerID" : "f145131d524bf190986e5de600a326e32a27beca",
       "triggerType" : "PUSH"
     }, {
       "hash" : "0b008f807b42aa6fb2686922bb88d63e84a1f299",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=993",
       "triggerID" : "0b008f807b42aa6fb2686922bb88d63e84a1f299",
       "triggerType" : "PUSH"
     }, {
       "hash" : "65c01ad9f2b33d36d4890355feef91844438e732",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=995",
       "triggerID" : "65c01ad9f2b33d36d4890355feef91844438e732",
       "triggerType" : "PUSH"
     }, {
       "hash" : "955493fdc93842e8c99529cec35e39cecdbe9638",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=996",
       "triggerID" : "955493fdc93842e8c99529cec35e39cecdbe9638",
       "triggerType" : "PUSH"
     }, {
       "hash" : "533f00e5546da0026bbe51e1c030ddd1c19fdd08",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "533f00e5546da0026bbe51e1c030ddd1c19fdd08",
       "triggerType" : "PUSH"
     }, {
       "hash" : "05777be67e37f3a6a8eaa59e54d9e17e5fc6ba9b",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1016",
       "triggerID" : "05777be67e37f3a6a8eaa59e54d9e17e5fc6ba9b",
       "triggerType" : "PUSH"
     }, {
       "hash" : "a2ee90c549332a7f17a7d9b2799849b77c08a8e6",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "a2ee90c549332a7f17a7d9b2799849b77c08a8e6",
       "triggerType" : "PUSH"
     }, {
       "hash" : "05ce568b0b6ba14da9ca40d26b39ba478837d747",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1017",
       "triggerID" : "05ce568b0b6ba14da9ca40d26b39ba478837d747",
       "triggerType" : "PUSH"
     }, {
       "hash" : "840e3b33d39b1f86367bb6b0e65eadeb33de7765",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1025",
       "triggerID" : "840e3b33d39b1f86367bb6b0e65eadeb33de7765",
       "triggerType" : "PUSH"
     }, {
       "hash" : "8d2e2dd83f51d03e890d424a51d3e0e84e904a43",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1031",
       "triggerID" : "8d2e2dd83f51d03e890d424a51d3e0e84e904a43",
       "triggerType" : "PUSH"
     }, {
       "hash" : "688381914087ff11acf2f8c9372dbf39afa0bd40",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1035",
       "triggerID" : "688381914087ff11acf2f8c9372dbf39afa0bd40",
       "triggerType" : "PUSH"
     }, {
       "hash" : "fc2c138901e82ed5f3fbf363c2a0ce8140ade050",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "fc2c138901e82ed5f3fbf363c2a0ce8140ade050",
       "triggerType" : "PUSH"
     }, {
       "hash" : "6ed9e1cb1e5a191752d6d39c7a9d5d41840f0cdf",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1039",
       "triggerID" : "6ed9e1cb1e5a191752d6d39c7a9d5d41840f0cdf",
       "triggerType" : "PUSH"
     }, {
       "hash" : "a510a7de1458bb996df5bbb341e8d010e4c91381",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "a510a7de1458bb996df5bbb341e8d010e4c91381",
       "triggerType" : "PUSH"
     }, {
       "hash" : "49677397cd117228742205944606f7a789090376",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1042",
       "triggerID" : "49677397cd117228742205944606f7a789090376",
       "triggerType" : "PUSH"
     }, {
       "hash" : "1630db0a37422aed942ff280560a9d6329df5015",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1044",
       "triggerID" : "1630db0a37422aed942ff280560a9d6329df5015",
       "triggerType" : "PUSH"
     }, {
       "hash" : "f11c76301d4ab524db0af57aa831be3a898ea0fb",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1048",
       "triggerID" : "f11c76301d4ab524db0af57aa831be3a898ea0fb",
       "triggerType" : "PUSH"
     }, {
       "hash" : "d1a0dc4c2017853a7ba0c3302f886043a63f29da",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1050",
       "triggerID" : "d1a0dc4c2017853a7ba0c3302f886043a63f29da",
       "triggerType" : "PUSH"
     }, {
       "hash" : "6a4c15ec39ded1f90b113b272554217698a98e1c",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1087",
       "triggerID" : "6a4c15ec39ded1f90b113b272554217698a98e1c",
       "triggerType" : "PUSH"
     }, {
       "hash" : "288e730d645a5e58e50a87ef5f44ba11a8fb0533",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1092",
       "triggerID" : "288e730d645a5e58e50a87ef5f44ba11a8fb0533",
       "triggerType" : "PUSH"
     }, {
       "hash" : "70a5c21321a0da3a0be8d85e206c24befa7b89f1",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1095",
       "triggerID" : "70a5c21321a0da3a0be8d85e206c24befa7b89f1",
       "triggerType" : "PUSH"
     }, {
       "hash" : "f072a986114949c5e426ce64a5f5568fc718e25d",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "f072a986114949c5e426ce64a5f5568fc718e25d",
       "triggerType" : "PUSH"
     }, {
       "hash" : "45b27475b1e8806ec8c566d6f151f4bc79c8549d",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1096",
       "triggerID" : "45b27475b1e8806ec8c566d6f151f4bc79c8549d",
       "triggerType" : "PUSH"
     }, {
       "hash" : "3c62ef22f6cde28110e5eb2fca8760a008d77723",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "3c62ef22f6cde28110e5eb2fca8760a008d77723",
       "triggerType" : "PUSH"
     }, {
       "hash" : "dc785fa04e542b2ccc45494e0292f5153c7ecea8",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1097",
       "triggerID" : "dc785fa04e542b2ccc45494e0292f5153c7ecea8",
       "triggerType" : "PUSH"
     }, {
       "hash" : "585ddb8049b9b9ccb057027e265b08a1b586df8c",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "585ddb8049b9b9ccb057027e265b08a1b586df8c",
       "triggerType" : "PUSH"
     }, {
       "hash" : "b62ff8675beed2c4487c788b9ca1a0ad4271b913",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "b62ff8675beed2c4487c788b9ca1a0ad4271b913",
       "triggerType" : "PUSH"
     }, {
       "hash" : "2d46048a7402907f348cc96da2839ba1b3e4932a",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1099",
       "triggerID" : "2d46048a7402907f348cc96da2839ba1b3e4932a",
       "triggerType" : "PUSH"
     }, {
       "hash" : "004fb6ebf23f796a4b77650e0530f0f1145e8b91",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1101",
       "triggerID" : "004fb6ebf23f796a4b77650e0530f0f1145e8b91",
       "triggerType" : "PUSH"
     }, {
       "hash" : "af0c33333b5e9ae6d59f20a6db1d7c016ea015d3",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1165",
       "triggerID" : "af0c33333b5e9ae6d59f20a6db1d7c016ea015d3",
       "triggerType" : "PUSH"
     }, {
       "hash" : "da8111118d71306500229619d3b7996fcd2526e8",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "da8111118d71306500229619d3b7996fcd2526e8",
       "triggerType" : "PUSH"
     }, {
       "hash" : "83257e2b9bd3c426cf90e11003a5f244a750d7fa",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1167",
       "triggerID" : "83257e2b9bd3c426cf90e11003a5f244a750d7fa",
       "triggerType" : "PUSH"
     }, {
       "hash" : "e0de408f1b8153fdaaea4c87b27a7b69213b75b5",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1177",
       "triggerID" : "e0de408f1b8153fdaaea4c87b27a7b69213b75b5",
       "triggerType" : "PUSH"
     }, {
       "hash" : "3f7951f7ad7931f5dc0b977d7c33bc5429623535",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1180",
       "triggerID" : "3f7951f7ad7931f5dc0b977d7c33bc5429623535",
       "triggerType" : "PUSH"
     }, {
       "hash" : "9cd6f06a753b63630e8245d51338d2bf9ab86b84",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "9cd6f06a753b63630e8245d51338d2bf9ab86b84",
       "triggerType" : "PUSH"
     }, {
       "hash" : "7380b902731a92e0645c02397e14591a05798620",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "7380b902731a92e0645c02397e14591a05798620",
       "triggerType" : "PUSH"
     }, {
       "hash" : "0bd5cca01a541845081d5430d48b021d4bdf0969",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "0bd5cca01a541845081d5430d48b021d4bdf0969",
       "triggerType" : "PUSH"
     }, {
       "hash" : "8f4e175ef554f015e34e0968b656b5de714ab430",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1182",
       "triggerID" : "8f4e175ef554f015e34e0968b656b5de714ab430",
       "triggerType" : "PUSH"
     }, {
       "hash" : "2023f3c882d0c2f5b96e28d2f21812dc0c64aaee",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "2023f3c882d0c2f5b96e28d2f21812dc0c64aaee",
       "triggerType" : "PUSH"
     }, {
       "hash" : "8394acf060d5c3953dbd67cf930b9b1d3b92d8b5",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1198",
       "triggerID" : "8394acf060d5c3953dbd67cf930b9b1d3b92d8b5",
       "triggerType" : "PUSH"
     }, {
       "hash" : "7ec42c105971bff463e359019405a7f4e7947d86",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "7ec42c105971bff463e359019405a7f4e7947d86",
       "triggerType" : "PUSH"
     }, {
       "hash" : "4b930786a7b2de09ac1653cd884ee76a7d1b2ca5",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1202",
       "triggerID" : "4b930786a7b2de09ac1653cd884ee76a7d1b2ca5",
       "triggerType" : "PUSH"
     }, {
       "hash" : "0e9a648c4e03b5a74d6a0085831805041a9a26a2",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1216",
       "triggerID" : "0e9a648c4e03b5a74d6a0085831805041a9a26a2",
       "triggerType" : "PUSH"
     }, {
       "hash" : "326a8729a5d1dd88bc474a5c273aec6ca219f298",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "326a8729a5d1dd88bc474a5c273aec6ca219f298",
       "triggerType" : "PUSH"
     }, {
       "hash" : "cad2f761fc50fba28e2fb3595e4b1270e39ac48a",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1221",
       "triggerID" : "cad2f761fc50fba28e2fb3595e4b1270e39ac48a",
       "triggerType" : "PUSH"
     }, {
       "hash" : "d485a50aec34bc5707377afe96d696772832558e",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "d485a50aec34bc5707377afe96d696772832558e",
       "triggerType" : "PUSH"
     }, {
       "hash" : "38a98106f19fe4e51632a2c4289ec0f01805ba5e",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1224",
       "triggerID" : "38a98106f19fe4e51632a2c4289ec0f01805ba5e",
       "triggerType" : "PUSH"
     }, {
       "hash" : "7a6116cfbf4eba54e77194664225ed1c372d71be",
       "status" : "FAILURE",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1250",
       "triggerID" : "7a6116cfbf4eba54e77194664225ed1c372d71be",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 9e668f65c6d4f73c76573236f8fa507dfb80edac UNKNOWN
   * 533f00e5546da0026bbe51e1c030ddd1c19fdd08 UNKNOWN
   * a2ee90c549332a7f17a7d9b2799849b77c08a8e6 UNKNOWN
   * fc2c138901e82ed5f3fbf363c2a0ce8140ade050 UNKNOWN
   * a510a7de1458bb996df5bbb341e8d010e4c91381 UNKNOWN
   * f072a986114949c5e426ce64a5f5568fc718e25d UNKNOWN
   * 3c62ef22f6cde28110e5eb2fca8760a008d77723 UNKNOWN
   * 585ddb8049b9b9ccb057027e265b08a1b586df8c UNKNOWN
   * b62ff8675beed2c4487c788b9ca1a0ad4271b913 UNKNOWN
   * da8111118d71306500229619d3b7996fcd2526e8 UNKNOWN
   * 9cd6f06a753b63630e8245d51338d2bf9ab86b84 UNKNOWN
   * 7380b902731a92e0645c02397e14591a05798620 UNKNOWN
   * 0bd5cca01a541845081d5430d48b021d4bdf0969 UNKNOWN
   * 2023f3c882d0c2f5b96e28d2f21812dc0c64aaee UNKNOWN
   * 7ec42c105971bff463e359019405a7f4e7947d86 UNKNOWN
   * 326a8729a5d1dd88bc474a5c273aec6ca219f298 UNKNOWN
   * d485a50aec34bc5707377afe96d696772832558e UNKNOWN
   * 7a6116cfbf4eba54e77194664225ed1c372d71be Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1250) 
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] [HUDI-8141] Incremental Query with Completion Time [hudi]

Posted by "yihua (via GitHub)" <gi...@apache.org>.

yihua commented on code in PR #11947:
URL: https://github.com/apache/hudi/pull/11947#discussion_r1806033883


##########
hudi-utilities/src/main/java/org/apache/hudi/utilities/sources/helpers/IncrSourceHelper.java:
##########
@@ -174,6 +168,63 @@ public static QueryInfo generateQueryInfo(JavaSparkContext jssc, String srcBaseP
     }
   }
 
+  public static IncrementalQueryAnalyzer getIncrementalQueryAnalyzer(
+      JavaSparkContext jssc,
+      String srcPath,
+      Option<String> lastCkptStr,
+      MissingCheckpointStrategy missingCheckpointStrategy,
+      int numInstantsFromConfig,
+      Option<SourceProfile<Integer>> latestSourceProfile) {
+    HoodieTableMetaClient metaClient = HoodieTableMetaClient.builder()
+        .setConf(HadoopFSUtils.getStorageConfWithCopy(jssc.hadoopConfiguration()))
+        .setBasePath(srcPath)
+        .setLoadActiveTimelineOnLoad(true)
+        .build();
+
+    String beginCompletionTime;

Review Comment:
   Fixed.



##########
hudi-utilities/src/main/java/org/apache/hudi/utilities/sources/helpers/IncrSourceHelper.java:
##########
@@ -174,6 +168,63 @@ public static QueryInfo generateQueryInfo(JavaSparkContext jssc, String srcBaseP
     }
   }
 
+  public static IncrementalQueryAnalyzer getIncrementalQueryAnalyzer(
+      JavaSparkContext jssc,
+      String srcPath,
+      Option<String> lastCkptStr,
+      MissingCheckpointStrategy missingCheckpointStrategy,
+      int numInstantsFromConfig,
+      Option<SourceProfile<Integer>> latestSourceProfile) {
+    HoodieTableMetaClient metaClient = HoodieTableMetaClient.builder()
+        .setConf(HadoopFSUtils.getStorageConfWithCopy(jssc.hadoopConfiguration()))
+        .setBasePath(srcPath)
+        .setLoadActiveTimelineOnLoad(true)
+        .build();
+
+    String beginCompletionTime;
+    RangeType rangeType;
+
+    if (lastCkptStr.isPresent() && !lastCkptStr.get().isEmpty()) {
+      beginCompletionTime = lastCkptStr.get();
+      rangeType = RangeType.OPEN_CLOSED;
+    } else if (missingCheckpointStrategy != null) {
+      rangeType = RangeType.CLOSED_CLOSED;
+      switch (missingCheckpointStrategy) {
+        case READ_UPTO_LATEST_COMMIT:
+          beginCompletionTime = DEFAULT_BEGIN_TIMESTAMP;
+          // disrespect numInstantsFromConfig when reading up to latest
+          numInstantsFromConfig = -1;
+          break;
+        case READ_LATEST:
+          // rely on IncrementalQueryAnalyzer to use the latest completed instant
+          beginCompletionTime = null;
+          break;
+        default:
+          throw new IllegalArgumentException("Unknown missing checkpoint strategy: " + missingCheckpointStrategy);
+      }
+    } else {
+      throw new IllegalArgumentException("Missing begin instant for incremental pull. For reading from latest "

Review Comment:
   Fixed.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] [HUDI-8141] Incremental Query with Completion Time [hudi]

Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.

hudi-bot commented on PR #11947:
URL: https://github.com/apache/hudi/pull/11947#issuecomment-2422015269

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "090f256a315cbbf0c6a54d93cc2739dd693360e6",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=484",
       "triggerID" : "090f256a315cbbf0c6a54d93cc2739dd693360e6",
       "triggerType" : "PUSH"
     }, {
       "hash" : "0df5649013c1cf9e782fcbb67b8a2e7174f94512",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=985",
       "triggerID" : "0df5649013c1cf9e782fcbb67b8a2e7174f94512",
       "triggerType" : "PUSH"
     }, {
       "hash" : "d52e0128f409918644255abbe4376dc4a780bfc8",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=986",
       "triggerID" : "d52e0128f409918644255abbe4376dc4a780bfc8",
       "triggerType" : "PUSH"
     }, {
       "hash" : "9e668f65c6d4f73c76573236f8fa507dfb80edac",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "9e668f65c6d4f73c76573236f8fa507dfb80edac",
       "triggerType" : "PUSH"
     }, {
       "hash" : "f145131d524bf190986e5de600a326e32a27beca",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=987",
       "triggerID" : "f145131d524bf190986e5de600a326e32a27beca",
       "triggerType" : "PUSH"
     }, {
       "hash" : "0b008f807b42aa6fb2686922bb88d63e84a1f299",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=993",
       "triggerID" : "0b008f807b42aa6fb2686922bb88d63e84a1f299",
       "triggerType" : "PUSH"
     }, {
       "hash" : "65c01ad9f2b33d36d4890355feef91844438e732",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=995",
       "triggerID" : "65c01ad9f2b33d36d4890355feef91844438e732",
       "triggerType" : "PUSH"
     }, {
       "hash" : "955493fdc93842e8c99529cec35e39cecdbe9638",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=996",
       "triggerID" : "955493fdc93842e8c99529cec35e39cecdbe9638",
       "triggerType" : "PUSH"
     }, {
       "hash" : "533f00e5546da0026bbe51e1c030ddd1c19fdd08",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "533f00e5546da0026bbe51e1c030ddd1c19fdd08",
       "triggerType" : "PUSH"
     }, {
       "hash" : "05777be67e37f3a6a8eaa59e54d9e17e5fc6ba9b",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1016",
       "triggerID" : "05777be67e37f3a6a8eaa59e54d9e17e5fc6ba9b",
       "triggerType" : "PUSH"
     }, {
       "hash" : "a2ee90c549332a7f17a7d9b2799849b77c08a8e6",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "a2ee90c549332a7f17a7d9b2799849b77c08a8e6",
       "triggerType" : "PUSH"
     }, {
       "hash" : "05ce568b0b6ba14da9ca40d26b39ba478837d747",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1017",
       "triggerID" : "05ce568b0b6ba14da9ca40d26b39ba478837d747",
       "triggerType" : "PUSH"
     }, {
       "hash" : "840e3b33d39b1f86367bb6b0e65eadeb33de7765",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1025",
       "triggerID" : "840e3b33d39b1f86367bb6b0e65eadeb33de7765",
       "triggerType" : "PUSH"
     }, {
       "hash" : "8d2e2dd83f51d03e890d424a51d3e0e84e904a43",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1031",
       "triggerID" : "8d2e2dd83f51d03e890d424a51d3e0e84e904a43",
       "triggerType" : "PUSH"
     }, {
       "hash" : "688381914087ff11acf2f8c9372dbf39afa0bd40",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1035",
       "triggerID" : "688381914087ff11acf2f8c9372dbf39afa0bd40",
       "triggerType" : "PUSH"
     }, {
       "hash" : "fc2c138901e82ed5f3fbf363c2a0ce8140ade050",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "fc2c138901e82ed5f3fbf363c2a0ce8140ade050",
       "triggerType" : "PUSH"
     }, {
       "hash" : "6ed9e1cb1e5a191752d6d39c7a9d5d41840f0cdf",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1039",
       "triggerID" : "6ed9e1cb1e5a191752d6d39c7a9d5d41840f0cdf",
       "triggerType" : "PUSH"
     }, {
       "hash" : "a510a7de1458bb996df5bbb341e8d010e4c91381",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "a510a7de1458bb996df5bbb341e8d010e4c91381",
       "triggerType" : "PUSH"
     }, {
       "hash" : "49677397cd117228742205944606f7a789090376",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1042",
       "triggerID" : "49677397cd117228742205944606f7a789090376",
       "triggerType" : "PUSH"
     }, {
       "hash" : "1630db0a37422aed942ff280560a9d6329df5015",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1044",
       "triggerID" : "1630db0a37422aed942ff280560a9d6329df5015",
       "triggerType" : "PUSH"
     }, {
       "hash" : "f11c76301d4ab524db0af57aa831be3a898ea0fb",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1048",
       "triggerID" : "f11c76301d4ab524db0af57aa831be3a898ea0fb",
       "triggerType" : "PUSH"
     }, {
       "hash" : "d1a0dc4c2017853a7ba0c3302f886043a63f29da",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1050",
       "triggerID" : "d1a0dc4c2017853a7ba0c3302f886043a63f29da",
       "triggerType" : "PUSH"
     }, {
       "hash" : "6a4c15ec39ded1f90b113b272554217698a98e1c",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1087",
       "triggerID" : "6a4c15ec39ded1f90b113b272554217698a98e1c",
       "triggerType" : "PUSH"
     }, {
       "hash" : "288e730d645a5e58e50a87ef5f44ba11a8fb0533",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1092",
       "triggerID" : "288e730d645a5e58e50a87ef5f44ba11a8fb0533",
       "triggerType" : "PUSH"
     }, {
       "hash" : "70a5c21321a0da3a0be8d85e206c24befa7b89f1",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1095",
       "triggerID" : "70a5c21321a0da3a0be8d85e206c24befa7b89f1",
       "triggerType" : "PUSH"
     }, {
       "hash" : "f072a986114949c5e426ce64a5f5568fc718e25d",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "f072a986114949c5e426ce64a5f5568fc718e25d",
       "triggerType" : "PUSH"
     }, {
       "hash" : "45b27475b1e8806ec8c566d6f151f4bc79c8549d",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1096",
       "triggerID" : "45b27475b1e8806ec8c566d6f151f4bc79c8549d",
       "triggerType" : "PUSH"
     }, {
       "hash" : "3c62ef22f6cde28110e5eb2fca8760a008d77723",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "3c62ef22f6cde28110e5eb2fca8760a008d77723",
       "triggerType" : "PUSH"
     }, {
       "hash" : "dc785fa04e542b2ccc45494e0292f5153c7ecea8",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1097",
       "triggerID" : "dc785fa04e542b2ccc45494e0292f5153c7ecea8",
       "triggerType" : "PUSH"
     }, {
       "hash" : "585ddb8049b9b9ccb057027e265b08a1b586df8c",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "585ddb8049b9b9ccb057027e265b08a1b586df8c",
       "triggerType" : "PUSH"
     }, {
       "hash" : "b62ff8675beed2c4487c788b9ca1a0ad4271b913",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "b62ff8675beed2c4487c788b9ca1a0ad4271b913",
       "triggerType" : "PUSH"
     }, {
       "hash" : "2d46048a7402907f348cc96da2839ba1b3e4932a",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1099",
       "triggerID" : "2d46048a7402907f348cc96da2839ba1b3e4932a",
       "triggerType" : "PUSH"
     }, {
       "hash" : "004fb6ebf23f796a4b77650e0530f0f1145e8b91",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1101",
       "triggerID" : "004fb6ebf23f796a4b77650e0530f0f1145e8b91",
       "triggerType" : "PUSH"
     }, {
       "hash" : "af0c33333b5e9ae6d59f20a6db1d7c016ea015d3",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1165",
       "triggerID" : "af0c33333b5e9ae6d59f20a6db1d7c016ea015d3",
       "triggerType" : "PUSH"
     }, {
       "hash" : "da8111118d71306500229619d3b7996fcd2526e8",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "da8111118d71306500229619d3b7996fcd2526e8",
       "triggerType" : "PUSH"
     }, {
       "hash" : "83257e2b9bd3c426cf90e11003a5f244a750d7fa",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1167",
       "triggerID" : "83257e2b9bd3c426cf90e11003a5f244a750d7fa",
       "triggerType" : "PUSH"
     }, {
       "hash" : "e0de408f1b8153fdaaea4c87b27a7b69213b75b5",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1177",
       "triggerID" : "e0de408f1b8153fdaaea4c87b27a7b69213b75b5",
       "triggerType" : "PUSH"
     }, {
       "hash" : "3f7951f7ad7931f5dc0b977d7c33bc5429623535",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1180",
       "triggerID" : "3f7951f7ad7931f5dc0b977d7c33bc5429623535",
       "triggerType" : "PUSH"
     }, {
       "hash" : "9cd6f06a753b63630e8245d51338d2bf9ab86b84",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "9cd6f06a753b63630e8245d51338d2bf9ab86b84",
       "triggerType" : "PUSH"
     }, {
       "hash" : "7380b902731a92e0645c02397e14591a05798620",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "7380b902731a92e0645c02397e14591a05798620",
       "triggerType" : "PUSH"
     }, {
       "hash" : "0bd5cca01a541845081d5430d48b021d4bdf0969",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "0bd5cca01a541845081d5430d48b021d4bdf0969",
       "triggerType" : "PUSH"
     }, {
       "hash" : "8f4e175ef554f015e34e0968b656b5de714ab430",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1182",
       "triggerID" : "8f4e175ef554f015e34e0968b656b5de714ab430",
       "triggerType" : "PUSH"
     }, {
       "hash" : "2023f3c882d0c2f5b96e28d2f21812dc0c64aaee",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "2023f3c882d0c2f5b96e28d2f21812dc0c64aaee",
       "triggerType" : "PUSH"
     }, {
       "hash" : "8394acf060d5c3953dbd67cf930b9b1d3b92d8b5",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1198",
       "triggerID" : "8394acf060d5c3953dbd67cf930b9b1d3b92d8b5",
       "triggerType" : "PUSH"
     }, {
       "hash" : "7ec42c105971bff463e359019405a7f4e7947d86",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "7ec42c105971bff463e359019405a7f4e7947d86",
       "triggerType" : "PUSH"
     }, {
       "hash" : "4b930786a7b2de09ac1653cd884ee76a7d1b2ca5",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1202",
       "triggerID" : "4b930786a7b2de09ac1653cd884ee76a7d1b2ca5",
       "triggerType" : "PUSH"
     }, {
       "hash" : "0e9a648c4e03b5a74d6a0085831805041a9a26a2",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1216",
       "triggerID" : "0e9a648c4e03b5a74d6a0085831805041a9a26a2",
       "triggerType" : "PUSH"
     }, {
       "hash" : "326a8729a5d1dd88bc474a5c273aec6ca219f298",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "326a8729a5d1dd88bc474a5c273aec6ca219f298",
       "triggerType" : "PUSH"
     }, {
       "hash" : "cad2f761fc50fba28e2fb3595e4b1270e39ac48a",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1221",
       "triggerID" : "cad2f761fc50fba28e2fb3595e4b1270e39ac48a",
       "triggerType" : "PUSH"
     }, {
       "hash" : "d485a50aec34bc5707377afe96d696772832558e",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "d485a50aec34bc5707377afe96d696772832558e",
       "triggerType" : "PUSH"
     }, {
       "hash" : "38a98106f19fe4e51632a2c4289ec0f01805ba5e",
       "status" : "SUCCESS",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1224",
       "triggerID" : "38a98106f19fe4e51632a2c4289ec0f01805ba5e",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 9e668f65c6d4f73c76573236f8fa507dfb80edac UNKNOWN
   * 533f00e5546da0026bbe51e1c030ddd1c19fdd08 UNKNOWN
   * a2ee90c549332a7f17a7d9b2799849b77c08a8e6 UNKNOWN
   * fc2c138901e82ed5f3fbf363c2a0ce8140ade050 UNKNOWN
   * a510a7de1458bb996df5bbb341e8d010e4c91381 UNKNOWN
   * f072a986114949c5e426ce64a5f5568fc718e25d UNKNOWN
   * 3c62ef22f6cde28110e5eb2fca8760a008d77723 UNKNOWN
   * 585ddb8049b9b9ccb057027e265b08a1b586df8c UNKNOWN
   * b62ff8675beed2c4487c788b9ca1a0ad4271b913 UNKNOWN
   * da8111118d71306500229619d3b7996fcd2526e8 UNKNOWN
   * 9cd6f06a753b63630e8245d51338d2bf9ab86b84 UNKNOWN
   * 7380b902731a92e0645c02397e14591a05798620 UNKNOWN
   * 0bd5cca01a541845081d5430d48b021d4bdf0969 UNKNOWN
   * 2023f3c882d0c2f5b96e28d2f21812dc0c64aaee UNKNOWN
   * 7ec42c105971bff463e359019405a7f4e7947d86 UNKNOWN
   * 326a8729a5d1dd88bc474a5c273aec6ca219f298 UNKNOWN
   * d485a50aec34bc5707377afe96d696772832558e UNKNOWN
   * 38a98106f19fe4e51632a2c4289ec0f01805ba5e Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1224) 
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] [HUDI-8141] Incremental Query with Completion Time [hudi]

Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.

hudi-bot commented on PR #11947:
URL: https://github.com/apache/hudi/pull/11947#issuecomment-2408825542

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "090f256a315cbbf0c6a54d93cc2739dd693360e6",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=484",
       "triggerID" : "090f256a315cbbf0c6a54d93cc2739dd693360e6",
       "triggerType" : "PUSH"
     }, {
       "hash" : "0df5649013c1cf9e782fcbb67b8a2e7174f94512",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=985",
       "triggerID" : "0df5649013c1cf9e782fcbb67b8a2e7174f94512",
       "triggerType" : "PUSH"
     }, {
       "hash" : "d52e0128f409918644255abbe4376dc4a780bfc8",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=986",
       "triggerID" : "d52e0128f409918644255abbe4376dc4a780bfc8",
       "triggerType" : "PUSH"
     }, {
       "hash" : "9e668f65c6d4f73c76573236f8fa507dfb80edac",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "9e668f65c6d4f73c76573236f8fa507dfb80edac",
       "triggerType" : "PUSH"
     }, {
       "hash" : "f145131d524bf190986e5de600a326e32a27beca",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=987",
       "triggerID" : "f145131d524bf190986e5de600a326e32a27beca",
       "triggerType" : "PUSH"
     }, {
       "hash" : "0b008f807b42aa6fb2686922bb88d63e84a1f299",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=993",
       "triggerID" : "0b008f807b42aa6fb2686922bb88d63e84a1f299",
       "triggerType" : "PUSH"
     }, {
       "hash" : "65c01ad9f2b33d36d4890355feef91844438e732",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=995",
       "triggerID" : "65c01ad9f2b33d36d4890355feef91844438e732",
       "triggerType" : "PUSH"
     }, {
       "hash" : "955493fdc93842e8c99529cec35e39cecdbe9638",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=996",
       "triggerID" : "955493fdc93842e8c99529cec35e39cecdbe9638",
       "triggerType" : "PUSH"
     }, {
       "hash" : "533f00e5546da0026bbe51e1c030ddd1c19fdd08",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "533f00e5546da0026bbe51e1c030ddd1c19fdd08",
       "triggerType" : "PUSH"
     }, {
       "hash" : "05777be67e37f3a6a8eaa59e54d9e17e5fc6ba9b",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1016",
       "triggerID" : "05777be67e37f3a6a8eaa59e54d9e17e5fc6ba9b",
       "triggerType" : "PUSH"
     }, {
       "hash" : "a2ee90c549332a7f17a7d9b2799849b77c08a8e6",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "a2ee90c549332a7f17a7d9b2799849b77c08a8e6",
       "triggerType" : "PUSH"
     }, {
       "hash" : "05ce568b0b6ba14da9ca40d26b39ba478837d747",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1017",
       "triggerID" : "05ce568b0b6ba14da9ca40d26b39ba478837d747",
       "triggerType" : "PUSH"
     }, {
       "hash" : "840e3b33d39b1f86367bb6b0e65eadeb33de7765",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1025",
       "triggerID" : "840e3b33d39b1f86367bb6b0e65eadeb33de7765",
       "triggerType" : "PUSH"
     }, {
       "hash" : "8d2e2dd83f51d03e890d424a51d3e0e84e904a43",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1031",
       "triggerID" : "8d2e2dd83f51d03e890d424a51d3e0e84e904a43",
       "triggerType" : "PUSH"
     }, {
       "hash" : "688381914087ff11acf2f8c9372dbf39afa0bd40",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1035",
       "triggerID" : "688381914087ff11acf2f8c9372dbf39afa0bd40",
       "triggerType" : "PUSH"
     }, {
       "hash" : "fc2c138901e82ed5f3fbf363c2a0ce8140ade050",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "fc2c138901e82ed5f3fbf363c2a0ce8140ade050",
       "triggerType" : "PUSH"
     }, {
       "hash" : "6ed9e1cb1e5a191752d6d39c7a9d5d41840f0cdf",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1039",
       "triggerID" : "6ed9e1cb1e5a191752d6d39c7a9d5d41840f0cdf",
       "triggerType" : "PUSH"
     }, {
       "hash" : "a510a7de1458bb996df5bbb341e8d010e4c91381",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "a510a7de1458bb996df5bbb341e8d010e4c91381",
       "triggerType" : "PUSH"
     }, {
       "hash" : "49677397cd117228742205944606f7a789090376",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1042",
       "triggerID" : "49677397cd117228742205944606f7a789090376",
       "triggerType" : "PUSH"
     }, {
       "hash" : "1630db0a37422aed942ff280560a9d6329df5015",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1044",
       "triggerID" : "1630db0a37422aed942ff280560a9d6329df5015",
       "triggerType" : "PUSH"
     }, {
       "hash" : "f11c76301d4ab524db0af57aa831be3a898ea0fb",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1048",
       "triggerID" : "f11c76301d4ab524db0af57aa831be3a898ea0fb",
       "triggerType" : "PUSH"
     }, {
       "hash" : "d1a0dc4c2017853a7ba0c3302f886043a63f29da",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1050",
       "triggerID" : "d1a0dc4c2017853a7ba0c3302f886043a63f29da",
       "triggerType" : "PUSH"
     }, {
       "hash" : "6a4c15ec39ded1f90b113b272554217698a98e1c",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1087",
       "triggerID" : "6a4c15ec39ded1f90b113b272554217698a98e1c",
       "triggerType" : "PUSH"
     }, {
       "hash" : "288e730d645a5e58e50a87ef5f44ba11a8fb0533",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1092",
       "triggerID" : "288e730d645a5e58e50a87ef5f44ba11a8fb0533",
       "triggerType" : "PUSH"
     }, {
       "hash" : "70a5c21321a0da3a0be8d85e206c24befa7b89f1",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1095",
       "triggerID" : "70a5c21321a0da3a0be8d85e206c24befa7b89f1",
       "triggerType" : "PUSH"
     }, {
       "hash" : "f072a986114949c5e426ce64a5f5568fc718e25d",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "f072a986114949c5e426ce64a5f5568fc718e25d",
       "triggerType" : "PUSH"
     }, {
       "hash" : "45b27475b1e8806ec8c566d6f151f4bc79c8549d",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1096",
       "triggerID" : "45b27475b1e8806ec8c566d6f151f4bc79c8549d",
       "triggerType" : "PUSH"
     }, {
       "hash" : "3c62ef22f6cde28110e5eb2fca8760a008d77723",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "3c62ef22f6cde28110e5eb2fca8760a008d77723",
       "triggerType" : "PUSH"
     }, {
       "hash" : "dc785fa04e542b2ccc45494e0292f5153c7ecea8",
       "status" : "FAILURE",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1097",
       "triggerID" : "dc785fa04e542b2ccc45494e0292f5153c7ecea8",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 9e668f65c6d4f73c76573236f8fa507dfb80edac UNKNOWN
   * 533f00e5546da0026bbe51e1c030ddd1c19fdd08 UNKNOWN
   * a2ee90c549332a7f17a7d9b2799849b77c08a8e6 UNKNOWN
   * fc2c138901e82ed5f3fbf363c2a0ce8140ade050 UNKNOWN
   * a510a7de1458bb996df5bbb341e8d010e4c91381 UNKNOWN
   * f072a986114949c5e426ce64a5f5568fc718e25d UNKNOWN
   * 3c62ef22f6cde28110e5eb2fca8760a008d77723 UNKNOWN
   * dc785fa04e542b2ccc45494e0292f5153c7ecea8 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1097) 
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] [HUDI-8141] Incremental Query with Completion Time [hudi]

Posted by "CTTY (via GitHub)" <gi...@apache.org>.

CTTY commented on code in PR #11947:
URL: https://github.com/apache/hudi/pull/11947#discussion_r1797786835


##########
hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/hudi/MergeOnReadIncrementalRelation.scala:
##########
@@ -94,9 +94,10 @@ case class MergeOnReadIncrementalRelation(override val sqlContext: SQLContext,
       tableState = tableState,
       mergeType = mergeType,
       fileSplits = fileSplits,
-      includeStartTime = includeStartTime,
+      includeStartTime = false,

Review Comment:
   Yes, we should get rid of it. Will remove this



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] [HUDI-8141] Incremental Query with Completion Time [hudi]

Posted by "CTTY (via GitHub)" <gi...@apache.org>.

CTTY commented on code in PR #11947:
URL: https://github.com/apache/hudi/pull/11947#discussion_r1798106737


##########
hudi-utilities/src/main/java/org/apache/hudi/utilities/sources/HoodieIncrSource.java:
##########
@@ -175,33 +181,72 @@ public Pair<Option<Dataset<Row>>, String> fetchNextBatch(Option<String> lastCkpt
     IncrSourceHelper.MissingCheckpointStrategy missingCheckpointStrategy =
         (containsConfigProperty(props, HoodieIncrSourceConfig.MISSING_CHECKPOINT_STRATEGY))
             ? IncrSourceHelper.MissingCheckpointStrategy.valueOf(
-            getStringWithAltKeys(props, HoodieIncrSourceConfig.MISSING_CHECKPOINT_STRATEGY))
+                getStringWithAltKeys(props, HoodieIncrSourceConfig.MISSING_CHECKPOINT_STRATEGY))
             : null;
     if (readLatestOnMissingCkpt) {
       missingCheckpointStrategy = IncrSourceHelper.MissingCheckpointStrategy.READ_LATEST;
     }
 
-    // Use begin Instant if set and non-empty
-    Option<String> beginInstant =
-        lastCkptStr.isPresent() ? lastCkptStr.get().isEmpty() ? Option.empty() : lastCkptStr : Option.empty();
+    HoodieTableMetaClient metaClient = HoodieTableMetaClient.builder()
+        .setConf(HadoopFSUtils.getStorageConfWithCopy(sparkContext.hadoopConfiguration()))
+        .setBasePath(srcPath)
+        .setLoadActiveTimelineOnLoad(true)
+        .build();
 
+    int numInstantsFromConfig = getIntWithAltKeys(props, HoodieIncrSourceConfig.NUM_INSTANTS_PER_FETCH);
+
+    String startTime;
+    if (lastCkptStr.isPresent() && !lastCkptStr.get().isEmpty()) {
+      startTime = lastCkptStr.get();
+    } else if (missingCheckpointStrategy != null) {
+      switch (missingCheckpointStrategy) {
+        case READ_UPTO_LATEST_COMMIT:
+          startTime = DEFAULT_BEGIN_TIMESTAMP;

Review Comment:
   Filed JIRA to migrate these logic to `QueryContext` and have `S3/GcsEventSource` switch to use `QueryContext`



##########
hudi-utilities/src/main/java/org/apache/hudi/utilities/sources/HoodieIncrSource.java:
##########
@@ -175,33 +181,72 @@ public Pair<Option<Dataset<Row>>, String> fetchNextBatch(Option<String> lastCkpt
     IncrSourceHelper.MissingCheckpointStrategy missingCheckpointStrategy =
         (containsConfigProperty(props, HoodieIncrSourceConfig.MISSING_CHECKPOINT_STRATEGY))
             ? IncrSourceHelper.MissingCheckpointStrategy.valueOf(
-            getStringWithAltKeys(props, HoodieIncrSourceConfig.MISSING_CHECKPOINT_STRATEGY))
+                getStringWithAltKeys(props, HoodieIncrSourceConfig.MISSING_CHECKPOINT_STRATEGY))
             : null;
     if (readLatestOnMissingCkpt) {
       missingCheckpointStrategy = IncrSourceHelper.MissingCheckpointStrategy.READ_LATEST;
     }
 
-    // Use begin Instant if set and non-empty
-    Option<String> beginInstant =
-        lastCkptStr.isPresent() ? lastCkptStr.get().isEmpty() ? Option.empty() : lastCkptStr : Option.empty();
+    HoodieTableMetaClient metaClient = HoodieTableMetaClient.builder()
+        .setConf(HadoopFSUtils.getStorageConfWithCopy(sparkContext.hadoopConfiguration()))
+        .setBasePath(srcPath)
+        .setLoadActiveTimelineOnLoad(true)
+        .build();
 
+    int numInstantsFromConfig = getIntWithAltKeys(props, HoodieIncrSourceConfig.NUM_INSTANTS_PER_FETCH);
+
+    String startTime;
+    if (lastCkptStr.isPresent() && !lastCkptStr.get().isEmpty()) {
+      startTime = lastCkptStr.get();
+    } else if (missingCheckpointStrategy != null) {
+      switch (missingCheckpointStrategy) {
+        case READ_UPTO_LATEST_COMMIT:
+          startTime = DEFAULT_BEGIN_TIMESTAMP;

Review Comment:
   Filed JIRA to migrate these logic to `QueryContext` and have `S3/GcsEventSource` switch to use `QueryContext`: https://issues.apache.org/jira/browse/HUDI-8354



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] [HUDI-8141] Incremental Query with Completion Time [hudi]

Posted by "CTTY (via GitHub)" <gi...@apache.org>.

CTTY commented on code in PR #11947:
URL: https://github.com/apache/hudi/pull/11947#discussion_r1798105503


##########
hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/hudi/HoodieMergeOnReadRDD.scala:
##########
@@ -82,7 +82,8 @@ class HoodieMergeOnReadRDD(@transient sc: SparkContext,
                            @transient fileSplits: Seq[HoodieMergeOnReadFileSplit],
                            includeStartTime: Boolean = false,
                            startTimestamp: String = null,
-                           endTimestamp: String = null)
+                           endTimestamp: String = null,
+                           includedTimestamps: Set[String] = null)

Review Comment:
   I've removed these unused variables in this PR



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] [HUDI-8141] Incremental Query with Completion Time [hudi]

Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.

hudi-bot commented on PR #11947:
URL: https://github.com/apache/hudi/pull/11947#issuecomment-2408682615

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "090f256a315cbbf0c6a54d93cc2739dd693360e6",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=484",
       "triggerID" : "090f256a315cbbf0c6a54d93cc2739dd693360e6",
       "triggerType" : "PUSH"
     }, {
       "hash" : "0df5649013c1cf9e782fcbb67b8a2e7174f94512",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=985",
       "triggerID" : "0df5649013c1cf9e782fcbb67b8a2e7174f94512",
       "triggerType" : "PUSH"
     }, {
       "hash" : "d52e0128f409918644255abbe4376dc4a780bfc8",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=986",
       "triggerID" : "d52e0128f409918644255abbe4376dc4a780bfc8",
       "triggerType" : "PUSH"
     }, {
       "hash" : "9e668f65c6d4f73c76573236f8fa507dfb80edac",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "9e668f65c6d4f73c76573236f8fa507dfb80edac",
       "triggerType" : "PUSH"
     }, {
       "hash" : "f145131d524bf190986e5de600a326e32a27beca",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=987",
       "triggerID" : "f145131d524bf190986e5de600a326e32a27beca",
       "triggerType" : "PUSH"
     }, {
       "hash" : "0b008f807b42aa6fb2686922bb88d63e84a1f299",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=993",
       "triggerID" : "0b008f807b42aa6fb2686922bb88d63e84a1f299",
       "triggerType" : "PUSH"
     }, {
       "hash" : "65c01ad9f2b33d36d4890355feef91844438e732",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=995",
       "triggerID" : "65c01ad9f2b33d36d4890355feef91844438e732",
       "triggerType" : "PUSH"
     }, {
       "hash" : "955493fdc93842e8c99529cec35e39cecdbe9638",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=996",
       "triggerID" : "955493fdc93842e8c99529cec35e39cecdbe9638",
       "triggerType" : "PUSH"
     }, {
       "hash" : "533f00e5546da0026bbe51e1c030ddd1c19fdd08",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "533f00e5546da0026bbe51e1c030ddd1c19fdd08",
       "triggerType" : "PUSH"
     }, {
       "hash" : "05777be67e37f3a6a8eaa59e54d9e17e5fc6ba9b",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1016",
       "triggerID" : "05777be67e37f3a6a8eaa59e54d9e17e5fc6ba9b",
       "triggerType" : "PUSH"
     }, {
       "hash" : "a2ee90c549332a7f17a7d9b2799849b77c08a8e6",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "a2ee90c549332a7f17a7d9b2799849b77c08a8e6",
       "triggerType" : "PUSH"
     }, {
       "hash" : "05ce568b0b6ba14da9ca40d26b39ba478837d747",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1017",
       "triggerID" : "05ce568b0b6ba14da9ca40d26b39ba478837d747",
       "triggerType" : "PUSH"
     }, {
       "hash" : "840e3b33d39b1f86367bb6b0e65eadeb33de7765",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1025",
       "triggerID" : "840e3b33d39b1f86367bb6b0e65eadeb33de7765",
       "triggerType" : "PUSH"
     }, {
       "hash" : "8d2e2dd83f51d03e890d424a51d3e0e84e904a43",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1031",
       "triggerID" : "8d2e2dd83f51d03e890d424a51d3e0e84e904a43",
       "triggerType" : "PUSH"
     }, {
       "hash" : "688381914087ff11acf2f8c9372dbf39afa0bd40",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1035",
       "triggerID" : "688381914087ff11acf2f8c9372dbf39afa0bd40",
       "triggerType" : "PUSH"
     }, {
       "hash" : "fc2c138901e82ed5f3fbf363c2a0ce8140ade050",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "fc2c138901e82ed5f3fbf363c2a0ce8140ade050",
       "triggerType" : "PUSH"
     }, {
       "hash" : "6ed9e1cb1e5a191752d6d39c7a9d5d41840f0cdf",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1039",
       "triggerID" : "6ed9e1cb1e5a191752d6d39c7a9d5d41840f0cdf",
       "triggerType" : "PUSH"
     }, {
       "hash" : "a510a7de1458bb996df5bbb341e8d010e4c91381",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "a510a7de1458bb996df5bbb341e8d010e4c91381",
       "triggerType" : "PUSH"
     }, {
       "hash" : "49677397cd117228742205944606f7a789090376",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1042",
       "triggerID" : "49677397cd117228742205944606f7a789090376",
       "triggerType" : "PUSH"
     }, {
       "hash" : "1630db0a37422aed942ff280560a9d6329df5015",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1044",
       "triggerID" : "1630db0a37422aed942ff280560a9d6329df5015",
       "triggerType" : "PUSH"
     }, {
       "hash" : "f11c76301d4ab524db0af57aa831be3a898ea0fb",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1048",
       "triggerID" : "f11c76301d4ab524db0af57aa831be3a898ea0fb",
       "triggerType" : "PUSH"
     }, {
       "hash" : "d1a0dc4c2017853a7ba0c3302f886043a63f29da",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1050",
       "triggerID" : "d1a0dc4c2017853a7ba0c3302f886043a63f29da",
       "triggerType" : "PUSH"
     }, {
       "hash" : "6a4c15ec39ded1f90b113b272554217698a98e1c",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1087",
       "triggerID" : "6a4c15ec39ded1f90b113b272554217698a98e1c",
       "triggerType" : "PUSH"
     }, {
       "hash" : "288e730d645a5e58e50a87ef5f44ba11a8fb0533",
       "status" : "FAILURE",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1092",
       "triggerID" : "288e730d645a5e58e50a87ef5f44ba11a8fb0533",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 9e668f65c6d4f73c76573236f8fa507dfb80edac UNKNOWN
   * 533f00e5546da0026bbe51e1c030ddd1c19fdd08 UNKNOWN
   * a2ee90c549332a7f17a7d9b2799849b77c08a8e6 UNKNOWN
   * fc2c138901e82ed5f3fbf363c2a0ce8140ade050 UNKNOWN
   * a510a7de1458bb996df5bbb341e8d010e4c91381 UNKNOWN
   * 288e730d645a5e58e50a87ef5f44ba11a8fb0533 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1092) 
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] [HUDI-8141] Incremental Query with Completion Time [hudi]

Posted by "yihua (via GitHub)" <gi...@apache.org>.

yihua commented on code in PR #11947:
URL: https://github.com/apache/hudi/pull/11947#discussion_r1797770211


##########
hudi-common/src/main/java/org/apache/hudi/common/table/read/IncrementalQueryAnalyzer.java:
##########
@@ -471,5 +500,20 @@ public HoodieTimeline getActiveTimeline() {
     public @Nullable HoodieTimeline getArchivedTimeline() {
       return archivedTimeline;
     }
+
+    public Option<String> getPredicateFilter() {
+      return predicateFilter;
+    }
+
+    public QueryContext withUpdatedEndInstant(String endInstant) {
+      return new QueryContext(
+          startInstant.get(),
+          endInstant,
+          instants,
+          archivedInstants,
+          activeInstants,
+          activeTimeline,archivedTimeline,
+          null);
+    }

Review Comment:
   Should we avoid this API to disallow changing the result?



##########
hudi-common/src/main/java/org/apache/hudi/common/table/timeline/CompletionTimeQueryView.java:
##########
@@ -223,6 +224,25 @@ public List<String> getStartTimes(
         InstantRange.RangeType.CLOSED_CLOSED, earliestInstantTimeFunc);
   }
 
+  /**
+   * Queries the instant start time with given completion time range.
+   *
+   * @param rangeStart              The query range start completion time.
+   * @param rangeEnd                The query range end completion time.
+   * @param rangeType               The range type.
+   *                                with the minimum completion time.
+   *
+   * @return The sorted instant time list.
+   */
+  @VisibleForTesting
+  public List<String> getStartTimes(
+      String rangeStart,
+      String rangeEnd,

Review Comment:
   ```suggestion
         String rangeStartCompletionTime,
         String rangeEndCompletionTime,
   ```



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] [HUDI-8141] Incremental Query with Completion Time [hudi]

Posted by "CTTY (via GitHub)" <gi...@apache.org>.

CTTY commented on code in PR #11947:
URL: https://github.com/apache/hudi/pull/11947#discussion_r1797849289


##########
hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/hudi/IncrementalRelation.scala:
##########
@@ -90,24 +78,25 @@ class IncrementalRelation(val sqlContext: SQLContext,
     throw new HoodieException("Incremental queries are not supported when meta fields are disabled")
   }
 
+  private val queryContext: IncrementalQueryAnalyzer.QueryContext =
+    IncrementalQueryAnalyzer.builder()
+      .metaClient(metaClient)
+      .startTime(optParams(DataSourceReadOptions.BEGIN_INSTANTTIME.key))
+      .endTime(optParams.getOrElse(DataSourceReadOptions.END_INSTANTTIME.key, null))
+      .rangeType(InstantRange.RangeType.OPEN_CLOSED)
+      .limit(optParams.getOrElse(
+        DataSourceReadOptions.INCREMENTAL_LIMIT.key,
+        DataSourceReadOptions.INCREMENTAL_LIMIT.defaultValue).toInt)
+      .build()
+      .analyze()
+
+  private val commitsToReturn = List.concat(

Review Comment:
   Good point, will sort this list based on completionTime



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] [HUDI-8141] Incremental Query with Completion Time [hudi]

Posted by "CTTY (via GitHub)" <gi...@apache.org>.

CTTY commented on code in PR #11947:
URL: https://github.com/apache/hudi/pull/11947#discussion_r1797843194


##########
hudi-spark-datasource/hudi-spark/src/main/java/org/apache/hudi/HoodieDataSourceHelpers.java:
##########
@@ -88,6 +97,16 @@ public static String latestCommit(HoodieStorage storage, String basePath) {
     return timeline.lastInstant().get().getTimestamp();
   }
 
+  public static String latestCommitCompletionTime(FileSystem fs, String basePath) {
+    HoodieTimeline timeline = allCompletedCommitsCompactions(fs, basePath);
+    return timeline.lastInstant().get().getCompletionTime();
+  }
+
+  public static String latestCommitCompletionTime(HoodieStorage storage, String basePath) {
+    HoodieTimeline timeline = allCompletedCommitsCompactions(storage, basePath);
+    return timeline.lastInstant().get().getCompletionTime();
+  }

Review Comment:
   Yes, I think it makes sense. I'll fix the completion time logic now, and will file a jira to move them to test utils because this will require changing a lot of classes.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] [HUDI-8141] Incremental Query with Completion Time [hudi]

Posted by "danny0405 (via GitHub)" <gi...@apache.org>.

danny0405 commented on code in PR #11947:
URL: https://github.com/apache/hudi/pull/11947#discussion_r1798655419


##########
hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/hudi/MergeOnReadIncrementalRelation.scala:
##########
@@ -201,55 +199,42 @@ trait HoodieIncrementalRelationTrait extends HoodieBaseRelation {
     val fallbackToFullTableScan = optParams.getOrElse(DataSourceReadOptions.INCREMENTAL_FALLBACK_TO_FULL_TABLE_SCAN.key,
       DataSourceReadOptions.INCREMENTAL_FALLBACK_TO_FULL_TABLE_SCAN.defaultValue).toBoolean
 
-    fallbackToFullTableScan && (startInstantArchived || endInstantArchived
+    fallbackToFullTableScan && (startInstantArchived
       || affectedFilesInCommits.asScala.exists(fileStatus => !metaClient.getStorage.exists(fileStatus.getPath)))
   }
 
-  protected lazy val includedCommits: immutable.Seq[HoodieInstant] = {
-    if (!startInstantArchived || !endInstantArchived) {
-      // If endTimestamp commit is not archived, will filter instants
-      // before endTimestamp.
-      if (hollowCommitHandling == USE_TRANSITION_TIME) {
-        super.timeline.findInstantsInRangeByCompletionTime(startTimestamp, endTimestamp).getInstants.asScala.toList
-      } else {
-        super.timeline.findInstantsInRange(startTimestamp, endTimestamp).getInstants.asScala.toList
-      }
-    } else {
-      super.timeline.getInstants.asScala.toList
-    }
-  }
+  protected lazy val queryContext: IncrementalQueryAnalyzer.QueryContext =
+    IncrementalQueryAnalyzer.builder()
+      .metaClient(metaClient)
+      .startTime(optParams(DataSourceReadOptions.BEGIN_INSTANTTIME.key))
+      .endTime(optParams.getOrElse(DataSourceReadOptions.END_INSTANTTIME.key, null))
+      .rangeType(InstantRange.RangeType.OPEN_CLOSED)

Review Comment:
   Why the start instant is hard coded to be ignored.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] [HUDI-8141] Incremental Query with Completion Time [hudi]

Posted by "danny0405 (via GitHub)" <gi...@apache.org>.

danny0405 commented on code in PR #11947:
URL: https://github.com/apache/hudi/pull/11947#discussion_r1798654248


##########
hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/hudi/IncrementalRelation.scala:
##########
@@ -90,24 +78,24 @@ class IncrementalRelation(val sqlContext: SQLContext,
     throw new HoodieException("Incremental queries are not supported when meta fields are disabled")
   }
 
+  private val queryContext: IncrementalQueryAnalyzer.QueryContext =
+    IncrementalQueryAnalyzer.builder()
+      .metaClient(metaClient)
+      .startTime(optParams(DataSourceReadOptions.BEGIN_INSTANTTIME.key))
+      .endTime(optParams.getOrElse(DataSourceReadOptions.END_INSTANTTIME.key, null))
+      .rangeType(InstantRange.RangeType.OPEN_CLOSED)

Review Comment:
   Why the start instant is hard code as ignored, for continuous streaming read, it may make sense but may not for batch incremental source.
   
   A more proper way is dy default to read the start instant always, and only ignore it when we are sure the read request comes from `DeltaStreamer`.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] [HUDI-8141] Incremental Query with Completion Time [hudi]

Posted by "danny0405 (via GitHub)" <gi...@apache.org>.

danny0405 commented on code in PR #11947:
URL: https://github.com/apache/hudi/pull/11947#discussion_r1798654248


##########
hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/hudi/IncrementalRelation.scala:
##########
@@ -90,24 +78,24 @@ class IncrementalRelation(val sqlContext: SQLContext,
     throw new HoodieException("Incremental queries are not supported when meta fields are disabled")
   }
 
+  private val queryContext: IncrementalQueryAnalyzer.QueryContext =
+    IncrementalQueryAnalyzer.builder()
+      .metaClient(metaClient)
+      .startTime(optParams(DataSourceReadOptions.BEGIN_INSTANTTIME.key))
+      .endTime(optParams.getOrElse(DataSourceReadOptions.END_INSTANTTIME.key, null))
+      .rangeType(InstantRange.RangeType.OPEN_CLOSED)

Review Comment:
   Why the start instant is hard code as ignored, for continuous streaming read, it may make sense but may not for batch incremental source.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] [HUDI-8141] Incremental Query with Completion Time [hudi]

Posted by "yihua (via GitHub)" <gi...@apache.org>.

yihua commented on code in PR #11947:
URL: https://github.com/apache/hudi/pull/11947#discussion_r1797486335


##########
hudi-utilities/src/main/java/org/apache/hudi/utilities/sources/HoodieIncrSource.java:
##########
@@ -175,33 +181,72 @@ public Pair<Option<Dataset<Row>>, String> fetchNextBatch(Option<String> lastCkpt
     IncrSourceHelper.MissingCheckpointStrategy missingCheckpointStrategy =
         (containsConfigProperty(props, HoodieIncrSourceConfig.MISSING_CHECKPOINT_STRATEGY))
             ? IncrSourceHelper.MissingCheckpointStrategy.valueOf(
-            getStringWithAltKeys(props, HoodieIncrSourceConfig.MISSING_CHECKPOINT_STRATEGY))
+                getStringWithAltKeys(props, HoodieIncrSourceConfig.MISSING_CHECKPOINT_STRATEGY))
             : null;
     if (readLatestOnMissingCkpt) {
       missingCheckpointStrategy = IncrSourceHelper.MissingCheckpointStrategy.READ_LATEST;
     }
 
-    // Use begin Instant if set and non-empty
-    Option<String> beginInstant =
-        lastCkptStr.isPresent() ? lastCkptStr.get().isEmpty() ? Option.empty() : lastCkptStr : Option.empty();
+    HoodieTableMetaClient metaClient = HoodieTableMetaClient.builder()
+        .setConf(HadoopFSUtils.getStorageConfWithCopy(sparkContext.hadoopConfiguration()))
+        .setBasePath(srcPath)
+        .setLoadActiveTimelineOnLoad(true)
+        .build();
 
+    int numInstantsFromConfig = getIntWithAltKeys(props, HoodieIncrSourceConfig.NUM_INSTANTS_PER_FETCH);
+
+    String startTime;
+    if (lastCkptStr.isPresent() && !lastCkptStr.get().isEmpty()) {
+      startTime = lastCkptStr.get();
+    } else if (missingCheckpointStrategy != null) {
+      switch (missingCheckpointStrategy) {
+        case READ_UPTO_LATEST_COMMIT:
+          startTime = DEFAULT_BEGIN_TIMESTAMP;

Review Comment:
   After reading more, I see that `IncrSourceHelper#generateQueryInfo` contains the following logic, introduced by #9336 and #9473.  Should we consider that too in the new completion-time-based logic?  Basically, the last checkpoint can be represented by `completionTime#key` for event incremental source.
   ```
       // When `beginInstantTime` is present, `previousInstantTime` is set to the completed commit before `beginInstantTime` if that exists.
       // If there is no completed commit before `beginInstantTime`, e.g., `beginInstantTime` is the first commit in the active timeline,
       // `previousInstantTime` is set to `DEFAULT_BEGIN_TIMESTAMP`.
       String previousInstantTime = DEFAULT_BEGIN_TIMESTAMP;
       if (!beginInstantTime.equals(DEFAULT_BEGIN_TIMESTAMP)) {
         Option<HoodieInstant> previousInstant = activeCommitTimeline.findInstantBefore(beginInstantTime);
         if (previousInstant.isPresent()) {
           previousInstantTime = previousInstant.get().getTimestamp();
         } else {
           // if begin instant time matches first entry in active timeline, we can set previous = beginInstantTime - 1
           if (activeCommitTimeline.filterCompletedInstants().firstInstant().isPresent()
               && activeCommitTimeline.filterCompletedInstants().firstInstant().get().getTimestamp().equals(beginInstantTime)) {
             previousInstantTime = String.valueOf(Long.parseLong(beginInstantTime) - 1);
           }
         }
       }
   
       if (missingCheckpointStrategy == MissingCheckpointStrategy.READ_LATEST || !activeCommitTimeline.isBeforeTimelineStarts(beginInstantTime)) {
         Option<HoodieInstant> nthInstant;
         // When we are in the upgrade code path from non-sourcelimit-based batching to sourcelimit-based batching, we need to avoid fetching the commit
         // that is read already. Else we will have duplicates in append-only use case if we use "findInstantsAfterOrEquals".
         // As soon as we have a new format of checkpoint and a key we will move to the new code of fetching the current commit as well.
         if (sourceLimitBasedBatching && lastCheckpointKey.isPresent()) {
           nthInstant = Option.fromJavaOptional(activeCommitTimeline
               .findInstantsAfterOrEquals(beginInstantTime, numInstantsPerFetch).getInstantsAsStream().reduce((x, y) -> y));
         } else {
           nthInstant = Option.fromJavaOptional(activeCommitTimeline
               .findInstantsAfter(beginInstantTime, numInstantsPerFetch).getInstantsAsStream().reduce((x, y) -> y));
         }
         return new QueryInfo(DataSourceReadOptions.QUERY_TYPE_INCREMENTAL_OPT_VAL(), previousInstantTime,
             beginInstantTime, nthInstant.map(HoodieInstant::getTimestamp).orElse(beginInstantTime),
             orderColumn, keyColumn, limitColumn);
       } else {
         // when MissingCheckpointStrategy is set to read everything until latest, trigger snapshot query.
         Option<HoodieInstant> lastInstant = activeCommitTimeline.lastInstant();
         return new QueryInfo(DataSourceReadOptions.QUERY_TYPE_SNAPSHOT_OPT_VAL(),
             previousInstantTime, beginInstantTime, lastInstant.get().getTimestamp(),
             orderColumn, keyColumn, limitColumn);
       }
   ```



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] [HUDI-8141] Incremental Query with Completion Time [hudi]

Posted by "CTTY (via GitHub)" <gi...@apache.org>.

CTTY commented on code in PR #11947:
URL: https://github.com/apache/hudi/pull/11947#discussion_r1797749269


##########
hudi-utilities/src/main/java/org/apache/hudi/utilities/sources/HoodieIncrSource.java:
##########
@@ -212,35 +257,43 @@ public Pair<Option<Dataset<Row>>, String> fetchNextBatch(Option<String> lastCkpt
           .collect(Collectors.toMap(Pair::getLeft, Pair::getRight));
       reader = reader.options(optionsMap);
     }
+    boolean shouldFullScan =
+        queryContext.getActiveTimeline().isBeforeTimelineStartsByCompletionTime(startTime)
+        && missingCheckpointStrategy == MissingCheckpointStrategy.READ_UPTO_LATEST_COMMIT;
     Dataset<Row> source;
-    // Do Incr pull. Set end instant if available
-    if (queryInfo.isIncremental()) {
-      source = reader
-          .options(readOpts)
-          .option(QUERY_TYPE().key(), QUERY_TYPE_INCREMENTAL_OPT_VAL())
-          .option(BEGIN_INSTANTTIME().key(), queryInfo.getStartInstant())
-          .option(END_INSTANTTIME().key(), queryInfo.getEndInstant())
-          .option(INCREMENTAL_FALLBACK_TO_FULL_TABLE_SCAN().key(),
-              props.getString(INCREMENTAL_FALLBACK_TO_FULL_TABLE_SCAN().key(),
-                  INCREMENTAL_FALLBACK_TO_FULL_TABLE_SCAN().defaultValue()))
-          .option(INCREMENTAL_READ_HANDLE_HOLLOW_COMMIT().key(), handlingMode.name())
-          .load(srcPath);
-    } else {
-      // if checkpoint is missing from source table, and if strategy is set to READ_UPTO_LATEST_COMMIT, we have to issue snapshot query
+    if (instantRange.isEmpty() || shouldFullScan) {
+      // snapshot query
       Dataset<Row> snapshot = reader
           .options(readOpts)
           .option(DataSourceReadOptions.QUERY_TYPE().key(), DataSourceReadOptions.QUERY_TYPE_SNAPSHOT_OPT_VAL())
           .load(srcPath);
       if (snapshotLoadQuerySplitter.isPresent()) {
-        queryInfo = snapshotLoadQuerySplitter.get().getNextCheckpoint(snapshot, queryInfo, sourceProfileSupplier);
+        queryContext = snapshotLoadQuerySplitter.get().getNextCheckpoint(snapshot, queryContext, sourceProfileSupplier);
+        // update endTime/next checkpoint
+        endTime = queryContext.getEndInstant().orElse(queryContext.getLastInstant());
       }
+      Set<String> validInstants = new HashSet<>(queryContext.getInstants());
+      snapshot = queryContext.getPredicateFilter().map(snapshot::filter).orElse(snapshot);
       source = snapshot
           // add filtering so that only interested records are returned.
-          .filter(String.format("%s > '%s'", HoodieRecord.COMMIT_TIME_METADATA_FIELD,
-              queryInfo.getStartInstant()))
+          // completion time comparison uses ( , ], but when comparing start time we need to use [, ]
+          .filter(String.format("%s >= '%s'", HoodieRecord.COMMIT_TIME_METADATA_FIELD,
+              queryContext.getStartInstant().get()))
           .filter(String.format("%s <= '%s'", HoodieRecord.COMMIT_TIME_METADATA_FIELD,
-              queryInfo.getEndInstant()));
-      source = queryInfo.getPredicateFilter().map(source::filter).orElse(source);
+              queryContext.getEndInstant().orElse(queryContext.getLastInstant())))
+          .filter((Row row) -> validInstants.contains(row.getAs(HoodieRecord.COMMIT_TIME_METADATA_FIELD)));

Review Comment:
   This cannot be pushed down to any existing Hudi incremental relation,because it's running a full table scan. This predicate has to be written like this because of the limitation of using Java. Maybe we could add a scala util like `JavaScalaConverters` to make this predicate more elegant? 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] [HUDI-8141] Incremental Query with Completion Time [hudi]

Posted by "yihua (via GitHub)" <gi...@apache.org>.

yihua commented on code in PR #11947:
URL: https://github.com/apache/hudi/pull/11947#discussion_r1797771592


##########
hudi-common/src/main/java/org/apache/hudi/common/table/timeline/CompletionTimeQueryView.java:
##########
@@ -223,6 +224,25 @@ public List<String> getStartTimes(
         InstantRange.RangeType.CLOSED_CLOSED, earliestInstantTimeFunc);
   }
 
+  /**
+   * Queries the instant start time with given completion time range.
+   *
+   * @param rangeStart              The query range start completion time.
+   * @param rangeEnd                The query range end completion time.
+   * @param rangeType               The range type.
+   *                                with the minimum completion time.
+   *
+   * @return The sorted instant time list.
+   */
+  @VisibleForTesting
+  public List<String> getStartTimes(
+      String rangeStart,
+      String rangeEnd,
+      RangeType rangeType) {
+    return getStartTimes(metaClient.getCommitsTimeline().filterCompletedInstants(), Option.ofNullable(rangeStart), Option.ofNullable(rangeEnd),
+        rangeType, s -> HoodieInstantTimeGenerator.instantTimeMinusMillis(s, MILLI_SECONDS_IN_ONE_DAY));

Review Comment:
   Could we extract `s -> HoodieInstantTimeGenerator.instantTimeMinusMillis(s, MILLI_SECONDS_IN_ONE_DAY)` to a final variable so the assumption is clear?  I think this is also copied from another place.



##########
hudi-common/src/main/java/org/apache/hudi/common/table/timeline/HoodieDefaultTimeline.java:
##########
@@ -532,6 +532,13 @@ public boolean isBeforeTimelineStarts(String instant) {
         && compareTimestamps(instant, LESSER_THAN, firstNonSavepointCommit.get().getTimestamp());
   }
 
+  @Override
+  public boolean isBeforeTimelineStartsByCompletionTime(String completionTime) {
+    Option<HoodieInstant> firstNonSavepointCommit = getFirstNonSavepointCommit();

Review Comment:
   `getFirstNonSavepointCommit()`uses the sorted instants by (start) instant time.  Here it needs the instant with the smallest completion time.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] [HUDI-8141] Incremental Query with Completion Time [hudi]

Posted by "yihua (via GitHub)" <gi...@apache.org>.

yihua commented on code in PR #11947:
URL: https://github.com/apache/hudi/pull/11947#discussion_r1797472112


##########
hudi-utilities/src/main/java/org/apache/hudi/utilities/sources/HoodieIncrSource.java:
##########
@@ -175,33 +181,72 @@ public Pair<Option<Dataset<Row>>, String> fetchNextBatch(Option<String> lastCkpt
     IncrSourceHelper.MissingCheckpointStrategy missingCheckpointStrategy =
         (containsConfigProperty(props, HoodieIncrSourceConfig.MISSING_CHECKPOINT_STRATEGY))
             ? IncrSourceHelper.MissingCheckpointStrategy.valueOf(
-            getStringWithAltKeys(props, HoodieIncrSourceConfig.MISSING_CHECKPOINT_STRATEGY))
+                getStringWithAltKeys(props, HoodieIncrSourceConfig.MISSING_CHECKPOINT_STRATEGY))
             : null;
     if (readLatestOnMissingCkpt) {
       missingCheckpointStrategy = IncrSourceHelper.MissingCheckpointStrategy.READ_LATEST;
     }
 
-    // Use begin Instant if set and non-empty
-    Option<String> beginInstant =
-        lastCkptStr.isPresent() ? lastCkptStr.get().isEmpty() ? Option.empty() : lastCkptStr : Option.empty();
+    HoodieTableMetaClient metaClient = HoodieTableMetaClient.builder()
+        .setConf(HadoopFSUtils.getStorageConfWithCopy(sparkContext.hadoopConfiguration()))
+        .setBasePath(srcPath)
+        .setLoadActiveTimelineOnLoad(true)
+        .build();
 
+    int numInstantsFromConfig = getIntWithAltKeys(props, HoodieIncrSourceConfig.NUM_INSTANTS_PER_FETCH);
+
+    String startTime;

Review Comment:
   ```suggestion
       String completionTimeToResumeFrom;
   ```



##########
hudi-utilities/src/main/java/org/apache/hudi/utilities/sources/HoodieIncrSource.java:
##########
@@ -175,33 +181,72 @@ public Pair<Option<Dataset<Row>>, String> fetchNextBatch(Option<String> lastCkpt
     IncrSourceHelper.MissingCheckpointStrategy missingCheckpointStrategy =
         (containsConfigProperty(props, HoodieIncrSourceConfig.MISSING_CHECKPOINT_STRATEGY))
             ? IncrSourceHelper.MissingCheckpointStrategy.valueOf(
-            getStringWithAltKeys(props, HoodieIncrSourceConfig.MISSING_CHECKPOINT_STRATEGY))
+                getStringWithAltKeys(props, HoodieIncrSourceConfig.MISSING_CHECKPOINT_STRATEGY))
             : null;
     if (readLatestOnMissingCkpt) {
       missingCheckpointStrategy = IncrSourceHelper.MissingCheckpointStrategy.READ_LATEST;
     }
 
-    // Use begin Instant if set and non-empty
-    Option<String> beginInstant =
-        lastCkptStr.isPresent() ? lastCkptStr.get().isEmpty() ? Option.empty() : lastCkptStr : Option.empty();
+    HoodieTableMetaClient metaClient = HoodieTableMetaClient.builder()
+        .setConf(HadoopFSUtils.getStorageConfWithCopy(sparkContext.hadoopConfiguration()))
+        .setBasePath(srcPath)
+        .setLoadActiveTimelineOnLoad(true)
+        .build();
 
+    int numInstantsFromConfig = getIntWithAltKeys(props, HoodieIncrSourceConfig.NUM_INSTANTS_PER_FETCH);
+
+    String startTime;

Review Comment:
   This can be a `Option<String>` (see my comment below).



##########
hudi-utilities/src/main/java/org/apache/hudi/utilities/sources/HoodieIncrSource.java:
##########
@@ -175,33 +181,72 @@ public Pair<Option<Dataset<Row>>, String> fetchNextBatch(Option<String> lastCkpt
     IncrSourceHelper.MissingCheckpointStrategy missingCheckpointStrategy =
         (containsConfigProperty(props, HoodieIncrSourceConfig.MISSING_CHECKPOINT_STRATEGY))
             ? IncrSourceHelper.MissingCheckpointStrategy.valueOf(
-            getStringWithAltKeys(props, HoodieIncrSourceConfig.MISSING_CHECKPOINT_STRATEGY))
+                getStringWithAltKeys(props, HoodieIncrSourceConfig.MISSING_CHECKPOINT_STRATEGY))
             : null;
     if (readLatestOnMissingCkpt) {
       missingCheckpointStrategy = IncrSourceHelper.MissingCheckpointStrategy.READ_LATEST;
     }
 
-    // Use begin Instant if set and non-empty
-    Option<String> beginInstant =
-        lastCkptStr.isPresent() ? lastCkptStr.get().isEmpty() ? Option.empty() : lastCkptStr : Option.empty();
+    HoodieTableMetaClient metaClient = HoodieTableMetaClient.builder()
+        .setConf(HadoopFSUtils.getStorageConfWithCopy(sparkContext.hadoopConfiguration()))
+        .setBasePath(srcPath)
+        .setLoadActiveTimelineOnLoad(true)
+        .build();
 
+    int numInstantsFromConfig = getIntWithAltKeys(props, HoodieIncrSourceConfig.NUM_INSTANTS_PER_FETCH);
+
+    String startTime;
+    if (lastCkptStr.isPresent() && !lastCkptStr.get().isEmpty()) {
+      startTime = lastCkptStr.get();
+    } else if (missingCheckpointStrategy != null) {
+      switch (missingCheckpointStrategy) {
+        case READ_UPTO_LATEST_COMMIT:
+          startTime = DEFAULT_BEGIN_TIMESTAMP;

Review Comment:
   To be consistent with the `IncrementalQueryAnalyzer` semantics, it's better to make `completionTimeToResumeFrom` `null` or `Option.empty` instead of `DEFAULT_BEGIN_TIMESTAMP` (we can remove `DEFAULT_BEGIN_TIMESTAMP` if not used anywhere else).  We pass `null`/`Option.empty` to `IncrementalQueryAnalyzer` as the begin time.



##########
hudi-utilities/src/main/java/org/apache/hudi/utilities/sources/HoodieIncrSource.java:
##########
@@ -175,33 +181,72 @@ public Pair<Option<Dataset<Row>>, String> fetchNextBatch(Option<String> lastCkpt
     IncrSourceHelper.MissingCheckpointStrategy missingCheckpointStrategy =
         (containsConfigProperty(props, HoodieIncrSourceConfig.MISSING_CHECKPOINT_STRATEGY))
             ? IncrSourceHelper.MissingCheckpointStrategy.valueOf(
-            getStringWithAltKeys(props, HoodieIncrSourceConfig.MISSING_CHECKPOINT_STRATEGY))
+                getStringWithAltKeys(props, HoodieIncrSourceConfig.MISSING_CHECKPOINT_STRATEGY))
             : null;
     if (readLatestOnMissingCkpt) {
       missingCheckpointStrategy = IncrSourceHelper.MissingCheckpointStrategy.READ_LATEST;
     }
 
-    // Use begin Instant if set and non-empty
-    Option<String> beginInstant =
-        lastCkptStr.isPresent() ? lastCkptStr.get().isEmpty() ? Option.empty() : lastCkptStr : Option.empty();
+    HoodieTableMetaClient metaClient = HoodieTableMetaClient.builder()
+        .setConf(HadoopFSUtils.getStorageConfWithCopy(sparkContext.hadoopConfiguration()))
+        .setBasePath(srcPath)
+        .setLoadActiveTimelineOnLoad(true)
+        .build();
 
+    int numInstantsFromConfig = getIntWithAltKeys(props, HoodieIncrSourceConfig.NUM_INSTANTS_PER_FETCH);
+
+    String startTime;
+    if (lastCkptStr.isPresent() && !lastCkptStr.get().isEmpty()) {
+      startTime = lastCkptStr.get();
+    } else if (missingCheckpointStrategy != null) {
+      switch (missingCheckpointStrategy) {
+        case READ_UPTO_LATEST_COMMIT:
+          startTime = DEFAULT_BEGIN_TIMESTAMP;
+          // disrespect numInstantsFromConfig when reading up to latest
+          numInstantsFromConfig = -1;
+          break;
+        case READ_LATEST:
+          Option<HoodieInstant> lastInstant = metaClient
+              .getCommitsAndCompactionTimeline()
+              .filterCompletedInstants()
+              .lastInstant();
+          startTime = lastInstant
+              .map(hoodieInstant -> getStrictlyLowerTimestamp(hoodieInstant.getCompletionTime()))
+              .orElse(DEFAULT_BEGIN_TIMESTAMP);
+          break;
+        default:
+          throw new IllegalArgumentException("Unknown missing checkpoint strategy: " + missingCheckpointStrategy);
+      }
+    } else {
+      throw new IllegalArgumentException("Missing begin instant for incremental pull. For reading from latest "
+          + "committed instant set hoodie.streamer.source.hoodieincr.missing.checkpoint.strategy to a valid value");
+    }
+
+    final int numInstantsFromConfigFinal = numInstantsFromConfig;
     // If source profile exists, use the numInstants from source profile.
-    final int numInstantsFromConfig = getIntWithAltKeys(props, HoodieIncrSourceConfig.NUM_INSTANTS_PER_FETCH);
     int numInstantsPerFetch = getLatestSourceProfile().map(sourceProfile -> {
       int numInstantsFromSourceProfile = sourceProfile.getSourceSpecificContext();
-      LOG.info("Overriding numInstantsPerFetch from source profile numInstantsFromSourceProfile {} , numInstantsFromConfig {}", numInstantsFromSourceProfile, numInstantsFromConfig);
+      LOG.info("Overriding numInstantsPerFetch from source profile numInstantsFromSourceProfile {} , numInstantsFromConfig {}",
+          numInstantsFromSourceProfile, numInstantsFromConfigFinal);
       return numInstantsFromSourceProfile;
     }).orElse(numInstantsFromConfig);
 
-    HollowCommitHandling handlingMode = getHollowCommitHandleMode(props);
-    QueryInfo queryInfo = generateQueryInfo(sparkContext, srcPath,
-        numInstantsPerFetch, beginInstant, missingCheckpointStrategy, handlingMode,
-        HoodieRecord.COMMIT_TIME_METADATA_FIELD, HoodieRecord.RECORD_KEY_METADATA_FIELD,
-        null, false, Option.empty());
+    IncrementalQueryAnalyzer analyzer = IncrementalQueryAnalyzer.builder()
+        .metaClient(metaClient)
+        .startTime(startTime)
+        .endTime(null)
+        .rangeType(RangeType.OPEN_CLOSED)
+        .limit(numInstantsPerFetch)
+        .build();
+
+    QueryContext queryContext = analyzer.analyze();
+    Option<InstantRange> instantRange = queryContext.getInstantRange();
 
-    if (queryInfo.areStartAndEndInstantsEqual()) {
+    String endTime;

Review Comment:
   ```suggestion
       String completionTimeToStopAt;
   ```



##########
hudi-utilities/src/main/java/org/apache/hudi/utilities/sources/HoodieIncrSource.java:
##########
@@ -175,33 +181,72 @@ public Pair<Option<Dataset<Row>>, String> fetchNextBatch(Option<String> lastCkpt
     IncrSourceHelper.MissingCheckpointStrategy missingCheckpointStrategy =
         (containsConfigProperty(props, HoodieIncrSourceConfig.MISSING_CHECKPOINT_STRATEGY))
             ? IncrSourceHelper.MissingCheckpointStrategy.valueOf(
-            getStringWithAltKeys(props, HoodieIncrSourceConfig.MISSING_CHECKPOINT_STRATEGY))
+                getStringWithAltKeys(props, HoodieIncrSourceConfig.MISSING_CHECKPOINT_STRATEGY))
             : null;
     if (readLatestOnMissingCkpt) {
       missingCheckpointStrategy = IncrSourceHelper.MissingCheckpointStrategy.READ_LATEST;
     }
 
-    // Use begin Instant if set and non-empty
-    Option<String> beginInstant =
-        lastCkptStr.isPresent() ? lastCkptStr.get().isEmpty() ? Option.empty() : lastCkptStr : Option.empty();
+    HoodieTableMetaClient metaClient = HoodieTableMetaClient.builder()
+        .setConf(HadoopFSUtils.getStorageConfWithCopy(sparkContext.hadoopConfiguration()))
+        .setBasePath(srcPath)
+        .setLoadActiveTimelineOnLoad(true)
+        .build();
 
+    int numInstantsFromConfig = getIntWithAltKeys(props, HoodieIncrSourceConfig.NUM_INSTANTS_PER_FETCH);
+
+    String startTime;
+    if (lastCkptStr.isPresent() && !lastCkptStr.get().isEmpty()) {
+      startTime = lastCkptStr.get();
+    } else if (missingCheckpointStrategy != null) {
+      switch (missingCheckpointStrategy) {
+        case READ_UPTO_LATEST_COMMIT:
+          startTime = DEFAULT_BEGIN_TIMESTAMP;
+          // disrespect numInstantsFromConfig when reading up to latest
+          numInstantsFromConfig = -1;
+          break;
+        case READ_LATEST:
+          Option<HoodieInstant> lastInstant = metaClient
+              .getCommitsAndCompactionTimeline()
+              .filterCompletedInstants()
+              .lastInstant();
+          startTime = lastInstant
+              .map(hoodieInstant -> getStrictlyLowerTimestamp(hoodieInstant.getCompletionTime()))
+              .orElse(DEFAULT_BEGIN_TIMESTAMP);

Review Comment:
   Similar here on using `null` or `Option.empty`.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] [HUDI-8141] WIP:Always use completion time to find begin Instant for incremental source [hudi]

Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.

hudi-bot commented on PR #11947:
URL: https://github.com/apache/hudi/pull/11947#issuecomment-2403784553

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "090f256a315cbbf0c6a54d93cc2739dd693360e6",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=484",
       "triggerID" : "090f256a315cbbf0c6a54d93cc2739dd693360e6",
       "triggerType" : "PUSH"
     }, {
       "hash" : "0df5649013c1cf9e782fcbb67b8a2e7174f94512",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=985",
       "triggerID" : "0df5649013c1cf9e782fcbb67b8a2e7174f94512",
       "triggerType" : "PUSH"
     }, {
       "hash" : "d52e0128f409918644255abbe4376dc4a780bfc8",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=986",
       "triggerID" : "d52e0128f409918644255abbe4376dc4a780bfc8",
       "triggerType" : "PUSH"
     }, {
       "hash" : "9e668f65c6d4f73c76573236f8fa507dfb80edac",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "9e668f65c6d4f73c76573236f8fa507dfb80edac",
       "triggerType" : "PUSH"
     }, {
       "hash" : "f145131d524bf190986e5de600a326e32a27beca",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=987",
       "triggerID" : "f145131d524bf190986e5de600a326e32a27beca",
       "triggerType" : "PUSH"
     }, {
       "hash" : "0b008f807b42aa6fb2686922bb88d63e84a1f299",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=993",
       "triggerID" : "0b008f807b42aa6fb2686922bb88d63e84a1f299",
       "triggerType" : "PUSH"
     }, {
       "hash" : "65c01ad9f2b33d36d4890355feef91844438e732",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=995",
       "triggerID" : "65c01ad9f2b33d36d4890355feef91844438e732",
       "triggerType" : "PUSH"
     }, {
       "hash" : "955493fdc93842e8c99529cec35e39cecdbe9638",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=996",
       "triggerID" : "955493fdc93842e8c99529cec35e39cecdbe9638",
       "triggerType" : "PUSH"
     }, {
       "hash" : "533f00e5546da0026bbe51e1c030ddd1c19fdd08",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "533f00e5546da0026bbe51e1c030ddd1c19fdd08",
       "triggerType" : "PUSH"
     }, {
       "hash" : "05777be67e37f3a6a8eaa59e54d9e17e5fc6ba9b",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1016",
       "triggerID" : "05777be67e37f3a6a8eaa59e54d9e17e5fc6ba9b",
       "triggerType" : "PUSH"
     }, {
       "hash" : "a2ee90c549332a7f17a7d9b2799849b77c08a8e6",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "a2ee90c549332a7f17a7d9b2799849b77c08a8e6",
       "triggerType" : "PUSH"
     }, {
       "hash" : "05ce568b0b6ba14da9ca40d26b39ba478837d747",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1017",
       "triggerID" : "05ce568b0b6ba14da9ca40d26b39ba478837d747",
       "triggerType" : "PUSH"
     }, {
       "hash" : "840e3b33d39b1f86367bb6b0e65eadeb33de7765",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1025",
       "triggerID" : "840e3b33d39b1f86367bb6b0e65eadeb33de7765",
       "triggerType" : "PUSH"
     }, {
       "hash" : "8d2e2dd83f51d03e890d424a51d3e0e84e904a43",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1031",
       "triggerID" : "8d2e2dd83f51d03e890d424a51d3e0e84e904a43",
       "triggerType" : "PUSH"
     }, {
       "hash" : "688381914087ff11acf2f8c9372dbf39afa0bd40",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1035",
       "triggerID" : "688381914087ff11acf2f8c9372dbf39afa0bd40",
       "triggerType" : "PUSH"
     }, {
       "hash" : "fc2c138901e82ed5f3fbf363c2a0ce8140ade050",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "fc2c138901e82ed5f3fbf363c2a0ce8140ade050",
       "triggerType" : "PUSH"
     }, {
       "hash" : "6ed9e1cb1e5a191752d6d39c7a9d5d41840f0cdf",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1039",
       "triggerID" : "6ed9e1cb1e5a191752d6d39c7a9d5d41840f0cdf",
       "triggerType" : "PUSH"
     }, {
       "hash" : "a510a7de1458bb996df5bbb341e8d010e4c91381",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "a510a7de1458bb996df5bbb341e8d010e4c91381",
       "triggerType" : "PUSH"
     }, {
       "hash" : "49677397cd117228742205944606f7a789090376",
       "status" : "CANCELED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1042",
       "triggerID" : "49677397cd117228742205944606f7a789090376",
       "triggerType" : "PUSH"
     }, {
       "hash" : "1630db0a37422aed942ff280560a9d6329df5015",
       "status" : "PENDING",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1044",
       "triggerID" : "1630db0a37422aed942ff280560a9d6329df5015",
       "triggerType" : "PUSH"
     }, {
       "hash" : "f11c76301d4ab524db0af57aa831be3a898ea0fb",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "f11c76301d4ab524db0af57aa831be3a898ea0fb",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 9e668f65c6d4f73c76573236f8fa507dfb80edac UNKNOWN
   * 533f00e5546da0026bbe51e1c030ddd1c19fdd08 UNKNOWN
   * a2ee90c549332a7f17a7d9b2799849b77c08a8e6 UNKNOWN
   * fc2c138901e82ed5f3fbf363c2a0ce8140ade050 UNKNOWN
   * a510a7de1458bb996df5bbb341e8d010e4c91381 UNKNOWN
   * 49677397cd117228742205944606f7a789090376 Azure: [CANCELED](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1042) 
   * 1630db0a37422aed942ff280560a9d6329df5015 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1044) 
   * f11c76301d4ab524db0af57aa831be3a898ea0fb UNKNOWN
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] [HUDI-8141] Always use completion time to find begin Instant for incremental source [hudi]

Posted by "CTTY (via GitHub)" <gi...@apache.org>.

CTTY commented on PR #11947:
URL: https://github.com/apache/hudi/pull/11947#issuecomment-2404114816

   <img width="521" alt="Screenshot 2024-10-09 at 11 05 43 PM" src="https://github.com/user-attachments/assets/ef3edf9d-57b8-4ce9-ad97-4f7c9b405552">
   There is a test failure in azure CI but I cannot reproduce the issue locally, seems like a flaky test


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] [HUDI-8141] WIP:Always use completion time to find begin Instant for incremental source [hudi]

Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.

hudi-bot commented on PR #11947:
URL: https://github.com/apache/hudi/pull/11947#issuecomment-2351153464

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "090f256a315cbbf0c6a54d93cc2739dd693360e6",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "090f256a315cbbf0c6a54d93cc2739dd693360e6",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 090f256a315cbbf0c6a54d93cc2739dd693360e6 UNKNOWN
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] [HUDI-8141] Incremental Query with Completion Time [hudi]

Posted by "yihua (via GitHub)" <gi...@apache.org>.

yihua commented on code in PR #11947:
URL: https://github.com/apache/hudi/pull/11947#discussion_r1802427060


##########
hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/spark/sql/hudi/streaming/HoodieStreamSource.scala:
##########
@@ -114,19 +99,14 @@ class HoodieStreamSource(
 
   private def getLatestOffset: Option[HoodieSourceOffset] = {
     metaClient.reloadActiveTimeline()
-    val filteredTimeline = handleHollowCommitIfNeeded(
-      metaClient.getActiveTimeline.filterCompletedInstants(), metaClient, hollowCommitHandling)
+    val filteredTimeline = metaClient.getActiveTimeline.filterCompletedInstants()
     filteredTimeline match {
       case activeInstants if !activeInstants.empty() =>
-        val timestamp = if (hollowCommitHandling == USE_TRANSITION_TIME) {
-          activeInstants.getInstantsOrderedByCompletionTime
-            .skip(activeInstants.countInstants() - 1)
-            .findFirst()
-            .get()
-            .getCompletionTime
-        } else {
-          activeInstants.lastInstant().get().getTimestamp
-        }
+        val timestamp = activeInstants.getInstantsOrderedByCompletionTime
+          .skip(activeInstants.countInstants() - 1)

Review Comment:
   Fixed.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] [HUDI-8141] Incremental Query with Completion Time [hudi]

Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.

hudi-bot commented on PR #11947:
URL: https://github.com/apache/hudi/pull/11947#issuecomment-2415970247

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "090f256a315cbbf0c6a54d93cc2739dd693360e6",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=484",
       "triggerID" : "090f256a315cbbf0c6a54d93cc2739dd693360e6",
       "triggerType" : "PUSH"
     }, {
       "hash" : "0df5649013c1cf9e782fcbb67b8a2e7174f94512",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=985",
       "triggerID" : "0df5649013c1cf9e782fcbb67b8a2e7174f94512",
       "triggerType" : "PUSH"
     }, {
       "hash" : "d52e0128f409918644255abbe4376dc4a780bfc8",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=986",
       "triggerID" : "d52e0128f409918644255abbe4376dc4a780bfc8",
       "triggerType" : "PUSH"
     }, {
       "hash" : "9e668f65c6d4f73c76573236f8fa507dfb80edac",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "9e668f65c6d4f73c76573236f8fa507dfb80edac",
       "triggerType" : "PUSH"
     }, {
       "hash" : "f145131d524bf190986e5de600a326e32a27beca",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=987",
       "triggerID" : "f145131d524bf190986e5de600a326e32a27beca",
       "triggerType" : "PUSH"
     }, {
       "hash" : "0b008f807b42aa6fb2686922bb88d63e84a1f299",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=993",
       "triggerID" : "0b008f807b42aa6fb2686922bb88d63e84a1f299",
       "triggerType" : "PUSH"
     }, {
       "hash" : "65c01ad9f2b33d36d4890355feef91844438e732",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=995",
       "triggerID" : "65c01ad9f2b33d36d4890355feef91844438e732",
       "triggerType" : "PUSH"
     }, {
       "hash" : "955493fdc93842e8c99529cec35e39cecdbe9638",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=996",
       "triggerID" : "955493fdc93842e8c99529cec35e39cecdbe9638",
       "triggerType" : "PUSH"
     }, {
       "hash" : "533f00e5546da0026bbe51e1c030ddd1c19fdd08",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "533f00e5546da0026bbe51e1c030ddd1c19fdd08",
       "triggerType" : "PUSH"
     }, {
       "hash" : "05777be67e37f3a6a8eaa59e54d9e17e5fc6ba9b",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1016",
       "triggerID" : "05777be67e37f3a6a8eaa59e54d9e17e5fc6ba9b",
       "triggerType" : "PUSH"
     }, {
       "hash" : "a2ee90c549332a7f17a7d9b2799849b77c08a8e6",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "a2ee90c549332a7f17a7d9b2799849b77c08a8e6",
       "triggerType" : "PUSH"
     }, {
       "hash" : "05ce568b0b6ba14da9ca40d26b39ba478837d747",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1017",
       "triggerID" : "05ce568b0b6ba14da9ca40d26b39ba478837d747",
       "triggerType" : "PUSH"
     }, {
       "hash" : "840e3b33d39b1f86367bb6b0e65eadeb33de7765",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1025",
       "triggerID" : "840e3b33d39b1f86367bb6b0e65eadeb33de7765",
       "triggerType" : "PUSH"
     }, {
       "hash" : "8d2e2dd83f51d03e890d424a51d3e0e84e904a43",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1031",
       "triggerID" : "8d2e2dd83f51d03e890d424a51d3e0e84e904a43",
       "triggerType" : "PUSH"
     }, {
       "hash" : "688381914087ff11acf2f8c9372dbf39afa0bd40",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1035",
       "triggerID" : "688381914087ff11acf2f8c9372dbf39afa0bd40",
       "triggerType" : "PUSH"
     }, {
       "hash" : "fc2c138901e82ed5f3fbf363c2a0ce8140ade050",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "fc2c138901e82ed5f3fbf363c2a0ce8140ade050",
       "triggerType" : "PUSH"
     }, {
       "hash" : "6ed9e1cb1e5a191752d6d39c7a9d5d41840f0cdf",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1039",
       "triggerID" : "6ed9e1cb1e5a191752d6d39c7a9d5d41840f0cdf",
       "triggerType" : "PUSH"
     }, {
       "hash" : "a510a7de1458bb996df5bbb341e8d010e4c91381",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "a510a7de1458bb996df5bbb341e8d010e4c91381",
       "triggerType" : "PUSH"
     }, {
       "hash" : "49677397cd117228742205944606f7a789090376",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1042",
       "triggerID" : "49677397cd117228742205944606f7a789090376",
       "triggerType" : "PUSH"
     }, {
       "hash" : "1630db0a37422aed942ff280560a9d6329df5015",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1044",
       "triggerID" : "1630db0a37422aed942ff280560a9d6329df5015",
       "triggerType" : "PUSH"
     }, {
       "hash" : "f11c76301d4ab524db0af57aa831be3a898ea0fb",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1048",
       "triggerID" : "f11c76301d4ab524db0af57aa831be3a898ea0fb",
       "triggerType" : "PUSH"
     }, {
       "hash" : "d1a0dc4c2017853a7ba0c3302f886043a63f29da",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1050",
       "triggerID" : "d1a0dc4c2017853a7ba0c3302f886043a63f29da",
       "triggerType" : "PUSH"
     }, {
       "hash" : "6a4c15ec39ded1f90b113b272554217698a98e1c",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1087",
       "triggerID" : "6a4c15ec39ded1f90b113b272554217698a98e1c",
       "triggerType" : "PUSH"
     }, {
       "hash" : "288e730d645a5e58e50a87ef5f44ba11a8fb0533",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1092",
       "triggerID" : "288e730d645a5e58e50a87ef5f44ba11a8fb0533",
       "triggerType" : "PUSH"
     }, {
       "hash" : "70a5c21321a0da3a0be8d85e206c24befa7b89f1",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1095",
       "triggerID" : "70a5c21321a0da3a0be8d85e206c24befa7b89f1",
       "triggerType" : "PUSH"
     }, {
       "hash" : "f072a986114949c5e426ce64a5f5568fc718e25d",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "f072a986114949c5e426ce64a5f5568fc718e25d",
       "triggerType" : "PUSH"
     }, {
       "hash" : "45b27475b1e8806ec8c566d6f151f4bc79c8549d",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1096",
       "triggerID" : "45b27475b1e8806ec8c566d6f151f4bc79c8549d",
       "triggerType" : "PUSH"
     }, {
       "hash" : "3c62ef22f6cde28110e5eb2fca8760a008d77723",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "3c62ef22f6cde28110e5eb2fca8760a008d77723",
       "triggerType" : "PUSH"
     }, {
       "hash" : "dc785fa04e542b2ccc45494e0292f5153c7ecea8",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1097",
       "triggerID" : "dc785fa04e542b2ccc45494e0292f5153c7ecea8",
       "triggerType" : "PUSH"
     }, {
       "hash" : "585ddb8049b9b9ccb057027e265b08a1b586df8c",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "585ddb8049b9b9ccb057027e265b08a1b586df8c",
       "triggerType" : "PUSH"
     }, {
       "hash" : "b62ff8675beed2c4487c788b9ca1a0ad4271b913",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "b62ff8675beed2c4487c788b9ca1a0ad4271b913",
       "triggerType" : "PUSH"
     }, {
       "hash" : "2d46048a7402907f348cc96da2839ba1b3e4932a",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1099",
       "triggerID" : "2d46048a7402907f348cc96da2839ba1b3e4932a",
       "triggerType" : "PUSH"
     }, {
       "hash" : "004fb6ebf23f796a4b77650e0530f0f1145e8b91",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1101",
       "triggerID" : "004fb6ebf23f796a4b77650e0530f0f1145e8b91",
       "triggerType" : "PUSH"
     }, {
       "hash" : "af0c33333b5e9ae6d59f20a6db1d7c016ea015d3",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1165",
       "triggerID" : "af0c33333b5e9ae6d59f20a6db1d7c016ea015d3",
       "triggerType" : "PUSH"
     }, {
       "hash" : "da8111118d71306500229619d3b7996fcd2526e8",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "da8111118d71306500229619d3b7996fcd2526e8",
       "triggerType" : "PUSH"
     }, {
       "hash" : "83257e2b9bd3c426cf90e11003a5f244a750d7fa",
       "status" : "FAILURE",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1167",
       "triggerID" : "83257e2b9bd3c426cf90e11003a5f244a750d7fa",
       "triggerType" : "PUSH"
     }, {
       "hash" : "e0de408f1b8153fdaaea4c87b27a7b69213b75b5",
       "status" : "PENDING",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1177",
       "triggerID" : "e0de408f1b8153fdaaea4c87b27a7b69213b75b5",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 9e668f65c6d4f73c76573236f8fa507dfb80edac UNKNOWN
   * 533f00e5546da0026bbe51e1c030ddd1c19fdd08 UNKNOWN
   * a2ee90c549332a7f17a7d9b2799849b77c08a8e6 UNKNOWN
   * fc2c138901e82ed5f3fbf363c2a0ce8140ade050 UNKNOWN
   * a510a7de1458bb996df5bbb341e8d010e4c91381 UNKNOWN
   * f072a986114949c5e426ce64a5f5568fc718e25d UNKNOWN
   * 3c62ef22f6cde28110e5eb2fca8760a008d77723 UNKNOWN
   * 585ddb8049b9b9ccb057027e265b08a1b586df8c UNKNOWN
   * b62ff8675beed2c4487c788b9ca1a0ad4271b913 UNKNOWN
   * da8111118d71306500229619d3b7996fcd2526e8 UNKNOWN
   * 83257e2b9bd3c426cf90e11003a5f244a750d7fa Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1167) 
   * e0de408f1b8153fdaaea4c87b27a7b69213b75b5 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1177) 
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] [HUDI-8141] Incremental Query with Completion Time [hudi]

Posted by "danny0405 (via GitHub)" <gi...@apache.org>.

danny0405 commented on code in PR #11947:
URL: https://github.com/apache/hudi/pull/11947#discussion_r1803968159


##########
hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/hudi/IncrementalRelation.scala:
##########
@@ -90,24 +78,24 @@ class IncrementalRelation(val sqlContext: SQLContext,
     throw new HoodieException("Incremental queries are not supported when meta fields are disabled")
   }
 
+  private val queryContext: IncrementalQueryAnalyzer.QueryContext =
+    IncrementalQueryAnalyzer.builder()
+      .metaClient(metaClient)
+      .startTime(optParams(DataSourceReadOptions.BEGIN_INSTANTTIME.key))
+      .endTime(optParams.getOrElse(DataSourceReadOptions.END_INSTANTTIME.key, null))
+      .rangeType(InstantRange.RangeType.OPEN_CLOSED)

Review Comment:
   Got it, why we need a separate query context for `IncrementalRelation` and `MergeOnReadIncrementalRelation` with hard code range type as `OPEN_CLOSED`, should we make the range type configurable for these two?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] [HUDI-8141] Incremental Query with Completion Time [hudi]

Posted by "danny0405 (via GitHub)" <gi...@apache.org>.

danny0405 commented on code in PR #11947:
URL: https://github.com/apache/hudi/pull/11947#discussion_r1806009707


##########
hudi-utilities/src/main/java/org/apache/hudi/utilities/sources/helpers/IncrSourceHelper.java:
##########
@@ -174,6 +168,63 @@ public static QueryInfo generateQueryInfo(JavaSparkContext jssc, String srcBaseP
     }
   }
 
+  public static IncrementalQueryAnalyzer getIncrementalQueryAnalyzer(
+      JavaSparkContext jssc,
+      String srcPath,
+      Option<String> lastCkptStr,
+      MissingCheckpointStrategy missingCheckpointStrategy,
+      int numInstantsFromConfig,
+      Option<SourceProfile<Integer>> latestSourceProfile) {
+    HoodieTableMetaClient metaClient = HoodieTableMetaClient.builder()
+        .setConf(HadoopFSUtils.getStorageConfWithCopy(jssc.hadoopConfiguration()))
+        .setBasePath(srcPath)
+        .setLoadActiveTimelineOnLoad(true)
+        .build();
+
+    String beginCompletionTime;

Review Comment:
   start



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] [HUDI-8141] Incremental Query with Completion Time [hudi]

Posted by "danny0405 (via GitHub)" <gi...@apache.org>.

danny0405 commented on code in PR #11947:
URL: https://github.com/apache/hudi/pull/11947#discussion_r1802202942


##########
hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/hudi/MergeOnReadIncrementalRelation.scala:
##########
@@ -201,55 +199,42 @@ trait HoodieIncrementalRelationTrait extends HoodieBaseRelation {
     val fallbackToFullTableScan = optParams.getOrElse(DataSourceReadOptions.INCREMENTAL_FALLBACK_TO_FULL_TABLE_SCAN.key,
       DataSourceReadOptions.INCREMENTAL_FALLBACK_TO_FULL_TABLE_SCAN.defaultValue).toBoolean
 
-    fallbackToFullTableScan && (startInstantArchived || endInstantArchived
+    fallbackToFullTableScan && (startInstantArchived
       || affectedFilesInCommits.asScala.exists(fileStatus => !metaClient.getStorage.exists(fileStatus.getPath)))
   }
 
-  protected lazy val includedCommits: immutable.Seq[HoodieInstant] = {
-    if (!startInstantArchived || !endInstantArchived) {
-      // If endTimestamp commit is not archived, will filter instants
-      // before endTimestamp.
-      if (hollowCommitHandling == USE_TRANSITION_TIME) {
-        super.timeline.findInstantsInRangeByCompletionTime(startTimestamp, endTimestamp).getInstants.asScala.toList
-      } else {
-        super.timeline.findInstantsInRange(startTimestamp, endTimestamp).getInstants.asScala.toList
-      }
-    } else {
-      super.timeline.getInstants.asScala.toList
-    }
-  }
+  protected lazy val queryContext: IncrementalQueryAnalyzer.QueryContext =
+    IncrementalQueryAnalyzer.builder()
+      .metaClient(metaClient)
+      .startTime(optParams(DataSourceReadOptions.BEGIN_INSTANTTIME.key))
+      .endTime(optParams.getOrElse(DataSourceReadOptions.END_INSTANTTIME.key, null))
+      .rangeType(InstantRange.RangeType.OPEN_CLOSED)

Review Comment:
   Not make sense from my side.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] [HUDI-8141] Incremental Query with Completion Time [hudi]

Posted by "yihua (via GitHub)" <gi...@apache.org>.

yihua commented on code in PR #11947:
URL: https://github.com/apache/hudi/pull/11947#discussion_r1802456538


##########
hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/hudi/IncrementalRelation.scala:
##########
@@ -90,24 +78,24 @@ class IncrementalRelation(val sqlContext: SQLContext,
     throw new HoodieException("Incremental queries are not supported when meta fields are disabled")
   }
 
+  private val queryContext: IncrementalQueryAnalyzer.QueryContext =
+    IncrementalQueryAnalyzer.builder()
+      .metaClient(metaClient)
+      .startTime(optParams(DataSourceReadOptions.BEGIN_INSTANTTIME.key))
+      .endTime(optParams.getOrElse(DataSourceReadOptions.END_INSTANTTIME.key, null))
+      .rangeType(InstantRange.RangeType.OPEN_CLOSED)

Review Comment:
   I've changed `IncrSourceHelper#getIncrementalQueryAnalyzer` to use different `RangeType` based on whether there is last checkpoint or not, so `CLOSED_CLOSED` is used for the first batch, while `OPEN_CLOSED` is used for subsequent batches.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] [HUDI-8141] Incremental Query with Completion Time [hudi]

Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.

hudi-bot commented on PR #11947:
URL: https://github.com/apache/hudi/pull/11947#issuecomment-2421749318

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "090f256a315cbbf0c6a54d93cc2739dd693360e6",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=484",
       "triggerID" : "090f256a315cbbf0c6a54d93cc2739dd693360e6",
       "triggerType" : "PUSH"
     }, {
       "hash" : "0df5649013c1cf9e782fcbb67b8a2e7174f94512",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=985",
       "triggerID" : "0df5649013c1cf9e782fcbb67b8a2e7174f94512",
       "triggerType" : "PUSH"
     }, {
       "hash" : "d52e0128f409918644255abbe4376dc4a780bfc8",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=986",
       "triggerID" : "d52e0128f409918644255abbe4376dc4a780bfc8",
       "triggerType" : "PUSH"
     }, {
       "hash" : "9e668f65c6d4f73c76573236f8fa507dfb80edac",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "9e668f65c6d4f73c76573236f8fa507dfb80edac",
       "triggerType" : "PUSH"
     }, {
       "hash" : "f145131d524bf190986e5de600a326e32a27beca",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=987",
       "triggerID" : "f145131d524bf190986e5de600a326e32a27beca",
       "triggerType" : "PUSH"
     }, {
       "hash" : "0b008f807b42aa6fb2686922bb88d63e84a1f299",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=993",
       "triggerID" : "0b008f807b42aa6fb2686922bb88d63e84a1f299",
       "triggerType" : "PUSH"
     }, {
       "hash" : "65c01ad9f2b33d36d4890355feef91844438e732",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=995",
       "triggerID" : "65c01ad9f2b33d36d4890355feef91844438e732",
       "triggerType" : "PUSH"
     }, {
       "hash" : "955493fdc93842e8c99529cec35e39cecdbe9638",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=996",
       "triggerID" : "955493fdc93842e8c99529cec35e39cecdbe9638",
       "triggerType" : "PUSH"
     }, {
       "hash" : "533f00e5546da0026bbe51e1c030ddd1c19fdd08",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "533f00e5546da0026bbe51e1c030ddd1c19fdd08",
       "triggerType" : "PUSH"
     }, {
       "hash" : "05777be67e37f3a6a8eaa59e54d9e17e5fc6ba9b",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1016",
       "triggerID" : "05777be67e37f3a6a8eaa59e54d9e17e5fc6ba9b",
       "triggerType" : "PUSH"
     }, {
       "hash" : "a2ee90c549332a7f17a7d9b2799849b77c08a8e6",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "a2ee90c549332a7f17a7d9b2799849b77c08a8e6",
       "triggerType" : "PUSH"
     }, {
       "hash" : "05ce568b0b6ba14da9ca40d26b39ba478837d747",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1017",
       "triggerID" : "05ce568b0b6ba14da9ca40d26b39ba478837d747",
       "triggerType" : "PUSH"
     }, {
       "hash" : "840e3b33d39b1f86367bb6b0e65eadeb33de7765",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1025",
       "triggerID" : "840e3b33d39b1f86367bb6b0e65eadeb33de7765",
       "triggerType" : "PUSH"
     }, {
       "hash" : "8d2e2dd83f51d03e890d424a51d3e0e84e904a43",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1031",
       "triggerID" : "8d2e2dd83f51d03e890d424a51d3e0e84e904a43",
       "triggerType" : "PUSH"
     }, {
       "hash" : "688381914087ff11acf2f8c9372dbf39afa0bd40",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1035",
       "triggerID" : "688381914087ff11acf2f8c9372dbf39afa0bd40",
       "triggerType" : "PUSH"
     }, {
       "hash" : "fc2c138901e82ed5f3fbf363c2a0ce8140ade050",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "fc2c138901e82ed5f3fbf363c2a0ce8140ade050",
       "triggerType" : "PUSH"
     }, {
       "hash" : "6ed9e1cb1e5a191752d6d39c7a9d5d41840f0cdf",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1039",
       "triggerID" : "6ed9e1cb1e5a191752d6d39c7a9d5d41840f0cdf",
       "triggerType" : "PUSH"
     }, {
       "hash" : "a510a7de1458bb996df5bbb341e8d010e4c91381",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "a510a7de1458bb996df5bbb341e8d010e4c91381",
       "triggerType" : "PUSH"
     }, {
       "hash" : "49677397cd117228742205944606f7a789090376",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1042",
       "triggerID" : "49677397cd117228742205944606f7a789090376",
       "triggerType" : "PUSH"
     }, {
       "hash" : "1630db0a37422aed942ff280560a9d6329df5015",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1044",
       "triggerID" : "1630db0a37422aed942ff280560a9d6329df5015",
       "triggerType" : "PUSH"
     }, {
       "hash" : "f11c76301d4ab524db0af57aa831be3a898ea0fb",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1048",
       "triggerID" : "f11c76301d4ab524db0af57aa831be3a898ea0fb",
       "triggerType" : "PUSH"
     }, {
       "hash" : "d1a0dc4c2017853a7ba0c3302f886043a63f29da",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1050",
       "triggerID" : "d1a0dc4c2017853a7ba0c3302f886043a63f29da",
       "triggerType" : "PUSH"
     }, {
       "hash" : "6a4c15ec39ded1f90b113b272554217698a98e1c",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1087",
       "triggerID" : "6a4c15ec39ded1f90b113b272554217698a98e1c",
       "triggerType" : "PUSH"
     }, {
       "hash" : "288e730d645a5e58e50a87ef5f44ba11a8fb0533",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1092",
       "triggerID" : "288e730d645a5e58e50a87ef5f44ba11a8fb0533",
       "triggerType" : "PUSH"
     }, {
       "hash" : "70a5c21321a0da3a0be8d85e206c24befa7b89f1",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1095",
       "triggerID" : "70a5c21321a0da3a0be8d85e206c24befa7b89f1",
       "triggerType" : "PUSH"
     }, {
       "hash" : "f072a986114949c5e426ce64a5f5568fc718e25d",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "f072a986114949c5e426ce64a5f5568fc718e25d",
       "triggerType" : "PUSH"
     }, {
       "hash" : "45b27475b1e8806ec8c566d6f151f4bc79c8549d",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1096",
       "triggerID" : "45b27475b1e8806ec8c566d6f151f4bc79c8549d",
       "triggerType" : "PUSH"
     }, {
       "hash" : "3c62ef22f6cde28110e5eb2fca8760a008d77723",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "3c62ef22f6cde28110e5eb2fca8760a008d77723",
       "triggerType" : "PUSH"
     }, {
       "hash" : "dc785fa04e542b2ccc45494e0292f5153c7ecea8",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1097",
       "triggerID" : "dc785fa04e542b2ccc45494e0292f5153c7ecea8",
       "triggerType" : "PUSH"
     }, {
       "hash" : "585ddb8049b9b9ccb057027e265b08a1b586df8c",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "585ddb8049b9b9ccb057027e265b08a1b586df8c",
       "triggerType" : "PUSH"
     }, {
       "hash" : "b62ff8675beed2c4487c788b9ca1a0ad4271b913",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "b62ff8675beed2c4487c788b9ca1a0ad4271b913",
       "triggerType" : "PUSH"
     }, {
       "hash" : "2d46048a7402907f348cc96da2839ba1b3e4932a",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1099",
       "triggerID" : "2d46048a7402907f348cc96da2839ba1b3e4932a",
       "triggerType" : "PUSH"
     }, {
       "hash" : "004fb6ebf23f796a4b77650e0530f0f1145e8b91",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1101",
       "triggerID" : "004fb6ebf23f796a4b77650e0530f0f1145e8b91",
       "triggerType" : "PUSH"
     }, {
       "hash" : "af0c33333b5e9ae6d59f20a6db1d7c016ea015d3",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1165",
       "triggerID" : "af0c33333b5e9ae6d59f20a6db1d7c016ea015d3",
       "triggerType" : "PUSH"
     }, {
       "hash" : "da8111118d71306500229619d3b7996fcd2526e8",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "da8111118d71306500229619d3b7996fcd2526e8",
       "triggerType" : "PUSH"
     }, {
       "hash" : "83257e2b9bd3c426cf90e11003a5f244a750d7fa",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1167",
       "triggerID" : "83257e2b9bd3c426cf90e11003a5f244a750d7fa",
       "triggerType" : "PUSH"
     }, {
       "hash" : "e0de408f1b8153fdaaea4c87b27a7b69213b75b5",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1177",
       "triggerID" : "e0de408f1b8153fdaaea4c87b27a7b69213b75b5",
       "triggerType" : "PUSH"
     }, {
       "hash" : "3f7951f7ad7931f5dc0b977d7c33bc5429623535",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1180",
       "triggerID" : "3f7951f7ad7931f5dc0b977d7c33bc5429623535",
       "triggerType" : "PUSH"
     }, {
       "hash" : "9cd6f06a753b63630e8245d51338d2bf9ab86b84",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "9cd6f06a753b63630e8245d51338d2bf9ab86b84",
       "triggerType" : "PUSH"
     }, {
       "hash" : "7380b902731a92e0645c02397e14591a05798620",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "7380b902731a92e0645c02397e14591a05798620",
       "triggerType" : "PUSH"
     }, {
       "hash" : "0bd5cca01a541845081d5430d48b021d4bdf0969",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "0bd5cca01a541845081d5430d48b021d4bdf0969",
       "triggerType" : "PUSH"
     }, {
       "hash" : "8f4e175ef554f015e34e0968b656b5de714ab430",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1182",
       "triggerID" : "8f4e175ef554f015e34e0968b656b5de714ab430",
       "triggerType" : "PUSH"
     }, {
       "hash" : "2023f3c882d0c2f5b96e28d2f21812dc0c64aaee",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "2023f3c882d0c2f5b96e28d2f21812dc0c64aaee",
       "triggerType" : "PUSH"
     }, {
       "hash" : "8394acf060d5c3953dbd67cf930b9b1d3b92d8b5",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1198",
       "triggerID" : "8394acf060d5c3953dbd67cf930b9b1d3b92d8b5",
       "triggerType" : "PUSH"
     }, {
       "hash" : "7ec42c105971bff463e359019405a7f4e7947d86",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "7ec42c105971bff463e359019405a7f4e7947d86",
       "triggerType" : "PUSH"
     }, {
       "hash" : "4b930786a7b2de09ac1653cd884ee76a7d1b2ca5",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1202",
       "triggerID" : "4b930786a7b2de09ac1653cd884ee76a7d1b2ca5",
       "triggerType" : "PUSH"
     }, {
       "hash" : "0e9a648c4e03b5a74d6a0085831805041a9a26a2",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1216",
       "triggerID" : "0e9a648c4e03b5a74d6a0085831805041a9a26a2",
       "triggerType" : "PUSH"
     }, {
       "hash" : "326a8729a5d1dd88bc474a5c273aec6ca219f298",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "326a8729a5d1dd88bc474a5c273aec6ca219f298",
       "triggerType" : "PUSH"
     }, {
       "hash" : "cad2f761fc50fba28e2fb3595e4b1270e39ac48a",
       "status" : "CANCELED",
       "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1221",
       "triggerID" : "cad2f761fc50fba28e2fb3595e4b1270e39ac48a",
       "triggerType" : "PUSH"
     }, {
       "hash" : "d485a50aec34bc5707377afe96d696772832558e",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "d485a50aec34bc5707377afe96d696772832558e",
       "triggerType" : "PUSH"
     }, {
       "hash" : "38a98106f19fe4e51632a2c4289ec0f01805ba5e",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "38a98106f19fe4e51632a2c4289ec0f01805ba5e",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 9e668f65c6d4f73c76573236f8fa507dfb80edac UNKNOWN
   * 533f00e5546da0026bbe51e1c030ddd1c19fdd08 UNKNOWN
   * a2ee90c549332a7f17a7d9b2799849b77c08a8e6 UNKNOWN
   * fc2c138901e82ed5f3fbf363c2a0ce8140ade050 UNKNOWN
   * a510a7de1458bb996df5bbb341e8d010e4c91381 UNKNOWN
   * f072a986114949c5e426ce64a5f5568fc718e25d UNKNOWN
   * 3c62ef22f6cde28110e5eb2fca8760a008d77723 UNKNOWN
   * 585ddb8049b9b9ccb057027e265b08a1b586df8c UNKNOWN
   * b62ff8675beed2c4487c788b9ca1a0ad4271b913 UNKNOWN
   * da8111118d71306500229619d3b7996fcd2526e8 UNKNOWN
   * 9cd6f06a753b63630e8245d51338d2bf9ab86b84 UNKNOWN
   * 7380b902731a92e0645c02397e14591a05798620 UNKNOWN
   * 0bd5cca01a541845081d5430d48b021d4bdf0969 UNKNOWN
   * 2023f3c882d0c2f5b96e28d2f21812dc0c64aaee UNKNOWN
   * 7ec42c105971bff463e359019405a7f4e7947d86 UNKNOWN
   * 326a8729a5d1dd88bc474a5c273aec6ca219f298 UNKNOWN
   * cad2f761fc50fba28e2fb3595e4b1270e39ac48a Azure: [CANCELED](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=1221) 
   * d485a50aec34bc5707377afe96d696772832558e UNKNOWN
   * 38a98106f19fe4e51632a2c4289ec0f01805ba5e UNKNOWN
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] [HUDI-8141] Incremental Query with Completion Time [hudi]

Posted by "danny0405 (via GitHub)" <gi...@apache.org>.

danny0405 commented on code in PR #11947:
URL: https://github.com/apache/hudi/pull/11947#discussion_r1804193065


##########
hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/hudi/DataSourceOptions.scala:
##########
@@ -108,7 +108,7 @@ object DataSourceReadOptions {
     .key("hoodie.datasource.read.begin.instanttime")
     .noDefaultValue()
     .withDocumentation("Required when `" + QUERY_TYPE.key() + "` is set to `" + QUERY_TYPE_INCREMENTAL_OPT_VAL + "`. Represents the instant time to start incrementally pulling data from. The instanttime here need not necessarily "
-      + "correspond to an instant on the timeline. New data written with an instant_time > BEGIN_INSTANTTIME are fetched out. "
+      + "correspond to an instant on the timeline. New data written with an instant_time >= BEGIN_INSTANTTIME are fetched out. "

Review Comment:
   with completion_time ?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] [HUDI-8141] Incremental Query with Completion Time [hudi]

Posted by "yihua (via GitHub)" <gi...@apache.org>.

yihua commented on code in PR #11947:
URL: https://github.com/apache/hudi/pull/11947#discussion_r1806030147


##########
hudi-utilities/src/main/java/org/apache/hudi/utilities/sources/HoodieIncrSource.java:
##########
@@ -212,35 +208,58 @@ public Pair<Option<Dataset<Row>>, String> fetchNextBatch(Option<String> lastCkpt
           .collect(Collectors.toMap(Pair::getLeft, Pair::getRight));
       reader = reader.options(optionsMap);
     }
+
+    boolean shouldFullScan =
+        missingCheckpointStrategy == MissingCheckpointStrategy.READ_UPTO_LATEST_COMMIT
+            && queryContext.getActiveTimeline()
+            .isBeforeTimelineStartsByCompletionTime(analyzer.getStartCompletionTime().get());
     Dataset<Row> source;
-    // Do Incr pull. Set end instant if available
-    if (queryInfo.isIncremental()) {
-      source = reader
-          .options(readOpts)
-          .option(QUERY_TYPE().key(), QUERY_TYPE_INCREMENTAL_OPT_VAL())
-          .option(BEGIN_INSTANTTIME().key(), queryInfo.getStartInstant())
-          .option(END_INSTANTTIME().key(), queryInfo.getEndInstant())
-          .option(INCREMENTAL_FALLBACK_TO_FULL_TABLE_SCAN().key(),
-              props.getString(INCREMENTAL_FALLBACK_TO_FULL_TABLE_SCAN().key(),
-                  INCREMENTAL_FALLBACK_TO_FULL_TABLE_SCAN().defaultValue()))
-          .option(INCREMENTAL_READ_HANDLE_HOLLOW_COMMIT().key(), handlingMode.name())
-          .load(srcPath);
-    } else {
-      // if checkpoint is missing from source table, and if strategy is set to READ_UPTO_LATEST_COMMIT, we have to issue snapshot query
+    if (instantRange.isEmpty() || shouldFullScan) {
+      // snapshot query
       Dataset<Row> snapshot = reader
           .options(readOpts)
           .option(DataSourceReadOptions.QUERY_TYPE().key(), DataSourceReadOptions.QUERY_TYPE_SNAPSHOT_OPT_VAL())
           .load(srcPath);
+
+      Option<String> predicate = Option.empty();
+      List<String> instantTimeList = queryContext.getInstantTimeList();
       if (snapshotLoadQuerySplitter.isPresent()) {
-        queryInfo = snapshotLoadQuerySplitter.get().getNextCheckpoint(snapshot, queryInfo, sourceProfileSupplier);
+        Option<SnapshotLoadQuerySplitter.CheckpointWithPredicates> newCheckpointAndPredicate =
+            snapshotLoadQuerySplitter.get().getNextCheckpointWithPredicates(snapshot, queryContext);
+        if (newCheckpointAndPredicate.isPresent()) {
+          endCompletionTime = newCheckpointAndPredicate.get().endInstant;
+          predicate = Option.of(newCheckpointAndPredicate.get().predicateFilter);
+          instantTimeList = queryContext.getInstants().stream()
+              .filter(instant -> HoodieTimeline.compareTimestamps(
+                  instant.getCompletionTime(), HoodieTimeline.LESSER_THAN_OR_EQUALS, newCheckpointAndPredicate.get().endInstant))
+              .map(HoodieInstant::getTimestamp)
+              .collect(Collectors.toList());
+        } else {
+          endCompletionTime = queryContext.getMaxCompletionTime();
+        }
       }
+
+      snapshot = predicate.map(snapshot::filter).orElse(snapshot);
       source = snapshot
           // add filtering so that only interested records are returned.
-          .filter(String.format("%s > '%s'", HoodieRecord.COMMIT_TIME_METADATA_FIELD,
-              queryInfo.getStartInstant()))
-          .filter(String.format("%s <= '%s'", HoodieRecord.COMMIT_TIME_METADATA_FIELD,
-              queryInfo.getEndInstant()));
-      source = queryInfo.getPredicateFilter().map(source::filter).orElse(source);
+          .filter(String.format("%s IN ('%s')", HoodieRecord.COMMIT_TIME_METADATA_FIELD,
+              String.join("','", instantTimeList)));
+    } else {
+      // normal incremental query
+      String beginCompletionTime = queryContext.getInstants().stream()
+          .min(HoodieInstant.COMPLETION_TIME_COMPARATOR)
+          .map(HoodieInstant::getCompletionTime)
+          .get();

Review Comment:
   Since this part of logic goes through the Spark datasource read which uses `CLOSED_CLOSED` range type, the inclusive start completion time is needed.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] [HUDI-8141] Incremental Query with Completion Time [hudi]

Posted by "yihua (via GitHub)" <gi...@apache.org>.

yihua commented on code in PR #11947:
URL: https://github.com/apache/hudi/pull/11947#discussion_r1806025849


##########
hudi-utilities/src/main/java/org/apache/hudi/utilities/sources/HoodieIncrSource.java:
##########
@@ -212,35 +208,58 @@ public Pair<Option<Dataset<Row>>, String> fetchNextBatch(Option<String> lastCkpt
           .collect(Collectors.toMap(Pair::getLeft, Pair::getRight));
       reader = reader.options(optionsMap);
     }
+
+    boolean shouldFullScan =
+        missingCheckpointStrategy == MissingCheckpointStrategy.READ_UPTO_LATEST_COMMIT
+            && queryContext.getActiveTimeline()
+            .isBeforeTimelineStartsByCompletionTime(analyzer.getStartCompletionTime().get());
     Dataset<Row> source;
-    // Do Incr pull. Set end instant if available
-    if (queryInfo.isIncremental()) {
-      source = reader
-          .options(readOpts)
-          .option(QUERY_TYPE().key(), QUERY_TYPE_INCREMENTAL_OPT_VAL())
-          .option(BEGIN_INSTANTTIME().key(), queryInfo.getStartInstant())
-          .option(END_INSTANTTIME().key(), queryInfo.getEndInstant())
-          .option(INCREMENTAL_FALLBACK_TO_FULL_TABLE_SCAN().key(),
-              props.getString(INCREMENTAL_FALLBACK_TO_FULL_TABLE_SCAN().key(),
-                  INCREMENTAL_FALLBACK_TO_FULL_TABLE_SCAN().defaultValue()))
-          .option(INCREMENTAL_READ_HANDLE_HOLLOW_COMMIT().key(), handlingMode.name())
-          .load(srcPath);
-    } else {
-      // if checkpoint is missing from source table, and if strategy is set to READ_UPTO_LATEST_COMMIT, we have to issue snapshot query
+    if (instantRange.isEmpty() || shouldFullScan) {
+      // snapshot query
       Dataset<Row> snapshot = reader
           .options(readOpts)
           .option(DataSourceReadOptions.QUERY_TYPE().key(), DataSourceReadOptions.QUERY_TYPE_SNAPSHOT_OPT_VAL())
           .load(srcPath);
+
+      Option<String> predicate = Option.empty();
+      List<String> instantTimeList = queryContext.getInstantTimeList();
       if (snapshotLoadQuerySplitter.isPresent()) {
-        queryInfo = snapshotLoadQuerySplitter.get().getNextCheckpoint(snapshot, queryInfo, sourceProfileSupplier);
+        Option<SnapshotLoadQuerySplitter.CheckpointWithPredicates> newCheckpointAndPredicate =
+            snapshotLoadQuerySplitter.get().getNextCheckpointWithPredicates(snapshot, queryContext);
+        if (newCheckpointAndPredicate.isPresent()) {
+          endCompletionTime = newCheckpointAndPredicate.get().endInstant;
+          predicate = Option.of(newCheckpointAndPredicate.get().predicateFilter);
+          instantTimeList = queryContext.getInstants().stream()
+              .filter(instant -> HoodieTimeline.compareTimestamps(
+                  instant.getCompletionTime(), HoodieTimeline.LESSER_THAN_OR_EQUALS, newCheckpointAndPredicate.get().endInstant))

Review Comment:
   The `snapshotLoadQuerySplitter.get().getNextCheckpointWithPredicates(snapshot, queryContext)` can return a different end completion time than what `queryContext` has, because of additional filtering, so we need to update the instant time list for `IN` predicate for the snapshot query.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org