You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by GitBox <gi...@apache.org> on 2021/10/18 16:15:01 UTC

[GitHub] [hudi] davehagman opened a new pull request #3820: [BUGFIX] Merge commit state from previous instant instead of current

davehagman opened a new pull request #3820:
URL: https://github.com/apache/hudi/pull/3820


   
   ## What is the purpose of the pull request
   In order to support multi-writer concurrency where one writer is the Deltastreamer, other writers must copy any checkpoint state from previous commits into their current one in order to prevent interleaved commits from crashing the deltastreamer. 
   
   The code that does this did the following (before this change):
   * Get all the keys for the *current* inflight commit metadata
   * Filter out any keys that are not specified in the metadata config (a list of keys to replace)
   * For keys that exist in the current metadata, pull the data for that key from the *previous* commit and replace the current commit's metadata property value with that value
   
   This does not work because a non-deltastreamer writer (such as a spark datasource writer) will never have the checkpoint key specified in its commit metadata (`deltastreamer.checkpoint.key`) which results in a commit in the timeline that does not have checkpoint state. If the deltastreamer tries to start from that commit it will fail.
   
   This fixes that by changing the keyset that is filtered from the current commit to the previous commit. This fixes two issues:
   1. Checkpoint state is copied over from a previous commit which was made by the deltastreamer
   2. If the deltastreamer process fails or is stopped for a prolonged period of time, the non-deltastreamer writers will continue to carry over the checkpoint state which will allow the deltastreamer to correctly start from its last known position
   
   
   ## Brief change log
   
   *(for example:)*
     - *Modify `TransactionUtils::overrideWithLatestCommitMetadata` to pull the keys from the last commit instead of the current commit*
   
   ## Verify this pull request
   * Manually verified the change by running multiple writers against the same table
   * Writer One: Deltastreamer, kafka source
   * Writer Two:  Spark datasource, event data from existing hudi table
   * Verified zero errors from deltastreamer over hundreds of interleaved commits
   * Shut down deltastreamer for a prolonged period, then verified that I could start it back up without losing its position in kafka (checkpoint state in tact on recent commits)
   
   ## Committer checklist
   
    - [x ] Has a corresponding JIRA in PR title & commit
    
    - [x] Commit message is descriptive of the change
    
    - [ ] CI is green
   
    - [ x] Necessary doc changes done or have another open PR
          
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [hudi] nsivabalan commented on pull request #3820: [HUDI-2579] Make deltastreamer checkpoint state merging more explicit

Posted by GitBox <gi...@apache.org>.
nsivabalan commented on pull request #3820:
URL: https://github.com/apache/hudi/pull/3820#issuecomment-961621624


   @n3nash : Is there any other purpose to the old way of copying commit metadata apart from copying over checkpoints? We can add the old config as an alternative to new one added. but old one is somewhat generic to copy any matching entries, where as with new config, we explicitly just copy over only the delta streamer checkpoint entry. 
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [hudi] nsivabalan commented on pull request #3820: [HUDI-2579] Make deltastreamer checkpoint state merging more explicit

Posted by GitBox <gi...@apache.org>.
nsivabalan commented on pull request #3820:
URL: https://github.com/apache/hudi/pull/3820#issuecomment-956855780


   @hudi-bot azure run


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [hudi] nsivabalan edited a comment on pull request #3820: [BUGFIX] Merge commit state from previous commit instead of current

Posted by GitBox <gi...@apache.org>.
nsivabalan edited a comment on pull request #3820:
URL: https://github.com/apache/hudi/pull/3820#issuecomment-946024720


   thanks a lot for fixing this Dave. I would like to propose something here. 
   I am wondering why do we need to retrofit copying over delta streamer checkpoint into hoodie.write.meta.key.prefixes. to me, this new requirement is very simple and not really tied to `hoodie.write.meta.key.prefixes`. 
   
   Let me propose something and see how that looks like.
   Introduce a new config called `hoodie.copy.over.deltastreamer.checkpoints`. we can brainstorm on actual naming later. 
   When set to true,
   within TransactionUtils::overrideWithLatestCommitMetadata
   ```
      fetch value of "deltastreamer.checkpoint.key" from last committed transaction and copy to cur inflight commit extra metadata.
   ```
   This is very tight and not error prone. Users don't need to set two different config as below which is not very intuitive as to why they need to do this. 
   ```
   hoodie.write.meta.key.prefixes = 'deltastreamer.checkpoint.key'
   ```
   and optionally 
   ```
   deltastreamer.checkpoint.key =. ""
   ```
   
   All users have to do is, for all of their spark writers, they need to set `hoodie.copy.over.deltastreamer.checkpoints` to true. 
   
   welcome thoughts @n3nash @vinothchandar @davehagman 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [hudi] hudi-bot edited a comment on pull request #3820: [HUDI-2579] Make deltastreamer checkpoint state merging more explicit

Posted by GitBox <gi...@apache.org>.
hudi-bot edited a comment on pull request #3820:
URL: https://github.com/apache/hudi/pull/3820#issuecomment-945958915


   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "8de6afb8a205a41de2a4b214c8982488b2b8ec19",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2708",
       "triggerID" : "8de6afb8a205a41de2a4b214c8982488b2b8ec19",
       "triggerType" : "PUSH"
     }, {
       "hash" : "4606a96bbce3b9638026eceac7ce0610f0b7045e",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2718",
       "triggerID" : "4606a96bbce3b9638026eceac7ce0610f0b7045e",
       "triggerType" : "PUSH"
     }, {
       "hash" : "c41001efeff2a7723ad45a637fb75336b58b294b",
       "status" : "FAILURE",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2726",
       "triggerID" : "c41001efeff2a7723ad45a637fb75336b58b294b",
       "triggerType" : "PUSH"
     }, {
       "hash" : "ba8f8b0059fd3dcd549edf2d675605711b74d266",
       "status" : "PENDING",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2836",
       "triggerID" : "ba8f8b0059fd3dcd549edf2d675605711b74d266",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * c41001efeff2a7723ad45a637fb75336b58b294b Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2726) 
   * ba8f8b0059fd3dcd549edf2d675605711b74d266 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2836) 
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run travis` re-run the last Travis build
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [hudi] hudi-bot edited a comment on pull request #3820: [HUDI-2579] Make deltastreamer checkpoint state merging more explicit

Posted by GitBox <gi...@apache.org>.
hudi-bot edited a comment on pull request #3820:
URL: https://github.com/apache/hudi/pull/3820#issuecomment-945958915


   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "8de6afb8a205a41de2a4b214c8982488b2b8ec19",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2708",
       "triggerID" : "8de6afb8a205a41de2a4b214c8982488b2b8ec19",
       "triggerType" : "PUSH"
     }, {
       "hash" : "4606a96bbce3b9638026eceac7ce0610f0b7045e",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2718",
       "triggerID" : "4606a96bbce3b9638026eceac7ce0610f0b7045e",
       "triggerType" : "PUSH"
     }, {
       "hash" : "c41001efeff2a7723ad45a637fb75336b58b294b",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2726",
       "triggerID" : "c41001efeff2a7723ad45a637fb75336b58b294b",
       "triggerType" : "PUSH"
     }, {
       "hash" : "ba8f8b0059fd3dcd549edf2d675605711b74d266",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2836",
       "triggerID" : "ba8f8b0059fd3dcd549edf2d675605711b74d266",
       "triggerType" : "PUSH"
     }, {
       "hash" : "1d031a881c677dee5e40a0119be17e6a35969f60",
       "status" : "FAILURE",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2854",
       "triggerID" : "1d031a881c677dee5e40a0119be17e6a35969f60",
       "triggerType" : "PUSH"
     }, {
       "hash" : "9e8a962c9aee646713ebca560be2cfd234f25b09",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "9e8a962c9aee646713ebca560be2cfd234f25b09",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 1d031a881c677dee5e40a0119be17e6a35969f60 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2854) 
   * 9e8a962c9aee646713ebca560be2cfd234f25b09 UNKNOWN
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run travis` re-run the last Travis build
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [hudi] hudi-bot edited a comment on pull request #3820: [HUDI-2579] Make deltastreamer checkpoint state merging more explicit

Posted by GitBox <gi...@apache.org>.
hudi-bot edited a comment on pull request #3820:
URL: https://github.com/apache/hudi/pull/3820#issuecomment-945958915


   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "8de6afb8a205a41de2a4b214c8982488b2b8ec19",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2708",
       "triggerID" : "8de6afb8a205a41de2a4b214c8982488b2b8ec19",
       "triggerType" : "PUSH"
     }, {
       "hash" : "4606a96bbce3b9638026eceac7ce0610f0b7045e",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2718",
       "triggerID" : "4606a96bbce3b9638026eceac7ce0610f0b7045e",
       "triggerType" : "PUSH"
     }, {
       "hash" : "c41001efeff2a7723ad45a637fb75336b58b294b",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2726",
       "triggerID" : "c41001efeff2a7723ad45a637fb75336b58b294b",
       "triggerType" : "PUSH"
     }, {
       "hash" : "ba8f8b0059fd3dcd549edf2d675605711b74d266",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2836",
       "triggerID" : "ba8f8b0059fd3dcd549edf2d675605711b74d266",
       "triggerType" : "PUSH"
     }, {
       "hash" : "1d031a881c677dee5e40a0119be17e6a35969f60",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2854",
       "triggerID" : "1d031a881c677dee5e40a0119be17e6a35969f60",
       "triggerType" : "PUSH"
     }, {
       "hash" : "9e8a962c9aee646713ebca560be2cfd234f25b09",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2910",
       "triggerID" : "9e8a962c9aee646713ebca560be2cfd234f25b09",
       "triggerType" : "PUSH"
     }, {
       "hash" : "4007755f0bb30705d3cc342f11ecfcebd3a108e9",
       "status" : "SUCCESS",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2911",
       "triggerID" : "4007755f0bb30705d3cc342f11ecfcebd3a108e9",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 4007755f0bb30705d3cc342f11ecfcebd3a108e9 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2911) 
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run travis` re-run the last Travis build
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [hudi] hudi-bot edited a comment on pull request #3820: [HUDI-2579] Make deltastreamer checkpoint state merging more explicit

Posted by GitBox <gi...@apache.org>.
hudi-bot edited a comment on pull request #3820:
URL: https://github.com/apache/hudi/pull/3820#issuecomment-945958915


   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "8de6afb8a205a41de2a4b214c8982488b2b8ec19",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2708",
       "triggerID" : "8de6afb8a205a41de2a4b214c8982488b2b8ec19",
       "triggerType" : "PUSH"
     }, {
       "hash" : "4606a96bbce3b9638026eceac7ce0610f0b7045e",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2718",
       "triggerID" : "4606a96bbce3b9638026eceac7ce0610f0b7045e",
       "triggerType" : "PUSH"
     }, {
       "hash" : "c41001efeff2a7723ad45a637fb75336b58b294b",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2726",
       "triggerID" : "c41001efeff2a7723ad45a637fb75336b58b294b",
       "triggerType" : "PUSH"
     }, {
       "hash" : "ba8f8b0059fd3dcd549edf2d675605711b74d266",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2836",
       "triggerID" : "ba8f8b0059fd3dcd549edf2d675605711b74d266",
       "triggerType" : "PUSH"
     }, {
       "hash" : "1d031a881c677dee5e40a0119be17e6a35969f60",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2854",
       "triggerID" : "1d031a881c677dee5e40a0119be17e6a35969f60",
       "triggerType" : "PUSH"
     }, {
       "hash" : "9e8a962c9aee646713ebca560be2cfd234f25b09",
       "status" : "SUCCESS",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2910",
       "triggerID" : "9e8a962c9aee646713ebca560be2cfd234f25b09",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 9e8a962c9aee646713ebca560be2cfd234f25b09 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2910) 
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run travis` re-run the last Travis build
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [hudi] hudi-bot edited a comment on pull request #3820: [HUDI-2579] Make deltastreamer checkpoint state merging more explicit

Posted by GitBox <gi...@apache.org>.
hudi-bot edited a comment on pull request #3820:
URL: https://github.com/apache/hudi/pull/3820#issuecomment-945958915


   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "8de6afb8a205a41de2a4b214c8982488b2b8ec19",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2708",
       "triggerID" : "8de6afb8a205a41de2a4b214c8982488b2b8ec19",
       "triggerType" : "PUSH"
     }, {
       "hash" : "4606a96bbce3b9638026eceac7ce0610f0b7045e",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2718",
       "triggerID" : "4606a96bbce3b9638026eceac7ce0610f0b7045e",
       "triggerType" : "PUSH"
     }, {
       "hash" : "c41001efeff2a7723ad45a637fb75336b58b294b",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2726",
       "triggerID" : "c41001efeff2a7723ad45a637fb75336b58b294b",
       "triggerType" : "PUSH"
     }, {
       "hash" : "ba8f8b0059fd3dcd549edf2d675605711b74d266",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2836",
       "triggerID" : "ba8f8b0059fd3dcd549edf2d675605711b74d266",
       "triggerType" : "PUSH"
     }, {
       "hash" : "1d031a881c677dee5e40a0119be17e6a35969f60",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2854",
       "triggerID" : "1d031a881c677dee5e40a0119be17e6a35969f60",
       "triggerType" : "PUSH"
     }, {
       "hash" : "9e8a962c9aee646713ebca560be2cfd234f25b09",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2910",
       "triggerID" : "9e8a962c9aee646713ebca560be2cfd234f25b09",
       "triggerType" : "PUSH"
     }, {
       "hash" : "4007755f0bb30705d3cc342f11ecfcebd3a108e9",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2911",
       "triggerID" : "4007755f0bb30705d3cc342f11ecfcebd3a108e9",
       "triggerType" : "PUSH"
     }, {
       "hash" : "255b35f65537a3adaedb14b25bb53d98beadb371",
       "status" : "SUCCESS",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2914",
       "triggerID" : "255b35f65537a3adaedb14b25bb53d98beadb371",
       "triggerType" : "PUSH"
     }, {
       "hash" : "d36815988f20c005b439af98d2a43bd9c145f7aa",
       "status" : "PENDING",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3020",
       "triggerID" : "d36815988f20c005b439af98d2a43bd9c145f7aa",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 255b35f65537a3adaedb14b25bb53d98beadb371 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2914) 
   * d36815988f20c005b439af98d2a43bd9c145f7aa Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3020) 
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run travis` re-run the last Travis build
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [hudi] hudi-bot commented on pull request #3820: [HUDI-2579] Make deltastreamer checkpoint state merging more explicit

Posted by GitBox <gi...@apache.org>.
hudi-bot commented on pull request #3820:
URL: https://github.com/apache/hudi/pull/3820#issuecomment-964289064


   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "8de6afb8a205a41de2a4b214c8982488b2b8ec19",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2708",
       "triggerID" : "8de6afb8a205a41de2a4b214c8982488b2b8ec19",
       "triggerType" : "PUSH"
     }, {
       "hash" : "4606a96bbce3b9638026eceac7ce0610f0b7045e",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2718",
       "triggerID" : "4606a96bbce3b9638026eceac7ce0610f0b7045e",
       "triggerType" : "PUSH"
     }, {
       "hash" : "c41001efeff2a7723ad45a637fb75336b58b294b",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2726",
       "triggerID" : "c41001efeff2a7723ad45a637fb75336b58b294b",
       "triggerType" : "PUSH"
     }, {
       "hash" : "ba8f8b0059fd3dcd549edf2d675605711b74d266",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2836",
       "triggerID" : "ba8f8b0059fd3dcd549edf2d675605711b74d266",
       "triggerType" : "PUSH"
     }, {
       "hash" : "1d031a881c677dee5e40a0119be17e6a35969f60",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2854",
       "triggerID" : "1d031a881c677dee5e40a0119be17e6a35969f60",
       "triggerType" : "PUSH"
     }, {
       "hash" : "9e8a962c9aee646713ebca560be2cfd234f25b09",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2910",
       "triggerID" : "9e8a962c9aee646713ebca560be2cfd234f25b09",
       "triggerType" : "PUSH"
     }, {
       "hash" : "4007755f0bb30705d3cc342f11ecfcebd3a108e9",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2911",
       "triggerID" : "4007755f0bb30705d3cc342f11ecfcebd3a108e9",
       "triggerType" : "PUSH"
     }, {
       "hash" : "255b35f65537a3adaedb14b25bb53d98beadb371",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2914",
       "triggerID" : "255b35f65537a3adaedb14b25bb53d98beadb371",
       "triggerType" : "PUSH"
     }, {
       "hash" : "d36815988f20c005b439af98d2a43bd9c145f7aa",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3020",
       "triggerID" : "d36815988f20c005b439af98d2a43bd9c145f7aa",
       "triggerType" : "PUSH"
     }, {
       "hash" : "98f42d69bed5505c9a1fa78e54da281fec73bd42",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3070",
       "triggerID" : "98f42d69bed5505c9a1fa78e54da281fec73bd42",
       "triggerType" : "PUSH"
     }, {
       "hash" : "1c139b8fb0405745bfea474365363715101e0323",
       "status" : "SUCCESS",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3249",
       "triggerID" : "1c139b8fb0405745bfea474365363715101e0323",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 1c139b8fb0405745bfea474365363715101e0323 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3249) 
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [hudi] n3nash commented on pull request #3820: [HUDI-2579] Make deltastreamer checkpoint state merging more explicit

Posted by GitBox <gi...@apache.org>.
n3nash commented on pull request #3820:
URL: https://github.com/apache/hudi/pull/3820#issuecomment-961618542


   @jsbali This may affect use cases at Uber, FYI 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [hudi] hudi-bot removed a comment on pull request #3820: [HUDI-2579] Make deltastreamer checkpoint state merging more explicit

Posted by GitBox <gi...@apache.org>.
hudi-bot removed a comment on pull request #3820:
URL: https://github.com/apache/hudi/pull/3820#issuecomment-945958915


   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "8de6afb8a205a41de2a4b214c8982488b2b8ec19",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2708",
       "triggerID" : "8de6afb8a205a41de2a4b214c8982488b2b8ec19",
       "triggerType" : "PUSH"
     }, {
       "hash" : "4606a96bbce3b9638026eceac7ce0610f0b7045e",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2718",
       "triggerID" : "4606a96bbce3b9638026eceac7ce0610f0b7045e",
       "triggerType" : "PUSH"
     }, {
       "hash" : "c41001efeff2a7723ad45a637fb75336b58b294b",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2726",
       "triggerID" : "c41001efeff2a7723ad45a637fb75336b58b294b",
       "triggerType" : "PUSH"
     }, {
       "hash" : "ba8f8b0059fd3dcd549edf2d675605711b74d266",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2836",
       "triggerID" : "ba8f8b0059fd3dcd549edf2d675605711b74d266",
       "triggerType" : "PUSH"
     }, {
       "hash" : "1d031a881c677dee5e40a0119be17e6a35969f60",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2854",
       "triggerID" : "1d031a881c677dee5e40a0119be17e6a35969f60",
       "triggerType" : "PUSH"
     }, {
       "hash" : "9e8a962c9aee646713ebca560be2cfd234f25b09",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2910",
       "triggerID" : "9e8a962c9aee646713ebca560be2cfd234f25b09",
       "triggerType" : "PUSH"
     }, {
       "hash" : "4007755f0bb30705d3cc342f11ecfcebd3a108e9",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2911",
       "triggerID" : "4007755f0bb30705d3cc342f11ecfcebd3a108e9",
       "triggerType" : "PUSH"
     }, {
       "hash" : "255b35f65537a3adaedb14b25bb53d98beadb371",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2914",
       "triggerID" : "255b35f65537a3adaedb14b25bb53d98beadb371",
       "triggerType" : "PUSH"
     }, {
       "hash" : "d36815988f20c005b439af98d2a43bd9c145f7aa",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3020",
       "triggerID" : "d36815988f20c005b439af98d2a43bd9c145f7aa",
       "triggerType" : "PUSH"
     }, {
       "hash" : "98f42d69bed5505c9a1fa78e54da281fec73bd42",
       "status" : "SUCCESS",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3070",
       "triggerID" : "98f42d69bed5505c9a1fa78e54da281fec73bd42",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 98f42d69bed5505c9a1fa78e54da281fec73bd42 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3070) 
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run travis` re-run the last Travis build
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [hudi] hudi-bot commented on pull request #3820: [HUDI-2579] Make deltastreamer checkpoint state merging more explicit

Posted by GitBox <gi...@apache.org>.
hudi-bot commented on pull request #3820:
URL: https://github.com/apache/hudi/pull/3820#issuecomment-961588596


   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "8de6afb8a205a41de2a4b214c8982488b2b8ec19",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2708",
       "triggerID" : "8de6afb8a205a41de2a4b214c8982488b2b8ec19",
       "triggerType" : "PUSH"
     }, {
       "hash" : "4606a96bbce3b9638026eceac7ce0610f0b7045e",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2718",
       "triggerID" : "4606a96bbce3b9638026eceac7ce0610f0b7045e",
       "triggerType" : "PUSH"
     }, {
       "hash" : "c41001efeff2a7723ad45a637fb75336b58b294b",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2726",
       "triggerID" : "c41001efeff2a7723ad45a637fb75336b58b294b",
       "triggerType" : "PUSH"
     }, {
       "hash" : "ba8f8b0059fd3dcd549edf2d675605711b74d266",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2836",
       "triggerID" : "ba8f8b0059fd3dcd549edf2d675605711b74d266",
       "triggerType" : "PUSH"
     }, {
       "hash" : "1d031a881c677dee5e40a0119be17e6a35969f60",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2854",
       "triggerID" : "1d031a881c677dee5e40a0119be17e6a35969f60",
       "triggerType" : "PUSH"
     }, {
       "hash" : "9e8a962c9aee646713ebca560be2cfd234f25b09",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2910",
       "triggerID" : "9e8a962c9aee646713ebca560be2cfd234f25b09",
       "triggerType" : "PUSH"
     }, {
       "hash" : "4007755f0bb30705d3cc342f11ecfcebd3a108e9",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2911",
       "triggerID" : "4007755f0bb30705d3cc342f11ecfcebd3a108e9",
       "triggerType" : "PUSH"
     }, {
       "hash" : "255b35f65537a3adaedb14b25bb53d98beadb371",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2914",
       "triggerID" : "255b35f65537a3adaedb14b25bb53d98beadb371",
       "triggerType" : "PUSH"
     }, {
       "hash" : "d36815988f20c005b439af98d2a43bd9c145f7aa",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3020",
       "triggerID" : "d36815988f20c005b439af98d2a43bd9c145f7aa",
       "triggerType" : "PUSH"
     }, {
       "hash" : "98f42d69bed5505c9a1fa78e54da281fec73bd42",
       "status" : "SUCCESS",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3070",
       "triggerID" : "98f42d69bed5505c9a1fa78e54da281fec73bd42",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 98f42d69bed5505c9a1fa78e54da281fec73bd42 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3070) 
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [hudi] hudi-bot edited a comment on pull request #3820: [HUDI-2579] Make deltastreamer checkpoint state merging more explicit

Posted by GitBox <gi...@apache.org>.
hudi-bot edited a comment on pull request #3820:
URL: https://github.com/apache/hudi/pull/3820#issuecomment-945958915


   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "8de6afb8a205a41de2a4b214c8982488b2b8ec19",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2708",
       "triggerID" : "8de6afb8a205a41de2a4b214c8982488b2b8ec19",
       "triggerType" : "PUSH"
     }, {
       "hash" : "4606a96bbce3b9638026eceac7ce0610f0b7045e",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2718",
       "triggerID" : "4606a96bbce3b9638026eceac7ce0610f0b7045e",
       "triggerType" : "PUSH"
     }, {
       "hash" : "c41001efeff2a7723ad45a637fb75336b58b294b",
       "status" : "FAILURE",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2726",
       "triggerID" : "c41001efeff2a7723ad45a637fb75336b58b294b",
       "triggerType" : "PUSH"
     }, {
       "hash" : "ba8f8b0059fd3dcd549edf2d675605711b74d266",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "ba8f8b0059fd3dcd549edf2d675605711b74d266",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * c41001efeff2a7723ad45a637fb75336b58b294b Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2726) 
   * ba8f8b0059fd3dcd549edf2d675605711b74d266 UNKNOWN
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run travis` re-run the last Travis build
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [hudi] n3nash commented on pull request #3820: [HUDI-2579] Make deltastreamer checkpoint state merging more explicit

Posted by GitBox <gi...@apache.org>.
n3nash commented on pull request #3820:
URL: https://github.com/apache/hudi/pull/3820#issuecomment-961618542


   @jsbali This may affect use cases at Uber, FYI 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [hudi] hudi-bot edited a comment on pull request #3820: [HUDI-2579] Make deltastreamer checkpoint state merging more explicit

Posted by GitBox <gi...@apache.org>.
hudi-bot edited a comment on pull request #3820:
URL: https://github.com/apache/hudi/pull/3820#issuecomment-945958915


   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "8de6afb8a205a41de2a4b214c8982488b2b8ec19",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2708",
       "triggerID" : "8de6afb8a205a41de2a4b214c8982488b2b8ec19",
       "triggerType" : "PUSH"
     }, {
       "hash" : "4606a96bbce3b9638026eceac7ce0610f0b7045e",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2718",
       "triggerID" : "4606a96bbce3b9638026eceac7ce0610f0b7045e",
       "triggerType" : "PUSH"
     }, {
       "hash" : "c41001efeff2a7723ad45a637fb75336b58b294b",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2726",
       "triggerID" : "c41001efeff2a7723ad45a637fb75336b58b294b",
       "triggerType" : "PUSH"
     }, {
       "hash" : "ba8f8b0059fd3dcd549edf2d675605711b74d266",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2836",
       "triggerID" : "ba8f8b0059fd3dcd549edf2d675605711b74d266",
       "triggerType" : "PUSH"
     }, {
       "hash" : "1d031a881c677dee5e40a0119be17e6a35969f60",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2854",
       "triggerID" : "1d031a881c677dee5e40a0119be17e6a35969f60",
       "triggerType" : "PUSH"
     }, {
       "hash" : "9e8a962c9aee646713ebca560be2cfd234f25b09",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2910",
       "triggerID" : "9e8a962c9aee646713ebca560be2cfd234f25b09",
       "triggerType" : "PUSH"
     }, {
       "hash" : "4007755f0bb30705d3cc342f11ecfcebd3a108e9",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2911",
       "triggerID" : "4007755f0bb30705d3cc342f11ecfcebd3a108e9",
       "triggerType" : "PUSH"
     }, {
       "hash" : "255b35f65537a3adaedb14b25bb53d98beadb371",
       "status" : "SUCCESS",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2914",
       "triggerID" : "255b35f65537a3adaedb14b25bb53d98beadb371",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 255b35f65537a3adaedb14b25bb53d98beadb371 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2914) 
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run travis` re-run the last Travis build
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [hudi] davehagman commented on pull request #3820: [HUDI-2579] Make deltastreamer checkpoint state merging more explicit

Posted by GitBox <gi...@apache.org>.
davehagman commented on pull request #3820:
URL: https://github.com/apache/hudi/pull/3820#issuecomment-953257614


   A test was added to verify that checkpoint state gets added from the previous completed commit. I ended up just mocking `TransactionUtils` because trying to use the public method `resolveWriteConflictIfAny` would be very difficult since the `hudi-client-common` module does not have any table implementations. 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [hudi] hudi-bot removed a comment on pull request #3820: [HUDI-2579] Make deltastreamer checkpoint state merging more explicit

Posted by GitBox <gi...@apache.org>.
hudi-bot removed a comment on pull request #3820:
URL: https://github.com/apache/hudi/pull/3820#issuecomment-945958915


   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "8de6afb8a205a41de2a4b214c8982488b2b8ec19",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2708",
       "triggerID" : "8de6afb8a205a41de2a4b214c8982488b2b8ec19",
       "triggerType" : "PUSH"
     }, {
       "hash" : "4606a96bbce3b9638026eceac7ce0610f0b7045e",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2718",
       "triggerID" : "4606a96bbce3b9638026eceac7ce0610f0b7045e",
       "triggerType" : "PUSH"
     }, {
       "hash" : "c41001efeff2a7723ad45a637fb75336b58b294b",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2726",
       "triggerID" : "c41001efeff2a7723ad45a637fb75336b58b294b",
       "triggerType" : "PUSH"
     }, {
       "hash" : "ba8f8b0059fd3dcd549edf2d675605711b74d266",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2836",
       "triggerID" : "ba8f8b0059fd3dcd549edf2d675605711b74d266",
       "triggerType" : "PUSH"
     }, {
       "hash" : "1d031a881c677dee5e40a0119be17e6a35969f60",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2854",
       "triggerID" : "1d031a881c677dee5e40a0119be17e6a35969f60",
       "triggerType" : "PUSH"
     }, {
       "hash" : "9e8a962c9aee646713ebca560be2cfd234f25b09",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2910",
       "triggerID" : "9e8a962c9aee646713ebca560be2cfd234f25b09",
       "triggerType" : "PUSH"
     }, {
       "hash" : "4007755f0bb30705d3cc342f11ecfcebd3a108e9",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2911",
       "triggerID" : "4007755f0bb30705d3cc342f11ecfcebd3a108e9",
       "triggerType" : "PUSH"
     }, {
       "hash" : "255b35f65537a3adaedb14b25bb53d98beadb371",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2914",
       "triggerID" : "255b35f65537a3adaedb14b25bb53d98beadb371",
       "triggerType" : "PUSH"
     }, {
       "hash" : "d36815988f20c005b439af98d2a43bd9c145f7aa",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3020",
       "triggerID" : "d36815988f20c005b439af98d2a43bd9c145f7aa",
       "triggerType" : "PUSH"
     }, {
       "hash" : "98f42d69bed5505c9a1fa78e54da281fec73bd42",
       "status" : "SUCCESS",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3070",
       "triggerID" : "98f42d69bed5505c9a1fa78e54da281fec73bd42",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 98f42d69bed5505c9a1fa78e54da281fec73bd42 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3070) 
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run travis` re-run the last Travis build
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [hudi] nsivabalan commented on pull request #3820: [HUDI-2579] Make deltastreamer checkpoint state merging more explicit

Posted by GitBox <gi...@apache.org>.
nsivabalan commented on pull request #3820:
URL: https://github.com/apache/hudi/pull/3820#issuecomment-962526138


   oh, btw, I checked the code in TransactionUtils.overrideWithLatestCommitMetadata. This will be called only when config.getWriteConcurrencyMode().supportsOptimisticConcurrencyControl() is set to true. so, not applicable for single writer mode. 
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [hudi] davehagman commented on pull request #3820: [HUDI-2579] Make deltastreamer checkpoint state merging more explicit

Posted by GitBox <gi...@apache.org>.
davehagman commented on pull request #3820:
URL: https://github.com/apache/hudi/pull/3820#issuecomment-947736703


   @nsivabalan This is ready for review


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [hudi] davehagman commented on pull request #3820: [BUGFIX] Merge commit state from previous commit instead of current

Posted by GitBox <gi...@apache.org>.
davehagman commented on pull request #3820:
URL: https://github.com/apache/hudi/pull/3820#issuecomment-946035803


   I like that idea a lot. It reduces the chance of error as well. Here are some thoughts:
   
   > a new config called `hoodie.copy.over.deltastreamer.checkpoints`
   
   Since this is very specific to multi-writer/OCC what about putting it under the `concurrency` namespace? Something like `hoodie.write.concurrency.merge.deltastreamer.state`. This also removes the implementation detail of "checkpoint" in favor of a generalized "state" which will allow us to extend this to other keys in the future if necessary without needing more configs. 
   
   > fetch value of "deltastreamer.checkpoint.key" from last committed transaction and copy to cur inflight commit extra metadata.
   
   Yea we can even re-use the existing code (still need my fix) that merges a key from the previous instant's metadata to the inflight (current) one. Now we will just make this access private and only expose a new method which is specific to copying over checkpoint state if the above config is set. Something like:
   `TransactionUtils.mergeCheckpointStateFromPreviousCommit(thisInstant, previousCommit)`
    
   this will ultimately just call the existing `overrideWithLatestCommitMetadata` (now private) specifically with the metadata key `deltastreamer.checkpoint.key`, successfully abstracting details and removing the need for users to know anything about the internal state of commits.
   
   Thoughts?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [hudi] nsivabalan commented on pull request #3820: [HUDI-2579] Make deltastreamer checkpoint state merging more explicit

Posted by GitBox <gi...@apache.org>.
nsivabalan commented on pull request #3820:
URL: https://github.com/apache/hudi/pull/3820#issuecomment-964609888


   Added a tracking ticket https://issues.apache.org/jira/browse/HUDI-2719
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [hudi] hudi-bot removed a comment on pull request #3820: [HUDI-2579] Make deltastreamer checkpoint state merging more explicit

Posted by GitBox <gi...@apache.org>.
hudi-bot removed a comment on pull request #3820:
URL: https://github.com/apache/hudi/pull/3820#issuecomment-964233492


   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "8de6afb8a205a41de2a4b214c8982488b2b8ec19",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2708",
       "triggerID" : "8de6afb8a205a41de2a4b214c8982488b2b8ec19",
       "triggerType" : "PUSH"
     }, {
       "hash" : "4606a96bbce3b9638026eceac7ce0610f0b7045e",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2718",
       "triggerID" : "4606a96bbce3b9638026eceac7ce0610f0b7045e",
       "triggerType" : "PUSH"
     }, {
       "hash" : "c41001efeff2a7723ad45a637fb75336b58b294b",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2726",
       "triggerID" : "c41001efeff2a7723ad45a637fb75336b58b294b",
       "triggerType" : "PUSH"
     }, {
       "hash" : "ba8f8b0059fd3dcd549edf2d675605711b74d266",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2836",
       "triggerID" : "ba8f8b0059fd3dcd549edf2d675605711b74d266",
       "triggerType" : "PUSH"
     }, {
       "hash" : "1d031a881c677dee5e40a0119be17e6a35969f60",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2854",
       "triggerID" : "1d031a881c677dee5e40a0119be17e6a35969f60",
       "triggerType" : "PUSH"
     }, {
       "hash" : "9e8a962c9aee646713ebca560be2cfd234f25b09",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2910",
       "triggerID" : "9e8a962c9aee646713ebca560be2cfd234f25b09",
       "triggerType" : "PUSH"
     }, {
       "hash" : "4007755f0bb30705d3cc342f11ecfcebd3a108e9",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2911",
       "triggerID" : "4007755f0bb30705d3cc342f11ecfcebd3a108e9",
       "triggerType" : "PUSH"
     }, {
       "hash" : "255b35f65537a3adaedb14b25bb53d98beadb371",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2914",
       "triggerID" : "255b35f65537a3adaedb14b25bb53d98beadb371",
       "triggerType" : "PUSH"
     }, {
       "hash" : "d36815988f20c005b439af98d2a43bd9c145f7aa",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3020",
       "triggerID" : "d36815988f20c005b439af98d2a43bd9c145f7aa",
       "triggerType" : "PUSH"
     }, {
       "hash" : "98f42d69bed5505c9a1fa78e54da281fec73bd42",
       "status" : "SUCCESS",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3070",
       "triggerID" : "98f42d69bed5505c9a1fa78e54da281fec73bd42",
       "triggerType" : "PUSH"
     }, {
       "hash" : "1c139b8fb0405745bfea474365363715101e0323",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "1c139b8fb0405745bfea474365363715101e0323",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 98f42d69bed5505c9a1fa78e54da281fec73bd42 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3070) 
   * 1c139b8fb0405745bfea474365363715101e0323 UNKNOWN
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [hudi] nsivabalan commented on pull request #3820: [HUDI-2579] Make deltastreamer checkpoint state merging more explicit

Posted by GitBox <gi...@apache.org>.
nsivabalan commented on pull request #3820:
URL: https://github.com/apache/hudi/pull/3820#issuecomment-956855780


   @hudi-bot azure run


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [hudi] hudi-bot edited a comment on pull request #3820: [HUDI-2579] Make deltastreamer checkpoint state merging more explicit

Posted by GitBox <gi...@apache.org>.
hudi-bot edited a comment on pull request #3820:
URL: https://github.com/apache/hudi/pull/3820#issuecomment-945958915


   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "8de6afb8a205a41de2a4b214c8982488b2b8ec19",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2708",
       "triggerID" : "8de6afb8a205a41de2a4b214c8982488b2b8ec19",
       "triggerType" : "PUSH"
     }, {
       "hash" : "4606a96bbce3b9638026eceac7ce0610f0b7045e",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2718",
       "triggerID" : "4606a96bbce3b9638026eceac7ce0610f0b7045e",
       "triggerType" : "PUSH"
     }, {
       "hash" : "c41001efeff2a7723ad45a637fb75336b58b294b",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2726",
       "triggerID" : "c41001efeff2a7723ad45a637fb75336b58b294b",
       "triggerType" : "PUSH"
     }, {
       "hash" : "ba8f8b0059fd3dcd549edf2d675605711b74d266",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2836",
       "triggerID" : "ba8f8b0059fd3dcd549edf2d675605711b74d266",
       "triggerType" : "PUSH"
     }, {
       "hash" : "1d031a881c677dee5e40a0119be17e6a35969f60",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2854",
       "triggerID" : "1d031a881c677dee5e40a0119be17e6a35969f60",
       "triggerType" : "PUSH"
     }, {
       "hash" : "9e8a962c9aee646713ebca560be2cfd234f25b09",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2910",
       "triggerID" : "9e8a962c9aee646713ebca560be2cfd234f25b09",
       "triggerType" : "PUSH"
     }, {
       "hash" : "4007755f0bb30705d3cc342f11ecfcebd3a108e9",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2911",
       "triggerID" : "4007755f0bb30705d3cc342f11ecfcebd3a108e9",
       "triggerType" : "PUSH"
     }, {
       "hash" : "255b35f65537a3adaedb14b25bb53d98beadb371",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2914",
       "triggerID" : "255b35f65537a3adaedb14b25bb53d98beadb371",
       "triggerType" : "PUSH"
     }, {
       "hash" : "d36815988f20c005b439af98d2a43bd9c145f7aa",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3020",
       "triggerID" : "d36815988f20c005b439af98d2a43bd9c145f7aa",
       "triggerType" : "PUSH"
     }, {
       "hash" : "98f42d69bed5505c9a1fa78e54da281fec73bd42",
       "status" : "SUCCESS",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3070",
       "triggerID" : "98f42d69bed5505c9a1fa78e54da281fec73bd42",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 98f42d69bed5505c9a1fa78e54da281fec73bd42 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3070) 
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run travis` re-run the last Travis build
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [hudi] hudi-bot edited a comment on pull request #3820: [HUDI-2579] Make deltastreamer checkpoint state merging more explicit

Posted by GitBox <gi...@apache.org>.
hudi-bot edited a comment on pull request #3820:
URL: https://github.com/apache/hudi/pull/3820#issuecomment-945958915






-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [hudi] hudi-bot edited a comment on pull request #3820: [HUDI-2579] Make deltastreamer checkpoint state merging more explicit

Posted by GitBox <gi...@apache.org>.
hudi-bot edited a comment on pull request #3820:
URL: https://github.com/apache/hudi/pull/3820#issuecomment-945958915


   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "8de6afb8a205a41de2a4b214c8982488b2b8ec19",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2708",
       "triggerID" : "8de6afb8a205a41de2a4b214c8982488b2b8ec19",
       "triggerType" : "PUSH"
     }, {
       "hash" : "4606a96bbce3b9638026eceac7ce0610f0b7045e",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2718",
       "triggerID" : "4606a96bbce3b9638026eceac7ce0610f0b7045e",
       "triggerType" : "PUSH"
     }, {
       "hash" : "c41001efeff2a7723ad45a637fb75336b58b294b",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2726",
       "triggerID" : "c41001efeff2a7723ad45a637fb75336b58b294b",
       "triggerType" : "PUSH"
     }, {
       "hash" : "ba8f8b0059fd3dcd549edf2d675605711b74d266",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2836",
       "triggerID" : "ba8f8b0059fd3dcd549edf2d675605711b74d266",
       "triggerType" : "PUSH"
     }, {
       "hash" : "1d031a881c677dee5e40a0119be17e6a35969f60",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2854",
       "triggerID" : "1d031a881c677dee5e40a0119be17e6a35969f60",
       "triggerType" : "PUSH"
     }, {
       "hash" : "9e8a962c9aee646713ebca560be2cfd234f25b09",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2910",
       "triggerID" : "9e8a962c9aee646713ebca560be2cfd234f25b09",
       "triggerType" : "PUSH"
     }, {
       "hash" : "4007755f0bb30705d3cc342f11ecfcebd3a108e9",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2911",
       "triggerID" : "4007755f0bb30705d3cc342f11ecfcebd3a108e9",
       "triggerType" : "PUSH"
     }, {
       "hash" : "255b35f65537a3adaedb14b25bb53d98beadb371",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2914",
       "triggerID" : "255b35f65537a3adaedb14b25bb53d98beadb371",
       "triggerType" : "PUSH"
     }, {
       "hash" : "d36815988f20c005b439af98d2a43bd9c145f7aa",
       "status" : "FAILURE",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3020",
       "triggerID" : "d36815988f20c005b439af98d2a43bd9c145f7aa",
       "triggerType" : "PUSH"
     }, {
       "hash" : "98f42d69bed5505c9a1fa78e54da281fec73bd42",
       "status" : "PENDING",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3070",
       "triggerID" : "98f42d69bed5505c9a1fa78e54da281fec73bd42",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * d36815988f20c005b439af98d2a43bd9c145f7aa Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3020) 
   * 98f42d69bed5505c9a1fa78e54da281fec73bd42 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3070) 
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run travis` re-run the last Travis build
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [hudi] nsivabalan commented on pull request #3820: [HUDI-2579] Make deltastreamer checkpoint state merging more explicit

Posted by GitBox <gi...@apache.org>.
nsivabalan commented on pull request #3820:
URL: https://github.com/apache/hudi/pull/3820#issuecomment-964236722


   @n3nash : After some thinking, here is my take. 
   Old config is used for two purpose. 
   1. Carrying over deltastreamer checkpoint. 
   2. Carrying over any regex (prefix) based extra metadata from previous commit to current one. 
   
   And this is used only within multi-writer code path. I don't see this being used in single writer. 
   Given we have fixed the (1) above with new config, and multi-writer is an evolving feature, I prefer to go ahead w/ new config. We can add the old config(`hoodie.write.meta.key.prefixes`) as an alternative to the new one, but only (1) will be taken care. We will definitely call this out in release notes. 
   
   If you feel we definitely need support for (2) above, let me know. I fill file a follow up ticket and work on it. but I prefer to fix for both single writer and multi-writer. not sure why we had a constraint only with multiwriter. If you can find resources to work on it, I am good too. Hoping to close this out sooner. Let me know.
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [hudi] hudi-bot edited a comment on pull request #3820: [BUGFIX] Merge commit state from previous commit instead of current

Posted by GitBox <gi...@apache.org>.
hudi-bot edited a comment on pull request #3820:
URL: https://github.com/apache/hudi/pull/3820#issuecomment-945958915


   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "8de6afb8a205a41de2a4b214c8982488b2b8ec19",
       "status" : "PENDING",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2708",
       "triggerID" : "8de6afb8a205a41de2a4b214c8982488b2b8ec19",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 8de6afb8a205a41de2a4b214c8982488b2b8ec19 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2708) 
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run travis` re-run the last Travis build
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [hudi] nsivabalan commented on pull request #3820: [BUGFIX] Merge commit state from previous commit instead of current

Posted by GitBox <gi...@apache.org>.
nsivabalan commented on pull request #3820:
URL: https://github.com/apache/hudi/pull/3820#issuecomment-946024720


   thanks a lot for fixing this Dave. I would like to propose something here. 
   I am wondering why do we need to retrofit copying over delta streamer checkpoint into hoodie.write.meta.key.prefixes. to me, this new requirement is very simple and not really tied to `hoodie.write.meta.key.prefixes`. 
   
   Let me propose something and see how that looks like.
   Introduce a new config called `hoodie.copy.over.deltastreamer.checkpoints`. we can brainstorm on actual naming later. 
   When set to true,
   within TransactionUtils::overrideWithLatestCommitMetadata
   ```
      fetch value of "deltastreamer.checkpoint.key" from last committed transaction and copy to cur inflight commit extra metadata.
   ```
   This is very tight and not error prone. Users don't need to set two different config as below which is not very intuitive as to why they need to do this. 
   ```
   hoodie.write.meta.key.prefixes = 'deltastreamer.checkpoint.key'
   ```
   and optionally 
   ```
   deltastreamer.checkpoint.key =. ""
   ```
   
   All users have to do is, for all of their spark writers, they need to set `hoodie.copy.over.deltastreamer.checkpoints` to true. 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [hudi] hudi-bot edited a comment on pull request #3820: [HUDI-2579] Merge commit state from previous commit instead of current

Posted by GitBox <gi...@apache.org>.
hudi-bot edited a comment on pull request #3820:
URL: https://github.com/apache/hudi/pull/3820#issuecomment-945958915


   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "8de6afb8a205a41de2a4b214c8982488b2b8ec19",
       "status" : "SUCCESS",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2708",
       "triggerID" : "8de6afb8a205a41de2a4b214c8982488b2b8ec19",
       "triggerType" : "PUSH"
     }, {
       "hash" : "4606a96bbce3b9638026eceac7ce0610f0b7045e",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "4606a96bbce3b9638026eceac7ce0610f0b7045e",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 8de6afb8a205a41de2a4b214c8982488b2b8ec19 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2708) 
   * 4606a96bbce3b9638026eceac7ce0610f0b7045e UNKNOWN
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run travis` re-run the last Travis build
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [hudi] nsivabalan commented on a change in pull request #3820: [HUDI-2579] Make deltastreamer checkpoint state merging more explicit

Posted by GitBox <gi...@apache.org>.
nsivabalan commented on a change in pull request #3820:
URL: https://github.com/apache/hudi/pull/3820#discussion_r733013939



##########
File path: hudi-client/hudi-client-common/src/main/java/org/apache/hudi/config/HoodieWriteConfig.java
##########
@@ -368,11 +372,13 @@
           + "OPTIMISTIC_CONCURRENCY_CONTROL: Multiple writers can operate on the table and exactly one of them succeed "
           + "if a conflict (writes affect the same file group) is detected.");
 
-  public static final ConfigProperty<String> WRITE_META_KEY_PREFIXES = ConfigProperty
-      .key("hoodie.write.meta.key.prefixes")
-      .defaultValue("")
-      .withDocumentation("Comma separated metadata key prefixes to override from latest commit "
-          + "during overlapping commits via multi writing");
+  public static final ConfigProperty<Boolean> WRITE_CONCURRENCY_MERGE_DELTASTREAMER_STATE = ConfigProperty
+          .key("hoodie.write.concurrency.merge.deltastreamer.state")
+          .defaultValue(false)
+          .withDocumentation("If enabled, this writer will merge Deltastreamer state "
+                  + "from the previous checkpoint in order to allow both realtime "
+                  + "and batch writers to ingest into a single table. "
+                  + "This should not be enabled on Deltastreamer writers.");

Review comment:
       do you think we can add in a validation withiin Deltastreamer code base that this config is not enabled. would be good to tighten it and not leave it to end user. 




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [hudi] hudi-bot edited a comment on pull request #3820: [HUDI-2579] Make deltastreamer checkpoint state merging more explicit

Posted by GitBox <gi...@apache.org>.
hudi-bot edited a comment on pull request #3820:
URL: https://github.com/apache/hudi/pull/3820#issuecomment-945958915


   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "8de6afb8a205a41de2a4b214c8982488b2b8ec19",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2708",
       "triggerID" : "8de6afb8a205a41de2a4b214c8982488b2b8ec19",
       "triggerType" : "PUSH"
     }, {
       "hash" : "4606a96bbce3b9638026eceac7ce0610f0b7045e",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2718",
       "triggerID" : "4606a96bbce3b9638026eceac7ce0610f0b7045e",
       "triggerType" : "PUSH"
     }, {
       "hash" : "c41001efeff2a7723ad45a637fb75336b58b294b",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2726",
       "triggerID" : "c41001efeff2a7723ad45a637fb75336b58b294b",
       "triggerType" : "PUSH"
     }, {
       "hash" : "ba8f8b0059fd3dcd549edf2d675605711b74d266",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2836",
       "triggerID" : "ba8f8b0059fd3dcd549edf2d675605711b74d266",
       "triggerType" : "PUSH"
     }, {
       "hash" : "1d031a881c677dee5e40a0119be17e6a35969f60",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2854",
       "triggerID" : "1d031a881c677dee5e40a0119be17e6a35969f60",
       "triggerType" : "PUSH"
     }, {
       "hash" : "9e8a962c9aee646713ebca560be2cfd234f25b09",
       "status" : "SUCCESS",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2910",
       "triggerID" : "9e8a962c9aee646713ebca560be2cfd234f25b09",
       "triggerType" : "PUSH"
     }, {
       "hash" : "4007755f0bb30705d3cc342f11ecfcebd3a108e9",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "4007755f0bb30705d3cc342f11ecfcebd3a108e9",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 9e8a962c9aee646713ebca560be2cfd234f25b09 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2910) 
   * 4007755f0bb30705d3cc342f11ecfcebd3a108e9 UNKNOWN
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run travis` re-run the last Travis build
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [hudi] davehagman commented on a change in pull request #3820: [HUDI-2579] Make deltastreamer checkpoint state merging more explicit

Posted by GitBox <gi...@apache.org>.
davehagman commented on a change in pull request #3820:
URL: https://github.com/apache/hudi/pull/3820#discussion_r733064710



##########
File path: hudi-client/hudi-client-common/src/main/java/org/apache/hudi/config/HoodieWriteConfig.java
##########
@@ -368,11 +372,13 @@
           + "OPTIMISTIC_CONCURRENCY_CONTROL: Multiple writers can operate on the table and exactly one of them succeed "
           + "if a conflict (writes affect the same file group) is detected.");
 
-  public static final ConfigProperty<String> WRITE_META_KEY_PREFIXES = ConfigProperty
-      .key("hoodie.write.meta.key.prefixes")
-      .defaultValue("")
-      .withDocumentation("Comma separated metadata key prefixes to override from latest commit "
-          + "during overlapping commits via multi writing");
+  public static final ConfigProperty<Boolean> WRITE_CONCURRENCY_MERGE_DELTASTREAMER_STATE = ConfigProperty
+          .key("hoodie.write.concurrency.merge.deltastreamer.state")
+          .defaultValue(false)
+          .withDocumentation("If enabled, this writer will merge Deltastreamer state "
+                  + "from the previous checkpoint in order to allow both realtime "
+                  + "and batch writers to ingest into a single table. "
+                  + "This should not be enabled on Deltastreamer writers.");

Review comment:
       That's a good idea. Let me add that




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [hudi] hudi-bot edited a comment on pull request #3820: [HUDI-2579] Make deltastreamer checkpoint state merging more explicit

Posted by GitBox <gi...@apache.org>.
hudi-bot edited a comment on pull request #3820:
URL: https://github.com/apache/hudi/pull/3820#issuecomment-945958915






-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [hudi] nsivabalan commented on pull request #3820: [HUDI-2579] Make deltastreamer checkpoint state merging more explicit

Posted by GitBox <gi...@apache.org>.
nsivabalan commented on pull request #3820:
URL: https://github.com/apache/hudi/pull/3820#issuecomment-956855780


   @hudi-bot azure run


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [hudi] hudi-bot removed a comment on pull request #3820: [HUDI-2579] Make deltastreamer checkpoint state merging more explicit

Posted by GitBox <gi...@apache.org>.
hudi-bot removed a comment on pull request #3820:
URL: https://github.com/apache/hudi/pull/3820#issuecomment-945958915


   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "8de6afb8a205a41de2a4b214c8982488b2b8ec19",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2708",
       "triggerID" : "8de6afb8a205a41de2a4b214c8982488b2b8ec19",
       "triggerType" : "PUSH"
     }, {
       "hash" : "4606a96bbce3b9638026eceac7ce0610f0b7045e",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2718",
       "triggerID" : "4606a96bbce3b9638026eceac7ce0610f0b7045e",
       "triggerType" : "PUSH"
     }, {
       "hash" : "c41001efeff2a7723ad45a637fb75336b58b294b",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2726",
       "triggerID" : "c41001efeff2a7723ad45a637fb75336b58b294b",
       "triggerType" : "PUSH"
     }, {
       "hash" : "ba8f8b0059fd3dcd549edf2d675605711b74d266",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2836",
       "triggerID" : "ba8f8b0059fd3dcd549edf2d675605711b74d266",
       "triggerType" : "PUSH"
     }, {
       "hash" : "1d031a881c677dee5e40a0119be17e6a35969f60",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2854",
       "triggerID" : "1d031a881c677dee5e40a0119be17e6a35969f60",
       "triggerType" : "PUSH"
     }, {
       "hash" : "9e8a962c9aee646713ebca560be2cfd234f25b09",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2910",
       "triggerID" : "9e8a962c9aee646713ebca560be2cfd234f25b09",
       "triggerType" : "PUSH"
     }, {
       "hash" : "4007755f0bb30705d3cc342f11ecfcebd3a108e9",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2911",
       "triggerID" : "4007755f0bb30705d3cc342f11ecfcebd3a108e9",
       "triggerType" : "PUSH"
     }, {
       "hash" : "255b35f65537a3adaedb14b25bb53d98beadb371",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2914",
       "triggerID" : "255b35f65537a3adaedb14b25bb53d98beadb371",
       "triggerType" : "PUSH"
     }, {
       "hash" : "d36815988f20c005b439af98d2a43bd9c145f7aa",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3020",
       "triggerID" : "d36815988f20c005b439af98d2a43bd9c145f7aa",
       "triggerType" : "PUSH"
     }, {
       "hash" : "98f42d69bed5505c9a1fa78e54da281fec73bd42",
       "status" : "SUCCESS",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3070",
       "triggerID" : "98f42d69bed5505c9a1fa78e54da281fec73bd42",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 98f42d69bed5505c9a1fa78e54da281fec73bd42 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3070) 
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run travis` re-run the last Travis build
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [hudi] nsivabalan edited a comment on pull request #3820: [BUGFIX] Merge commit state from previous commit instead of current

Posted by GitBox <gi...@apache.org>.
nsivabalan edited a comment on pull request #3820:
URL: https://github.com/apache/hudi/pull/3820#issuecomment-946024720


   thanks a lot for fixing this Dave. I would like to propose something here. 
   I am wondering why do we need to retrofit copying over delta streamer checkpoint into logic meant for hoodie.write.meta.key.prefixes. to me, this new requirement is very simple and not really tied to `hoodie.write.meta.key.prefixes`. 
   
   Let me propose something and see how that looks like.
   Introduce a new config called `hoodie.copy.over.deltastreamer.checkpoints`. we can brainstorm on actual naming later. 
   When set to true,
   within TransactionUtils::overrideWithLatestCommitMetadata
   ```
      fetch value of "deltastreamer.checkpoint.key" from last committed transaction and copy to cur inflight commit extra metadata.
   ```
   This is very tight and not error prone. Users don't need to set two different config as below which is not very intuitive as to why they need to do this. 
   ```
   hoodie.write.meta.key.prefixes = 'deltastreamer.checkpoint.key'
   ```
   and optionally 
   ```
   deltastreamer.checkpoint.key =. ""
   ```
   
   All users have to do is, for all of their spark writers, they need to set `hoodie.copy.over.deltastreamer.checkpoints` to true. 
   
   welcome thoughts @n3nash @vinothchandar @davehagman 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [hudi] hudi-bot edited a comment on pull request #3820: [HUDI-2579] Make deltastreamer checkpoint state merging more explicit

Posted by GitBox <gi...@apache.org>.
hudi-bot edited a comment on pull request #3820:
URL: https://github.com/apache/hudi/pull/3820#issuecomment-945958915


   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "8de6afb8a205a41de2a4b214c8982488b2b8ec19",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2708",
       "triggerID" : "8de6afb8a205a41de2a4b214c8982488b2b8ec19",
       "triggerType" : "PUSH"
     }, {
       "hash" : "4606a96bbce3b9638026eceac7ce0610f0b7045e",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2718",
       "triggerID" : "4606a96bbce3b9638026eceac7ce0610f0b7045e",
       "triggerType" : "PUSH"
     }, {
       "hash" : "c41001efeff2a7723ad45a637fb75336b58b294b",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2726",
       "triggerID" : "c41001efeff2a7723ad45a637fb75336b58b294b",
       "triggerType" : "PUSH"
     }, {
       "hash" : "ba8f8b0059fd3dcd549edf2d675605711b74d266",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2836",
       "triggerID" : "ba8f8b0059fd3dcd549edf2d675605711b74d266",
       "triggerType" : "PUSH"
     }, {
       "hash" : "1d031a881c677dee5e40a0119be17e6a35969f60",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2854",
       "triggerID" : "1d031a881c677dee5e40a0119be17e6a35969f60",
       "triggerType" : "PUSH"
     }, {
       "hash" : "9e8a962c9aee646713ebca560be2cfd234f25b09",
       "status" : "SUCCESS",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2910",
       "triggerID" : "9e8a962c9aee646713ebca560be2cfd234f25b09",
       "triggerType" : "PUSH"
     }, {
       "hash" : "4007755f0bb30705d3cc342f11ecfcebd3a108e9",
       "status" : "PENDING",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2911",
       "triggerID" : "4007755f0bb30705d3cc342f11ecfcebd3a108e9",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 9e8a962c9aee646713ebca560be2cfd234f25b09 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2910) 
   * 4007755f0bb30705d3cc342f11ecfcebd3a108e9 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2911) 
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run travis` re-run the last Travis build
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [hudi] nsivabalan commented on pull request #3820: [HUDI-2579] Make deltastreamer checkpoint state merging more explicit

Posted by GitBox <gi...@apache.org>.
nsivabalan commented on pull request #3820:
URL: https://github.com/apache/hudi/pull/3820#issuecomment-961621624






-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [hudi] nsivabalan merged pull request #3820: [HUDI-2579] Make deltastreamer checkpoint state merging more explicit

Posted by GitBox <gi...@apache.org>.
nsivabalan merged pull request #3820:
URL: https://github.com/apache/hudi/pull/3820


   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [hudi] nsivabalan commented on pull request #3820: [HUDI-2579] Make deltastreamer checkpoint state merging more explicit

Posted by GitBox <gi...@apache.org>.
nsivabalan commented on pull request #3820:
URL: https://github.com/apache/hudi/pull/3820#issuecomment-964237370


   @hudi-bot azure run


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [hudi] hudi-bot edited a comment on pull request #3820: [HUDI-2579] Make deltastreamer checkpoint state merging more explicit

Posted by GitBox <gi...@apache.org>.
hudi-bot edited a comment on pull request #3820:
URL: https://github.com/apache/hudi/pull/3820#issuecomment-945958915


   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "8de6afb8a205a41de2a4b214c8982488b2b8ec19",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2708",
       "triggerID" : "8de6afb8a205a41de2a4b214c8982488b2b8ec19",
       "triggerType" : "PUSH"
     }, {
       "hash" : "4606a96bbce3b9638026eceac7ce0610f0b7045e",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2718",
       "triggerID" : "4606a96bbce3b9638026eceac7ce0610f0b7045e",
       "triggerType" : "PUSH"
     }, {
       "hash" : "c41001efeff2a7723ad45a637fb75336b58b294b",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2726",
       "triggerID" : "c41001efeff2a7723ad45a637fb75336b58b294b",
       "triggerType" : "PUSH"
     }, {
       "hash" : "ba8f8b0059fd3dcd549edf2d675605711b74d266",
       "status" : "FAILURE",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2836",
       "triggerID" : "ba8f8b0059fd3dcd549edf2d675605711b74d266",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * ba8f8b0059fd3dcd549edf2d675605711b74d266 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2836) 
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run travis` re-run the last Travis build
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [hudi] hudi-bot edited a comment on pull request #3820: [HUDI-2579] Make deltastreamer checkpoint state merging more explicit

Posted by GitBox <gi...@apache.org>.
hudi-bot edited a comment on pull request #3820:
URL: https://github.com/apache/hudi/pull/3820#issuecomment-945958915


   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "8de6afb8a205a41de2a4b214c8982488b2b8ec19",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2708",
       "triggerID" : "8de6afb8a205a41de2a4b214c8982488b2b8ec19",
       "triggerType" : "PUSH"
     }, {
       "hash" : "4606a96bbce3b9638026eceac7ce0610f0b7045e",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2718",
       "triggerID" : "4606a96bbce3b9638026eceac7ce0610f0b7045e",
       "triggerType" : "PUSH"
     }, {
       "hash" : "c41001efeff2a7723ad45a637fb75336b58b294b",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2726",
       "triggerID" : "c41001efeff2a7723ad45a637fb75336b58b294b",
       "triggerType" : "PUSH"
     }, {
       "hash" : "ba8f8b0059fd3dcd549edf2d675605711b74d266",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2836",
       "triggerID" : "ba8f8b0059fd3dcd549edf2d675605711b74d266",
       "triggerType" : "PUSH"
     }, {
       "hash" : "1d031a881c677dee5e40a0119be17e6a35969f60",
       "status" : "FAILURE",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2854",
       "triggerID" : "1d031a881c677dee5e40a0119be17e6a35969f60",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 1d031a881c677dee5e40a0119be17e6a35969f60 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2854) 
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run travis` re-run the last Travis build
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [hudi] hudi-bot edited a comment on pull request #3820: [HUDI-2579] Make deltastreamer checkpoint state merging more explicit

Posted by GitBox <gi...@apache.org>.
hudi-bot edited a comment on pull request #3820:
URL: https://github.com/apache/hudi/pull/3820#issuecomment-945958915


   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "8de6afb8a205a41de2a4b214c8982488b2b8ec19",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2708",
       "triggerID" : "8de6afb8a205a41de2a4b214c8982488b2b8ec19",
       "triggerType" : "PUSH"
     }, {
       "hash" : "4606a96bbce3b9638026eceac7ce0610f0b7045e",
       "status" : "FAILURE",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2718",
       "triggerID" : "4606a96bbce3b9638026eceac7ce0610f0b7045e",
       "triggerType" : "PUSH"
     }, {
       "hash" : "c41001efeff2a7723ad45a637fb75336b58b294b",
       "status" : "PENDING",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2726",
       "triggerID" : "c41001efeff2a7723ad45a637fb75336b58b294b",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 4606a96bbce3b9638026eceac7ce0610f0b7045e Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2718) 
   * c41001efeff2a7723ad45a637fb75336b58b294b Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2726) 
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run travis` re-run the last Travis build
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [hudi] davehagman commented on pull request #3820: [BUGFIX] Merge commit state from previous commit instead of current

Posted by GitBox <gi...@apache.org>.
davehagman commented on pull request #3820:
URL: https://github.com/apache/hudi/pull/3820#issuecomment-945961702


   I also noticed that there isn't any documentation around `hoodie.write.meta.key.prefixes` config in the multi-writer docs. We should add something about it since it is very important if you're multi-writer table includes a deltastreamer. Thoughts?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [hudi] hudi-bot edited a comment on pull request #3820: [HUDI-2579] Merge commit state from previous commit instead of current

Posted by GitBox <gi...@apache.org>.
hudi-bot edited a comment on pull request #3820:
URL: https://github.com/apache/hudi/pull/3820#issuecomment-945958915


   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "8de6afb8a205a41de2a4b214c8982488b2b8ec19",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2708",
       "triggerID" : "8de6afb8a205a41de2a4b214c8982488b2b8ec19",
       "triggerType" : "PUSH"
     }, {
       "hash" : "4606a96bbce3b9638026eceac7ce0610f0b7045e",
       "status" : "FAILURE",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2718",
       "triggerID" : "4606a96bbce3b9638026eceac7ce0610f0b7045e",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 4606a96bbce3b9638026eceac7ce0610f0b7045e Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2718) 
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run travis` re-run the last Travis build
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [hudi] hudi-bot edited a comment on pull request #3820: [HUDI-2579] Merge commit state from previous commit instead of current

Posted by GitBox <gi...@apache.org>.
hudi-bot edited a comment on pull request #3820:
URL: https://github.com/apache/hudi/pull/3820#issuecomment-945958915


   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "8de6afb8a205a41de2a4b214c8982488b2b8ec19",
       "status" : "SUCCESS",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2708",
       "triggerID" : "8de6afb8a205a41de2a4b214c8982488b2b8ec19",
       "triggerType" : "PUSH"
     }, {
       "hash" : "4606a96bbce3b9638026eceac7ce0610f0b7045e",
       "status" : "PENDING",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2718",
       "triggerID" : "4606a96bbce3b9638026eceac7ce0610f0b7045e",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 8de6afb8a205a41de2a4b214c8982488b2b8ec19 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2708) 
   * 4606a96bbce3b9638026eceac7ce0610f0b7045e Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2718) 
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run travis` re-run the last Travis build
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [hudi] nsivabalan commented on pull request #3820: [BUGFIX] Merge commit state from previous commit instead of current

Posted by GitBox <gi...@apache.org>.
nsivabalan commented on pull request #3820:
URL: https://github.com/apache/hudi/pull/3820#issuecomment-946312014


   @davehagman let's proceed with the approach you suggested. If others have any thoughts, I can take it up in a follow up PR. but lets proceed with this for now. 
   One more request: 
   Do add a unit test for the changes in TransactionUtils. should be easy to add one. 
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [hudi] hudi-bot edited a comment on pull request #3820: [HUDI-2579] Make deltastreamer checkpoint state merging more explicit

Posted by GitBox <gi...@apache.org>.
hudi-bot edited a comment on pull request #3820:
URL: https://github.com/apache/hudi/pull/3820#issuecomment-945958915


   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "8de6afb8a205a41de2a4b214c8982488b2b8ec19",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2708",
       "triggerID" : "8de6afb8a205a41de2a4b214c8982488b2b8ec19",
       "triggerType" : "PUSH"
     }, {
       "hash" : "4606a96bbce3b9638026eceac7ce0610f0b7045e",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2718",
       "triggerID" : "4606a96bbce3b9638026eceac7ce0610f0b7045e",
       "triggerType" : "PUSH"
     }, {
       "hash" : "c41001efeff2a7723ad45a637fb75336b58b294b",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2726",
       "triggerID" : "c41001efeff2a7723ad45a637fb75336b58b294b",
       "triggerType" : "PUSH"
     }, {
       "hash" : "ba8f8b0059fd3dcd549edf2d675605711b74d266",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2836",
       "triggerID" : "ba8f8b0059fd3dcd549edf2d675605711b74d266",
       "triggerType" : "PUSH"
     }, {
       "hash" : "1d031a881c677dee5e40a0119be17e6a35969f60",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2854",
       "triggerID" : "1d031a881c677dee5e40a0119be17e6a35969f60",
       "triggerType" : "PUSH"
     }, {
       "hash" : "9e8a962c9aee646713ebca560be2cfd234f25b09",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2910",
       "triggerID" : "9e8a962c9aee646713ebca560be2cfd234f25b09",
       "triggerType" : "PUSH"
     }, {
       "hash" : "4007755f0bb30705d3cc342f11ecfcebd3a108e9",
       "status" : "SUCCESS",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2911",
       "triggerID" : "4007755f0bb30705d3cc342f11ecfcebd3a108e9",
       "triggerType" : "PUSH"
     }, {
       "hash" : "255b35f65537a3adaedb14b25bb53d98beadb371",
       "status" : "PENDING",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2914",
       "triggerID" : "255b35f65537a3adaedb14b25bb53d98beadb371",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 4007755f0bb30705d3cc342f11ecfcebd3a108e9 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2911) 
   * 255b35f65537a3adaedb14b25bb53d98beadb371 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2914) 
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run travis` re-run the last Travis build
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [hudi] davehagman commented on a change in pull request #3820: [HUDI-2579] Make deltastreamer checkpoint state merging more explicit

Posted by GitBox <gi...@apache.org>.
davehagman commented on a change in pull request #3820:
URL: https://github.com/apache/hudi/pull/3820#discussion_r737560726



##########
File path: hudi-client/hudi-client-common/src/main/java/org/apache/hudi/client/utils/TransactionUtils.java
##########
@@ -111,16 +118,27 @@
     }
   }
 
-  // override the current metadata with the metadata from the latest instant for the specified key prefixes
-  private static void overrideWithLatestCommitMetadata(HoodieTableMetaClient metaClient, Option<HoodieCommitMetadata> thisMetadata,
-      Option<HoodieInstant> thisInstant, List<String> keyPrefixes) {
+  private static void mergeCheckpointStateFromPreviousCommit(HoodieTableMetaClient metaClient, Option<HoodieCommitMetadata> thisMetadata) {
+    overrideWithLatestCommitMetadata(metaClient, thisMetadata, Collections.singletonList(HoodieWriteConfig.DELTASTREAMER_CHECKPOINT_KEY));
+  }
+
+  /**
+   * Generic method allowing us to override the current metadata with the metadata from
+   * the latest instant for the specified key prefixes
+   * @param metaClient
+   * @param thisMetadata
+   * @param keyPrefixes The key prefixes to merge from the previous commit
+   */
+  private static void overrideWithLatestCommitMetadata(HoodieTableMetaClient metaClient,

Review comment:
       > Were you able to write unit tests for mergeCheckpointStateFromPreviousCommit atleast. you can mock getLastCompletedTxnInstantAndMetadata if need be to make it easier.
   
   Working on this now




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [hudi] hudi-bot edited a comment on pull request #3820: [HUDI-2579] Make deltastreamer checkpoint state merging more explicit

Posted by GitBox <gi...@apache.org>.
hudi-bot edited a comment on pull request #3820:
URL: https://github.com/apache/hudi/pull/3820#issuecomment-945958915


   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "8de6afb8a205a41de2a4b214c8982488b2b8ec19",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2708",
       "triggerID" : "8de6afb8a205a41de2a4b214c8982488b2b8ec19",
       "triggerType" : "PUSH"
     }, {
       "hash" : "4606a96bbce3b9638026eceac7ce0610f0b7045e",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2718",
       "triggerID" : "4606a96bbce3b9638026eceac7ce0610f0b7045e",
       "triggerType" : "PUSH"
     }, {
       "hash" : "c41001efeff2a7723ad45a637fb75336b58b294b",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2726",
       "triggerID" : "c41001efeff2a7723ad45a637fb75336b58b294b",
       "triggerType" : "PUSH"
     }, {
       "hash" : "ba8f8b0059fd3dcd549edf2d675605711b74d266",
       "status" : "FAILURE",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2836",
       "triggerID" : "ba8f8b0059fd3dcd549edf2d675605711b74d266",
       "triggerType" : "PUSH"
     }, {
       "hash" : "1d031a881c677dee5e40a0119be17e6a35969f60",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "1d031a881c677dee5e40a0119be17e6a35969f60",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * ba8f8b0059fd3dcd549edf2d675605711b74d266 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2836) 
   * 1d031a881c677dee5e40a0119be17e6a35969f60 UNKNOWN
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run travis` re-run the last Travis build
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [hudi] nsivabalan commented on pull request #3820: [HUDI-2579] Make deltastreamer checkpoint state merging more explicit

Posted by GitBox <gi...@apache.org>.
nsivabalan commented on pull request #3820:
URL: https://github.com/apache/hudi/pull/3820#issuecomment-951022359


   @n3nash : This patch is removing a config used in previous releases wrt config used for multi writer for checkpoint migration. would it suffice if we add a section in migration regarding the same. appreciate your inputs here.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [hudi] davehagman commented on a change in pull request #3820: [HUDI-2579] Make deltastreamer checkpoint state merging more explicit

Posted by GitBox <gi...@apache.org>.
davehagman commented on a change in pull request #3820:
URL: https://github.com/apache/hudi/pull/3820#discussion_r735757338



##########
File path: hudi-client/hudi-client-common/src/main/java/org/apache/hudi/client/utils/TransactionUtils.java
##########
@@ -111,16 +118,27 @@
     }
   }
 
-  // override the current metadata with the metadata from the latest instant for the specified key prefixes
-  private static void overrideWithLatestCommitMetadata(HoodieTableMetaClient metaClient, Option<HoodieCommitMetadata> thisMetadata,
-      Option<HoodieInstant> thisInstant, List<String> keyPrefixes) {
+  private static void mergeCheckpointStateFromPreviousCommit(HoodieTableMetaClient metaClient, Option<HoodieCommitMetadata> thisMetadata) {
+    overrideWithLatestCommitMetadata(metaClient, thisMetadata, Collections.singletonList(HoodieWriteConfig.DELTASTREAMER_CHECKPOINT_KEY));
+  }
+
+  /**
+   * Generic method allowing us to override the current metadata with the metadata from
+   * the latest instant for the specified key prefixes
+   * @param metaClient
+   * @param thisMetadata
+   * @param keyPrefixes The key prefixes to merge from the previous commit
+   */
+  private static void overrideWithLatestCommitMetadata(HoodieTableMetaClient metaClient,

Review comment:
       I only kept it because it does dry-up the solution overall since any metadata merge operation is going to use this same process. Merging state specific to the deltastreamer is just one example of this. Does that make sense? Should I still remove it?

##########
File path: hudi-client/hudi-client-common/src/main/java/org/apache/hudi/config/HoodieWriteConfig.java
##########
@@ -368,11 +372,13 @@
           + "OPTIMISTIC_CONCURRENCY_CONTROL: Multiple writers can operate on the table and exactly one of them succeed "
           + "if a conflict (writes affect the same file group) is detected.");
 
-  public static final ConfigProperty<String> WRITE_META_KEY_PREFIXES = ConfigProperty
-      .key("hoodie.write.meta.key.prefixes")
-      .defaultValue("")
-      .withDocumentation("Comma separated metadata key prefixes to override from latest commit "
-          + "during overlapping commits via multi writing");
+  public static final ConfigProperty<Boolean> WRITE_CONCURRENCY_MERGE_DELTASTREAMER_STATE = ConfigProperty
+          .key("hoodie.write.concurrency.merge.deltastreamer.state")
+          .defaultValue(false)
+          .withDocumentation("If enabled, this writer will merge Deltastreamer state "
+                  + "from the previous checkpoint in order to allow both realtime "
+                  + "and batch writers to ingest into a single table. "
+                  + "This should not be enabled on Deltastreamer writers.");

Review comment:
       Done!




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [hudi] hudi-bot commented on pull request #3820: [HUDI-2579] Make deltastreamer checkpoint state merging more explicit

Posted by GitBox <gi...@apache.org>.
hudi-bot commented on pull request #3820:
URL: https://github.com/apache/hudi/pull/3820#issuecomment-961588596






-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [hudi] nsivabalan edited a comment on pull request #3820: [HUDI-2579] Make deltastreamer checkpoint state merging more explicit

Posted by GitBox <gi...@apache.org>.
nsivabalan edited a comment on pull request #3820:
URL: https://github.com/apache/hudi/pull/3820#issuecomment-962460574


   @n3nash : hmmm, the new config may not meet the 2nd requirement. it just takes care of 1st one i.e. copying over deltastreamer checkpoint from previous commit. Do you think one of you folks from uber can take this up immediately as a follow up. Or you wish to get it fixed in this patch itself? Even if we support the 2nd, I feel we could still have the new way of copying over deltastreamer checkpoint. From a user's standpoint who is interested in multi-writer, the new config is more clear and crisp that way. 
   Let me know your opinion. 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [hudi] nsivabalan commented on pull request #3820: [HUDI-2579] Make deltastreamer checkpoint state merging more explicit

Posted by GitBox <gi...@apache.org>.
nsivabalan commented on pull request #3820:
URL: https://github.com/apache/hudi/pull/3820#issuecomment-962460574


   @n3nash : hmmm, the new config may not meet the 2nd requirement. it just takes care of 1st one i.e. copying over deltastreamer checkpoint from previous commit. Do you think one of you folks from uber can take this up immediately as a follow up. Or you wish to get it fixed in this patch itself? 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [hudi] nsivabalan commented on pull request #3820: [HUDI-2579] Make deltastreamer checkpoint state merging more explicit

Posted by GitBox <gi...@apache.org>.
nsivabalan commented on pull request #3820:
URL: https://github.com/apache/hudi/pull/3820#issuecomment-951478890


   @hudi-bot azure run


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [hudi] hudi-bot edited a comment on pull request #3820: [HUDI-2579] Make deltastreamer checkpoint state merging more explicit

Posted by GitBox <gi...@apache.org>.
hudi-bot edited a comment on pull request #3820:
URL: https://github.com/apache/hudi/pull/3820#issuecomment-945958915


   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "8de6afb8a205a41de2a4b214c8982488b2b8ec19",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2708",
       "triggerID" : "8de6afb8a205a41de2a4b214c8982488b2b8ec19",
       "triggerType" : "PUSH"
     }, {
       "hash" : "4606a96bbce3b9638026eceac7ce0610f0b7045e",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2718",
       "triggerID" : "4606a96bbce3b9638026eceac7ce0610f0b7045e",
       "triggerType" : "PUSH"
     }, {
       "hash" : "c41001efeff2a7723ad45a637fb75336b58b294b",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2726",
       "triggerID" : "c41001efeff2a7723ad45a637fb75336b58b294b",
       "triggerType" : "PUSH"
     }, {
       "hash" : "ba8f8b0059fd3dcd549edf2d675605711b74d266",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2836",
       "triggerID" : "ba8f8b0059fd3dcd549edf2d675605711b74d266",
       "triggerType" : "PUSH"
     }, {
       "hash" : "1d031a881c677dee5e40a0119be17e6a35969f60",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2854",
       "triggerID" : "1d031a881c677dee5e40a0119be17e6a35969f60",
       "triggerType" : "PUSH"
     }, {
       "hash" : "9e8a962c9aee646713ebca560be2cfd234f25b09",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2910",
       "triggerID" : "9e8a962c9aee646713ebca560be2cfd234f25b09",
       "triggerType" : "PUSH"
     }, {
       "hash" : "4007755f0bb30705d3cc342f11ecfcebd3a108e9",
       "status" : "SUCCESS",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2911",
       "triggerID" : "4007755f0bb30705d3cc342f11ecfcebd3a108e9",
       "triggerType" : "PUSH"
     }, {
       "hash" : "255b35f65537a3adaedb14b25bb53d98beadb371",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "255b35f65537a3adaedb14b25bb53d98beadb371",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 4007755f0bb30705d3cc342f11ecfcebd3a108e9 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2911) 
   * 255b35f65537a3adaedb14b25bb53d98beadb371 UNKNOWN
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run travis` re-run the last Travis build
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [hudi] hudi-bot edited a comment on pull request #3820: [HUDI-2579] Make deltastreamer checkpoint state merging more explicit

Posted by GitBox <gi...@apache.org>.
hudi-bot edited a comment on pull request #3820:
URL: https://github.com/apache/hudi/pull/3820#issuecomment-945958915


   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "8de6afb8a205a41de2a4b214c8982488b2b8ec19",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2708",
       "triggerID" : "8de6afb8a205a41de2a4b214c8982488b2b8ec19",
       "triggerType" : "PUSH"
     }, {
       "hash" : "4606a96bbce3b9638026eceac7ce0610f0b7045e",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2718",
       "triggerID" : "4606a96bbce3b9638026eceac7ce0610f0b7045e",
       "triggerType" : "PUSH"
     }, {
       "hash" : "c41001efeff2a7723ad45a637fb75336b58b294b",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2726",
       "triggerID" : "c41001efeff2a7723ad45a637fb75336b58b294b",
       "triggerType" : "PUSH"
     }, {
       "hash" : "ba8f8b0059fd3dcd549edf2d675605711b74d266",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2836",
       "triggerID" : "ba8f8b0059fd3dcd549edf2d675605711b74d266",
       "triggerType" : "PUSH"
     }, {
       "hash" : "1d031a881c677dee5e40a0119be17e6a35969f60",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2854",
       "triggerID" : "1d031a881c677dee5e40a0119be17e6a35969f60",
       "triggerType" : "PUSH"
     }, {
       "hash" : "9e8a962c9aee646713ebca560be2cfd234f25b09",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2910",
       "triggerID" : "9e8a962c9aee646713ebca560be2cfd234f25b09",
       "triggerType" : "PUSH"
     }, {
       "hash" : "4007755f0bb30705d3cc342f11ecfcebd3a108e9",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2911",
       "triggerID" : "4007755f0bb30705d3cc342f11ecfcebd3a108e9",
       "triggerType" : "PUSH"
     }, {
       "hash" : "255b35f65537a3adaedb14b25bb53d98beadb371",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2914",
       "triggerID" : "255b35f65537a3adaedb14b25bb53d98beadb371",
       "triggerType" : "PUSH"
     }, {
       "hash" : "d36815988f20c005b439af98d2a43bd9c145f7aa",
       "status" : "FAILURE",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3020",
       "triggerID" : "d36815988f20c005b439af98d2a43bd9c145f7aa",
       "triggerType" : "PUSH"
     }, {
       "hash" : "98f42d69bed5505c9a1fa78e54da281fec73bd42",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "98f42d69bed5505c9a1fa78e54da281fec73bd42",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * d36815988f20c005b439af98d2a43bd9c145f7aa Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3020) 
   * 98f42d69bed5505c9a1fa78e54da281fec73bd42 UNKNOWN
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run travis` re-run the last Travis build
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [hudi] hudi-bot removed a comment on pull request #3820: [HUDI-2579] Make deltastreamer checkpoint state merging more explicit

Posted by GitBox <gi...@apache.org>.
hudi-bot removed a comment on pull request #3820:
URL: https://github.com/apache/hudi/pull/3820#issuecomment-961588596


   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "8de6afb8a205a41de2a4b214c8982488b2b8ec19",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2708",
       "triggerID" : "8de6afb8a205a41de2a4b214c8982488b2b8ec19",
       "triggerType" : "PUSH"
     }, {
       "hash" : "4606a96bbce3b9638026eceac7ce0610f0b7045e",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2718",
       "triggerID" : "4606a96bbce3b9638026eceac7ce0610f0b7045e",
       "triggerType" : "PUSH"
     }, {
       "hash" : "c41001efeff2a7723ad45a637fb75336b58b294b",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2726",
       "triggerID" : "c41001efeff2a7723ad45a637fb75336b58b294b",
       "triggerType" : "PUSH"
     }, {
       "hash" : "ba8f8b0059fd3dcd549edf2d675605711b74d266",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2836",
       "triggerID" : "ba8f8b0059fd3dcd549edf2d675605711b74d266",
       "triggerType" : "PUSH"
     }, {
       "hash" : "1d031a881c677dee5e40a0119be17e6a35969f60",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2854",
       "triggerID" : "1d031a881c677dee5e40a0119be17e6a35969f60",
       "triggerType" : "PUSH"
     }, {
       "hash" : "9e8a962c9aee646713ebca560be2cfd234f25b09",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2910",
       "triggerID" : "9e8a962c9aee646713ebca560be2cfd234f25b09",
       "triggerType" : "PUSH"
     }, {
       "hash" : "4007755f0bb30705d3cc342f11ecfcebd3a108e9",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2911",
       "triggerID" : "4007755f0bb30705d3cc342f11ecfcebd3a108e9",
       "triggerType" : "PUSH"
     }, {
       "hash" : "255b35f65537a3adaedb14b25bb53d98beadb371",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2914",
       "triggerID" : "255b35f65537a3adaedb14b25bb53d98beadb371",
       "triggerType" : "PUSH"
     }, {
       "hash" : "d36815988f20c005b439af98d2a43bd9c145f7aa",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3020",
       "triggerID" : "d36815988f20c005b439af98d2a43bd9c145f7aa",
       "triggerType" : "PUSH"
     }, {
       "hash" : "98f42d69bed5505c9a1fa78e54da281fec73bd42",
       "status" : "SUCCESS",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3070",
       "triggerID" : "98f42d69bed5505c9a1fa78e54da281fec73bd42",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 98f42d69bed5505c9a1fa78e54da281fec73bd42 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3070) 
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [hudi] nsivabalan commented on pull request #3820: [BUGFIX] Merge commit state from previous commit instead of current

Posted by GitBox <gi...@apache.org>.
nsivabalan commented on pull request #3820:
URL: https://github.com/apache/hudi/pull/3820#issuecomment-946148718


   yeah, the naming looks fine by me. 
   
   btw, Can you please attach jira ticket to PR. prefix w/ ticket id. Especially for bugs, we need a tracking ticket. 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [hudi] hudi-bot edited a comment on pull request #3820: [BUGFIX] Merge commit state from previous commit instead of current

Posted by GitBox <gi...@apache.org>.
hudi-bot edited a comment on pull request #3820:
URL: https://github.com/apache/hudi/pull/3820#issuecomment-945958915


   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "8de6afb8a205a41de2a4b214c8982488b2b8ec19",
       "status" : "SUCCESS",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2708",
       "triggerID" : "8de6afb8a205a41de2a4b214c8982488b2b8ec19",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 8de6afb8a205a41de2a4b214c8982488b2b8ec19 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2708) 
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run travis` re-run the last Travis build
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [hudi] hudi-bot commented on pull request #3820: [BUGFIX] Merge commit state from previous commit instead of current

Posted by GitBox <gi...@apache.org>.
hudi-bot commented on pull request #3820:
URL: https://github.com/apache/hudi/pull/3820#issuecomment-945958915


   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "8de6afb8a205a41de2a4b214c8982488b2b8ec19",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "8de6afb8a205a41de2a4b214c8982488b2b8ec19",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 8de6afb8a205a41de2a4b214c8982488b2b8ec19 UNKNOWN
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run travis` re-run the last Travis build
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [hudi] hudi-bot edited a comment on pull request #3820: [HUDI-2579] Make deltastreamer checkpoint state merging more explicit

Posted by GitBox <gi...@apache.org>.
hudi-bot edited a comment on pull request #3820:
URL: https://github.com/apache/hudi/pull/3820#issuecomment-945958915


   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "8de6afb8a205a41de2a4b214c8982488b2b8ec19",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2708",
       "triggerID" : "8de6afb8a205a41de2a4b214c8982488b2b8ec19",
       "triggerType" : "PUSH"
     }, {
       "hash" : "4606a96bbce3b9638026eceac7ce0610f0b7045e",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2718",
       "triggerID" : "4606a96bbce3b9638026eceac7ce0610f0b7045e",
       "triggerType" : "PUSH"
     }, {
       "hash" : "c41001efeff2a7723ad45a637fb75336b58b294b",
       "status" : "FAILURE",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2726",
       "triggerID" : "c41001efeff2a7723ad45a637fb75336b58b294b",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * c41001efeff2a7723ad45a637fb75336b58b294b Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2726) 
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run travis` re-run the last Travis build
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [hudi] nsivabalan commented on pull request #3820: [HUDI-2579] Make deltastreamer checkpoint state merging more explicit

Posted by GitBox <gi...@apache.org>.
nsivabalan commented on pull request #3820:
URL: https://github.com/apache/hudi/pull/3820#issuecomment-947721497


   wanted to think about backwards compatability. if someone was using similar set up before (090) and now upgrades, they might have to fix their config. we can call it out in migration section. just leaving a comment for reminder.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [hudi] hudi-bot edited a comment on pull request #3820: [HUDI-2579] Make deltastreamer checkpoint state merging more explicit

Posted by GitBox <gi...@apache.org>.
hudi-bot edited a comment on pull request #3820:
URL: https://github.com/apache/hudi/pull/3820#issuecomment-945958915


   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "8de6afb8a205a41de2a4b214c8982488b2b8ec19",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2708",
       "triggerID" : "8de6afb8a205a41de2a4b214c8982488b2b8ec19",
       "triggerType" : "PUSH"
     }, {
       "hash" : "4606a96bbce3b9638026eceac7ce0610f0b7045e",
       "status" : "FAILURE",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2718",
       "triggerID" : "4606a96bbce3b9638026eceac7ce0610f0b7045e",
       "triggerType" : "PUSH"
     }, {
       "hash" : "c41001efeff2a7723ad45a637fb75336b58b294b",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "c41001efeff2a7723ad45a637fb75336b58b294b",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 4606a96bbce3b9638026eceac7ce0610f0b7045e Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2718) 
   * c41001efeff2a7723ad45a637fb75336b58b294b UNKNOWN
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run travis` re-run the last Travis build
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [hudi] nsivabalan commented on a change in pull request #3820: [HUDI-2579] Make deltastreamer checkpoint state merging more explicit

Posted by GitBox <gi...@apache.org>.
nsivabalan commented on a change in pull request #3820:
URL: https://github.com/apache/hudi/pull/3820#discussion_r733011813



##########
File path: hudi-client/hudi-client-common/src/main/java/org/apache/hudi/client/utils/TransactionUtils.java
##########
@@ -111,16 +118,27 @@
     }
   }
 
-  // override the current metadata with the metadata from the latest instant for the specified key prefixes
-  private static void overrideWithLatestCommitMetadata(HoodieTableMetaClient metaClient, Option<HoodieCommitMetadata> thisMetadata,
-      Option<HoodieInstant> thisInstant, List<String> keyPrefixes) {
+  private static void mergeCheckpointStateFromPreviousCommit(HoodieTableMetaClient metaClient, Option<HoodieCommitMetadata> thisMetadata) {
+    overrideWithLatestCommitMetadata(metaClient, thisMetadata, Collections.singletonList(HoodieWriteConfig.DELTASTREAMER_CHECKPOINT_KEY));
+  }
+
+  /**
+   * Generic method allowing us to override the current metadata with the metadata from
+   * the latest instant for the specified key prefixes
+   * @param metaClient
+   * @param thisMetadata
+   * @param keyPrefixes The key prefixes to merge from the previous commit
+   */
+  private static void overrideWithLatestCommitMetadata(HoodieTableMetaClient metaClient,

Review comment:
       I don't see a need for this method. we can just embed within mergeCheckpointStateFromPreviousCommit. basically, rename overrideWithLatestCommitMetadata to mergeCheckpointStateFromPreviousCommit.

##########
File path: hudi-client/hudi-client-common/src/main/java/org/apache/hudi/config/HoodieWriteConfig.java
##########
@@ -368,11 +372,13 @@
           + "OPTIMISTIC_CONCURRENCY_CONTROL: Multiple writers can operate on the table and exactly one of them succeed "
           + "if a conflict (writes affect the same file group) is detected.");
 
-  public static final ConfigProperty<String> WRITE_META_KEY_PREFIXES = ConfigProperty
-      .key("hoodie.write.meta.key.prefixes")
-      .defaultValue("")
-      .withDocumentation("Comma separated metadata key prefixes to override from latest commit "
-          + "during overlapping commits via multi writing");
+  public static final ConfigProperty<Boolean> WRITE_CONCURRENCY_MERGE_DELTASTREAMER_STATE = ConfigProperty
+          .key("hoodie.write.concurrency.merge.deltastreamer.state")
+          .defaultValue(false)
+          .withDocumentation("If enabled, this writer will merge Deltastreamer state "
+                  + "from the previous checkpoint in order to allow both realtime "
+                  + "and batch writers to ingest into a single table. "
+                  + "This should not be enabled on Deltastreamer writers.");

Review comment:
       do you think we can add in a validation withiin Deltastreamer code base that this config is not enabled. would be good to tighten it and leave it to end user. 




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [hudi] davehagman commented on pull request #3820: [HUDI-2579] Merge commit state from previous commit instead of current

Posted by GitBox <gi...@apache.org>.
davehagman commented on pull request #3820:
URL: https://github.com/apache/hudi/pull/3820#issuecomment-946759009


   @nsivabalan The metadata prefixes config is now unused with this PR ([here](https://github.com/apache/hudi/blob/master/hudi-client/hudi-client-common/src/main/java/org/apache/hudi/config/HoodieWriteConfig.java#L371)). Should we just remove this config? Are there any other use-cases for it?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [hudi] hudi-bot edited a comment on pull request #3820: [HUDI-2579] Make deltastreamer checkpoint state merging more explicit

Posted by GitBox <gi...@apache.org>.
hudi-bot edited a comment on pull request #3820:
URL: https://github.com/apache/hudi/pull/3820#issuecomment-945958915


   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "8de6afb8a205a41de2a4b214c8982488b2b8ec19",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2708",
       "triggerID" : "8de6afb8a205a41de2a4b214c8982488b2b8ec19",
       "triggerType" : "PUSH"
     }, {
       "hash" : "4606a96bbce3b9638026eceac7ce0610f0b7045e",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2718",
       "triggerID" : "4606a96bbce3b9638026eceac7ce0610f0b7045e",
       "triggerType" : "PUSH"
     }, {
       "hash" : "c41001efeff2a7723ad45a637fb75336b58b294b",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2726",
       "triggerID" : "c41001efeff2a7723ad45a637fb75336b58b294b",
       "triggerType" : "PUSH"
     }, {
       "hash" : "ba8f8b0059fd3dcd549edf2d675605711b74d266",
       "status" : "FAILURE",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2836",
       "triggerID" : "ba8f8b0059fd3dcd549edf2d675605711b74d266",
       "triggerType" : "PUSH"
     }, {
       "hash" : "1d031a881c677dee5e40a0119be17e6a35969f60",
       "status" : "PENDING",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2854",
       "triggerID" : "1d031a881c677dee5e40a0119be17e6a35969f60",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * ba8f8b0059fd3dcd549edf2d675605711b74d266 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2836) 
   * 1d031a881c677dee5e40a0119be17e6a35969f60 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2854) 
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run travis` re-run the last Travis build
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [hudi] davehagman commented on pull request #3820: [HUDI-2579] Make deltastreamer checkpoint state merging more explicit

Posted by GitBox <gi...@apache.org>.
davehagman commented on pull request #3820:
URL: https://github.com/apache/hudi/pull/3820#issuecomment-953014932


   > Were you able to write unit tests for mergeCheckpointStateFromPreviousCommit atleast. you can mock getLastCompletedTxnInstantAndMetadata if need be to make it easier.
   
   Working on this now


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [hudi] hudi-bot edited a comment on pull request #3820: [HUDI-2579] Make deltastreamer checkpoint state merging more explicit

Posted by GitBox <gi...@apache.org>.
hudi-bot edited a comment on pull request #3820:
URL: https://github.com/apache/hudi/pull/3820#issuecomment-945958915


   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "8de6afb8a205a41de2a4b214c8982488b2b8ec19",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2708",
       "triggerID" : "8de6afb8a205a41de2a4b214c8982488b2b8ec19",
       "triggerType" : "PUSH"
     }, {
       "hash" : "4606a96bbce3b9638026eceac7ce0610f0b7045e",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2718",
       "triggerID" : "4606a96bbce3b9638026eceac7ce0610f0b7045e",
       "triggerType" : "PUSH"
     }, {
       "hash" : "c41001efeff2a7723ad45a637fb75336b58b294b",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2726",
       "triggerID" : "c41001efeff2a7723ad45a637fb75336b58b294b",
       "triggerType" : "PUSH"
     }, {
       "hash" : "ba8f8b0059fd3dcd549edf2d675605711b74d266",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2836",
       "triggerID" : "ba8f8b0059fd3dcd549edf2d675605711b74d266",
       "triggerType" : "PUSH"
     }, {
       "hash" : "1d031a881c677dee5e40a0119be17e6a35969f60",
       "status" : "FAILURE",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2854",
       "triggerID" : "1d031a881c677dee5e40a0119be17e6a35969f60",
       "triggerType" : "PUSH"
     }, {
       "hash" : "9e8a962c9aee646713ebca560be2cfd234f25b09",
       "status" : "PENDING",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2910",
       "triggerID" : "9e8a962c9aee646713ebca560be2cfd234f25b09",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 1d031a881c677dee5e40a0119be17e6a35969f60 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2854) 
   * 9e8a962c9aee646713ebca560be2cfd234f25b09 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2910) 
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run travis` re-run the last Travis build
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [hudi] hudi-bot edited a comment on pull request #3820: [HUDI-2579] Make deltastreamer checkpoint state merging more explicit

Posted by GitBox <gi...@apache.org>.
hudi-bot edited a comment on pull request #3820:
URL: https://github.com/apache/hudi/pull/3820#issuecomment-945958915


   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "8de6afb8a205a41de2a4b214c8982488b2b8ec19",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2708",
       "triggerID" : "8de6afb8a205a41de2a4b214c8982488b2b8ec19",
       "triggerType" : "PUSH"
     }, {
       "hash" : "4606a96bbce3b9638026eceac7ce0610f0b7045e",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2718",
       "triggerID" : "4606a96bbce3b9638026eceac7ce0610f0b7045e",
       "triggerType" : "PUSH"
     }, {
       "hash" : "c41001efeff2a7723ad45a637fb75336b58b294b",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2726",
       "triggerID" : "c41001efeff2a7723ad45a637fb75336b58b294b",
       "triggerType" : "PUSH"
     }, {
       "hash" : "ba8f8b0059fd3dcd549edf2d675605711b74d266",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2836",
       "triggerID" : "ba8f8b0059fd3dcd549edf2d675605711b74d266",
       "triggerType" : "PUSH"
     }, {
       "hash" : "1d031a881c677dee5e40a0119be17e6a35969f60",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2854",
       "triggerID" : "1d031a881c677dee5e40a0119be17e6a35969f60",
       "triggerType" : "PUSH"
     }, {
       "hash" : "9e8a962c9aee646713ebca560be2cfd234f25b09",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2910",
       "triggerID" : "9e8a962c9aee646713ebca560be2cfd234f25b09",
       "triggerType" : "PUSH"
     }, {
       "hash" : "4007755f0bb30705d3cc342f11ecfcebd3a108e9",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2911",
       "triggerID" : "4007755f0bb30705d3cc342f11ecfcebd3a108e9",
       "triggerType" : "PUSH"
     }, {
       "hash" : "255b35f65537a3adaedb14b25bb53d98beadb371",
       "status" : "SUCCESS",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2914",
       "triggerID" : "255b35f65537a3adaedb14b25bb53d98beadb371",
       "triggerType" : "PUSH"
     }, {
       "hash" : "d36815988f20c005b439af98d2a43bd9c145f7aa",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "d36815988f20c005b439af98d2a43bd9c145f7aa",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 255b35f65537a3adaedb14b25bb53d98beadb371 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2914) 
   * d36815988f20c005b439af98d2a43bd9c145f7aa UNKNOWN
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run travis` re-run the last Travis build
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [hudi] hudi-bot edited a comment on pull request #3820: [HUDI-2579] Make deltastreamer checkpoint state merging more explicit

Posted by GitBox <gi...@apache.org>.
hudi-bot edited a comment on pull request #3820:
URL: https://github.com/apache/hudi/pull/3820#issuecomment-945958915


   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "8de6afb8a205a41de2a4b214c8982488b2b8ec19",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2708",
       "triggerID" : "8de6afb8a205a41de2a4b214c8982488b2b8ec19",
       "triggerType" : "PUSH"
     }, {
       "hash" : "4606a96bbce3b9638026eceac7ce0610f0b7045e",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2718",
       "triggerID" : "4606a96bbce3b9638026eceac7ce0610f0b7045e",
       "triggerType" : "PUSH"
     }, {
       "hash" : "c41001efeff2a7723ad45a637fb75336b58b294b",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2726",
       "triggerID" : "c41001efeff2a7723ad45a637fb75336b58b294b",
       "triggerType" : "PUSH"
     }, {
       "hash" : "ba8f8b0059fd3dcd549edf2d675605711b74d266",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2836",
       "triggerID" : "ba8f8b0059fd3dcd549edf2d675605711b74d266",
       "triggerType" : "PUSH"
     }, {
       "hash" : "1d031a881c677dee5e40a0119be17e6a35969f60",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2854",
       "triggerID" : "1d031a881c677dee5e40a0119be17e6a35969f60",
       "triggerType" : "PUSH"
     }, {
       "hash" : "9e8a962c9aee646713ebca560be2cfd234f25b09",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2910",
       "triggerID" : "9e8a962c9aee646713ebca560be2cfd234f25b09",
       "triggerType" : "PUSH"
     }, {
       "hash" : "4007755f0bb30705d3cc342f11ecfcebd3a108e9",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2911",
       "triggerID" : "4007755f0bb30705d3cc342f11ecfcebd3a108e9",
       "triggerType" : "PUSH"
     }, {
       "hash" : "255b35f65537a3adaedb14b25bb53d98beadb371",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2914",
       "triggerID" : "255b35f65537a3adaedb14b25bb53d98beadb371",
       "triggerType" : "PUSH"
     }, {
       "hash" : "d36815988f20c005b439af98d2a43bd9c145f7aa",
       "status" : "FAILURE",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3020",
       "triggerID" : "d36815988f20c005b439af98d2a43bd9c145f7aa",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * d36815988f20c005b439af98d2a43bd9c145f7aa Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3020) 
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run travis` re-run the last Travis build
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [hudi] n3nash commented on pull request #3820: [HUDI-2579] Make deltastreamer checkpoint state merging more explicit

Posted by GitBox <gi...@apache.org>.
n3nash commented on pull request #3820:
URL: https://github.com/apache/hudi/pull/3820#issuecomment-962452394


   @nsivabalan i think there are 2 reasons for the old way of copying 1) to carry over during multiwriter 2) even if people are not using multiwriter they want to carry over values from one commit to next for their own debugging purposes.
   
   If you are able to serve either/and use case above, it should be fine


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [hudi] nsivabalan edited a comment on pull request #3820: [HUDI-2579] Make deltastreamer checkpoint state merging more explicit

Posted by GitBox <gi...@apache.org>.
nsivabalan edited a comment on pull request #3820:
URL: https://github.com/apache/hudi/pull/3820#issuecomment-962460574


   @n3nash : hmmm, the new config may not meet the 2nd requirement. it just takes care of 1st one i.e. copying over deltastreamer checkpoint from previous commit. Do you think one of you folks from uber can take this up immediately as a follow up. Or you wish to get it fixed in this patch itself? Even if we support the 2nd, I feel we could still have the new way of copying over deltastreamer checkpoint. From a user's standpoint who is interested in multi-writer, configs are more clear and crisp that way. 
   Let me know your opinion. 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [hudi] n3nash commented on pull request #3820: [HUDI-2579] Make deltastreamer checkpoint state merging more explicit

Posted by GitBox <gi...@apache.org>.
n3nash commented on pull request #3820:
URL: https://github.com/apache/hudi/pull/3820#issuecomment-961618542


   @jsbali This may affect use cases at Uber, FYI 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [hudi] hudi-bot commented on pull request #3820: [HUDI-2579] Make deltastreamer checkpoint state merging more explicit

Posted by GitBox <gi...@apache.org>.
hudi-bot commented on pull request #3820:
URL: https://github.com/apache/hudi/pull/3820#issuecomment-964236135


   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "8de6afb8a205a41de2a4b214c8982488b2b8ec19",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2708",
       "triggerID" : "8de6afb8a205a41de2a4b214c8982488b2b8ec19",
       "triggerType" : "PUSH"
     }, {
       "hash" : "4606a96bbce3b9638026eceac7ce0610f0b7045e",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2718",
       "triggerID" : "4606a96bbce3b9638026eceac7ce0610f0b7045e",
       "triggerType" : "PUSH"
     }, {
       "hash" : "c41001efeff2a7723ad45a637fb75336b58b294b",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2726",
       "triggerID" : "c41001efeff2a7723ad45a637fb75336b58b294b",
       "triggerType" : "PUSH"
     }, {
       "hash" : "ba8f8b0059fd3dcd549edf2d675605711b74d266",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2836",
       "triggerID" : "ba8f8b0059fd3dcd549edf2d675605711b74d266",
       "triggerType" : "PUSH"
     }, {
       "hash" : "1d031a881c677dee5e40a0119be17e6a35969f60",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2854",
       "triggerID" : "1d031a881c677dee5e40a0119be17e6a35969f60",
       "triggerType" : "PUSH"
     }, {
       "hash" : "9e8a962c9aee646713ebca560be2cfd234f25b09",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2910",
       "triggerID" : "9e8a962c9aee646713ebca560be2cfd234f25b09",
       "triggerType" : "PUSH"
     }, {
       "hash" : "4007755f0bb30705d3cc342f11ecfcebd3a108e9",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2911",
       "triggerID" : "4007755f0bb30705d3cc342f11ecfcebd3a108e9",
       "triggerType" : "PUSH"
     }, {
       "hash" : "255b35f65537a3adaedb14b25bb53d98beadb371",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2914",
       "triggerID" : "255b35f65537a3adaedb14b25bb53d98beadb371",
       "triggerType" : "PUSH"
     }, {
       "hash" : "d36815988f20c005b439af98d2a43bd9c145f7aa",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3020",
       "triggerID" : "d36815988f20c005b439af98d2a43bd9c145f7aa",
       "triggerType" : "PUSH"
     }, {
       "hash" : "98f42d69bed5505c9a1fa78e54da281fec73bd42",
       "status" : "SUCCESS",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3070",
       "triggerID" : "98f42d69bed5505c9a1fa78e54da281fec73bd42",
       "triggerType" : "PUSH"
     }, {
       "hash" : "1c139b8fb0405745bfea474365363715101e0323",
       "status" : "PENDING",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3249",
       "triggerID" : "1c139b8fb0405745bfea474365363715101e0323",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 98f42d69bed5505c9a1fa78e54da281fec73bd42 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3070) 
   * 1c139b8fb0405745bfea474365363715101e0323 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3249) 
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [hudi] hudi-bot commented on pull request #3820: [HUDI-2579] Make deltastreamer checkpoint state merging more explicit

Posted by GitBox <gi...@apache.org>.
hudi-bot commented on pull request #3820:
URL: https://github.com/apache/hudi/pull/3820#issuecomment-964233492


   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "8de6afb8a205a41de2a4b214c8982488b2b8ec19",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2708",
       "triggerID" : "8de6afb8a205a41de2a4b214c8982488b2b8ec19",
       "triggerType" : "PUSH"
     }, {
       "hash" : "4606a96bbce3b9638026eceac7ce0610f0b7045e",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2718",
       "triggerID" : "4606a96bbce3b9638026eceac7ce0610f0b7045e",
       "triggerType" : "PUSH"
     }, {
       "hash" : "c41001efeff2a7723ad45a637fb75336b58b294b",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2726",
       "triggerID" : "c41001efeff2a7723ad45a637fb75336b58b294b",
       "triggerType" : "PUSH"
     }, {
       "hash" : "ba8f8b0059fd3dcd549edf2d675605711b74d266",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2836",
       "triggerID" : "ba8f8b0059fd3dcd549edf2d675605711b74d266",
       "triggerType" : "PUSH"
     }, {
       "hash" : "1d031a881c677dee5e40a0119be17e6a35969f60",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2854",
       "triggerID" : "1d031a881c677dee5e40a0119be17e6a35969f60",
       "triggerType" : "PUSH"
     }, {
       "hash" : "9e8a962c9aee646713ebca560be2cfd234f25b09",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2910",
       "triggerID" : "9e8a962c9aee646713ebca560be2cfd234f25b09",
       "triggerType" : "PUSH"
     }, {
       "hash" : "4007755f0bb30705d3cc342f11ecfcebd3a108e9",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2911",
       "triggerID" : "4007755f0bb30705d3cc342f11ecfcebd3a108e9",
       "triggerType" : "PUSH"
     }, {
       "hash" : "255b35f65537a3adaedb14b25bb53d98beadb371",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2914",
       "triggerID" : "255b35f65537a3adaedb14b25bb53d98beadb371",
       "triggerType" : "PUSH"
     }, {
       "hash" : "d36815988f20c005b439af98d2a43bd9c145f7aa",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3020",
       "triggerID" : "d36815988f20c005b439af98d2a43bd9c145f7aa",
       "triggerType" : "PUSH"
     }, {
       "hash" : "98f42d69bed5505c9a1fa78e54da281fec73bd42",
       "status" : "SUCCESS",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3070",
       "triggerID" : "98f42d69bed5505c9a1fa78e54da281fec73bd42",
       "triggerType" : "PUSH"
     }, {
       "hash" : "1c139b8fb0405745bfea474365363715101e0323",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "1c139b8fb0405745bfea474365363715101e0323",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 98f42d69bed5505c9a1fa78e54da281fec73bd42 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3070) 
   * 1c139b8fb0405745bfea474365363715101e0323 UNKNOWN
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [hudi] hudi-bot removed a comment on pull request #3820: [HUDI-2579] Make deltastreamer checkpoint state merging more explicit

Posted by GitBox <gi...@apache.org>.
hudi-bot removed a comment on pull request #3820:
URL: https://github.com/apache/hudi/pull/3820#issuecomment-964236135


   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "8de6afb8a205a41de2a4b214c8982488b2b8ec19",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2708",
       "triggerID" : "8de6afb8a205a41de2a4b214c8982488b2b8ec19",
       "triggerType" : "PUSH"
     }, {
       "hash" : "4606a96bbce3b9638026eceac7ce0610f0b7045e",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2718",
       "triggerID" : "4606a96bbce3b9638026eceac7ce0610f0b7045e",
       "triggerType" : "PUSH"
     }, {
       "hash" : "c41001efeff2a7723ad45a637fb75336b58b294b",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2726",
       "triggerID" : "c41001efeff2a7723ad45a637fb75336b58b294b",
       "triggerType" : "PUSH"
     }, {
       "hash" : "ba8f8b0059fd3dcd549edf2d675605711b74d266",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2836",
       "triggerID" : "ba8f8b0059fd3dcd549edf2d675605711b74d266",
       "triggerType" : "PUSH"
     }, {
       "hash" : "1d031a881c677dee5e40a0119be17e6a35969f60",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2854",
       "triggerID" : "1d031a881c677dee5e40a0119be17e6a35969f60",
       "triggerType" : "PUSH"
     }, {
       "hash" : "9e8a962c9aee646713ebca560be2cfd234f25b09",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2910",
       "triggerID" : "9e8a962c9aee646713ebca560be2cfd234f25b09",
       "triggerType" : "PUSH"
     }, {
       "hash" : "4007755f0bb30705d3cc342f11ecfcebd3a108e9",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2911",
       "triggerID" : "4007755f0bb30705d3cc342f11ecfcebd3a108e9",
       "triggerType" : "PUSH"
     }, {
       "hash" : "255b35f65537a3adaedb14b25bb53d98beadb371",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=2914",
       "triggerID" : "255b35f65537a3adaedb14b25bb53d98beadb371",
       "triggerType" : "PUSH"
     }, {
       "hash" : "d36815988f20c005b439af98d2a43bd9c145f7aa",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3020",
       "triggerID" : "d36815988f20c005b439af98d2a43bd9c145f7aa",
       "triggerType" : "PUSH"
     }, {
       "hash" : "98f42d69bed5505c9a1fa78e54da281fec73bd42",
       "status" : "SUCCESS",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3070",
       "triggerID" : "98f42d69bed5505c9a1fa78e54da281fec73bd42",
       "triggerType" : "PUSH"
     }, {
       "hash" : "1c139b8fb0405745bfea474365363715101e0323",
       "status" : "PENDING",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3249",
       "triggerID" : "1c139b8fb0405745bfea474365363715101e0323",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 98f42d69bed5505c9a1fa78e54da281fec73bd42 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3070) 
   * 1c139b8fb0405745bfea474365363715101e0323 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3249) 
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [hudi] davehagman commented on pull request #3820: [BUGFIX] Merge commit state from previous commit instead of current

Posted by GitBox <gi...@apache.org>.
davehagman commented on pull request #3820:
URL: https://github.com/apache/hudi/pull/3820#issuecomment-945952200


   Also worth noting that the config used to determine which keys are merged from the past commit into the current one is generic (`hoodie.write.meta.key.prefixes`). At the moment I know of only one use for this which is to copy over the checkpoint data from deltastreamer commits but with this fix we can copy over any data from the previous commit to the current one. 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org