You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by GitBox <gi...@apache.org> on 2022/02/23 01:05:48 UTC

[GitHub] [hudi] waywtdcc opened a new pull request #4874: 1.The global index is enabled regardless of changlog

waywtdcc opened a new pull request #4874:
URL: https://github.com/apache/hudi/pull/4874


   ## *Tips*
   - *Thank you very much for contributing to Apache Hudi.*
   - *Please review https://hudi.apache.org/contribute/how-to-contribute before opening a pull request.*
   
   ## What is the purpose of the pull request
   
   Handling the logic error of global index startup leads to the problem of partition table data redundancy when both changglobal and global index are enabled. Reference issue #4868 
   
   ## Brief change log
   
     - *Modify BucketAssignFunction Constructor  in org.apache.hudi.sink.partitioner.BucketAssignFunction*
   
   ## Verify this pull request
   
   It can solve issue #4868 
   ### before
   ![image](https://user-images.githubusercontent.com/59957056/155245820-de49ad72-6f55-4695-a36d-b560dab1d041.png)
   
   ### update the data
   `
   update users_cdc3 
   set date_str = '2022-1-25' , name3  = 'ccccccddddeee'
   where id = 462;
   `
   
   ### result
   ![image](https://user-images.githubusercontent.com/59957056/155245879-3f231eba-6be2-476a-847c-16e93cf212ab.png)
   
   ## Committer checklist
   
    - [ ] Has a corresponding JIRA in PR title & commit
    
    - [ ] Commit message is descriptive of the change
    
    - [ ] CI is green
   
    - [ ] Necessary doc changes done or have another open PR
          
    - [ ] For large changes, please consider breaking it into sub-tasks under an umbrella JIRA.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [hudi] danny0405 commented on pull request #4874: [HUDI-3487] The global index is enabled regardless of changlog

Posted by GitBox <gi...@apache.org>.
danny0405 commented on pull request #4874:
URL: https://github.com/apache/hudi/pull/4874#issuecomment-1058990931


   The changelog mode has retraction message itself, so no need to additional retraction for the deletions of different partition.
   
   The right solution for spark source is to adapter to the change flag. 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [hudi] hudi-bot removed a comment on pull request #4874: The global index is enabled regardless of changlog

Posted by GitBox <gi...@apache.org>.
hudi-bot removed a comment on pull request #4874:
URL: https://github.com/apache/hudi/pull/4874#issuecomment-1048363382


   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "f1357d86a7bc7619aa9fb5de0bb1189731695efd",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "f1357d86a7bc7619aa9fb5de0bb1189731695efd",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * f1357d86a7bc7619aa9fb5de0bb1189731695efd UNKNOWN
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [hudi] waywtdcc commented on pull request #4874: [HUDI-3487] The global index is enabled regardless of changlog

Posted by GitBox <gi...@apache.org>.
waywtdcc commented on pull request #4874:
URL: https://github.com/apache/hudi/pull/4874#issuecomment-1061433397


   > The changelog mode has retraction message itself, so no need to additional retraction for the deletions of different partition.
   > 
   > The right solution for spark source is to adapter to the change flag.
   
   Will these long - U operation records be automatically merged and deleted? I can see that there are still - U records that have not been deleted for a long time


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [hudi] danny0405 commented on pull request #4874: [HUDI-3487] The global index is enabled regardless of changlog

Posted by GitBox <gi...@apache.org>.
danny0405 commented on pull request #4874:
URL: https://github.com/apache/hudi/pull/4874#issuecomment-1061545989


   > > The changelog mode has retraction message itself, so no need to additional retraction for the deletions of different partition.
   > > The right solution for spark source is to adapter to the change flag.
   > 
   > Will these long - U operation records be automatically merged and deleted? I can see that there are still - U records that have not been deleted for a long time
   
   -U expects to be okey, -D record may be recognized as an INSERT.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [hudi] hudi-bot commented on pull request #4874: The global index is enabled regardless of changlog

Posted by GitBox <gi...@apache.org>.
hudi-bot commented on pull request #4874:
URL: https://github.com/apache/hudi/pull/4874#issuecomment-1048365404


   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "f1357d86a7bc7619aa9fb5de0bb1189731695efd",
       "status" : "PENDING",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6210",
       "triggerID" : "f1357d86a7bc7619aa9fb5de0bb1189731695efd",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * f1357d86a7bc7619aa9fb5de0bb1189731695efd Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6210) 
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [hudi] waywtdcc commented on pull request #4874: The global index is enabled regardless of changlog

Posted by GitBox <gi...@apache.org>.
waywtdcc commented on pull request #4874:
URL: https://github.com/apache/hudi/pull/4874#issuecomment-1048400229


   @danny0405 please review it.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [hudi] hudi-bot commented on pull request #4874: The global index is enabled regardless of changlog

Posted by GitBox <gi...@apache.org>.
hudi-bot commented on pull request #4874:
URL: https://github.com/apache/hudi/pull/4874#issuecomment-1048410402


   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "f1357d86a7bc7619aa9fb5de0bb1189731695efd",
       "status" : "SUCCESS",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6210",
       "triggerID" : "f1357d86a7bc7619aa9fb5de0bb1189731695efd",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * f1357d86a7bc7619aa9fb5de0bb1189731695efd Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6210) 
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [hudi] waywtdcc closed pull request #4874: [HUDI-3487] The global index is enabled regardless of changlog

Posted by GitBox <gi...@apache.org>.
waywtdcc closed pull request #4874:
URL: https://github.com/apache/hudi/pull/4874


   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [hudi] hudi-bot commented on pull request #4874: The global index is enabled regardless of changlog

Posted by GitBox <gi...@apache.org>.
hudi-bot commented on pull request #4874:
URL: https://github.com/apache/hudi/pull/4874#issuecomment-1048389207


   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "f1357d86a7bc7619aa9fb5de0bb1189731695efd",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "f1357d86a7bc7619aa9fb5de0bb1189731695efd",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * f1357d86a7bc7619aa9fb5de0bb1189731695efd UNKNOWN
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [hudi] hudi-bot removed a comment on pull request #4874: The global index is enabled regardless of changlog

Posted by GitBox <gi...@apache.org>.
hudi-bot removed a comment on pull request #4874:
URL: https://github.com/apache/hudi/pull/4874#issuecomment-1048389207


   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "f1357d86a7bc7619aa9fb5de0bb1189731695efd",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "f1357d86a7bc7619aa9fb5de0bb1189731695efd",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * f1357d86a7bc7619aa9fb5de0bb1189731695efd UNKNOWN
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [hudi] hudi-bot removed a comment on pull request #4874: The global index is enabled regardless of changlog

Posted by GitBox <gi...@apache.org>.
hudi-bot removed a comment on pull request #4874:
URL: https://github.com/apache/hudi/pull/4874#issuecomment-1048365404


   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "f1357d86a7bc7619aa9fb5de0bb1189731695efd",
       "status" : "PENDING",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6210",
       "triggerID" : "f1357d86a7bc7619aa9fb5de0bb1189731695efd",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * f1357d86a7bc7619aa9fb5de0bb1189731695efd Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6210) 
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [hudi] hudi-bot commented on pull request #4874: The global index is enabled regardless of changlog

Posted by GitBox <gi...@apache.org>.
hudi-bot commented on pull request #4874:
URL: https://github.com/apache/hudi/pull/4874#issuecomment-1048363382


   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "f1357d86a7bc7619aa9fb5de0bb1189731695efd",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "f1357d86a7bc7619aa9fb5de0bb1189731695efd",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * f1357d86a7bc7619aa9fb5de0bb1189731695efd UNKNOWN
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org