You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by GitBox <gi...@apache.org> on 2023/01/18 19:41:58 UTC

[GitHub] [hudi] nsivabalan opened a new pull request, #7699: [HUDI-5535] Support any record key generation along w/ any partition path generation

nsivabalan opened a new pull request, #7699:
URL: https://github.com/apache/hudi/pull/7699

   ### Change Logs
   
   As of now, record key generation and partition path generation are tightly coupled based on the key gen class used. But these could be de-coupled. For eg, we can't have a complex record key generation with non-partitioned dataset since the NonpartitionedKeyGen uses simple key generation logic for record key generation. So, this patch introduces a separate RecordKeyGenerator interface and uses a factory to determine the record key generation strategy. Users don't need to set any additional param. 
   
   After this patch, this is how the record key generation strategy is determined. 
   Users will have to configure the right value for "hoodie.datasource.write.recordkey.field" or enable "hoodie.auto.generate.record.keys".
   a. If "hoodie.auto.generate.record.keys" is enabled, record keys will be auto generated internally and users don't need to set any value for "hoodie.datasource.write.recordkey.field".
   b. else, if single field is set for "hoodie.datasource.write.recordkey.field", simple record key generation. 
   c. else its deduced as multi record key generation(complex). 
   Above logic applies to any key gen class we have in hudi. 
   
   ### Impact
   
   Enables users to choose any record key generation along w/ any partition path generation strategy. 
   
   ### Risk level (write none, low medium or high below)
   
   low. 
   
   ### Documentation Update
   
   _Describe any necessary documentation update if there is any new feature, config, or user-facing change_
   
   - _The config description must be updated if new configs are added or the default value of the configs are changed_
   - _Any new feature or user-facing change requires updating the Hudi website. Please create a Jira ticket, attach the
     ticket number here and follow the [instruction](https://hudi.apache.org/contribute/developer-setup#website) to make
     changes to the website._
   
   ### Contributor's checklist
   
   - [ ] Read through [contributor's guide](https://hudi.apache.org/contribute/how-to-contribute)
   - [ ] Change Logs and Impact were stated clearly
   - [ ] Adequate tests were added if applicable
   - [ ] CI passed
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [hudi] hudi-bot commented on pull request #7699: [HUDI-5535] Support any record key generation along w/ any partition path generation

Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #7699:
URL: https://github.com/apache/hudi/pull/7699#issuecomment-1387708077

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "560fdab3f9d409d4f34ca383f285ef4da8a76b66",
       "status" : "PENDING",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14412",
       "triggerID" : "560fdab3f9d409d4f34ca383f285ef4da8a76b66",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 560fdab3f9d409d4f34ca383f285ef4da8a76b66 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14412) 
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [hudi] nsivabalan closed pull request #7699: [HUDI-5535] Support any record key generation along w/ any partition path generation

Posted by "nsivabalan (via GitHub)" <gi...@apache.org>.
nsivabalan closed pull request #7699: [HUDI-5535] Support any record key generation along w/ any partition path generation
URL: https://github.com/apache/hudi/pull/7699


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [hudi] hudi-bot commented on pull request #7699: [HUDI-5535] Support any record key generation along w/ any partition path generation

Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #7699:
URL: https://github.com/apache/hudi/pull/7699#issuecomment-1387691257

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "560fdab3f9d409d4f34ca383f285ef4da8a76b66",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "560fdab3f9d409d4f34ca383f285ef4da8a76b66",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 560fdab3f9d409d4f34ca383f285ef4da8a76b66 UNKNOWN
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [hudi] hudi-bot commented on pull request #7699: [HUDI-5535] Support any record key generation along w/ any partition path generation

Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #7699:
URL: https://github.com/apache/hudi/pull/7699#issuecomment-1396426891

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "560fdab3f9d409d4f34ca383f285ef4da8a76b66",
       "status" : "FAILURE",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14412",
       "triggerID" : "560fdab3f9d409d4f34ca383f285ef4da8a76b66",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 560fdab3f9d409d4f34ca383f285ef4da8a76b66 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14412) 
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org