You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by "lvhu-goodluck (via GitHub)" <gi...@apache.org> on 2023/02/16 13:40:42 UTC

[GitHub] [hudi] lvhu-goodluck opened a new pull request, #7984: hash partition in spark datasoure

lvhu-goodluck opened a new pull request, #7984:
URL: https://github.com/apache/hudi/pull/7984

   ### Change Logs
   
   Add hash partition in spark datasoure.
    It is often difficult to find an appropriate partition key in the existing data. Hash partitioning can easily solve this problem.
   
   How to use hash partition in spark data source can refer to hudi/hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/hudi/DataSourceOptions.scala#testHashPartition
   
   ### Impact
   
   No public API or user-facing feature change or any performance impact if the hash partition parameters are not specified.
   
   When hash.partition.fields is specified and partition.fields contains _hoodie_hash_partition, a column named _hoodie_hash_partition will be added in this table as one of the partition key.
   
   If predicates of hash.partition.fields appear in the query statement, the _hoodie_hash_partition = X predicate will be automatically added to the query statement for partition pruning.
   
   ### Risk level (write none, low medium or high below)
   
   Low .
   
   ### Documentation Update
   
   To be continued.
   
   ### Contributor's checklist
   
   - [ ] Read through [contributor's guide](https://hudi.apache.org/contribute/how-to-contribute)
   - [ ] Change Logs and Impact were stated clearly
   - [ ] Adequate tests were added if applicable
   - [ ] CI passed
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [hudi] hudi-bot commented on pull request #7984: hash partition in spark datasoure

Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #7984:
URL: https://github.com/apache/hudi/pull/7984#issuecomment-1434165539

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "3c081bf5181aec3b882f72abe208d6b5781d6628",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "3c081bf5181aec3b882f72abe208d6b5781d6628",
       "triggerType" : "PUSH"
     }, {
       "hash" : "d879e42e4c1d0f0cd061fa3d7cde6c0156ac9075",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15249",
       "triggerID" : "d879e42e4c1d0f0cd061fa3d7cde6c0156ac9075",
       "triggerType" : "PUSH"
     }, {
       "hash" : "3f01ba85e876c493eb4526269e5c39f2140198da",
       "status" : "CANCELED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15250",
       "triggerID" : "3f01ba85e876c493eb4526269e5c39f2140198da",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 3c081bf5181aec3b882f72abe208d6b5781d6628 UNKNOWN
   * 3f01ba85e876c493eb4526269e5c39f2140198da Azure: [CANCELED](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15250) 
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [hudi] lvhu-goodluck commented on pull request #7984: hash partition in spark datasoure

Posted by "lvhu-goodluck (via GitHub)" <gi...@apache.org>.
lvhu-goodluck commented on PR #7984:
URL: https://github.com/apache/hudi/pull/7984#issuecomment-1434042785

   > Thanks for your contribution @lvhu-goodluck!
   
   Thanks for your suggestion. I don't know How to create mini-rfc. Would you please give me some more help.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [hudi] lvhu-goodluck commented on pull request #7984: hash partition in spark datasoure

Posted by "lvhu-goodluck (via GitHub)" <gi...@apache.org>.
lvhu-goodluck commented on PR #7984:
URL: https://github.com/apache/hudi/pull/7984#issuecomment-1435835223

   > Thanks for your contribution @lvhu-goodluck!
   > 
   > I'd suggest we first go with mini-RFC to align on the goals and the technical approach there. That would save us a lot of time in avoiding back-and-forth on the implementation PR.
   
   Thanks a lot. I'll do it right now.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [hudi] lvhu-goodluck closed pull request #7984: hash partition in spark datasoure

Posted by "lvhu-goodluck (via GitHub)" <gi...@apache.org>.
lvhu-goodluck closed pull request #7984: hash partition in spark datasoure
URL: https://github.com/apache/hudi/pull/7984


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [hudi] vinothchandar commented on pull request #7984: hash partition in spark datasoure

Posted by "vinothchandar (via GitHub)" <gi...@apache.org>.
vinothchandar commented on PR #7984:
URL: https://github.com/apache/hudi/pull/7984#issuecomment-1471373856

   @lvhu-goodluck Apologize for the delays here. Most of us were heads down with 0.13. Let me respond on the dev list.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [hudi] lvhu-goodluck commented on pull request #7984: hash partition in spark datasoure

Posted by "lvhu-goodluck (via GitHub)" <gi...@apache.org>.
lvhu-goodluck commented on PR #7984:
URL: https://github.com/apache/hudi/pull/7984#issuecomment-1469593533

   > @lvhu-goodluck : Can you add the link to mini-RFC when it is ready ?
   
   Yes, I can. I have started  a discussion by sending an email to dev@hudi.apache.org with subject line <Hudi hash partition table>, No one has replied yet.    I don't know when the mini-rfc will be created


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [hudi] hudi-bot commented on pull request #7984: hash partition in spark datasoure

Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #7984:
URL: https://github.com/apache/hudi/pull/7984#issuecomment-1433288499

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "3c081bf5181aec3b882f72abe208d6b5781d6628",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "3c081bf5181aec3b882f72abe208d6b5781d6628",
       "triggerType" : "PUSH"
     }, {
       "hash" : "d879e42e4c1d0f0cd061fa3d7cde6c0156ac9075",
       "status" : "CANCELED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15249",
       "triggerID" : "d879e42e4c1d0f0cd061fa3d7cde6c0156ac9075",
       "triggerType" : "PUSH"
     }, {
       "hash" : "3f01ba85e876c493eb4526269e5c39f2140198da",
       "status" : "PENDING",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15250",
       "triggerID" : "3f01ba85e876c493eb4526269e5c39f2140198da",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 3c081bf5181aec3b882f72abe208d6b5781d6628 UNKNOWN
   * d879e42e4c1d0f0cd061fa3d7cde6c0156ac9075 Azure: [CANCELED](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15249) 
   * 3f01ba85e876c493eb4526269e5c39f2140198da Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15250) 
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [hudi] alexeykudinkin commented on pull request #7984: hash partition in spark datasoure

Posted by "alexeykudinkin (via GitHub)" <gi...@apache.org>.
alexeykudinkin commented on PR #7984:
URL: https://github.com/apache/hudi/pull/7984#issuecomment-1433600947

   Thanks for your contribution @lvhu-goodluck!
   
   I'd suggest we first go with mini-RFC to align on the goals and the technical approach there. That would save us a lot of time in avoiding back-and-forth on the implementation PR.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [hudi] hudi-bot commented on pull request #7984: hash partition in spark datasoure

Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #7984:
URL: https://github.com/apache/hudi/pull/7984#issuecomment-1433204793

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "3c081bf5181aec3b882f72abe208d6b5781d6628",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "3c081bf5181aec3b882f72abe208d6b5781d6628",
       "triggerType" : "PUSH"
     }, {
       "hash" : "d879e42e4c1d0f0cd061fa3d7cde6c0156ac9075",
       "status" : "CANCELED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15249",
       "triggerID" : "d879e42e4c1d0f0cd061fa3d7cde6c0156ac9075",
       "triggerType" : "PUSH"
     }, {
       "hash" : "3f01ba85e876c493eb4526269e5c39f2140198da",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "3f01ba85e876c493eb4526269e5c39f2140198da",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 3c081bf5181aec3b882f72abe208d6b5781d6628 UNKNOWN
   * d879e42e4c1d0f0cd061fa3d7cde6c0156ac9075 Azure: [CANCELED](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15249) 
   * 3f01ba85e876c493eb4526269e5c39f2140198da UNKNOWN
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [hudi] alexeykudinkin commented on pull request #7984: hash partition in spark datasoure

Posted by "alexeykudinkin (via GitHub)" <gi...@apache.org>.
alexeykudinkin commented on PR #7984:
URL: https://github.com/apache/hudi/pull/7984#issuecomment-1435126766

   @lvhu-goodluck please follow standard RFC process: https://hudi.apache.org/contribute/rfc-process/


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [hudi] bvaradar commented on pull request #7984: hash partition in spark datasoure

Posted by "bvaradar (via GitHub)" <gi...@apache.org>.
bvaradar commented on PR #7984:
URL: https://github.com/apache/hudi/pull/7984#issuecomment-1467287281

   @lvhu-goodluck : Can you add the link to mini-RFC when it is ready ? 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [hudi] hudi-bot commented on pull request #7984: hash partition in spark datasoure

Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #7984:
URL: https://github.com/apache/hudi/pull/7984#issuecomment-1433194133

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "3c081bf5181aec3b882f72abe208d6b5781d6628",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "3c081bf5181aec3b882f72abe208d6b5781d6628",
       "triggerType" : "PUSH"
     }, {
       "hash" : "d879e42e4c1d0f0cd061fa3d7cde6c0156ac9075",
       "status" : "PENDING",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15249",
       "triggerID" : "d879e42e4c1d0f0cd061fa3d7cde6c0156ac9075",
       "triggerType" : "PUSH"
     }, {
       "hash" : "3f01ba85e876c493eb4526269e5c39f2140198da",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "3f01ba85e876c493eb4526269e5c39f2140198da",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 3c081bf5181aec3b882f72abe208d6b5781d6628 UNKNOWN
   * d879e42e4c1d0f0cd061fa3d7cde6c0156ac9075 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=15249) 
   * 3f01ba85e876c493eb4526269e5c39f2140198da UNKNOWN
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [hudi] lvhu-goodluck commented on pull request #7984: hash partition in spark datasoure

Posted by "lvhu-goodluck (via GitHub)" <gi...@apache.org>.
lvhu-goodluck commented on PR #7984:
URL: https://github.com/apache/hudi/pull/7984#issuecomment-1434037828

   > mini
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [hudi] hudi-bot commented on pull request #7984: hash partition in spark datasoure

Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #7984:
URL: https://github.com/apache/hudi/pull/7984#issuecomment-1433120773

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "3c081bf5181aec3b882f72abe208d6b5781d6628",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "3c081bf5181aec3b882f72abe208d6b5781d6628",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 3c081bf5181aec3b882f72abe208d6b5781d6628 UNKNOWN
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [hudi] hudi-bot commented on pull request #7984: hash partition in spark datasoure

Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #7984:
URL: https://github.com/apache/hudi/pull/7984#issuecomment-1433182870

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "3c081bf5181aec3b882f72abe208d6b5781d6628",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "3c081bf5181aec3b882f72abe208d6b5781d6628",
       "triggerType" : "PUSH"
     }, {
       "hash" : "d879e42e4c1d0f0cd061fa3d7cde6c0156ac9075",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "d879e42e4c1d0f0cd061fa3d7cde6c0156ac9075",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 3c081bf5181aec3b882f72abe208d6b5781d6628 UNKNOWN
   * d879e42e4c1d0f0cd061fa3d7cde6c0156ac9075 UNKNOWN
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [hudi] abcyhq commented on pull request #7984: hash partition in spark datasoure

Posted by "abcyhq (via GitHub)" <gi...@apache.org>.
abcyhq commented on PR #7984:
URL: https://github.com/apache/hudi/pull/7984#issuecomment-1469645116

   I'm sure I can use this function. I'm very concerned about the progress.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org