You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by GitBox <gi...@apache.org> on 2022/11/17 20:39:50 UTC

[GitHub] [hudi] alexeykudinkin opened a new pull request, #7233: [MINOR] Fixing `FileIndex` impls to properly batch partitions listing

alexeykudinkin opened a new pull request, #7233:
URL: https://github.com/apache/hudi/pull/7233

   ### Change Logs
   
   While rebasing Hudi's FileIndex implementations to do lazy-listing in https://github.com/apache/hudi/pull/6680, it was  unfortunately overlooked that it was issuing point-wise lookup operations when listing actual partitions instead of doing batch operations (thanks to @psendyk for noticing and reporting this!).
   
   ### Impact
   
   This should considerably reduce file-listing latency, especially when using MT.
   
   ### Risk level (write none, low medium or high below)
   
   Low
   
   ### Documentation Update
   
   N/A 
   
   ### Contributor's checklist
   
   - [ ] Read through [contributor's guide](https://hudi.apache.org/contribute/how-to-contribute)
   - [ ] Change Logs and Impact were stated clearly
   - [ ] Adequate tests were added if applicable
   - [ ] CI passed
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [hudi] hudi-bot commented on pull request #7233: [MINOR] Fixing `FileIndex` impls to properly batch partitions listing

Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #7233:
URL: https://github.com/apache/hudi/pull/7233#issuecomment-1319182072

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "2e116e401016c1313e22a2e6b2f964481772a603",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "2e116e401016c1313e22a2e6b2f964481772a603",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 2e116e401016c1313e22a2e6b2f964481772a603 UNKNOWN
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [hudi] hudi-bot commented on pull request #7233: [MINOR] Fixing `FileIndex` impls to properly batch partitions listing

Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #7233:
URL: https://github.com/apache/hudi/pull/7233#issuecomment-1319400746

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "2e116e401016c1313e22a2e6b2f964481772a603",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=13088",
       "triggerID" : "2e116e401016c1313e22a2e6b2f964481772a603",
       "triggerType" : "PUSH"
     }, {
       "hash" : "5450282b8c66ef4f0aaceac40f6635de48304684",
       "status" : "SUCCESS",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=13089",
       "triggerID" : "5450282b8c66ef4f0aaceac40f6635de48304684",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 5450282b8c66ef4f0aaceac40f6635de48304684 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=13089) 
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [hudi] hudi-bot commented on pull request #7233: [MINOR] Fixing `FileIndex` impls to properly batch partitions listing

Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #7233:
URL: https://github.com/apache/hudi/pull/7233#issuecomment-1319187460

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "2e116e401016c1313e22a2e6b2f964481772a603",
       "status" : "PENDING",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=13088",
       "triggerID" : "2e116e401016c1313e22a2e6b2f964481772a603",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 2e116e401016c1313e22a2e6b2f964481772a603 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=13088) 
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [hudi] hudi-bot commented on pull request #7233: [MINOR] Fixing `FileIndex` impls to properly batch partitions listing

Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #7233:
URL: https://github.com/apache/hudi/pull/7233#issuecomment-1319264657

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "2e116e401016c1313e22a2e6b2f964481772a603",
       "status" : "PENDING",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=13088",
       "triggerID" : "2e116e401016c1313e22a2e6b2f964481772a603",
       "triggerType" : "PUSH"
     }, {
       "hash" : "5450282b8c66ef4f0aaceac40f6635de48304684",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "5450282b8c66ef4f0aaceac40f6635de48304684",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 2e116e401016c1313e22a2e6b2f964481772a603 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=13088) 
   * 5450282b8c66ef4f0aaceac40f6635de48304684 UNKNOWN
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [hudi] psendyk commented on pull request #7233: [MINOR] Fixing `FileIndex` impls to properly batch partitions listing

Posted by GitBox <gi...@apache.org>.
psendyk commented on PR #7233:
URL: https://github.com/apache/hudi/pull/7233#issuecomment-1319460200

   Thanks for fixing it @alexeykudinkin! 🙌 I just tested your changes and it works great, all files are listed in a single MT lookup.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [hudi] xushiyan merged pull request #7233: [HUDI-4812] Fixing `FileIndex` impls to properly batch partitions listing

Posted by GitBox <gi...@apache.org>.
xushiyan merged PR #7233:
URL: https://github.com/apache/hudi/pull/7233


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [hudi] hudi-bot commented on pull request #7233: [MINOR] Fixing `FileIndex` impls to properly batch partitions listing

Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #7233:
URL: https://github.com/apache/hudi/pull/7233#issuecomment-1319270827

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "2e116e401016c1313e22a2e6b2f964481772a603",
       "status" : "CANCELED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=13088",
       "triggerID" : "2e116e401016c1313e22a2e6b2f964481772a603",
       "triggerType" : "PUSH"
     }, {
       "hash" : "5450282b8c66ef4f0aaceac40f6635de48304684",
       "status" : "PENDING",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=13089",
       "triggerID" : "5450282b8c66ef4f0aaceac40f6635de48304684",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 2e116e401016c1313e22a2e6b2f964481772a603 Azure: [CANCELED](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=13088) 
   * 5450282b8c66ef4f0aaceac40f6635de48304684 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=13089) 
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org