You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by "xushiyan (via GitHub)" <gi...@apache.org> on 2023/05/27 09:25:36 UTC

[GitHub] [hudi] xushiyan opened a new pull request, #8831: [HUDI-5321] Prepend partition path for custom partitioner

xushiyan opened a new pull request, #8831:
URL: https://github.com/apache/hudi/pull/8831

   ### Change Logs
   
   For these Partitioners
   
   - RowCustomColumnsSortPartitioner
   - RDDCustomColumnsSortPartitioner
   - JavaCustomColumnsSortPartitioner
   
   when the sort column names don't start with the partitionPath, Parquet writers could be closed prematurely when writing files, which lead to a lot of small files. This patch prepend partition path columns when applicable to prevent it.
   
   Re-work from #7413, which had conflicts that are hard to rebase.
   
   ### Impact
   
   Bulkinsert and clustering perf
   
   ### Risk level
   
   Low.
   
   ### Documentation Update
   
   NA
   
   ### Contributor's checklist
   
   - [ ] Read through [contributor's guide](https://hudi.apache.org/contribute/how-to-contribute)
   - [ ] Change Logs and Impact were stated clearly
   - [ ] Adequate tests were added if applicable
   - [ ] CI passed
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [hudi] xushiyan merged pull request #8831: [HUDI-5321] Prepend partition path for custom partitioner

Posted by "xushiyan (via GitHub)" <gi...@apache.org>.
xushiyan merged PR #8831:
URL: https://github.com/apache/hudi/pull/8831


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [hudi] hudi-bot commented on pull request #8831: [HUDI-5321] Prepend partition path for custom partitioner

Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #8831:
URL: https://github.com/apache/hudi/pull/8831#issuecomment-1565308378

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "205c52c75d39af9758ed5c93eb898c0abe710eee",
       "status" : "PENDING",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17438",
       "triggerID" : "205c52c75d39af9758ed5c93eb898c0abe710eee",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 205c52c75d39af9758ed5c93eb898c0abe710eee Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17438) 
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [hudi] hudi-bot commented on pull request #8831: [HUDI-5321] Prepend partition path for custom partitioner

Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #8831:
URL: https://github.com/apache/hudi/pull/8831#issuecomment-1565423337

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "205c52c75d39af9758ed5c93eb898c0abe710eee",
       "status" : "PENDING",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17438",
       "triggerID" : "205c52c75d39af9758ed5c93eb898c0abe710eee",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 205c52c75d39af9758ed5c93eb898c0abe710eee Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17438) 
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [hudi] hudi-bot commented on pull request #8831: [HUDI-5321] Prepend partition path for custom partitioner

Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #8831:
URL: https://github.com/apache/hudi/pull/8831#issuecomment-1565306625

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "205c52c75d39af9758ed5c93eb898c0abe710eee",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "205c52c75d39af9758ed5c93eb898c0abe710eee",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 205c52c75d39af9758ed5c93eb898c0abe710eee UNKNOWN
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [hudi] hudi-bot commented on pull request #8831: [HUDI-5321] Prepend partition path for custom partitioner

Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #8831:
URL: https://github.com/apache/hudi/pull/8831#issuecomment-1565421243

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "205c52c75d39af9758ed5c93eb898c0abe710eee",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "205c52c75d39af9758ed5c93eb898c0abe710eee",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 205c52c75d39af9758ed5c93eb898c0abe710eee UNKNOWN
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [hudi] hudi-bot commented on pull request #8831: [HUDI-5321] Prepend partition path for custom partitioner

Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #8831:
URL: https://github.com/apache/hudi/pull/8831#issuecomment-1565584235

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "205c52c75d39af9758ed5c93eb898c0abe710eee",
       "status" : "SUCCESS",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17438",
       "triggerID" : "205c52c75d39af9758ed5c93eb898c0abe710eee",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 205c52c75d39af9758ed5c93eb898c0abe710eee Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17438) 
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [hudi] hudi-bot commented on pull request #8831: [HUDI-5321] Prepend partition path for custom partitioner

Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #8831:
URL: https://github.com/apache/hudi/pull/8831#issuecomment-1565392470

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "205c52c75d39af9758ed5c93eb898c0abe710eee",
       "status" : "FAILURE",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17438",
       "triggerID" : "205c52c75d39af9758ed5c93eb898c0abe710eee",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 205c52c75d39af9758ed5c93eb898c0abe710eee Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17438) 
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [hudi] xushiyan commented on pull request #8831: [HUDI-5321] Prepend partition path for custom partitioner

Posted by "xushiyan (via GitHub)" <gi...@apache.org>.
xushiyan commented on PR #8831:
URL: https://github.com/apache/hudi/pull/8831#issuecomment-1565305815

   @boneanxs Pls also take a look.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [hudi] xushiyan commented on pull request #8831: [HUDI-5321] Prepend partition path for custom partitioner

Posted by "xushiyan (via GitHub)" <gi...@apache.org>.
xushiyan commented on PR #8831:
URL: https://github.com/apache/hudi/pull/8831#issuecomment-1566591464

   > The change looks good, but can you help me understand why does parquet writer get closed when sort columns does not have partition path field? The sort partitioners simply repartition the records based on sort columns, why should it matter if partition path field is part of those sort columns or not?
   > 
   > Also, if we now additionally add partition path field to sort columns then isn't that gonna add to the clustering time?
   
   These partitioners are doing global sort for all incoming records during repartitioning, which leads to putting records with different physical partition paths in the same spark partition when not first sorted by the physical partition fields. During bulk insert, write handles are re-created if encountered records in different partition paths. That's what it means by "parquet writer closed prematurely". It should only improve the perf when writing using these partitioner for clustering, as global sorting happens anyway.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org