You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by GitBox <gi...@apache.org> on 2022/09/26 04:03:47 UTC

[GitHub] [hudi] SteNicholas opened a new pull request, #6792: [HUDI-4914] Managed memory weight should be set when sort clustering is enabled

SteNicholas opened a new pull request, #6792:
URL: https://github.com/apache/hudi/pull/6792

   ### Change Logs
   
   Managed memory weight should be set when sort clustering is enabled, otherwise the fraction of memory to allocate is 0 that throws the following exception when initialzing the sorter.
   
   - Sets the managed memory weight when sort clustering is enabled in Pipelines#cluster()
   - Initializes the `BinaryExternalSorter` in `ClusteringOperator#doClustering()`.
   
   ### Impact
   
   When sort clustering is enabled, the `ClusteringOperator` could initialize the `BinaryExternalSorter`.
   
   **Risk level: none | low | medium | high**
   
   high.
   
   - Introduces the `testWriteCopyOnWriteWithSortClustering` in `ITTestDataStreamWrite` to verify whether the sort clustering could work normally.
   
   ### Contributor's checklist
   
   - [x] Read through [contributor's guide](https://hudi.apache.org/contribute/how-to-contribute)
   - [x] Change Logs and Impact were stated clearly
   - [x] Adequate tests were added if applicable
   - [x] CI passed
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [hudi] danny0405 merged pull request #6792: [HUDI-4914] Managed memory weight should be set when sort clustering is enabled

Posted by GitBox <gi...@apache.org>.
danny0405 merged PR #6792:
URL: https://github.com/apache/hudi/pull/6792


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [hudi] hudi-bot commented on pull request #6792: [HUDI-4914] Managed memory weight should be set when sort clustering is enabled

Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #6792:
URL: https://github.com/apache/hudi/pull/6792#issuecomment-1257447082

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "37f80daa46eb8dc8524e43f99da5cff1e421e9bc",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "37f80daa46eb8dc8524e43f99da5cff1e421e9bc",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 37f80daa46eb8dc8524e43f99da5cff1e421e9bc UNKNOWN
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [hudi] hudi-bot commented on pull request #6792: [HUDI-4914] Managed memory weight should be set when sort clustering is enabled

Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #6792:
URL: https://github.com/apache/hudi/pull/6792#issuecomment-1257535744

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "37f80daa46eb8dc8524e43f99da5cff1e421e9bc",
       "status" : "PENDING",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11722",
       "triggerID" : "37f80daa46eb8dc8524e43f99da5cff1e421e9bc",
       "triggerType" : "PUSH"
     }, {
       "hash" : "b6858043288be4b64a6f552a4768d99debf3bbea",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "b6858043288be4b64a6f552a4768d99debf3bbea",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 37f80daa46eb8dc8524e43f99da5cff1e421e9bc Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11722) 
   * b6858043288be4b64a6f552a4768d99debf3bbea UNKNOWN
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [hudi] hudi-bot commented on pull request #6792: [HUDI-4914] Managed memory weight should be set when sort clustering is enabled

Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #6792:
URL: https://github.com/apache/hudi/pull/6792#issuecomment-1257887030

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "37f80daa46eb8dc8524e43f99da5cff1e421e9bc",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11722",
       "triggerID" : "37f80daa46eb8dc8524e43f99da5cff1e421e9bc",
       "triggerType" : "PUSH"
     }, {
       "hash" : "b6858043288be4b64a6f552a4768d99debf3bbea",
       "status" : "FAILURE",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11729",
       "triggerID" : "b6858043288be4b64a6f552a4768d99debf3bbea",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * b6858043288be4b64a6f552a4768d99debf3bbea Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11729) 
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [hudi] hudi-bot commented on pull request #6792: [HUDI-4914] Managed memory weight should be set when sort clustering is enabled

Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #6792:
URL: https://github.com/apache/hudi/pull/6792#issuecomment-1257486477

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "37f80daa46eb8dc8524e43f99da5cff1e421e9bc",
       "status" : "PENDING",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11722",
       "triggerID" : "37f80daa46eb8dc8524e43f99da5cff1e421e9bc",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 37f80daa46eb8dc8524e43f99da5cff1e421e9bc Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11722) 
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [hudi] danny0405 commented on pull request #6792: [HUDI-4914] Managed memory weight should be set when sort clustering is enabled

Posted by GitBox <gi...@apache.org>.
danny0405 commented on PR #6792:
URL: https://github.com/apache/hudi/pull/6792#issuecomment-1257953084

   The hudi-flink IT has passed, so i would just merge it ~


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [hudi] SteNicholas commented on pull request #6792: [HUDI-4914] Managed memory weight should be set when sort clustering is enabled

Posted by GitBox <gi...@apache.org>.
SteNicholas commented on PR #6792:
URL: https://github.com/apache/hudi/pull/6792#issuecomment-1257517439

   @danny0405, could you please help to review the sort clustering bugfix?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [hudi] hudi-bot commented on pull request #6792: [HUDI-4914] Managed memory weight should be set when sort clustering is enabled

Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #6792:
URL: https://github.com/apache/hudi/pull/6792#issuecomment-1257540555

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "37f80daa46eb8dc8524e43f99da5cff1e421e9bc",
       "status" : "CANCELED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11722",
       "triggerID" : "37f80daa46eb8dc8524e43f99da5cff1e421e9bc",
       "triggerType" : "PUSH"
     }, {
       "hash" : "b6858043288be4b64a6f552a4768d99debf3bbea",
       "status" : "PENDING",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11729",
       "triggerID" : "b6858043288be4b64a6f552a4768d99debf3bbea",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 37f80daa46eb8dc8524e43f99da5cff1e421e9bc Azure: [CANCELED](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11722) 
   * b6858043288be4b64a6f552a4768d99debf3bbea Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=11729) 
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org