You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@uniffle.apache.org by GitBox <gi...@apache.org> on 2022/11/05 12:34:54 UTC

[GitHub] [incubator-uniffle] zuston opened a new issue, #302: [Subtask][Improvement][AQE] Introduce the new MergedShuffleDataSegment to reduce number of rpc

zuston opened a new issue, #302:
URL: https://github.com/apache/incubator-uniffle/issues/302

   ### Code of Conduct
   
   - [X] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct)
   
   
   ### Search before asking
   
   - [X] I have searched in the [issues](https://github.com/apache/incubator-uniffle/issues) and found no similar issues.
   
   
   ### Describe the subtask
   
   Currently, when using the `LocalOrderSegmentSplitter`, it will split multiple `ShuffleDataSegment` by three factors.
   
   1. taskId in [startMapId, endMapId) taskIds bitmap
   2. ShuffleDataSegment size should < readBufferSize
   3. ShuffleDataSegment's blocks should be continuous
   
   In some cases, these requirements will make the data size very small (<14M). So we could combine some ShuffleDataSegment to single one data-fetching request to reduce the RPC interaction number.
   
   ### Parent issue
   
   #137 
   
   ### Are you willing to submit PR?
   
   - [X] Yes I am willing to submit a PR!


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscribe@uniffle.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [incubator-uniffle] zuston commented on issue #302: [Subtask][Improvement][AQE] Introduce the new MergedShuffleDataSegment to reduce number of rpc

Posted by GitBox <gi...@apache.org>.
zuston commented on issue #302:
URL: https://github.com/apache/incubator-uniffle/issues/302#issuecomment-1304668075

   Yes, maybe we should introduce global order to compare with other distribution types in different data size
   
   but this pr is still meaningful for local order


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscribe@uniffle.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [incubator-uniffle] jerqi commented on issue #302: [Subtask][Improvement][AQE] Introduce the new MergedShuffleDataSegment to reduce number of rpc

Posted by GitBox <gi...@apache.org>.
jerqi commented on issue #302:
URL: https://github.com/apache/incubator-uniffle/issues/302#issuecomment-1304572496

   Maybe global order will be a better solution for this case.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscribe@uniffle.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org