You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@uniffle.apache.org by "rickyma (via GitHub)" <gi...@apache.org> on 2024/04/11 13:05:44 UTC
[I] [Improvement] Optimize CompositeByteBuf initialization for better performance when getting memory data [incubator-uniffle]
rickyma opened a new issue, #1642:
URL: https://github.com/apache/incubator-uniffle/issues/1642
### Code of Conduct
- [X] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct)
### Search before asking
- [X] I have searched in the [issues](https://github.com/apache/incubator-uniffle/issues?q=is%3Aissue) and found no similar issues.
### What would you like to be improved?
Optimize the initialization of `CompositeByteBuf` by creating an accurately sized `CompositeByteBuf` to improve performance and reduce memory allocation and resizing overhead.
I think the following code could be optimized:
```
public synchronized ShuffleDataResult getShuffleData(
long lastBlockId, int readBufferSize, Roaring64NavigableMap expectedTaskIds) {
try {
List<BufferSegment> bufferSegments = Lists.newArrayList();
List<ShufflePartitionedBlock> readBlocks = Lists.newArrayList();
updateBufferSegmentsAndResultBlocks(
lastBlockId, readBufferSize, bufferSegments, readBlocks, expectedTaskIds);
if (!bufferSegments.isEmpty()) {
CompositeByteBuf byteBuf =
new CompositeByteBuf(
NettyUtils.getNettyBufferAllocator(),
true,
Constants.COMPOSITE_BYTE_BUF_MAX_COMPONENTS);
// copy result data
updateShuffleData(readBlocks, byteBuf);
return new ShuffleDataResult(byteBuf, bufferSegments);
}
} catch (Exception e) {
LOG.error("Exception happened when getShuffleData in buffer", e);
}
return new ShuffleDataResult();
}
```
### How should we improve?
_No response_
### Are you willing to submit PR?
- [X] Yes I am willing to submit a PR!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For additional commands, e-mail: issues-help@uniffle.apache.org
Re: [I] [Improvement] Optimize CompositeByteBuf initialization for better performance when getting memory data [incubator-uniffle]
Posted by "rickyma (via GitHub)" <gi...@apache.org>.
rickyma commented on issue #1642:
URL: https://github.com/apache/incubator-uniffle/issues/1642#issuecomment-2049919884
My mistake. It cannot be optimized this way.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For additional commands, e-mail: issues-help@uniffle.apache.org
Re: [I] [Improvement] Optimize CompositeByteBuf initialization for better performance when getting memory data [incubator-uniffle]
Posted by "rickyma (via GitHub)" <gi...@apache.org>.
rickyma closed issue #1642: [Improvement] Optimize CompositeByteBuf initialization for better performance when getting memory data
URL: https://github.com/apache/incubator-uniffle/issues/1642
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For additional commands, e-mail: issues-help@uniffle.apache.org