You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@uniffle.apache.org by "rickyma (via GitHub)" <gi...@apache.org> on 2024/06/11 09:40:01 UTC
[PR] [#1773] improvement(client): Reduce chunkSize of gRPC Netty's PooledByteBufAllocator to reduce memory usage [incubator-uniffle]
rickyma opened a new pull request, #1775:
URL: https://github.com/apache/incubator-uniffle/pull/1775
### What changes were proposed in this pull request?
Introduce configs to reduce the chunkSize of the gRPC internal Netty's PooledByteBufAllocator to reduce memory usage.
### Why are the changes needed?
For https://github.com/apache/incubator-uniffle/issues/1773.
### Does this PR introduce _any_ user-facing change?
No.
### How was this patch tested?
Existing tests.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For additional commands, e-mail: issues-help@uniffle.apache.org
Re: [PR] [#1773] improvement(client): Reduce chunkSize of gRPC Netty's PooledByteBufAllocator to reduce memory usage [incubator-uniffle]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #1775:
URL: https://github.com/apache/incubator-uniffle/pull/1775#issuecomment-2160389438
## Test Results
2 433 files ±0 2 433 suites ±0 5h 0m 29s :stopwatch: +36s
934 tests ±0 933 :white_check_mark: ±0 1 :zzz: ±0 0 :x: ±0
10 828 runs ±0 10 814 :white_check_mark: ±0 14 :zzz: ±0 0 :x: ±0
Results for commit 7901d994. ± Comparison against base commit 1c72cb74.
[test-results]:data:application/gzip;base64,H4sIAJEmaGYC/03MTQ6DIBCG4asY1l3MACr0Mg3CmJD60yCsTO9eMEpdfg/Du7PRT7SxZ8OlEI+GbcnH+3YpmOjXJQsq4DpTfo/lQgt5rdeWrD1I/OntP+VXhdH4KQNUoBDWcEpIS2kiKK7OeUWzoax2Vivcqse+R+06zz7mwXoN6LSWKKm3hgZhkMBxGFroyGnsFHGQLbLvD+5VHJ0SAQAA
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For additional commands, e-mail: issues-help@uniffle.apache.org
Re: [PR] [#1773] improvement(client): Reduce chunkSize of gRPC Netty's PooledByteBufAllocator to reduce memory usage [incubator-uniffle]
Posted by "rickyma (via GitHub)" <gi...@apache.org>.
rickyma commented on PR #1775:
URL: https://github.com/apache/incubator-uniffle/pull/1775#issuecomment-2160853528
PTAL. After this PR, I will do the same for the server-side. @zuston @jerqi
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For additional commands, e-mail: issues-help@uniffle.apache.org
Re: [PR] [#1773] improvement(client): Reduce chunkSize of gRPC Netty's PooledByteBufAllocator to reduce memory usage [incubator-uniffle]
Posted by "jerqi (via GitHub)" <gi...@apache.org>.
jerqi merged PR #1775:
URL: https://github.com/apache/incubator-uniffle/pull/1775
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For additional commands, e-mail: issues-help@uniffle.apache.org
Re: [PR] [#1773] improvement(client): Reduce chunkSize of gRPC Netty's PooledByteBufAllocator to reduce memory usage [incubator-uniffle]
Posted by "rickyma (via GitHub)" <gi...@apache.org>.
rickyma commented on PR #1775:
URL: https://github.com/apache/incubator-uniffle/pull/1775#issuecomment-2160358545
Why are the default values set this way? Please read this.
Currently, the most commonly used grpc interface should be the pre-allocated memory interface (requireBuffer), and its request object structure is as follows:
```
message RequireBufferRequest {
int32 requireSize = 1;
string appId = 2;
int32 shuffleId = 3;
repeated int32
partitionIds = 4;
}
```
Now, we can know the entire Proto object's size estimation in the Java heap:
Object overhead: 16 bytes
requireSize field: 4 bytes
appId field: about 130 bytes
shuffleId field: 4 bytes
partitionIds field: 16 bytes (assuming an empty array)
Total size: 16 + 4 + 130 + 4 + 16 = 170 bytes.
So it can be understood that the size of **_the RequireBufferRequest object is basically equivalent to the size of the partitionIds array_**, with each int32 element occupying 4 bytes. Then we can know the approximate correspondence between the array size and the RequireBufferRequest object size:
| Number of partitions | Occupied size |
| -------- | -------- |
| 100 | About0.4KB |
| 1,000 | About4KB |
| 10,000 | About40KB |
| 100,000 | About400KB |
| 1,000,000| About4MB |
In order to consider the versatility, our chunkSize is set to 32KB, which can basically cover most of the normal number of partition scenarios.
Therefore, because the minimum pageSize of Netty is 4096, the pageSize can be set to 4KB, and the maxOrder can be set to 3, because 4KB * 2^3 = 32KB, which can get the chunkSize of 32KB. This setting is reasonable because the smallest memory block is 4KB, which will not cause too much memory fragmentation and can meet most memory requirements.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For additional commands, e-mail: issues-help@uniffle.apache.org
Re: [PR] [#1773] improvement(client): Reduce chunkSize of gRPC Netty's PooledByteBufAllocator to reduce memory usage [incubator-uniffle]
Posted by "rickyma (via GitHub)" <gi...@apache.org>.
rickyma commented on PR #1775:
URL: https://github.com/apache/incubator-uniffle/pull/1775#issuecomment-2160585190
Before optimization:
usedDirectMemoryByGrpcNetty: 2696937751
After optimization:
usedDirectMemoryByGrpcNetty: 28705047
The optimization effect is very obvious, basically reducing by an order of magnitude, from using hundreds of MB to only occupying tens of MB.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For additional commands, e-mail: issues-help@uniffle.apache.org