You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@bookkeeper.apache.org by "wenbingshen (via GitHub)" <gi...@apache.org> on 2023/04/11 09:34:14 UTC
[GitHub] [bookkeeper] wenbingshen opened a new pull request, #3913: Enable PCBC completionObjects autoShrink to reduce memory usage and gc
wenbingshen opened a new pull request, #3913:
URL: https://github.com/apache/bookkeeper/pull/3913
### Motivation
PerChannelBookieClient completionObjects occupy a lot of heap space and cannot be recycled.
The figure below shows that the internal table array of ConcurrentOpenHashMap has used space size=0, but the array length is still 16384, and the memory overhead is 65552bytes.
ConcurrentOpenHashMap default DefaultConcurrencyLevel=16. We have hundreds of bookie nodes. Due to the feature of bookie polling and writing, the client and server have long connection characteristics, which will as a result, the memory usage of about 65552 * 16 * 1776 = 1.74GB cannot be recycled, and the space take up by these tables is all size=0.
When the throughput of the pulsar cluster increases and the bookie cluster expands, these memory usage will also increase. In addition to some other memory usage, the pulsar broker will continuously generate full gc.
### Changes
I think adding autoShrink to completionObjects can reduce this part of memory usage and reduce the frequency of Full GC.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@bookkeeper.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [bookkeeper] wenbingshen commented on pull request #3913: Enable PCBC completionObjects autoShrink to reduce memory usage and gc
Posted by "wenbingshen (via GitHub)" <gi...@apache.org>.
wenbingshen commented on PR #3913:
URL: https://github.com/apache/bookkeeper/pull/3913#issuecomment-1515640324
@merlimat @eolivelli @dlg99 @zymap Can you help take a look at this pr. Thanks.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@bookkeeper.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [bookkeeper] hangc0276 merged pull request #3913: Enable PCBC completionObjects autoShrink to reduce memory usage and gc
Posted by "hangc0276 (via GitHub)" <gi...@apache.org>.
hangc0276 merged PR #3913:
URL: https://github.com/apache/bookkeeper/pull/3913
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@bookkeeper.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [bookkeeper] horizonzy commented on pull request #3913: Enable PCBC completionObjects autoShrink to reduce memory usage and gc
Posted by "horizonzy (via GitHub)" <gi...@apache.org>.
horizonzy commented on PR #3913:
URL: https://github.com/apache/bookkeeper/pull/3913#issuecomment-1507843649
Thanks, I got it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@bookkeeper.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [bookkeeper] horizonzy commented on pull request #3913: Enable PCBC completionObjects autoShrink to reduce memory usage and gc
Posted by "horizonzy (via GitHub)" <gi...@apache.org>.
horizonzy commented on PR #3913:
URL: https://github.com/apache/bookkeeper/pull/3913#issuecomment-1507302969
> In our pulsar broker, such an array would occupy about = 65552 * 16 * 1776 = 1.74GB
Why multiply 16.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@bookkeeper.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [bookkeeper] wenbingshen commented on pull request #3913: Enable PCBC completionObjects autoShrink to reduce memory usage and gc
Posted by "wenbingshen (via GitHub)" <gi...@apache.org>.
wenbingshen commented on PR #3913:
URL: https://github.com/apache/bookkeeper/pull/3913#issuecomment-1506868164
> I have a question about this. The memory occupation is not about the array size. The key(CompletionKey) and the value(CompletionValue) is the occupier. As long as the key and the value is removed, the memory occupation will be decrease. We make the array size autoShrink didn't help the GC
@horizonzy The layout of the array object in memory, when pointer compression is enabled, includes
array object header + length * 4 (reference) + length * (single element size)
We have an object array with a size of 16384, and the space it occupies in memory is = 8 + 4 + 4 + 4 * 16384 = 65552
In our pulsar broker, such an array would occupy about = 65552 * 16 * 1776 = 1.74GB
If we can turn on autoShrink, in the case of size=0, the array size will shrink to 24.
And the space it occupies in memory is = 8 + 4 + 4 + 4 * 24 = 112
In our pulsar broker, such an array would occupy about = 112 * 16 * 1776 = 3108KB
This way we can reclaim a lot of space in memory.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@bookkeeper.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [bookkeeper] horizonzy commented on pull request #3913: Enable PCBC completionObjects autoShrink to reduce memory usage and gc
Posted by "horizonzy (via GitHub)" <gi...@apache.org>.
horizonzy commented on PR #3913:
URL: https://github.com/apache/bookkeeper/pull/3913#issuecomment-1505400156
> ConcurrentOpenHashMap default DefaultConcurrencyLevel=16. We have hundreds of bookie nodes. Due to the feature of bookie polling and writing, the client and server have long connection characteristics, which will as a result, the memory usage of about 65552 * 16 * 1776 = 1.74GB cannot be recycled, and the space take up by these tables is all size=0 (The broker's owner topic has drifted to other brokers due to Full GC).
I have a question about this. The memory occupation is not about the array size. The key(CompletionKey) and the value(CompletionValue) is the occupier. As long as the key and the value is removed, the memory occupation will be decrease.
We make the array size autoShrink didn't help the GC
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@bookkeeper.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [bookkeeper] wenbingshen commented on pull request #3913: Enable PCBC completionObjects autoShrink to reduce memory usage and gc
Posted by "wenbingshen (via GitHub)" <gi...@apache.org>.
wenbingshen commented on PR #3913:
URL: https://github.com/apache/bookkeeper/pull/3913#issuecomment-1507841889
> Why multiply 16.
@horizonzy ConcurrentOpenHashMap default DefaultConcurrencyLevel=16
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@bookkeeper.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org