You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pulsar.apache.org by "TakaHiR07 (via GitHub)" <gi...@apache.org> on 2025/05/28 02:45:39 UTC
[I] [Bug] Memory Leak In Netty Recycler of Bookie Client [pulsar]
TakaHiR07 opened a new issue, #24355:
URL: https://github.com/apache/pulsar/issues/24355
### Search before reporting
- [x] I searched in the [issues](https://github.com/apache/pulsar/issues) and found nothing similar.
### Read release policy
- [x] I understand that [unsupported versions](https://pulsar.apache.org/contribute/release-policy/#supported-versions) don't get bug fixes. I will attempt to reproduce the issue on a supported version of Pulsar client and Pulsar broker.
### User environment
pulsar-2.9.x and pulsar-3.0.x both have this memory leak
### Issue Description
After running the broker for a long time, it is found that the broker heap memory and zgc time keep increasing.
After dumping the heap memory, we found that reason is in netty recycler which is use for cache of bookie client variable. The recycler used memory keep increasing.
As seen in the heapdump, there are so many LocalPools in one FastThreadLocalThread, and the consumerBuffer contain so many reference in one LocalPool.
Our setting is io.netty.recycler.maxCapacityPerThread=1024, PerchannelBookieClient number is 16 * 500=8000. And if change to io.netty.recycler.maxCapacityPerThread=0, the memory leak issue is fixed, but the write and read performance would decrease.




### Error messages
```text
```
### Reproducing the issue
continue running broker, start a perf produce process with large qps, normal throughput can reproduce.
### Additional information
_No response_
### Are you willing to submit a PR?
- [ ] I'm willing to submit a PR!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug] Memory Leak In Netty Recycler of Bookie Client [pulsar]
Posted by "TakaHiR07 (via GitHub)" <gi...@apache.org>.
TakaHiR07 commented on issue #24355:
URL: https://github.com/apache/pulsar/issues/24355#issuecomment-2915621825
> select `ChannelOutboundBuffer` objects in your heap dump
seems no this object

--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug] Memory Leak In Netty Recycler of Bookie Client [pulsar]
Posted by "TakaHiR07 (via GitHub)" <gi...@apache.org>.
TakaHiR07 closed issue #24355: [Bug] Memory Leak In Netty Recycler of Bookie Client
URL: https://github.com/apache/pulsar/issues/24355
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug] Memory Leak In Netty Recycler of Bookie Client [pulsar]
Posted by "zymap (via GitHub)" <gi...@apache.org>.
zymap commented on issue #24355:
URL: https://github.com/apache/pulsar/issues/24355#issuecomment-2942531932
@TakaHiR07 Any pictures show after that fix?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug] Memory Leak In Netty Recycler of Bookie Client [pulsar]
Posted by "TakaHiR07 (via GitHub)" <gi...@apache.org>.
TakaHiR07 commented on issue #24355:
URL: https://github.com/apache/pulsar/issues/24355#issuecomment-2939201235
Close this issue since has been fixed in https://github.com/apache/bookkeeper/pull/4609 and https://github.com/apache/bookkeeper/pull/4610
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug] Memory Leak In Netty Recycler of Bookie Client [pulsar]
Posted by "dao-jun (via GitHub)" <gi...@apache.org>.
dao-jun commented on issue #24355:
URL: https://github.com/apache/pulsar/issues/24355#issuecomment-2915225876
all your bookies works fine?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug] Memory Leak In Netty Recycler of Bookie Client [pulsar]
Posted by "dao-jun (via GitHub)" <gi...@apache.org>.
dao-jun commented on issue #24355:
URL: https://github.com/apache/pulsar/issues/24355#issuecomment-2915601494
select `ChannelOutboundBuffer` objects in your heap dump
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug] Memory Leak In Netty Recycler of Bookie Client [pulsar]
Posted by "TakaHiR07 (via GitHub)" <gi...@apache.org>.
TakaHiR07 commented on issue #24355:
URL: https://github.com/apache/pulsar/issues/24355#issuecomment-2915541813
> Does the direct memory keeps increasing?
it seems increase

--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug] Memory Leak In Netty Recycler of Bookie Client [pulsar]
Posted by "TakaHiR07 (via GitHub)" <gi...@apache.org>.
TakaHiR07 commented on issue #24355:
URL: https://github.com/apache/pulsar/issues/24355#issuecomment-2921469740
I have found the reason of the oom risk, you can check it again @dao-jun
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug] Memory Leak In Netty Recycler of Bookie Client [pulsar]
Posted by "TakaHiR07 (via GitHub)" <gi...@apache.org>.
TakaHiR07 commented on issue #24355:
URL: https://github.com/apache/pulsar/issues/24355#issuecomment-2942592859
> [@TakaHiR07](https://github.com/TakaHiR07) Any pictures show the JVM and GC after that fix?
In test cluster.
Before fix:


After fix:


--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug] Memory Leak In Netty Recycler of Bookie Client [pulsar]
Posted by "TakaHiR07 (via GitHub)" <gi...@apache.org>.
TakaHiR07 commented on issue #24355:
URL: https://github.com/apache/pulsar/issues/24355#issuecomment-2915253142
> Are there any bookies being quarantined?
a little.

--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug] Memory Leak In Netty Recycler of Bookie Client [pulsar]
Posted by "dao-jun (via GitHub)" <gi...@apache.org>.
dao-jun commented on issue #24355:
URL: https://github.com/apache/pulsar/issues/24355#issuecomment-2915530892
Does the direct memory keeps increasing?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug] Memory Leak In Netty Recycler of Bookie Client [pulsar]
Posted by "dao-jun (via GitHub)" <gi...@apache.org>.
dao-jun commented on issue #24355:
URL: https://github.com/apache/pulsar/issues/24355#issuecomment-2915230488
Are there any bookies being quarantined?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug] Memory Leak In Netty Recycler of Bookie Client [pulsar]
Posted by "dao-jun (via GitHub)" <gi...@apache.org>.
dao-jun commented on issue #24355:
URL: https://github.com/apache/pulsar/issues/24355#issuecomment-2915700677
how many instances of `ChannelOutboundBuffer` class?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org