You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pulsar.apache.org by "TakaHiR07 (via GitHub)" <gi...@apache.org> on 2025/05/28 02:45:39 UTC

[I] [Bug] Memory Leak In Netty Recycler of Bookie Client [pulsar]

TakaHiR07 opened a new issue, #24355:
URL: https://github.com/apache/pulsar/issues/24355

   ### Search before reporting
   
   - [x] I searched in the [issues](https://github.com/apache/pulsar/issues) and found nothing similar.
   
   
   ### Read release policy
   
   - [x] I understand that [unsupported versions](https://pulsar.apache.org/contribute/release-policy/#supported-versions) don't get bug fixes. I will attempt to reproduce the issue on a supported version of Pulsar client and Pulsar broker.
   
   
   ### User environment
   
   pulsar-2.9.x and pulsar-3.0.x both have this memory leak
   
   ### Issue Description
   
   After running the broker for a long time, it is found that the broker heap memory  and zgc time keep increasing. 
   After dumping the heap memory, we found that reason is in netty recycler which is use for cache of bookie client variable. The recycler used memory keep increasing. 
   
   As seen in the heapdump, there are so many LocalPools in one FastThreadLocalThread, and the consumerBuffer contain so many reference in one LocalPool.
   
   Our setting is io.netty.recycler.maxCapacityPerThread=1024, PerchannelBookieClient number is 16 * 500=8000.  And if change to io.netty.recycler.maxCapacityPerThread=0, the memory leak issue is fixed, but the write and read performance would decrease.
   
   ![Image](https://github.com/user-attachments/assets/49037835-fb80-4503-ad96-2ae6175b9830)
   ![Image](https://github.com/user-attachments/assets/428c5d39-4774-49a4-b676-96181449e72d)
   
   ![Image](https://github.com/user-attachments/assets/83ffa9c5-6b18-4325-8f5e-1afa8eadf628)
   
   ![Image](https://github.com/user-attachments/assets/f5780d09-8630-4f50-bbea-5bf6f2cb29cf)
   
   ### Error messages
   
   ```text
   
   ```
   
   ### Reproducing the issue
   
   continue running broker, start a perf produce process with large qps, normal throughput can reproduce.
   
   ### Additional information
   
   _No response_
   
   ### Are you willing to submit a PR?
   
   - [ ] I'm willing to submit a PR!


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] [Bug] Memory Leak In Netty Recycler of Bookie Client [pulsar]

Posted by "TakaHiR07 (via GitHub)" <gi...@apache.org>.
TakaHiR07 commented on issue #24355:
URL: https://github.com/apache/pulsar/issues/24355#issuecomment-2915621825

   
   
   
   > select `ChannelOutboundBuffer` objects in your heap dump
   
   seems no this object
   ![Image](https://github.com/user-attachments/assets/811580b1-f0ad-4431-b4a8-bcea5bd5181c)


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] [Bug] Memory Leak In Netty Recycler of Bookie Client [pulsar]

Posted by "TakaHiR07 (via GitHub)" <gi...@apache.org>.
TakaHiR07 closed issue #24355: [Bug] Memory Leak In Netty Recycler of Bookie Client
URL: https://github.com/apache/pulsar/issues/24355


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] [Bug] Memory Leak In Netty Recycler of Bookie Client [pulsar]

Posted by "zymap (via GitHub)" <gi...@apache.org>.
zymap commented on issue #24355:
URL: https://github.com/apache/pulsar/issues/24355#issuecomment-2942531932

   @TakaHiR07 Any pictures show after that fix?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] [Bug] Memory Leak In Netty Recycler of Bookie Client [pulsar]

Posted by "TakaHiR07 (via GitHub)" <gi...@apache.org>.
TakaHiR07 commented on issue #24355:
URL: https://github.com/apache/pulsar/issues/24355#issuecomment-2939201235

   Close this issue since has been fixed in https://github.com/apache/bookkeeper/pull/4609 and https://github.com/apache/bookkeeper/pull/4610


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] [Bug] Memory Leak In Netty Recycler of Bookie Client [pulsar]

Posted by "dao-jun (via GitHub)" <gi...@apache.org>.
dao-jun commented on issue #24355:
URL: https://github.com/apache/pulsar/issues/24355#issuecomment-2915225876

   all your bookies works fine?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] [Bug] Memory Leak In Netty Recycler of Bookie Client [pulsar]

Posted by "dao-jun (via GitHub)" <gi...@apache.org>.
dao-jun commented on issue #24355:
URL: https://github.com/apache/pulsar/issues/24355#issuecomment-2915601494

   select `ChannelOutboundBuffer` objects in your heap dump


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] [Bug] Memory Leak In Netty Recycler of Bookie Client [pulsar]

Posted by "TakaHiR07 (via GitHub)" <gi...@apache.org>.
TakaHiR07 commented on issue #24355:
URL: https://github.com/apache/pulsar/issues/24355#issuecomment-2915541813

   > Does the direct memory keeps increasing?
   
   it seems increase
   ![Image](https://github.com/user-attachments/assets/15e88927-d1af-4657-96f2-6f13814203f7)


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] [Bug] Memory Leak In Netty Recycler of Bookie Client [pulsar]

Posted by "TakaHiR07 (via GitHub)" <gi...@apache.org>.
TakaHiR07 commented on issue #24355:
URL: https://github.com/apache/pulsar/issues/24355#issuecomment-2921469740

   I have found the reason of the oom risk, you can check it again @dao-jun 
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] [Bug] Memory Leak In Netty Recycler of Bookie Client [pulsar]

Posted by "TakaHiR07 (via GitHub)" <gi...@apache.org>.
TakaHiR07 commented on issue #24355:
URL: https://github.com/apache/pulsar/issues/24355#issuecomment-2942592859

   > [@TakaHiR07](https://github.com/TakaHiR07) Any pictures show the JVM and GC after that fix?
   
   In test cluster.
   
   Before fix: 
   
   ![Image](https://github.com/user-attachments/assets/2369ed89-b211-4765-a23b-92d15ba1a682)
   
   ![Image](https://github.com/user-attachments/assets/2e6321a7-26cf-4d36-a19c-1006da502eb9)
   
   
   After fix:
   
   ![Image](https://github.com/user-attachments/assets/45534eea-10d9-4b64-911c-dffe3712b236)
   
   ![Image](https://github.com/user-attachments/assets/732c6b2d-52b7-449f-ab3c-4a89859f287b)


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] [Bug] Memory Leak In Netty Recycler of Bookie Client [pulsar]

Posted by "TakaHiR07 (via GitHub)" <gi...@apache.org>.
TakaHiR07 commented on issue #24355:
URL: https://github.com/apache/pulsar/issues/24355#issuecomment-2915253142

   
   
   
   
   > Are there any bookies being quarantined?
   
   a little. 
   ![Image](https://github.com/user-attachments/assets/5b151837-da11-42b0-9dfe-26ffe2cea8fc)


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] [Bug] Memory Leak In Netty Recycler of Bookie Client [pulsar]

Posted by "dao-jun (via GitHub)" <gi...@apache.org>.
dao-jun commented on issue #24355:
URL: https://github.com/apache/pulsar/issues/24355#issuecomment-2915530892

   Does the direct memory keeps increasing?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] [Bug] Memory Leak In Netty Recycler of Bookie Client [pulsar]

Posted by "dao-jun (via GitHub)" <gi...@apache.org>.
dao-jun commented on issue #24355:
URL: https://github.com/apache/pulsar/issues/24355#issuecomment-2915230488

   Are there any bookies being quarantined?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] [Bug] Memory Leak In Netty Recycler of Bookie Client [pulsar]

Posted by "dao-jun (via GitHub)" <gi...@apache.org>.
dao-jun commented on issue #24355:
URL: https://github.com/apache/pulsar/issues/24355#issuecomment-2915700677

   how many instances of `ChannelOutboundBuffer` class? 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org