You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pulsar.apache.org by "shibd (via GitHub)" <gi...@apache.org> on 2024/07/05 19:07:03 UTC

[I] High GC pause cases high publish latency [pulsar]

shibd opened a new issue, #22866:
URL: https://github.com/apache/pulsar/issues/22866

   ### Search before asking
   
   - [X] I searched in the [issues](https://github.com/apache/pulsar/issues) and found nothing similar.
   
   
   ### Motivation
   
   We have observed that using Individual ack frequently results in young generation GC and relatively high GC pauses.
   
   <img width="668" alt="image" src="https://github.com/apache/pulsar/assets/33416836/0a2634a4-39f0-4d4d-b3a1-5e952173f321">
   
   - Resource: k8s cluster
     - broker(1.5CPU)(3CPU 8Gi)
     - bookie(2.5CPU 8Gi) 250Gi gp3 ssd
     - zookeeper(1CPU 4Gi)
   - Version: 3.2.2
   
   Performance cmd:
   ```
   bin/pulsar-perf consume test-c 1200 -n 10 -st Shared -t 10
   bin/pulsar-perf produce test-p -s 100 -time 1200 -n 10 -t 10 -ml 256M
   ```
   
   
   ### Solution
   
   The issue should be caused by the large long[] allocated by bitmap. The allocation invoking chain.
   
   <img width="1797" alt="image" src="https://github.com/apache/pulsar/assets/33416836/6d559f3d-e0fa-4725-85ae-21a8bd9083f5">
   
   <img width="1702" alt="image" src="https://github.com/apache/pulsar/assets/33416836/18bda1be-6dd0-4e6f-bfc3-6d17b7b60c8d">
   
   
   Maybe this is a normal phenomenon and can be solved by increasing the CPU, I would like to share it here to see if there are any similar problems and suggestions in the community.
   
   ### Alternatives
   
   _No response_
   
   ### Anything else?
   
   _No response_
   
   ### Are you willing to submit a PR?
   
   - [ ] I'm willing to submit a PR!


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [I] High GC pause cases high publish latency [pulsar]

Posted by "lhotari (via GitHub)" <gi...@apache.org>.
lhotari commented on issue #22866:
URL: https://github.com/apache/pulsar/issues/22866#issuecomment-2154013458

   Btw. The metric might be wrong. It's not necessarily pause time, but GC time that happens in the background.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [I] High GC pause cases high publish latency [pulsar]

Posted by "lhotari (via GitHub)" <gi...@apache.org>.
lhotari commented on issue #22866:
URL: https://github.com/apache/pulsar/issues/22866#issuecomment-2167218549

   > I think so. But we might need to improve here to avoid allocating a very large `long[]` to give GC pressure. :)
   
   
   One alternative would be to switch to use https://github.com/lemire/RoaringBitmap/ . 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [I] High GC pause cases high publish latency [pulsar]

Posted by "mattisonchao (via GitHub)" <gi...@apache.org>.
mattisonchao commented on issue #22866:
URL: https://github.com/apache/pulsar/issues/22866#issuecomment-2154038308

   
   >Btw. The metric might be wrong. It's not necessarily pause time, but GC time that happens in the background. See https://github.com/apache/pulsar/pull/18891
   
   
   I think so. But we might need to improve here to avoid allocating a very large `long[]` to give GC pressure. :) 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [I] High GC pause cases high publish latency [pulsar]

Posted by "dao-jun (via GitHub)" <gi...@apache.org>.
dao-jun closed issue #22866: High GC pause cases high publish latency
URL: https://github.com/apache/pulsar/issues/22866


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org