You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pulsar.apache.org by "lhotari (via GitHub)" <gi...@apache.org> on 2025/01/18 01:22:39 UTC

[I] [Bug] pulsar-io EventLoopGroup is used in dispatchers to execute synchronized code and CPU intensive work [pulsar]

lhotari opened a new issue, #23865:
URL: https://github.com/apache/pulsar/issues/23865

   ### Search before asking
   
   - [x] I searched in the [issues](https://github.com/apache/pulsar/issues) and found nothing similar.
   
   
   ### Read release policy
   
   - [x] I understand that unsupported versions don't get bug fixes. I will attempt to reproduce the issue on a supported version of Pulsar client and Pulsar broker.
   
   
   ### Version
   
   All released versions
   
   ### Minimal reproduce step
   
   In dispatchers, `topic.getBrokerService().executor()`, the "pulsar-io" EventLoopGroup is used to execute synchronized code and CPU intensive work. The "pulsar-io" EventLoopGroup is intended to be the worker for Pulsar's ServerCnx Netty handler so that network connections remain responsive. Since dispatcher implementations contain multiple synchronized methods, blocking operations could impact all connections that are bound to the same IO thread. 
   
   Another potential problem that the solution causes unnecessary context switching since the current solution doesn't use a single thread for the scheduled jobs and the work that it performs.
   
   ### What did you expect to see?
   
   A pinned executor to be used for executing all operations for a dispatcher or replicator.
   
   ### What did you see instead?
   
   EventLoopGroup is used as executor and the thread could change for each execution.
   
   ### Anything else?
   
   _No response_
   
   ### Are you willing to submit a PR?
   
   - [x] I'm willing to submit a PR!


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [I] [Bug] pulsar-io EventLoopGroup is used in dispatchers to execute synchronized code and CPU intensive work [pulsar]

Posted by "lhotari (via GitHub)" <gi...@apache.org>.
lhotari commented on issue #23865:
URL: https://github.com/apache/pulsar/issues/23865#issuecomment-3207059519

   this is a real performance issue with Pulsar. In #24623, I have added a solution for profiling test cases with Async Profiler with JFR integrated to it. It's possible to analyze the results with [Eclipse Mission Control](https://adoptium.net/jmc).
   
   The lock analysis shows that dispatchers contribute to most blocking on pulsar-io threads:
   <img width="1089" height="1408" alt="Image" src="https://github.com/user-attachments/assets/f3af93eb-469b-4fe4-a6a2-3b3990cfce99" />
   
   This is a problem since Netty IO threads shouldn't ever be blocked since one thread is used across multiple connections to handle the asynchronous IO events. Blocking will cause a "noisy neighbor" problem across connections.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [I] [Bug] pulsar-io EventLoopGroup is used in dispatchers to execute synchronized code and CPU intensive work [pulsar]

Posted by "merlimat (via GitHub)" <gi...@apache.org>.
merlimat commented on issue #23865:
URL: https://github.com/apache/pulsar/issues/23865#issuecomment-6380956649

   Still present on master. `BrokerService.executor()` returns the Netty worker group (`pulsar-broker/src/main/java/org/apache/pulsar/broker/service/BrokerService.java:3622`), and the dispatchers still schedule their synchronized read and flow work on it, for example the read reschedule in `PersistentDispatcherMultipleConsumers.java:602-604` and the flow handling at `:330`/`:378`, plus `PersistentDispatcherSingleActiveConsumer.java:139`/`:439`/`:554` and `PersistentReplicator.java:340`/`:365`. Only the dispatch step itself now uses a pinned topic-ordered thread (`PersistentDispatcherMultipleConsumers.java:182`). So blocking in these methods can still stall every connection bound to the same pulsar-io thread. The likely fix is to run each dispatcher's and replicator's scheduled work on one thread chosen from the topic-ordered executor, as the dispatch step already does.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org