[jira] [Updated] (AMQ-5486) Thread synchronization overhead is unexpectedly high

["Benjamin Huang (JIRA)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/AMQ-5486?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Benjamin  Huang updated AMQ-5486:
---------------------------------
    Attachment: nio_worker_blocked_frequently.png
                nio_worker_blocked_2.png
                nio_worker_blocked_1.png

> Thread synchronization overhead is unexpectedly high
> ----------------------------------------------------
>
>                 Key: AMQ-5486
>                 URL: https://issues.apache.org/jira/browse/AMQ-5486
>             Project: ActiveMQ
>          Issue Type: Bug
>          Components: Broker
>    Affects Versions: 5.9.1
>         Environment: UbuntuServer 12.04 x86_64, Linux Kernel 3.20.23, OpenJDK 1.7.0_51 64-Bit Server VM, Xms 2G, Xms 8G, CPU: E5-2620 v2 X 2, 64 GB RAM
>            Reporter: Benjamin  Huang
>         Attachments: nio_worker_blocked_1.png, nio_worker_blocked_2.png, nio_worker_blocked_frequently.png
>
>
> There's about 20 topics with virtual topic enabled, hundreds of comsumers/producers connected to MQ on NIO transport connector. During the run there're about 12000 msg flow in per second, not a very high rate, but ActiveMQ consumes a lot of CPU resource (about 600%~1000%). To find out what's the most CPU consuming code path, I use JProfiler to dig into the process. 
> Among all the NIO worker threads, most of them were frequently blocked and did a little job between the 'unblocked' time. While they're expected spend most of their time slices on waiting for work item and processed them.
> !nio_worker_blocked_frequently.png!
> After reviewing the monitor usage history and stats, I think these NIO workers were competing fiercely with each other on executing a synchronized method (DestinationMap::get), which is also the most hot spot in the program . I also notice that the caller AbstractRegion::getDestinations acquires a read lock before calling it, so I guess this could be a left out, read lock is the actual lock type required here.
> !monitor_usage_stats.png!
> !nio_worker_blocked_1.png!
> !nio_worker_blocked_2.png!
> As a rookie to Java, it's too difficult for me to list all critical sections between NIO workers, or between NIO workers and BrokerService which adds up to the overall synchronize overhead. So I attach the relevant info, with the hope of finding a feasible solution to this.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)