You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@activemq.apache.org by Netlancer <ne...@mobigraph.co> on 2014/06/02 05:06:40 UTC
ActiveMQ JVM crashes when high number of concurrent connections get
terminated
Hi,
We have activemq running behind an LB. we have close to 55K
connections to activemq. during cases of more than 1000 connections gets
terminated concurrently, we observe a huge spike in the number of
threads created.following which the jvm crashes. on further
investigation, these threads are pertained to executors
SelectorManager - defaultExecutor - labelled with "ActiveMQ NIO Worker"
- the executor has maximum thread pool size of Integer.MAX_VALUE
TaskRunnerFactory - defaultExecutor - labelled as "ActiveMQ Task" - this
executor as well has Integer.MAX_VALUE as the max pool size.
BrokerService.getTaskRunnerFactory - labelled as "ActiveMQ
BrokerService[%s] Task-" - this executor is also created with
Integer.MAX_VALUE
the configured persistence store is leveldb and dedicatedtaskrunner is
disabled. The JVM is running with 8GB RAM with the max heap at 5G.
following snippet from the hs_pid crash file
Java Threads: ( => current thread )
0x00007f1e42863800 JavaThread "ActiveMQ NIO Worker 9053" daemon
[_thread_blocked, id=10690, stack(0x00007f1d6fa53000,0x00007f1d6fa8c000)]
0x00007f1ea40dc000 JavaThread "ActiveMQ NIO Worker 9047" daemon
[_thread_blocked, id=10689, stack(0x00007f1d6fa8c000,0x00007f1d6fac5000)]
0x00007f1eac116800 JavaThread "ActiveMQ NIO Worker 9054" daemon
[_thread_blocked, id=10688, stack(0x00007f1d6fac5000,0x00007f1d6fafe000)]
0x00007f1df43f5800 JavaThread "ActiveMQ NIO Worker 9046" daemon
[_thread_blocked, id=10687, stack(0x00007f1d6fafe000,0x00007f1d6fb37000)]
0x00007f1e42870000 JavaThread "ActiveMQ NIO Worker 9056" daemon
[_thread_blocked, id=10686, stack(0x00007f1d6fb37000,0x00007f1d6fb70000)]
0x00007f1ef6054800 JavaThread "ActiveMQ NIO Worker 9107" daemon
[_thread_blocked, id=10685, stack(0x00007f1d6fb70000,0x00007f1d6fba9000)]
0x00007f1ef46af800 JavaThread "ActiveMQ NIO Worker 9059" daemon
[_thread_blocked, id=10684, stack(0x00007f1d6fba9000,0x00007f1d6fbe2000)]
0x00007f1e240cd000 JavaThread "ActiveMQ NIO Worker 9041" daemon
[_thread_blocked, id=10683, stack(0x00007f1d6fbe2000,0x00007f1d6fc1b000)]
0x00007f1e0016a800 JavaThread "ActiveMQ NIO Worker 9060" daemon
[_thread_blocked, id=10682, stack(0x00007f1d6fc1b000,0x00007f1d6fc54000)]
0x00007f1e5019d800 JavaThread "ActiveMQ NIO Worker 9061" daemon
[_thread_blocked, id=10681, stack(0x00007f1d6fc54000,0x00007f1d6fc8d000)]
0x00007f1e8006f000 JavaThread "ActiveMQ NIO Worker 9039" daemon
[_thread_blocked, id=10680, stack(0x00007f1d6fc8d000,0x00007f1d6fcc6000)]
0x00007f1e4c7df800 JavaThread "ActiveMQ NIO Worker 9101" daemon
[_thread_blocked, id=10679, stack(0x00007f1d6fcc6000,0x00007f1d6fcff000)]
0x00007f1da6794800 JavaThread "ActiveMQ BrokerService[MQMaster]
Task-1781" daemon [_thread_blocked, id=8302,
stack(0x00007f1db5a51000,0x00007f1db5a8a000)]
0x00007f1da6791000 JavaThread "ActiveMQ BrokerService[MQMaster]
Task-1780" daemon [_thread_blocked, id=8298,
stack(0x00007f1dc6d4c000,0x00007f1dc6d85000)]
0x00007f1da6793000 JavaThread "ActiveMQ BrokerService[MQMaster]
Task-1779" daemon [_thread_blocked, id=8296,
stack(0x00007f1dc6e30000,0x00007f1dc6e69000)]
0x00007f1da6790000 JavaThread "ActiveMQ BrokerService[MQMaster]
Task-1778" daemon [_thread_blocked, id=8293,
stack(0x00007f1db6903000,0x00007f1db693c000)]
0x00007f1da678f000 JavaThread "ActiveMQ BrokerService[MQMaster]
Task-1777" daemon [_thread_blocked, id=8292,
stack(0x00007f1db9dc8000,0x00007f1db9e01000)]
Please, can someone provide inputs on why this is happening.
Thanks
nl
Re: ActiveMQ JVM crashes when high number of concurrent connections
get terminated
Posted by Netlancer <ne...@mobigraph.co>.
Hi,
Please, can you let me know what is the effect of making
SelectorManager ThreadPoolExecutor as bounded ?
Thanks,
nl
On 02.06.2014 09.26, Netlancer wrote:
> Hi,
>
> yes. we increased the heap size to 8G in a machine which has 16GB
> RAM. the limits.conf is configured with 300000 as soft/hard limit and
> file-max is the same as hard limit. I have attached the hs_err file .
>
> Thanks,
>
> On 02.06.2014 08.52, Gaurav Sharma wrote:
>> If possible, pls provide some more info to help diagnose this. What's
>> the output of /etc/security/limits.conf and file-max value in
>> /etc/sysctl.conf?
>>
>> Is 5G the max you can allocate to amq's jvm?
>>
>> Sent from my iPhone
>>
>> On Jun 1, 2014, at 20:06, Netlancer <ne...@mobigraph.co> wrote:
>>
>> Hi,
>>
>> We have activemq running behind an LB. we have close to 55K
>> connections to activemq. during cases of more than 1000 connections
>> gets terminated concurrently, we observe a huge spike in the number
>> of threads created.following which the jvm crashes. on further
>> investigation, these threads are pertained to executors
>>
>> SelectorManager - defaultExecutor - labelled with "ActiveMQ NIO
>> Worker" - the executor has maximum thread pool size of Integer.MAX_VALUE
>> TaskRunnerFactory - defaultExecutor - labelled as "ActiveMQ Task" -
>> this executor as well has Integer.MAX_VALUE as the max pool size.
>> BrokerService.getTaskRunnerFactory - labelled as "ActiveMQ
>> BrokerService[%s] Task-" - this executor is also created with
>> Integer.MAX_VALUE
>>
>> the configured persistence store is leveldb and dedicatedtaskrunner
>> is disabled. The JVM is running with 8GB RAM with the max heap at 5G.
>>
>> following snippet from the hs_pid crash file
>>
>> Java Threads: ( => current thread )
>> 0x00007f1e42863800 JavaThread "ActiveMQ NIO Worker 9053" daemon
>> [_thread_blocked, id=10690,
>> stack(0x00007f1d6fa53000,0x00007f1d6fa8c000)]
>> 0x00007f1ea40dc000 JavaThread "ActiveMQ NIO Worker 9047" daemon
>> [_thread_blocked, id=10689,
>> stack(0x00007f1d6fa8c000,0x00007f1d6fac5000)]
>> 0x00007f1eac116800 JavaThread "ActiveMQ NIO Worker 9054" daemon
>> [_thread_blocked, id=10688,
>> stack(0x00007f1d6fac5000,0x00007f1d6fafe000)]
>> 0x00007f1df43f5800 JavaThread "ActiveMQ NIO Worker 9046" daemon
>> [_thread_blocked, id=10687,
>> stack(0x00007f1d6fafe000,0x00007f1d6fb37000)]
>> 0x00007f1e42870000 JavaThread "ActiveMQ NIO Worker 9056" daemon
>> [_thread_blocked, id=10686,
>> stack(0x00007f1d6fb37000,0x00007f1d6fb70000)]
>> 0x00007f1ef6054800 JavaThread "ActiveMQ NIO Worker 9107" daemon
>> [_thread_blocked, id=10685,
>> stack(0x00007f1d6fb70000,0x00007f1d6fba9000)]
>> 0x00007f1ef46af800 JavaThread "ActiveMQ NIO Worker 9059" daemon
>> [_thread_blocked, id=10684,
>> stack(0x00007f1d6fba9000,0x00007f1d6fbe2000)]
>> 0x00007f1e240cd000 JavaThread "ActiveMQ NIO Worker 9041" daemon
>> [_thread_blocked, id=10683,
>> stack(0x00007f1d6fbe2000,0x00007f1d6fc1b000)]
>> 0x00007f1e0016a800 JavaThread "ActiveMQ NIO Worker 9060" daemon
>> [_thread_blocked, id=10682,
>> stack(0x00007f1d6fc1b000,0x00007f1d6fc54000)]
>> 0x00007f1e5019d800 JavaThread "ActiveMQ NIO Worker 9061" daemon
>> [_thread_blocked, id=10681,
>> stack(0x00007f1d6fc54000,0x00007f1d6fc8d000)]
>> 0x00007f1e8006f000 JavaThread "ActiveMQ NIO Worker 9039" daemon
>> [_thread_blocked, id=10680,
>> stack(0x00007f1d6fc8d000,0x00007f1d6fcc6000)]
>> 0x00007f1e4c7df800 JavaThread "ActiveMQ NIO Worker 9101" daemon
>> [_thread_blocked, id=10679,
>> stack(0x00007f1d6fcc6000,0x00007f1d6fcff000)]
>>
>> 0x00007f1da6794800 JavaThread "ActiveMQ BrokerService[MQMaster]
>> Task-1781" daemon [_thread_blocked, id=8302,
>> stack(0x00007f1db5a51000,0x00007f1db5a8a000)]
>> 0x00007f1da6791000 JavaThread "ActiveMQ BrokerService[MQMaster]
>> Task-1780" daemon [_thread_blocked, id=8298,
>> stack(0x00007f1dc6d4c000,0x00007f1dc6d85000)]
>> 0x00007f1da6793000 JavaThread "ActiveMQ BrokerService[MQMaster]
>> Task-1779" daemon [_thread_blocked, id=8296,
>> stack(0x00007f1dc6e30000,0x00007f1dc6e69000)]
>> 0x00007f1da6790000 JavaThread "ActiveMQ BrokerService[MQMaster]
>> Task-1778" daemon [_thread_blocked, id=8293,
>> stack(0x00007f1db6903000,0x00007f1db693c000)]
>> 0x00007f1da678f000 JavaThread "ActiveMQ BrokerService[MQMaster]
>> Task-1777" daemon [_thread_blocked, id=8292,
>> stack(0x00007f1db9dc8000,0x00007f1db9e01000)]
>>
>> Please, can someone provide inputs on why this is happening.
>>
>> Thanks
>> nl
>>
>>
>>
>>
>>
>
Re: ActiveMQ JVM crashes when high number of concurrent connections
get terminated
Posted by Netlancer <ne...@mobigraph.co>.
Hi,
yes. we increased the heap size to 8G in a machine which has 16GB
RAM. the limits.conf is configured with 300000 as soft/hard limit and
file-max is the same as hard limit. I have attached the hs_err file .
Thanks,
On 02.06.2014 08.52, Gaurav Sharma wrote:
> If possible, pls provide some more info to help diagnose this. What's the output of /etc/security/limits.conf and file-max value in /etc/sysctl.conf?
>
> Is 5G the max you can allocate to amq's jvm?
>
> Sent from my iPhone
>
> On Jun 1, 2014, at 20:06, Netlancer <ne...@mobigraph.co> wrote:
>
> Hi,
>
> We have activemq running behind an LB. we have close to 55K connections to activemq. during cases of more than 1000 connections gets terminated concurrently, we observe a huge spike in the number of threads created.following which the jvm crashes. on further investigation, these threads are pertained to executors
>
> SelectorManager - defaultExecutor - labelled with "ActiveMQ NIO Worker" - the executor has maximum thread pool size of Integer.MAX_VALUE
> TaskRunnerFactory - defaultExecutor - labelled as "ActiveMQ Task" - this executor as well has Integer.MAX_VALUE as the max pool size.
> BrokerService.getTaskRunnerFactory - labelled as "ActiveMQ BrokerService[%s] Task-" - this executor is also created with Integer.MAX_VALUE
>
> the configured persistence store is leveldb and dedicatedtaskrunner is disabled. The JVM is running with 8GB RAM with the max heap at 5G.
>
> following snippet from the hs_pid crash file
>
> Java Threads: ( => current thread )
> 0x00007f1e42863800 JavaThread "ActiveMQ NIO Worker 9053" daemon [_thread_blocked, id=10690, stack(0x00007f1d6fa53000,0x00007f1d6fa8c000)]
> 0x00007f1ea40dc000 JavaThread "ActiveMQ NIO Worker 9047" daemon [_thread_blocked, id=10689, stack(0x00007f1d6fa8c000,0x00007f1d6fac5000)]
> 0x00007f1eac116800 JavaThread "ActiveMQ NIO Worker 9054" daemon [_thread_blocked, id=10688, stack(0x00007f1d6fac5000,0x00007f1d6fafe000)]
> 0x00007f1df43f5800 JavaThread "ActiveMQ NIO Worker 9046" daemon [_thread_blocked, id=10687, stack(0x00007f1d6fafe000,0x00007f1d6fb37000)]
> 0x00007f1e42870000 JavaThread "ActiveMQ NIO Worker 9056" daemon [_thread_blocked, id=10686, stack(0x00007f1d6fb37000,0x00007f1d6fb70000)]
> 0x00007f1ef6054800 JavaThread "ActiveMQ NIO Worker 9107" daemon [_thread_blocked, id=10685, stack(0x00007f1d6fb70000,0x00007f1d6fba9000)]
> 0x00007f1ef46af800 JavaThread "ActiveMQ NIO Worker 9059" daemon [_thread_blocked, id=10684, stack(0x00007f1d6fba9000,0x00007f1d6fbe2000)]
> 0x00007f1e240cd000 JavaThread "ActiveMQ NIO Worker 9041" daemon [_thread_blocked, id=10683, stack(0x00007f1d6fbe2000,0x00007f1d6fc1b000)]
> 0x00007f1e0016a800 JavaThread "ActiveMQ NIO Worker 9060" daemon [_thread_blocked, id=10682, stack(0x00007f1d6fc1b000,0x00007f1d6fc54000)]
> 0x00007f1e5019d800 JavaThread "ActiveMQ NIO Worker 9061" daemon [_thread_blocked, id=10681, stack(0x00007f1d6fc54000,0x00007f1d6fc8d000)]
> 0x00007f1e8006f000 JavaThread "ActiveMQ NIO Worker 9039" daemon [_thread_blocked, id=10680, stack(0x00007f1d6fc8d000,0x00007f1d6fcc6000)]
> 0x00007f1e4c7df800 JavaThread "ActiveMQ NIO Worker 9101" daemon [_thread_blocked, id=10679, stack(0x00007f1d6fcc6000,0x00007f1d6fcff000)]
>
> 0x00007f1da6794800 JavaThread "ActiveMQ BrokerService[MQMaster] Task-1781" daemon [_thread_blocked, id=8302, stack(0x00007f1db5a51000,0x00007f1db5a8a000)]
> 0x00007f1da6791000 JavaThread "ActiveMQ BrokerService[MQMaster] Task-1780" daemon [_thread_blocked, id=8298, stack(0x00007f1dc6d4c000,0x00007f1dc6d85000)]
> 0x00007f1da6793000 JavaThread "ActiveMQ BrokerService[MQMaster] Task-1779" daemon [_thread_blocked, id=8296, stack(0x00007f1dc6e30000,0x00007f1dc6e69000)]
> 0x00007f1da6790000 JavaThread "ActiveMQ BrokerService[MQMaster] Task-1778" daemon [_thread_blocked, id=8293, stack(0x00007f1db6903000,0x00007f1db693c000)]
> 0x00007f1da678f000 JavaThread "ActiveMQ BrokerService[MQMaster] Task-1777" daemon [_thread_blocked, id=8292, stack(0x00007f1db9dc8000,0x00007f1db9e01000)]
>
> Please, can someone provide inputs on why this is happening.
>
> Thanks
> nl
>
>
>
>
>
Re: ActiveMQ JVM crashes when high number of concurrent connections get terminated
Posted by Gaurav Sharma <ga...@gmail.com>.
If possible, pls provide some more info to help diagnose this. What's the output of /etc/security/limits.conf and file-max value in /etc/sysctl.conf?
Is 5G the max you can allocate to amq's jvm?
Sent from my iPhone
On Jun 1, 2014, at 20:06, Netlancer <ne...@mobigraph.co> wrote:
Hi,
We have activemq running behind an LB. we have close to 55K connections to activemq. during cases of more than 1000 connections gets terminated concurrently, we observe a huge spike in the number of threads created.following which the jvm crashes. on further investigation, these threads are pertained to executors
SelectorManager - defaultExecutor - labelled with "ActiveMQ NIO Worker" - the executor has maximum thread pool size of Integer.MAX_VALUE
TaskRunnerFactory - defaultExecutor - labelled as "ActiveMQ Task" - this executor as well has Integer.MAX_VALUE as the max pool size.
BrokerService.getTaskRunnerFactory - labelled as "ActiveMQ BrokerService[%s] Task-" - this executor is also created with Integer.MAX_VALUE
the configured persistence store is leveldb and dedicatedtaskrunner is disabled. The JVM is running with 8GB RAM with the max heap at 5G.
following snippet from the hs_pid crash file
Java Threads: ( => current thread )
0x00007f1e42863800 JavaThread "ActiveMQ NIO Worker 9053" daemon [_thread_blocked, id=10690, stack(0x00007f1d6fa53000,0x00007f1d6fa8c000)]
0x00007f1ea40dc000 JavaThread "ActiveMQ NIO Worker 9047" daemon [_thread_blocked, id=10689, stack(0x00007f1d6fa8c000,0x00007f1d6fac5000)]
0x00007f1eac116800 JavaThread "ActiveMQ NIO Worker 9054" daemon [_thread_blocked, id=10688, stack(0x00007f1d6fac5000,0x00007f1d6fafe000)]
0x00007f1df43f5800 JavaThread "ActiveMQ NIO Worker 9046" daemon [_thread_blocked, id=10687, stack(0x00007f1d6fafe000,0x00007f1d6fb37000)]
0x00007f1e42870000 JavaThread "ActiveMQ NIO Worker 9056" daemon [_thread_blocked, id=10686, stack(0x00007f1d6fb37000,0x00007f1d6fb70000)]
0x00007f1ef6054800 JavaThread "ActiveMQ NIO Worker 9107" daemon [_thread_blocked, id=10685, stack(0x00007f1d6fb70000,0x00007f1d6fba9000)]
0x00007f1ef46af800 JavaThread "ActiveMQ NIO Worker 9059" daemon [_thread_blocked, id=10684, stack(0x00007f1d6fba9000,0x00007f1d6fbe2000)]
0x00007f1e240cd000 JavaThread "ActiveMQ NIO Worker 9041" daemon [_thread_blocked, id=10683, stack(0x00007f1d6fbe2000,0x00007f1d6fc1b000)]
0x00007f1e0016a800 JavaThread "ActiveMQ NIO Worker 9060" daemon [_thread_blocked, id=10682, stack(0x00007f1d6fc1b000,0x00007f1d6fc54000)]
0x00007f1e5019d800 JavaThread "ActiveMQ NIO Worker 9061" daemon [_thread_blocked, id=10681, stack(0x00007f1d6fc54000,0x00007f1d6fc8d000)]
0x00007f1e8006f000 JavaThread "ActiveMQ NIO Worker 9039" daemon [_thread_blocked, id=10680, stack(0x00007f1d6fc8d000,0x00007f1d6fcc6000)]
0x00007f1e4c7df800 JavaThread "ActiveMQ NIO Worker 9101" daemon [_thread_blocked, id=10679, stack(0x00007f1d6fcc6000,0x00007f1d6fcff000)]
0x00007f1da6794800 JavaThread "ActiveMQ BrokerService[MQMaster] Task-1781" daemon [_thread_blocked, id=8302, stack(0x00007f1db5a51000,0x00007f1db5a8a000)]
0x00007f1da6791000 JavaThread "ActiveMQ BrokerService[MQMaster] Task-1780" daemon [_thread_blocked, id=8298, stack(0x00007f1dc6d4c000,0x00007f1dc6d85000)]
0x00007f1da6793000 JavaThread "ActiveMQ BrokerService[MQMaster] Task-1779" daemon [_thread_blocked, id=8296, stack(0x00007f1dc6e30000,0x00007f1dc6e69000)]
0x00007f1da6790000 JavaThread "ActiveMQ BrokerService[MQMaster] Task-1778" daemon [_thread_blocked, id=8293, stack(0x00007f1db6903000,0x00007f1db693c000)]
0x00007f1da678f000 JavaThread "ActiveMQ BrokerService[MQMaster] Task-1777" daemon [_thread_blocked, id=8292, stack(0x00007f1db9dc8000,0x00007f1db9e01000)]
Please, can someone provide inputs on why this is happening.
Thanks
nl