You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@activemq.apache.org by David Delbecq <de...@oma.be> on 2010/01/15 10:24:25 UTC
why is activeMQ stalled?
Hello,
can someone explain me why my activeMQ client is stalled in such
situation (i have 10 threads blocked like this having as consequence a
complete blocking of my client application):
"http-8081-Processor6" daemon prio=10 tid=00841c38 nid=512
lwp_id=2426611 waiting on condition [236f3000..236f153c]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <40ae1828> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1889)
at
java.util.concurrent.ArrayBlockingQueue.take(ArrayBlockingQueue.java:317)
at
org.apache.activemq.transport.FutureResponse.getResult(FutureResponse.java:40)
at
org.apache.activemq.transport.ResponseCorrelator.request(ResponseCorrelator.java:80)
at
org.apache.activemq.ActiveMQConnection.syncSendPacket(ActiveMQConnection.java:1233)
at
org.apache.activemq.ActiveMQSession.send(ActiveMQSession.java:1659)
- locked <40ae01e8> (a java.lang.Object)
at
org.apache.activemq.ActiveMQMessageProducer.send(ActiveMQMessageProducer.java:227)
at
org.apache.activemq.ActiveMQMessageProducerSupport.send(ActiveMQMessageProducerSupport.java:241)
at be.rmi.shark.JMSEventManager.sendEvent(JMSEventManager.java:146)
considering that the activeMQ lists size, as referenced by activeMQ
admin interface are as follow:
Name Number Of Pending Messages Number Of Consumers Messages
Sent Messages Received Views Operations
WF.Regraph 0 1 1714 1720 Browse Send To Purge Delete
WF.Notify 0 1 1696 1742 Browse Send To Purge Delete
WF.IncomingEvents 0 0 0 0 Browse Send To Purge Delete
Is this to be considered normal that i must restart activeMQ producers
on a regular basis because of those this stalling? Where can i find
clear documentation on proper configuration of activeMQ to prevent this
stalling? Thank you.
--
David Delbecq
ICT
Institut Royal Météorologique
Ext:557
Re: why is activeMQ stalled?
Posted by David Delbecq <de...@oma.be>.
Le 15/01/10 11:50, Rob Davies a écrit :
> David - can you upgrade 5.3 ?
Not immediatly, but worth a try
>
> On 15 Jan 2010, at 10:40, David Delbecq wrote:
>
>> Le 15/01/10 11:25, Gary Tully a écrit :
>>> A thread dump of the broker jvm, when the clients are stalled, would
>>> help
>>> diagnose what is going on. btw, what activemq version are you using?
>>>
>> BrokerService - ActiveMQ 5.2.0 JMS Message Broker
>> (localhost) is starting
>>
>> I'll come back with broker dump when there is a problem again. This can
>> be in several hours or within a few weeks. However i'll try to force a
>> stalling next week...
>>
>>
>>> 2010/1/15 David Delbecq <de...@oma.be>
>>>
>>>
>>>> Le 15/01/10 10:24, Dejan Bosanac a écrit :
>>>>
>>>>> Hi David,
>>>>>
>>>>> you're probably blocked by "producer flow control". Take a look here
>>>>> for more info: http://activemq.apache.org/producer-flow-control.html
>>>>>
>>>>> Cheers
>>>>>
>>>> I already took a look at it, read it three time to be sure i
>>>> understand.
>>>> It's quite technical but beyond that, i don't think this apply for me.
>>>> My queues are all empty!!! (they stalled when there were 50 messages
>>>> pending but where finally emptied), connections are not shared by
>>>> several queues and am not sure how to implement the cursor part of
>>>> documentation. There is no "slow consumer" that could influence it at
>>>> that right moment (customers are running an waiting for new
>>>> messages as
>>>> you can see in capture). Moreover my messages are of about 1k payload
>>>> (about 10 small text entries in a MapMessage), and my activeMQ
>>>> config is
>>>> like this
>>>> <policyEntry queue=">" memoryLimit="500mb"/>
>>>> <policyEntry topic=">" memoryLimit="500mb">
>>>> ....
>>>> <systemUsage>
>>>> <systemUsage>
>>>> <memoryUsage>
>>>> <memoryUsage limit="20 mb"/>
>>>> </memoryUsage>
>>>> <storeUsage>
>>>> <storeUsage limit="1 gb" name="foo"/>
>>>> </storeUsage>
>>>> <tempUsage>
>>>> <tempUsage limit="100 mb"/>
>>>> </tempUsage>
>>>> </systemUsage>
>>>> </systemUsage>
>>>>
>>>>
>>>> There is no way a bunch of 10 pending messages could fill a 1gb store,
>>>> isn't it? What config part of activemq do i need to change to ensure
>>>> that it can store at least about 10000 persistent messages (that makes
>>>> only 10Mb of datas!!) before failing? Currently it fails and is never
>>>> recovering without a restart of broker when there are only 50 small
>>>> pending messages in the queue. As a workaround i have set
>>>> "sendFailIfNoSpace" to true, so my client will fail instead of
>>>> stalling,
>>>> i'll see when this happen more quickly like that....
>>>>
>>>>
>>>> Here is broker part of my config file, can someone help me fix this?
>>>> Thank you:
>>>>
>>>>
>>>> <broker xmlns="http://activemq.apache.org/schema/core"
>>>> brokerName="localhost" dataDirectory="${activemq.base}/data"
>>>> persistent="true">
>>>> <plugins>
>>>> <simpleAuthenticationPlugin>
>>>> <users>
>>>> <authenticationUser username="admin" password="******"
>>>> groups="users,admins"/>
>>>> <authenticationUser username="sharkTool"
>>>> password="********"
>>>> groups="serverTools"/>
>>>> <authenticationUser username="adminConsole"
>>>> password="*********"
>>>> groups="serverTools"/>
>>>> </users>
>>>> </simpleAuthenticationPlugin>
>>>> <authorizationPlugin>
>>>> <map>
>>>> <authorizationMap>
>>>> <authorizationEntries>
>>>> <authorizationEntry queue=">" read="admins" write="admins"
>>>> admin="admins" />
>>>> <authorizationEntry queue="WF.>" read="users,serverTools"
>>>> write="serverTools" admin="admins,serverTools" />
>>>>
>>>> <authorizationEntry topic=">" read="admins" write="admins"
>>>> admin="admins" />
>>>> <authorizationEntry topic="WF.>" read="users,serverTools"
>>>> write="serverTools" admin="admins,serverTools" />
>>>>
>>>> <authorizationEntry topic="ActiveMQ.Advisory.>"
>>>> read="guests,users,serverTools" write="guests,users,serverTools"
>>>> admin="guests,users,serverTools"/>
>>>> </authorizationEntries>
>>>>
>>>> <!-- let's assign roles to temporary destinations. comment
>>>> this entry if we don't want any roles assigned to temp destinations
>>>> <tempDestinationAuthorizationEntry>
>>>> <tempDestinationAuthorizationEntry
>>>> read="tempDestinationAdmins" write="tempDestinationAdmins"
>>>> admin="tempDestinationAdmins"/>
>>>> </tempDestinationAuthorizationEntry> -->
>>>> </authorizationMap>
>>>> </map>
>>>> </authorizationPlugin>
>>>>
>>>>
>>>> </plugins>
>>>> <!-- Destination specific policies using destination names or
>>>> wildcards -->
>>>> <destinationPolicy>
>>>> <policyMap>
>>>> <policyEntries>
>>>> <policyEntry queue=">" memoryLimit="500mb"/>
>>>> <policyEntry topic=">" memoryLimit="500mb">
>>>> <!-- you can add other policies too such as these
>>>> <dispatchPolicy>
>>>> <strictOrderDispatchPolicy/>
>>>> </dispatchPolicy>
>>>> <subscriptionRecoveryPolicy>
>>>> <lastImageSubscriptionRecoveryPolicy/>
>>>> </subscriptionRecoveryPolicy>
>>>> -->
>>>> </policyEntry>
>>>> </policyEntries>
>>>> </policyMap>
>>>> </destinationPolicy>
>>>>
>>>> <!-- Use the following to configure how ActiveMQ is exposed in
>>>> JMX -->
>>>> <managementContext>
>>>> <managementContext createConnector="false"/>
>>>> </managementContext>
>>>>
>>>> <!-- The store and forward broker networks ActiveMQ will listen
>>>> to -->
>>>> <networkConnectors>
>>>> <!-- by default just auto discover the other brokers -->
>>>> <!-- <networkConnector name="default-nc"
>>>> uri="multicast://default"/>-->
>>>> <!-- Example of a static configuration:
>>>> <networkConnector name="host1 and host2"
>>>> uri="static://(tcp://host1:61616,tcp://host2:61616)"/>
>>>> -->
>>>> </networkConnectors>
>>>>
>>>> <persistenceAdapter>
>>>> <amqPersistenceAdapter syncOnWrite="false"
>>>> directory="${activemq.base}/data" maxFileLength="20 mb"/>
>>>> </persistenceAdapter>
>>>>
>>>>
>>>>
>>>> <sslContext>
>>>> <sslContext keyStore="file:${activemq.base}/conf/broker.ks"
>>>> keyStorePassword="password"
>>>> trustStore="file:${activemq.base}/conf/broker.ts"
>>>> trustStorePassword="password"/>
>>>> </sslContext>
>>>>
>>>> <!-- The maximum about of space the broker will use before
>>>> slowing down producers -->
>>>> <systemUsage>
>>>> <systemUsage>
>>>> <memoryUsage>
>>>> <memoryUsage limit="20 mb"/>
>>>> </memoryUsage>
>>>> <storeUsage>
>>>> <storeUsage limit="1 gb" name="foo"/>
>>>> </storeUsage>
>>>> <tempUsage>
>>>> <tempUsage limit="100 mb"/>
>>>> </tempUsage>
>>>> </systemUsage>
>>>> </systemUsage>
>>>>
>>>>
>>>> <!-- The transport connectors ActiveMQ will listen to -->
>>>> <transportConnectors>
>>>> <transportConnector name="openwire"
>>>> uri="tcp://localhost:61616"/>
>>>> <!-- <transportConnector name="ssl"
>>>> uri="ssl://localhost:61617"/>
>>>> <transportConnector name="stomp"
>>>> uri="stomp://localhost:61613"/>
>>>> <transportConnector name="xmpp"
>>>> uri="xmpp://localhost:61222"/> -->
>>>> </transportConnectors>
>>>> <destinations>
>>>> <queue physicalName="WF.Regraph" />
>>>> <queue physicalName="WF.Notify"/>
>>>> <topic physicalName="WF.Events"/>
>>>> </destinations>
>>>> <destinationInterceptors>
>>>> <virtualDestinationInterceptor>
>>>> <virtualDestinations>
>>>> <compositeQueue name="WF.IncomingEvents">
>>>> <forwardTo>
>>>> <queue physicalName="WF.Regraph"/>
>>>> <queue physicalName="WF.Notify"/>
>>>> <topic physicalName="WF.Events" />
>>>> </forwardTo>
>>>> </compositeQueue>
>>>> </virtualDestinations>
>>>> </virtualDestinationInterceptor>
>>>> </destinationInterceptors>
>>>>
>>>>
>>>> </broker>
>>>>
>>>>
>>>>
>>>>> --
>>>>>
>>>>>
>>>>> On Fri, Jan 15, 2010 at 10:24 AM, David Delbecq <delbd@oma.be
>>>>> <ma...@oma.be>> wrote:
>>>>>
>>>>> Hello,
>>>>>
>>>>> can someone explain me why my activeMQ client is stalled in such
>>>>> situation (i have 10 threads blocked like this having as
>>>>> consequence a complete blocking of my client application):
>>>>>
>>>>> "http-8081-Processor6" daemon prio=10 tid=00841c38 nid=512
>>>>> lwp_id=2426611 waiting on condition [236f3000..236f153c]
>>>>> java.lang.Thread.State: WAITING (parking)
>>>>> at sun.misc.Unsafe.park(Native Method)
>>>>> - parking to wait for <40ae1828> (a
>>>>>
>>>>>
>>>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>>>>
>>>>> at
>>>>> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
>>>>> at
>>>>>
>>>>>
>>>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1889)
>>>>
>>>>
>>>>> at
>>>>>
>>>>>
>>>> java.util.concurrent.ArrayBlockingQueue.take(ArrayBlockingQueue.java:317)
>>>>
>>>>
>>>>> at
>>>>>
>>>>>
>>>> org.apache.activemq.transport.FutureResponse.getResult(FutureResponse.java:40)
>>>>
>>>>
>>>>> at
>>>>>
>>>>>
>>>> org.apache.activemq.transport.ResponseCorrelator.request(ResponseCorrelator.java:80)
>>>>
>>>>
>>>>> at
>>>>>
>>>>>
>>>> org.apache.activemq.ActiveMQConnection.syncSendPacket(ActiveMQConnection.java:1233)
>>>>
>>>>
>>>>> at
>>>>>
>>>>> org.apache.activemq.ActiveMQSession.send(ActiveMQSession.java:1659)
>>>>> - locked <40ae01e8> (a java.lang.Object)
>>>>> at
>>>>>
>>>>>
>>>> org.apache.activemq.ActiveMQMessageProducer.send(ActiveMQMessageProducer.java:227)
>>>>
>>>>
>>>>> at
>>>>>
>>>>>
>>>> org.apache.activemq.ActiveMQMessageProducerSupport.send(ActiveMQMessageProducerSupport.java:241)
>>>>
>>>>
>>>>> at
>>>>> be.rmi.shark.JMSEventManager.sendEvent(JMSEventManager.java:146)
>>>>>
>>>>>
>>>>> considering that the activeMQ lists size, as referenced by
>>>>> activeMQ admin interface are as follow:
>>>>> Name Number Of Pending Messages Number Of Consumers
>>>>> Messages Sent Messages Received Views
>>>>> Operations
>>>>> WF.Regraph 0 1 1714 1720 Browse Send To
>>>>>
>>>> Purge Delete
>>>>
>>>>> WF.Notify 0 1 1696 1742 Browse Send To
>>>>>
>>>> Purge Delete
>>>>
>>>>> WF.IncomingEvents 0 0 0 0 Browse
>>>>>
>>>> Send To Purge Delete
>>>>
>>>>> Is this to be considered normal that i must restart activeMQ
>>>>> producers on a regular basis because of those this stalling? Where
>>>>> can i find clear documentation on proper configuration of activeMQ
>>>>> to prevent this stalling? Thank you.
>>>>>
>>>>> --
>>>>> David Delbecq
>>>>> ICT
>>>>> Institut Royal Météorologique
>>>>> Ext:557
>>>>>
>>>>>
>>>>>
>>>>>
>>>> --
>>>> David Delbecq
>>>> ICT
>>>> Institut Royal Météorologique
>>>> Ext:557
>>>>
>>>>
>>>>
>>>
>>>
>>>
>>
>>
>> --
>> David Delbecq
>> ICT
>> Institut Royal Météorologique
>> Ext:557
>>
>
> Rob Davies
> http://twitter.com/rajdavies
> I work here: http://fusesource.com
> My Blog: http://rajdavies.blogspot.com/
> I'm writing this: http://www.manning.com/snyder/
>
>
>
>
>
--
David Delbecq
ICT
Institut Royal Météorologique
Ext:557
Re: why is activeMQ stalled?
Posted by Rob Davies <ra...@gmail.com>.
David - can you upgrade 5.3 ?
On 15 Jan 2010, at 10:40, David Delbecq wrote:
> Le 15/01/10 11:25, Gary Tully a écrit :
>> A thread dump of the broker jvm, when the clients are stalled,
>> would help
>> diagnose what is going on. btw, what activemq version are you using?
>>
> BrokerService - ActiveMQ 5.2.0 JMS Message Broker
> (localhost) is starting
>
> I'll come back with broker dump when there is a problem again. This
> can
> be in several hours or within a few weeks. However i'll try to force a
> stalling next week...
>
>
>> 2010/1/15 David Delbecq <de...@oma.be>
>>
>>
>>> Le 15/01/10 10:24, Dejan Bosanac a écrit :
>>>
>>>> Hi David,
>>>>
>>>> you're probably blocked by "producer flow control". Take a look
>>>> here
>>>> for more info: http://activemq.apache.org/producer-flow-
>>>> control.html
>>>>
>>>> Cheers
>>>>
>>> I already took a look at it, read it three time to be sure i
>>> understand.
>>> It's quite technical but beyond that, i don't think this apply for
>>> me.
>>> My queues are all empty!!! (they stalled when there were 50 messages
>>> pending but where finally emptied), connections are not shared by
>>> several queues and am not sure how to implement the cursor part of
>>> documentation. There is no "slow consumer" that could influence it
>>> at
>>> that right moment (customers are running an waiting for new
>>> messages as
>>> you can see in capture). Moreover my messages are of about 1k
>>> payload
>>> (about 10 small text entries in a MapMessage), and my activeMQ
>>> config is
>>> like this
>>> <policyEntry queue=">" memoryLimit="500mb"/>
>>> <policyEntry topic=">" memoryLimit="500mb">
>>> ....
>>> <systemUsage>
>>> <systemUsage>
>>> <memoryUsage>
>>> <memoryUsage limit="20 mb"/>
>>> </memoryUsage>
>>> <storeUsage>
>>> <storeUsage limit="1 gb" name="foo"/>
>>> </storeUsage>
>>> <tempUsage>
>>> <tempUsage limit="100 mb"/>
>>> </tempUsage>
>>> </systemUsage>
>>> </systemUsage>
>>>
>>>
>>> There is no way a bunch of 10 pending messages could fill a 1gb
>>> store,
>>> isn't it? What config part of activemq do i need to change to ensure
>>> that it can store at least about 10000 persistent messages (that
>>> makes
>>> only 10Mb of datas!!) before failing? Currently it fails and is
>>> never
>>> recovering without a restart of broker when there are only 50 small
>>> pending messages in the queue. As a workaround i have set
>>> "sendFailIfNoSpace" to true, so my client will fail instead of
>>> stalling,
>>> i'll see when this happen more quickly like that....
>>>
>>>
>>> Here is broker part of my config file, can someone help me fix this?
>>> Thank you:
>>>
>>>
>>> <broker xmlns="http://activemq.apache.org/schema/core"
>>> brokerName="localhost" dataDirectory="${activemq.base}/data"
>>> persistent="true">
>>> <plugins>
>>> <simpleAuthenticationPlugin>
>>> <users>
>>> <authenticationUser username="admin" password="******"
>>> groups="users,admins"/>
>>> <authenticationUser username="sharkTool"
>>> password="********"
>>> groups="serverTools"/>
>>> <authenticationUser username="adminConsole"
>>> password="*********"
>>> groups="serverTools"/>
>>> </users>
>>> </simpleAuthenticationPlugin>
>>> <authorizationPlugin>
>>> <map>
>>> <authorizationMap>
>>> <authorizationEntries>
>>> <authorizationEntry queue=">" read="admins"
>>> write="admins"
>>> admin="admins" />
>>> <authorizationEntry queue="WF.>"
>>> read="users,serverTools"
>>> write="serverTools" admin="admins,serverTools" />
>>>
>>> <authorizationEntry topic=">" read="admins"
>>> write="admins"
>>> admin="admins" />
>>> <authorizationEntry topic="WF.>"
>>> read="users,serverTools"
>>> write="serverTools" admin="admins,serverTools" />
>>>
>>> <authorizationEntry topic="ActiveMQ.Advisory.>"
>>> read="guests,users,serverTools" write="guests,users,serverTools"
>>> admin="guests,users,serverTools"/>
>>> </authorizationEntries>
>>>
>>> <!-- let's assign roles to temporary destinations. comment
>>> this entry if we don't want any roles assigned to temp destinations
>>> <tempDestinationAuthorizationEntry>
>>> <tempDestinationAuthorizationEntry
>>> read="tempDestinationAdmins" write="tempDestinationAdmins"
>>> admin="tempDestinationAdmins"/>
>>> </tempDestinationAuthorizationEntry> -->
>>> </authorizationMap>
>>> </map>
>>> </authorizationPlugin>
>>>
>>>
>>> </plugins>
>>> <!-- Destination specific policies using destination names or
>>> wildcards -->
>>> <destinationPolicy>
>>> <policyMap>
>>> <policyEntries>
>>> <policyEntry queue=">" memoryLimit="500mb"/>
>>> <policyEntry topic=">" memoryLimit="500mb">
>>> <!-- you can add other policies too such as
>>> these
>>> <dispatchPolicy>
>>> <strictOrderDispatchPolicy/>
>>> </dispatchPolicy>
>>> <subscriptionRecoveryPolicy>
>>> <lastImageSubscriptionRecoveryPolicy/>
>>> </subscriptionRecoveryPolicy>
>>> -->
>>> </policyEntry>
>>> </policyEntries>
>>> </policyMap>
>>> </destinationPolicy>
>>>
>>> <!-- Use the following to configure how ActiveMQ is exposed in
>>> JMX -->
>>> <managementContext>
>>> <managementContext createConnector="false"/>
>>> </managementContext>
>>>
>>> <!-- The store and forward broker networks ActiveMQ will
>>> listen
>>> to -->
>>> <networkConnectors>
>>> <!-- by default just auto discover the other brokers -->
>>> <!-- <networkConnector name="default-nc"
>>> uri="multicast://default"/>-->
>>> <!-- Example of a static configuration:
>>> <networkConnector name="host1 and host2"
>>> uri="static://(tcp://host1:61616,tcp://host2:61616)"/>
>>> -->
>>> </networkConnectors>
>>>
>>> <persistenceAdapter>
>>> <amqPersistenceAdapter syncOnWrite="false"
>>> directory="${activemq.base}/data" maxFileLength="20 mb"/>
>>> </persistenceAdapter>
>>>
>>>
>>>
>>> <sslContext>
>>> <sslContext keyStore="file:${activemq.base}/conf/
>>> broker.ks"
>>> keyStorePassword="password"
>>> trustStore="file:${activemq.base}/conf/broker.ts"
>>> trustStorePassword="password"/>
>>> </sslContext>
>>>
>>> <!-- The maximum about of space the broker will use before
>>> slowing down producers -->
>>> <systemUsage>
>>> <systemUsage>
>>> <memoryUsage>
>>> <memoryUsage limit="20 mb"/>
>>> </memoryUsage>
>>> <storeUsage>
>>> <storeUsage limit="1 gb" name="foo"/>
>>> </storeUsage>
>>> <tempUsage>
>>> <tempUsage limit="100 mb"/>
>>> </tempUsage>
>>> </systemUsage>
>>> </systemUsage>
>>>
>>>
>>> <!-- The transport connectors ActiveMQ will listen to -->
>>> <transportConnectors>
>>> <transportConnector name="openwire"
>>> uri="tcp://localhost:61616"/>
>>> <!-- <transportConnector name="ssl"
>>> uri="ssl://localhost:61617"/>
>>> <transportConnector name="stomp" uri="stomp://localhost:
>>> 61613"/>
>>> <transportConnector name="xmpp"
>>> uri="xmpp://localhost:61222"/> -->
>>> </transportConnectors>
>>> <destinations>
>>> <queue physicalName="WF.Regraph" />
>>> <queue physicalName="WF.Notify"/>
>>> <topic physicalName="WF.Events"/>
>>> </destinations>
>>> <destinationInterceptors>
>>> <virtualDestinationInterceptor>
>>> <virtualDestinations>
>>> <compositeQueue name="WF.IncomingEvents">
>>> <forwardTo>
>>> <queue physicalName="WF.Regraph"/>
>>> <queue physicalName="WF.Notify"/>
>>> <topic physicalName="WF.Events" />
>>> </forwardTo>
>>> </compositeQueue>
>>> </virtualDestinations>
>>> </virtualDestinationInterceptor>
>>> </destinationInterceptors>
>>>
>>>
>>> </broker>
>>>
>>>
>>>
>>>> --
>>>>
>>>>
>>>> On Fri, Jan 15, 2010 at 10:24 AM, David Delbecq <delbd@oma.be
>>>> <ma...@oma.be>> wrote:
>>>>
>>>> Hello,
>>>>
>>>> can someone explain me why my activeMQ client is stalled in such
>>>> situation (i have 10 threads blocked like this having as
>>>> consequence a complete blocking of my client application):
>>>>
>>>> "http-8081-Processor6" daemon prio=10 tid=00841c38 nid=512
>>>> lwp_id=2426611 waiting on condition [236f3000..236f153c]
>>>> java.lang.Thread.State: WAITING (parking)
>>>> at sun.misc.Unsafe.park(Native Method)
>>>> - parking to wait for <40ae1828> (a
>>>>
>>>>
>>> java.util.concurrent.locks.AbstractQueuedSynchronizer
>>> $ConditionObject)
>>>
>>>> at
>>>> java.util.concurrent.locks.LockSupport.park(LockSupport.java:
>>>> 158)
>>>> at
>>>>
>>>>
>>> java.util.concurrent.locks.AbstractQueuedSynchronizer
>>> $ConditionObject.await(AbstractQueuedSynchronizer.java:1889)
>>>
>>>> at
>>>>
>>>>
>>> java
>>> .util.concurrent.ArrayBlockingQueue.take(ArrayBlockingQueue.java:
>>> 317)
>>>
>>>> at
>>>>
>>>>
>>> org
>>> .apache
>>> .activemq.transport.FutureResponse.getResult(FutureResponse.java:40)
>>>
>>>> at
>>>>
>>>>
>>> org
>>> .apache
>>> .activemq
>>> .transport.ResponseCorrelator.request(ResponseCorrelator.java:80)
>>>
>>>> at
>>>>
>>>>
>>> org
>>> .apache
>>> .activemq
>>> .ActiveMQConnection.syncSendPacket(ActiveMQConnection.java:1233)
>>>
>>>> at
>>>> org.apache.activemq.ActiveMQSession.send(ActiveMQSession.java:
>>>> 1659)
>>>> - locked <40ae01e8> (a java.lang.Object)
>>>> at
>>>>
>>>>
>>> org
>>> .apache
>>> .activemq
>>> .ActiveMQMessageProducer.send(ActiveMQMessageProducer.java:227)
>>>
>>>> at
>>>>
>>>>
>>> org
>>> .apache
>>> .activemq
>>> .ActiveMQMessageProducerSupport
>>> .send(ActiveMQMessageProducerSupport.java:241)
>>>
>>>> at
>>>> be.rmi.shark.JMSEventManager.sendEvent(JMSEventManager.java:146)
>>>>
>>>>
>>>> considering that the activeMQ lists size, as referenced by
>>>> activeMQ admin interface are as follow:
>>>> Name Number Of Pending Messages Number Of Consumers
>>>> Messages Sent Messages Received Views
>>>> Operations
>>>> WF.Regraph 0 1 1714 1720 Browse Send
>>>> To
>>>>
>>> Purge Delete
>>>
>>>> WF.Notify 0 1 1696 1742 Browse Send
>>>> To
>>>>
>>> Purge Delete
>>>
>>>> WF.IncomingEvents 0 0 0 0 Browse
>>>>
>>> Send To Purge Delete
>>>
>>>> Is this to be considered normal that i must restart activeMQ
>>>> producers on a regular basis because of those this stalling?
>>>> Where
>>>> can i find clear documentation on proper configuration of
>>>> activeMQ
>>>> to prevent this stalling? Thank you.
>>>>
>>>> --
>>>> David Delbecq
>>>> ICT
>>>> Institut Royal Météorologique
>>>> Ext:557
>>>>
>>>>
>>>>
>>>>
>>> --
>>> David Delbecq
>>> ICT
>>> Institut Royal Météorologique
>>> Ext:557
>>>
>>>
>>>
>>
>>
>>
>
>
> --
> David Delbecq
> ICT
> Institut Royal Météorologique
> Ext:557
>
Rob Davies
http://twitter.com/rajdavies
I work here: http://fusesource.com
My Blog: http://rajdavies.blogspot.com/
I'm writing this: http://www.manning.com/snyder/
Re: why is activeMQ stalled?
Posted by David Delbecq <de...@oma.be>.
Le 15/01/10 11:25, Gary Tully a écrit :
> A thread dump of the broker jvm, when the clients are stalled, would help
> diagnose what is going on. btw, what activemq version are you using?
>
BrokerService - ActiveMQ 5.2.0 JMS Message Broker
(localhost) is starting
I'll come back with broker dump when there is a problem again. This can
be in several hours or within a few weeks. However i'll try to force a
stalling next week...
> 2010/1/15 David Delbecq <de...@oma.be>
>
>
>> Le 15/01/10 10:24, Dejan Bosanac a écrit :
>>
>>> Hi David,
>>>
>>> you're probably blocked by "producer flow control". Take a look here
>>> for more info: http://activemq.apache.org/producer-flow-control.html
>>>
>>> Cheers
>>>
>> I already took a look at it, read it three time to be sure i understand.
>> It's quite technical but beyond that, i don't think this apply for me.
>> My queues are all empty!!! (they stalled when there were 50 messages
>> pending but where finally emptied), connections are not shared by
>> several queues and am not sure how to implement the cursor part of
>> documentation. There is no "slow consumer" that could influence it at
>> that right moment (customers are running an waiting for new messages as
>> you can see in capture). Moreover my messages are of about 1k payload
>> (about 10 small text entries in a MapMessage), and my activeMQ config is
>> like this
>> <policyEntry queue=">" memoryLimit="500mb"/>
>> <policyEntry topic=">" memoryLimit="500mb">
>> ....
>> <systemUsage>
>> <systemUsage>
>> <memoryUsage>
>> <memoryUsage limit="20 mb"/>
>> </memoryUsage>
>> <storeUsage>
>> <storeUsage limit="1 gb" name="foo"/>
>> </storeUsage>
>> <tempUsage>
>> <tempUsage limit="100 mb"/>
>> </tempUsage>
>> </systemUsage>
>> </systemUsage>
>>
>>
>> There is no way a bunch of 10 pending messages could fill a 1gb store,
>> isn't it? What config part of activemq do i need to change to ensure
>> that it can store at least about 10000 persistent messages (that makes
>> only 10Mb of datas!!) before failing? Currently it fails and is never
>> recovering without a restart of broker when there are only 50 small
>> pending messages in the queue. As a workaround i have set
>> "sendFailIfNoSpace" to true, so my client will fail instead of stalling,
>> i'll see when this happen more quickly like that....
>>
>>
>> Here is broker part of my config file, can someone help me fix this?
>> Thank you:
>>
>>
>> <broker xmlns="http://activemq.apache.org/schema/core"
>> brokerName="localhost" dataDirectory="${activemq.base}/data"
>> persistent="true">
>> <plugins>
>> <simpleAuthenticationPlugin>
>> <users>
>> <authenticationUser username="admin" password="******"
>> groups="users,admins"/>
>> <authenticationUser username="sharkTool" password="********"
>> groups="serverTools"/>
>> <authenticationUser username="adminConsole"
>> password="*********"
>> groups="serverTools"/>
>> </users>
>> </simpleAuthenticationPlugin>
>> <authorizationPlugin>
>> <map>
>> <authorizationMap>
>> <authorizationEntries>
>> <authorizationEntry queue=">" read="admins" write="admins"
>> admin="admins" />
>> <authorizationEntry queue="WF.>" read="users,serverTools"
>> write="serverTools" admin="admins,serverTools" />
>>
>> <authorizationEntry topic=">" read="admins" write="admins"
>> admin="admins" />
>> <authorizationEntry topic="WF.>" read="users,serverTools"
>> write="serverTools" admin="admins,serverTools" />
>>
>> <authorizationEntry topic="ActiveMQ.Advisory.>"
>> read="guests,users,serverTools" write="guests,users,serverTools"
>> admin="guests,users,serverTools"/>
>> </authorizationEntries>
>>
>> <!-- let's assign roles to temporary destinations. comment
>> this entry if we don't want any roles assigned to temp destinations
>> <tempDestinationAuthorizationEntry>
>> <tempDestinationAuthorizationEntry
>> read="tempDestinationAdmins" write="tempDestinationAdmins"
>> admin="tempDestinationAdmins"/>
>> </tempDestinationAuthorizationEntry> -->
>> </authorizationMap>
>> </map>
>> </authorizationPlugin>
>>
>>
>> </plugins>
>> <!-- Destination specific policies using destination names or
>> wildcards -->
>> <destinationPolicy>
>> <policyMap>
>> <policyEntries>
>> <policyEntry queue=">" memoryLimit="500mb"/>
>> <policyEntry topic=">" memoryLimit="500mb">
>> <!-- you can add other policies too such as these
>> <dispatchPolicy>
>> <strictOrderDispatchPolicy/>
>> </dispatchPolicy>
>> <subscriptionRecoveryPolicy>
>> <lastImageSubscriptionRecoveryPolicy/>
>> </subscriptionRecoveryPolicy>
>> -->
>> </policyEntry>
>> </policyEntries>
>> </policyMap>
>> </destinationPolicy>
>>
>> <!-- Use the following to configure how ActiveMQ is exposed in
>> JMX -->
>> <managementContext>
>> <managementContext createConnector="false"/>
>> </managementContext>
>>
>> <!-- The store and forward broker networks ActiveMQ will listen
>> to -->
>> <networkConnectors>
>> <!-- by default just auto discover the other brokers -->
>> <!-- <networkConnector name="default-nc"
>> uri="multicast://default"/>-->
>> <!-- Example of a static configuration:
>> <networkConnector name="host1 and host2"
>> uri="static://(tcp://host1:61616,tcp://host2:61616)"/>
>> -->
>> </networkConnectors>
>>
>> <persistenceAdapter>
>> <amqPersistenceAdapter syncOnWrite="false"
>> directory="${activemq.base}/data" maxFileLength="20 mb"/>
>> </persistenceAdapter>
>>
>>
>>
>> <sslContext>
>> <sslContext keyStore="file:${activemq.base}/conf/broker.ks"
>> keyStorePassword="password"
>> trustStore="file:${activemq.base}/conf/broker.ts"
>> trustStorePassword="password"/>
>> </sslContext>
>>
>> <!-- The maximum about of space the broker will use before
>> slowing down producers -->
>> <systemUsage>
>> <systemUsage>
>> <memoryUsage>
>> <memoryUsage limit="20 mb"/>
>> </memoryUsage>
>> <storeUsage>
>> <storeUsage limit="1 gb" name="foo"/>
>> </storeUsage>
>> <tempUsage>
>> <tempUsage limit="100 mb"/>
>> </tempUsage>
>> </systemUsage>
>> </systemUsage>
>>
>>
>> <!-- The transport connectors ActiveMQ will listen to -->
>> <transportConnectors>
>> <transportConnector name="openwire"
>> uri="tcp://localhost:61616"/>
>> <!-- <transportConnector name="ssl"
>> uri="ssl://localhost:61617"/>
>> <transportConnector name="stomp" uri="stomp://localhost:61613"/>
>> <transportConnector name="xmpp"
>> uri="xmpp://localhost:61222"/> -->
>> </transportConnectors>
>> <destinations>
>> <queue physicalName="WF.Regraph" />
>> <queue physicalName="WF.Notify"/>
>> <topic physicalName="WF.Events"/>
>> </destinations>
>> <destinationInterceptors>
>> <virtualDestinationInterceptor>
>> <virtualDestinations>
>> <compositeQueue name="WF.IncomingEvents">
>> <forwardTo>
>> <queue physicalName="WF.Regraph"/>
>> <queue physicalName="WF.Notify"/>
>> <topic physicalName="WF.Events" />
>> </forwardTo>
>> </compositeQueue>
>> </virtualDestinations>
>> </virtualDestinationInterceptor>
>> </destinationInterceptors>
>>
>>
>> </broker>
>>
>>
>>
>>> --
>>>
>>>
>>> On Fri, Jan 15, 2010 at 10:24 AM, David Delbecq <delbd@oma.be
>>> <ma...@oma.be>> wrote:
>>>
>>> Hello,
>>>
>>> can someone explain me why my activeMQ client is stalled in such
>>> situation (i have 10 threads blocked like this having as
>>> consequence a complete blocking of my client application):
>>>
>>> "http-8081-Processor6" daemon prio=10 tid=00841c38 nid=512
>>> lwp_id=2426611 waiting on condition [236f3000..236f153c]
>>> java.lang.Thread.State: WAITING (parking)
>>> at sun.misc.Unsafe.park(Native Method)
>>> - parking to wait for <40ae1828> (a
>>>
>>>
>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>>
>>> at
>>> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
>>> at
>>>
>>>
>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1889)
>>
>>> at
>>>
>>>
>> java.util.concurrent.ArrayBlockingQueue.take(ArrayBlockingQueue.java:317)
>>
>>> at
>>>
>>>
>> org.apache.activemq.transport.FutureResponse.getResult(FutureResponse.java:40)
>>
>>> at
>>>
>>>
>> org.apache.activemq.transport.ResponseCorrelator.request(ResponseCorrelator.java:80)
>>
>>> at
>>>
>>>
>> org.apache.activemq.ActiveMQConnection.syncSendPacket(ActiveMQConnection.java:1233)
>>
>>> at
>>> org.apache.activemq.ActiveMQSession.send(ActiveMQSession.java:1659)
>>> - locked <40ae01e8> (a java.lang.Object)
>>> at
>>>
>>>
>> org.apache.activemq.ActiveMQMessageProducer.send(ActiveMQMessageProducer.java:227)
>>
>>> at
>>>
>>>
>> org.apache.activemq.ActiveMQMessageProducerSupport.send(ActiveMQMessageProducerSupport.java:241)
>>
>>> at
>>> be.rmi.shark.JMSEventManager.sendEvent(JMSEventManager.java:146)
>>>
>>>
>>> considering that the activeMQ lists size, as referenced by
>>> activeMQ admin interface are as follow:
>>> Name Number Of Pending Messages Number Of Consumers
>>> Messages Sent Messages Received Views Operations
>>> WF.Regraph 0 1 1714 1720 Browse Send To
>>>
>> Purge Delete
>>
>>> WF.Notify 0 1 1696 1742 Browse Send To
>>>
>> Purge Delete
>>
>>> WF.IncomingEvents 0 0 0 0 Browse
>>>
>> Send To Purge Delete
>>
>>> Is this to be considered normal that i must restart activeMQ
>>> producers on a regular basis because of those this stalling? Where
>>> can i find clear documentation on proper configuration of activeMQ
>>> to prevent this stalling? Thank you.
>>>
>>> --
>>> David Delbecq
>>> ICT
>>> Institut Royal Météorologique
>>> Ext:557
>>>
>>>
>>>
>>>
>> --
>> David Delbecq
>> ICT
>> Institut Royal Météorologique
>> Ext:557
>>
>>
>>
>
>
>
--
David Delbecq
ICT
Institut Royal Météorologique
Ext:557
Re: why is activeMQ stalled?
Posted by Gary Tully <ga...@gmail.com>.
A thread dump of the broker jvm, when the clients are stalled, would help
diagnose what is going on. btw, what activemq version are you using?
2010/1/15 David Delbecq <de...@oma.be>
> Le 15/01/10 10:24, Dejan Bosanac a écrit :
> > Hi David,
> >
> > you're probably blocked by "producer flow control". Take a look here
> > for more info: http://activemq.apache.org/producer-flow-control.html
> >
> > Cheers
>
> I already took a look at it, read it three time to be sure i understand.
> It's quite technical but beyond that, i don't think this apply for me.
> My queues are all empty!!! (they stalled when there were 50 messages
> pending but where finally emptied), connections are not shared by
> several queues and am not sure how to implement the cursor part of
> documentation. There is no "slow consumer" that could influence it at
> that right moment (customers are running an waiting for new messages as
> you can see in capture). Moreover my messages are of about 1k payload
> (about 10 small text entries in a MapMessage), and my activeMQ config is
> like this
> <policyEntry queue=">" memoryLimit="500mb"/>
> <policyEntry topic=">" memoryLimit="500mb">
> ....
> <systemUsage>
> <systemUsage>
> <memoryUsage>
> <memoryUsage limit="20 mb"/>
> </memoryUsage>
> <storeUsage>
> <storeUsage limit="1 gb" name="foo"/>
> </storeUsage>
> <tempUsage>
> <tempUsage limit="100 mb"/>
> </tempUsage>
> </systemUsage>
> </systemUsage>
>
>
> There is no way a bunch of 10 pending messages could fill a 1gb store,
> isn't it? What config part of activemq do i need to change to ensure
> that it can store at least about 10000 persistent messages (that makes
> only 10Mb of datas!!) before failing? Currently it fails and is never
> recovering without a restart of broker when there are only 50 small
> pending messages in the queue. As a workaround i have set
> "sendFailIfNoSpace" to true, so my client will fail instead of stalling,
> i'll see when this happen more quickly like that....
>
>
> Here is broker part of my config file, can someone help me fix this?
> Thank you:
>
>
> <broker xmlns="http://activemq.apache.org/schema/core"
> brokerName="localhost" dataDirectory="${activemq.base}/data"
> persistent="true">
> <plugins>
> <simpleAuthenticationPlugin>
> <users>
> <authenticationUser username="admin" password="******"
> groups="users,admins"/>
> <authenticationUser username="sharkTool" password="********"
> groups="serverTools"/>
> <authenticationUser username="adminConsole"
> password="*********"
> groups="serverTools"/>
> </users>
> </simpleAuthenticationPlugin>
> <authorizationPlugin>
> <map>
> <authorizationMap>
> <authorizationEntries>
> <authorizationEntry queue=">" read="admins" write="admins"
> admin="admins" />
> <authorizationEntry queue="WF.>" read="users,serverTools"
> write="serverTools" admin="admins,serverTools" />
>
> <authorizationEntry topic=">" read="admins" write="admins"
> admin="admins" />
> <authorizationEntry topic="WF.>" read="users,serverTools"
> write="serverTools" admin="admins,serverTools" />
>
> <authorizationEntry topic="ActiveMQ.Advisory.>"
> read="guests,users,serverTools" write="guests,users,serverTools"
> admin="guests,users,serverTools"/>
> </authorizationEntries>
>
> <!-- let's assign roles to temporary destinations. comment
> this entry if we don't want any roles assigned to temp destinations
> <tempDestinationAuthorizationEntry>
> <tempDestinationAuthorizationEntry
> read="tempDestinationAdmins" write="tempDestinationAdmins"
> admin="tempDestinationAdmins"/>
> </tempDestinationAuthorizationEntry> -->
> </authorizationMap>
> </map>
> </authorizationPlugin>
>
>
> </plugins>
> <!-- Destination specific policies using destination names or
> wildcards -->
> <destinationPolicy>
> <policyMap>
> <policyEntries>
> <policyEntry queue=">" memoryLimit="500mb"/>
> <policyEntry topic=">" memoryLimit="500mb">
> <!-- you can add other policies too such as these
> <dispatchPolicy>
> <strictOrderDispatchPolicy/>
> </dispatchPolicy>
> <subscriptionRecoveryPolicy>
> <lastImageSubscriptionRecoveryPolicy/>
> </subscriptionRecoveryPolicy>
> -->
> </policyEntry>
> </policyEntries>
> </policyMap>
> </destinationPolicy>
>
> <!-- Use the following to configure how ActiveMQ is exposed in
> JMX -->
> <managementContext>
> <managementContext createConnector="false"/>
> </managementContext>
>
> <!-- The store and forward broker networks ActiveMQ will listen
> to -->
> <networkConnectors>
> <!-- by default just auto discover the other brokers -->
> <!-- <networkConnector name="default-nc"
> uri="multicast://default"/>-->
> <!-- Example of a static configuration:
> <networkConnector name="host1 and host2"
> uri="static://(tcp://host1:61616,tcp://host2:61616)"/>
> -->
> </networkConnectors>
>
> <persistenceAdapter>
> <amqPersistenceAdapter syncOnWrite="false"
> directory="${activemq.base}/data" maxFileLength="20 mb"/>
> </persistenceAdapter>
>
>
>
> <sslContext>
> <sslContext keyStore="file:${activemq.base}/conf/broker.ks"
> keyStorePassword="password"
> trustStore="file:${activemq.base}/conf/broker.ts"
> trustStorePassword="password"/>
> </sslContext>
>
> <!-- The maximum about of space the broker will use before
> slowing down producers -->
> <systemUsage>
> <systemUsage>
> <memoryUsage>
> <memoryUsage limit="20 mb"/>
> </memoryUsage>
> <storeUsage>
> <storeUsage limit="1 gb" name="foo"/>
> </storeUsage>
> <tempUsage>
> <tempUsage limit="100 mb"/>
> </tempUsage>
> </systemUsage>
> </systemUsage>
>
>
> <!-- The transport connectors ActiveMQ will listen to -->
> <transportConnectors>
> <transportConnector name="openwire"
> uri="tcp://localhost:61616"/>
> <!-- <transportConnector name="ssl"
> uri="ssl://localhost:61617"/>
> <transportConnector name="stomp" uri="stomp://localhost:61613"/>
> <transportConnector name="xmpp"
> uri="xmpp://localhost:61222"/> -->
> </transportConnectors>
> <destinations>
> <queue physicalName="WF.Regraph" />
> <queue physicalName="WF.Notify"/>
> <topic physicalName="WF.Events"/>
> </destinations>
> <destinationInterceptors>
> <virtualDestinationInterceptor>
> <virtualDestinations>
> <compositeQueue name="WF.IncomingEvents">
> <forwardTo>
> <queue physicalName="WF.Regraph"/>
> <queue physicalName="WF.Notify"/>
> <topic physicalName="WF.Events" />
> </forwardTo>
> </compositeQueue>
> </virtualDestinations>
> </virtualDestinationInterceptor>
> </destinationInterceptors>
>
>
> </broker>
>
>
> > --
> >
> >
> > On Fri, Jan 15, 2010 at 10:24 AM, David Delbecq <delbd@oma.be
> > <ma...@oma.be>> wrote:
> >
> > Hello,
> >
> > can someone explain me why my activeMQ client is stalled in such
> > situation (i have 10 threads blocked like this having as
> > consequence a complete blocking of my client application):
> >
> > "http-8081-Processor6" daemon prio=10 tid=00841c38 nid=512
> > lwp_id=2426611 waiting on condition [236f3000..236f153c]
> > java.lang.Thread.State: WAITING (parking)
> > at sun.misc.Unsafe.park(Native Method)
> > - parking to wait for <40ae1828> (a
> >
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> > at
> > java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> > at
> >
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1889)
> > at
> >
> java.util.concurrent.ArrayBlockingQueue.take(ArrayBlockingQueue.java:317)
> > at
> >
> org.apache.activemq.transport.FutureResponse.getResult(FutureResponse.java:40)
> > at
> >
> org.apache.activemq.transport.ResponseCorrelator.request(ResponseCorrelator.java:80)
> > at
> >
> org.apache.activemq.ActiveMQConnection.syncSendPacket(ActiveMQConnection.java:1233)
> > at
> > org.apache.activemq.ActiveMQSession.send(ActiveMQSession.java:1659)
> > - locked <40ae01e8> (a java.lang.Object)
> > at
> >
> org.apache.activemq.ActiveMQMessageProducer.send(ActiveMQMessageProducer.java:227)
> > at
> >
> org.apache.activemq.ActiveMQMessageProducerSupport.send(ActiveMQMessageProducerSupport.java:241)
> > at
> > be.rmi.shark.JMSEventManager.sendEvent(JMSEventManager.java:146)
> >
> >
> > considering that the activeMQ lists size, as referenced by
> > activeMQ admin interface are as follow:
> > Name Number Of Pending Messages Number Of Consumers
> > Messages Sent Messages Received Views Operations
> > WF.Regraph 0 1 1714 1720 Browse Send To
> Purge Delete
> > WF.Notify 0 1 1696 1742 Browse Send To
> Purge Delete
> > WF.IncomingEvents 0 0 0 0 Browse
> Send To Purge Delete
> >
> >
> > Is this to be considered normal that i must restart activeMQ
> > producers on a regular basis because of those this stalling? Where
> > can i find clear documentation on proper configuration of activeMQ
> > to prevent this stalling? Thank you.
> >
> > --
> > David Delbecq
> > ICT
> > Institut Royal Météorologique
> > Ext:557
> >
> >
> >
>
>
> --
> David Delbecq
> ICT
> Institut Royal Météorologique
> Ext:557
>
>
--
http://blog.garytully.com
Open Source Integration
http://fusesource.com
Re: why is activeMQ stalled?
Posted by David Delbecq <de...@oma.be>.
Le 15/01/10 10:24, Dejan Bosanac a écrit :
> Hi David,
>
> you're probably blocked by "producer flow control". Take a look here
> for more info: http://activemq.apache.org/producer-flow-control.html
>
> Cheers
I already took a look at it, read it three time to be sure i understand.
It's quite technical but beyond that, i don't think this apply for me.
My queues are all empty!!! (they stalled when there were 50 messages
pending but where finally emptied), connections are not shared by
several queues and am not sure how to implement the cursor part of
documentation. There is no "slow consumer" that could influence it at
that right moment (customers are running an waiting for new messages as
you can see in capture). Moreover my messages are of about 1k payload
(about 10 small text entries in a MapMessage), and my activeMQ config is
like this
<policyEntry queue=">" memoryLimit="500mb"/>
<policyEntry topic=">" memoryLimit="500mb">
....
<systemUsage>
<systemUsage>
<memoryUsage>
<memoryUsage limit="20 mb"/>
</memoryUsage>
<storeUsage>
<storeUsage limit="1 gb" name="foo"/>
</storeUsage>
<tempUsage>
<tempUsage limit="100 mb"/>
</tempUsage>
</systemUsage>
</systemUsage>
There is no way a bunch of 10 pending messages could fill a 1gb store,
isn't it? What config part of activemq do i need to change to ensure
that it can store at least about 10000 persistent messages (that makes
only 10Mb of datas!!) before failing? Currently it fails and is never
recovering without a restart of broker when there are only 50 small
pending messages in the queue. As a workaround i have set
"sendFailIfNoSpace" to true, so my client will fail instead of stalling,
i'll see when this happen more quickly like that....
Here is broker part of my config file, can someone help me fix this?
Thank you:
<broker xmlns="http://activemq.apache.org/schema/core"
brokerName="localhost" dataDirectory="${activemq.base}/data"
persistent="true">
<plugins>
<simpleAuthenticationPlugin>
<users>
<authenticationUser username="admin" password="******"
groups="users,admins"/>
<authenticationUser username="sharkTool" password="********"
groups="serverTools"/>
<authenticationUser username="adminConsole"
password="*********"
groups="serverTools"/>
</users>
</simpleAuthenticationPlugin>
<authorizationPlugin>
<map>
<authorizationMap>
<authorizationEntries>
<authorizationEntry queue=">" read="admins" write="admins"
admin="admins" />
<authorizationEntry queue="WF.>" read="users,serverTools"
write="serverTools" admin="admins,serverTools" />
<authorizationEntry topic=">" read="admins" write="admins"
admin="admins" />
<authorizationEntry topic="WF.>" read="users,serverTools"
write="serverTools" admin="admins,serverTools" />
<authorizationEntry topic="ActiveMQ.Advisory.>"
read="guests,users,serverTools" write="guests,users,serverTools"
admin="guests,users,serverTools"/>
</authorizationEntries>
<!-- let's assign roles to temporary destinations. comment
this entry if we don't want any roles assigned to temp destinations
<tempDestinationAuthorizationEntry>
<tempDestinationAuthorizationEntry
read="tempDestinationAdmins" write="tempDestinationAdmins"
admin="tempDestinationAdmins"/>
</tempDestinationAuthorizationEntry> -->
</authorizationMap>
</map>
</authorizationPlugin>
</plugins>
<!-- Destination specific policies using destination names or
wildcards -->
<destinationPolicy>
<policyMap>
<policyEntries>
<policyEntry queue=">" memoryLimit="500mb"/>
<policyEntry topic=">" memoryLimit="500mb">
<!-- you can add other policies too such as these
<dispatchPolicy>
<strictOrderDispatchPolicy/>
</dispatchPolicy>
<subscriptionRecoveryPolicy>
<lastImageSubscriptionRecoveryPolicy/>
</subscriptionRecoveryPolicy>
-->
</policyEntry>
</policyEntries>
</policyMap>
</destinationPolicy>
<!-- Use the following to configure how ActiveMQ is exposed in
JMX -->
<managementContext>
<managementContext createConnector="false"/>
</managementContext>
<!-- The store and forward broker networks ActiveMQ will listen
to -->
<networkConnectors>
<!-- by default just auto discover the other brokers -->
<!-- <networkConnector name="default-nc"
uri="multicast://default"/>-->
<!-- Example of a static configuration:
<networkConnector name="host1 and host2"
uri="static://(tcp://host1:61616,tcp://host2:61616)"/>
-->
</networkConnectors>
<persistenceAdapter>
<amqPersistenceAdapter syncOnWrite="false"
directory="${activemq.base}/data" maxFileLength="20 mb"/>
</persistenceAdapter>
<sslContext>
<sslContext keyStore="file:${activemq.base}/conf/broker.ks"
keyStorePassword="password"
trustStore="file:${activemq.base}/conf/broker.ts"
trustStorePassword="password"/>
</sslContext>
<!-- The maximum about of space the broker will use before
slowing down producers -->
<systemUsage>
<systemUsage>
<memoryUsage>
<memoryUsage limit="20 mb"/>
</memoryUsage>
<storeUsage>
<storeUsage limit="1 gb" name="foo"/>
</storeUsage>
<tempUsage>
<tempUsage limit="100 mb"/>
</tempUsage>
</systemUsage>
</systemUsage>
<!-- The transport connectors ActiveMQ will listen to -->
<transportConnectors>
<transportConnector name="openwire"
uri="tcp://localhost:61616"/>
<!-- <transportConnector name="ssl"
uri="ssl://localhost:61617"/>
<transportConnector name="stomp" uri="stomp://localhost:61613"/>
<transportConnector name="xmpp"
uri="xmpp://localhost:61222"/> -->
</transportConnectors>
<destinations>
<queue physicalName="WF.Regraph" />
<queue physicalName="WF.Notify"/>
<topic physicalName="WF.Events"/>
</destinations>
<destinationInterceptors>
<virtualDestinationInterceptor>
<virtualDestinations>
<compositeQueue name="WF.IncomingEvents">
<forwardTo>
<queue physicalName="WF.Regraph"/>
<queue physicalName="WF.Notify"/>
<topic physicalName="WF.Events" />
</forwardTo>
</compositeQueue>
</virtualDestinations>
</virtualDestinationInterceptor>
</destinationInterceptors>
</broker>
> --
>
>
> On Fri, Jan 15, 2010 at 10:24 AM, David Delbecq <delbd@oma.be
> <ma...@oma.be>> wrote:
>
> Hello,
>
> can someone explain me why my activeMQ client is stalled in such
> situation (i have 10 threads blocked like this having as
> consequence a complete blocking of my client application):
>
> "http-8081-Processor6" daemon prio=10 tid=00841c38 nid=512
> lwp_id=2426611 waiting on condition [236f3000..236f153c]
> java.lang.Thread.State: WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for <40ae1828> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> at
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1889)
> at
> java.util.concurrent.ArrayBlockingQueue.take(ArrayBlockingQueue.java:317)
> at
> org.apache.activemq.transport.FutureResponse.getResult(FutureResponse.java:40)
> at
> org.apache.activemq.transport.ResponseCorrelator.request(ResponseCorrelator.java:80)
> at
> org.apache.activemq.ActiveMQConnection.syncSendPacket(ActiveMQConnection.java:1233)
> at
> org.apache.activemq.ActiveMQSession.send(ActiveMQSession.java:1659)
> - locked <40ae01e8> (a java.lang.Object)
> at
> org.apache.activemq.ActiveMQMessageProducer.send(ActiveMQMessageProducer.java:227)
> at
> org.apache.activemq.ActiveMQMessageProducerSupport.send(ActiveMQMessageProducerSupport.java:241)
> at
> be.rmi.shark.JMSEventManager.sendEvent(JMSEventManager.java:146)
>
>
> considering that the activeMQ lists size, as referenced by
> activeMQ admin interface are as follow:
> Name Number Of Pending Messages Number Of Consumers
> Messages Sent Messages Received Views Operations
> WF.Regraph 0 1 1714 1720 Browse Send To Purge Delete
> WF.Notify 0 1 1696 1742 Browse Send To Purge Delete
> WF.IncomingEvents 0 0 0 0 Browse Send To Purge Delete
>
>
> Is this to be considered normal that i must restart activeMQ
> producers on a regular basis because of those this stalling? Where
> can i find clear documentation on proper configuration of activeMQ
> to prevent this stalling? Thank you.
>
> --
> David Delbecq
> ICT
> Institut Royal Météorologique
> Ext:557
>
>
>
--
David Delbecq
ICT
Institut Royal Météorologique
Ext:557
Re: why is activeMQ stalled?
Posted by Dejan Bosanac <de...@nighttale.net>.
Hi David,
you're probably blocked by "producer flow control". Take a look here for
more info: http://activemq.apache.org/producer-flow-control.html
Cheers
--
Dejan Bosanac - http://twitter.com/dejanb
Open Source Integration - http://fusesource.com/
ActiveMQ in Action - http://www.manning.com/snyder/
Blog - http://www.nighttale.net
On Fri, Jan 15, 2010 at 10:24 AM, David Delbecq <de...@oma.be> wrote:
> Hello,
>
> can someone explain me why my activeMQ client is stalled in such situation
> (i have 10 threads blocked like this having as consequence a complete
> blocking of my client application):
>
> "http-8081-Processor6" daemon prio=10 tid=00841c38 nid=512 lwp_id=2426611
> waiting on condition [236f3000..236f153c]
> java.lang.Thread.State: WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for <40ae1828> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> at
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1889)
> at
> java.util.concurrent.ArrayBlockingQueue.take(ArrayBlockingQueue.java:317)
> at
> org.apache.activemq.transport.FutureResponse.getResult(FutureResponse.java:40)
> at
> org.apache.activemq.transport.ResponseCorrelator.request(ResponseCorrelator.java:80)
> at
> org.apache.activemq.ActiveMQConnection.syncSendPacket(ActiveMQConnection.java:1233)
> at
> org.apache.activemq.ActiveMQSession.send(ActiveMQSession.java:1659)
> - locked <40ae01e8> (a java.lang.Object)
> at
> org.apache.activemq.ActiveMQMessageProducer.send(ActiveMQMessageProducer.java:227)
> at
> org.apache.activemq.ActiveMQMessageProducerSupport.send(ActiveMQMessageProducerSupport.java:241)
> at be.rmi.shark.JMSEventManager.sendEvent(JMSEventManager.java:146)
>
>
> considering that the activeMQ lists size, as referenced by activeMQ admin
> interface are as follow:
> Name Number Of Pending Messages Number Of Consumers Messages Sent Messages
> Received Views Operations WF.Regraph 0 1 1714 1720 Browse Send
> To Purge Delete WF.Notify 0 1 1696 1742 Browse Send To Purge Delete WF.IncomingEvents
> 0 0 0 0 Browse Send To Purge Delete
> Is this to be considered normal that i must restart activeMQ producers on a
> regular basis because of those this stalling? Where can i find clear
> documentation on proper configuration of activeMQ to prevent this stalling?
> Thank you.
>
> --
> David Delbecq
> ICT
> Institut Royal Météorologique
> Ext:557
>
>