You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@activemq.apache.org by Matthew Hixson <mh...@gmail.com> on 2010/09/09 23:33:16 UTC
5.4.0 stops accepting messages randomly
We have been running 5.4.0 in production for a few weeks and it can easily handle tens of thousands of messages a day, but then for no apparent reason it will cease accepting new messages from our producers. The consumers will drain the queue empty, but then nothing new gets accepted into Active MQ. Restarting Active MQ and then restarting all the producers and consumers allows things to proceed as normal.
Below is our activemq.xml.
Thanks for any help.
-M@
<!--
Licensed to the Apache Software Foundation (ASF) under one or more
contributor license agreements. See the NOTICE file distributed with
this work for additional information regarding copyright ownership.
The ASF licenses this file to You under the Apache License, Version 2.0
(the "License"); you may not use this file except in compliance with
the License. You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
-->
<beans
xmlns="http://www.springframework.org/schema/beans"
xmlns:amq="http://activemq.apache.org/schema/core"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://www.springframework.org/schema/beans http://www.springframework.org/schema/beans/spring-beans-2.0.xsd
http://activemq.apache.org/schema/core http://activemq.apache.org/schema/core/activemq-core.xsd">
<!-- Allows us to use system properties as variables in this configuration file -->
<bean class="org.springframework.beans.factory.config.PropertyPlaceholderConfigurer">
<property name="locations">
<value>file:${activemq.base}/conf/credentials.properties</value>
</property>
</bean>
<!--
The <broker> element is used to configure the ActiveMQ broker.
-->
<broker xmlns="http://activemq.apache.org/schema/core" brokerName="localhost" dataDirectory="${activemq.base}/data" destroyApplicationContextOnStop="true">
<!--
For better performances use VM cursor and small memory limit.
For more information, see:
http://activemq.apache.org/message-cursors.html
Also, if your producer is "hanging", it's probably due to producer flow control.
For more information, see:
http://activemq.apache.org/producer-flow-control.html
-->
<destinationPolicy>
<policyMap>
<policyEntries>
<policyEntry topic=">" producerFlowControl="false" memoryLimit="100mb">
<pendingSubscriberPolicy>
<vmCursor />
</pendingSubscriberPolicy>
</policyEntry>
<policyEntry queue=">" producerFlowControl="true" memoryLimit="1mb">
<!-- Use VM cursor for better latency
For more information, see:
http://activemq.apache.org/message-cursors.html
<pendingQueuePolicy>
<vmQueueCursor/>
</pendingQueuePolicy>
-->
</policyEntry>
</policyEntries>
</policyMap>
</destinationPolicy>
<!--
The managementContext is used to configure how ActiveMQ is exposed in
JMX. By default, ActiveMQ uses the MBean server that is started by
the JVM. For more information, see:
http://activemq.apache.org/jmx.html
-->
<managementContext>
<managementContext createConnector="false"/>
</managementContext>
<!--
Configure message persistence for the broker. The default persistence
mechanism is the KahaDB store (identified by the kahaDB tag).
For more information, see:
http://activemq.apache.org/persistence.html
-->
<persistenceAdapter>
<kahaDB directory="${activemq.base}/data/kahadb"/>
</persistenceAdapter>
<!--
The systemUsage controls the maximum amount of space the broker will
use before slowing down producers. For more information, see:
http://activemq.apache.org/producer-flow-control.html
<systemUsage>
<systemUsage>
<memoryUsage>
<memoryUsage limit="100 mb"/>
</memoryUsage>
<storeUsage>
<storeUsage limit="10 gb"/>
</storeUsage>
<tempUsage>
<tempUsage limit="100 mb"/>
</tempUsage>
</systemUsage>
</systemUsage>
-->
<!--
The transport connectors expose ActiveMQ over a given protocol to
clients and other brokers. For more information, see:
http://activemq.apache.org/configuring-transports.html
-->
<transportConnectors>
<transportConnector name="openwire" uri="tcp://0.0.0.0:61616"/>
</transportConnectors>
</broker>
<!--
Uncomment to enable Camel
Take a look at activemq-camel.xml for more details
<import resource="camel.xml"/>
-->
<!--
Enable web consoles, REST and Ajax APIs and demos
Take a look at activemq-jetty.xml for more details
-->
<import resource="jetty.xml"/>
</beans>
Re: 5.4.0 stops accepting messages randomly
Posted by Matthew Hixson <mh...@gmail.com>.
We did find this in the logs.
activemq.log:2010-09-08 10:55:02,555 | INFO | Usage Manager Memory Limit is full. Producer (ID:isp01-55443-1283902621643-0:0:1:1) stopped to prevent
flooding queue://mpqueue. See http://activemq.apache.org/producer-flow-control.html for more info (blocking for: 1s) | org.apache.activemq.broker.re
gion.Queue | ActiveMQ Transport: tcp:///10.13.85.14:52682
We have since increased the memory limit and we'll see if this happens again. If so I'll get a full thread dump.
Thanks,
-M@
On Sep 10, 2010, at 12:51 AM, Dejan Bosanac wrote:
> Can you turn on debug logging and post logs and thread dumps when the
> broker hangs. Also, do you see messages in logs about producer flow
> control being used?
>
>
> Cheers
> --
> Dejan Bosanac - http://twitter.com/dejanb
>
> Open Source Integration - http://fusesource.com/
> ActiveMQ in Action - http://www.manning.com/snyder/
> Blog - http://www.nighttale.net
>
>
>
> On Thu, Sep 9, 2010 at 11:33 PM, Matthew Hixson <mh...@gmail.com> wrote:
>> We have been running 5.4.0 in production for a few weeks and it can easily handle tens of thousands of messages a day, but then for no apparent reason it will cease accepting new messages from our producers. The consumers will drain the queue empty, but then nothing new gets accepted into Active MQ. Restarting Active MQ and then restarting all the producers and consumers allows things to proceed as normal.
>> Below is our activemq.xml.
>> Thanks for any help.
>> -M@
>>
>> <!--
>> Licensed to the Apache Software Foundation (ASF) under one or more
>> contributor license agreements. See the NOTICE file distributed with
>> this work for additional information regarding copyright ownership.
>> The ASF licenses this file to You under the Apache License, Version 2.0
>> (the "License"); you may not use this file except in compliance with
>> the License. You may obtain a copy of the License at
>>
>> http://www.apache.org/licenses/LICENSE-2.0
>>
>> Unless required by applicable law or agreed to in writing, software
>> distributed under the License is distributed on an "AS IS" BASIS,
>> WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
>> See the License for the specific language governing permissions and
>> limitations under the License.
>> -->
>> <beans
>> xmlns="http://www.springframework.org/schema/beans"
>> xmlns:amq="http://activemq.apache.org/schema/core"
>> xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
>> xsi:schemaLocation="http://www.springframework.org/schema/beans http://www.springframework.org/schema/beans/spring-beans-2.0.xsd
>> http://activemq.apache.org/schema/core http://activemq.apache.org/schema/core/activemq-core.xsd">
>>
>> <!-- Allows us to use system properties as variables in this configuration file -->
>> <bean class="org.springframework.beans.factory.config.PropertyPlaceholderConfigurer">
>> <property name="locations">
>> <value>file:${activemq.base}/conf/credentials.properties</value>
>> </property>
>> </bean>
>>
>> <!--
>> The <broker> element is used to configure the ActiveMQ broker.
>> -->
>> <broker xmlns="http://activemq.apache.org/schema/core" brokerName="localhost" dataDirectory="${activemq.base}/data" destroyApplicationContextOnStop="true">
>>
>> <!--
>> For better performances use VM cursor and small memory limit.
>> For more information, see:
>>
>> http://activemq.apache.org/message-cursors.html
>>
>> Also, if your producer is "hanging", it's probably due to producer flow control.
>> For more information, see:
>> http://activemq.apache.org/producer-flow-control.html
>> -->
>>
>> <destinationPolicy>
>> <policyMap>
>> <policyEntries>
>> <policyEntry topic=">" producerFlowControl="false" memoryLimit="100mb">
>> <pendingSubscriberPolicy>
>> <vmCursor />
>> </pendingSubscriberPolicy>
>> </policyEntry>
>> <policyEntry queue=">" producerFlowControl="true" memoryLimit="1mb">
>> <!-- Use VM cursor for better latency
>> For more information, see:
>>
>> http://activemq.apache.org/message-cursors.html
>>
>> <pendingQueuePolicy>
>> <vmQueueCursor/>
>> </pendingQueuePolicy>
>> -->
>> </policyEntry>
>> </policyEntries>
>> </policyMap>
>> </destinationPolicy>
>>
>>
>> <!--
>> The managementContext is used to configure how ActiveMQ is exposed in
>> JMX. By default, ActiveMQ uses the MBean server that is started by
>> the JVM. For more information, see:
>>
>> http://activemq.apache.org/jmx.html
>> -->
>> <managementContext>
>> <managementContext createConnector="false"/>
>> </managementContext>
>>
>> <!--
>> Configure message persistence for the broker. The default persistence
>> mechanism is the KahaDB store (identified by the kahaDB tag).
>> For more information, see:
>>
>> http://activemq.apache.org/persistence.html
>> -->
>> <persistenceAdapter>
>> <kahaDB directory="${activemq.base}/data/kahadb"/>
>> </persistenceAdapter>
>>
>>
>> <!--
>> The systemUsage controls the maximum amount of space the broker will
>> use before slowing down producers. For more information, see:
>>
>> http://activemq.apache.org/producer-flow-control.html
>>
>> <systemUsage>
>> <systemUsage>
>> <memoryUsage>
>> <memoryUsage limit="100 mb"/>
>> </memoryUsage>
>> <storeUsage>
>> <storeUsage limit="10 gb"/>
>> </storeUsage>
>> <tempUsage>
>> <tempUsage limit="100 mb"/>
>> </tempUsage>
>> </systemUsage>
>> </systemUsage>
>> -->
>>
>> <!--
>> The transport connectors expose ActiveMQ over a given protocol to
>> clients and other brokers. For more information, see:
>>
>> http://activemq.apache.org/configuring-transports.html
>> -->
>> <transportConnectors>
>> <transportConnector name="openwire" uri="tcp://0.0.0.0:61616"/>
>> </transportConnectors>
>>
>> </broker>
>>
>> <!--
>> Uncomment to enable Camel
>> Take a look at activemq-camel.xml for more details
>>
>> <import resource="camel.xml"/>
>> -->
>>
>> <!--
>> Enable web consoles, REST and Ajax APIs and demos
>> Take a look at activemq-jetty.xml for more details
>> -->
>> <import resource="jetty.xml"/>
>>
>> </beans>
Re: 5.4.0 stops accepting messages randomly
Posted by Dejan Bosanac <de...@nighttale.net>.
Can you turn on debug logging and post logs and thread dumps when the
broker hangs. Also, do you see messages in logs about producer flow
control being used?
Cheers
--
Dejan Bosanac - http://twitter.com/dejanb
Open Source Integration - http://fusesource.com/
ActiveMQ in Action - http://www.manning.com/snyder/
Blog - http://www.nighttale.net
On Thu, Sep 9, 2010 at 11:33 PM, Matthew Hixson <mh...@gmail.com> wrote:
> We have been running 5.4.0 in production for a few weeks and it can easily handle tens of thousands of messages a day, but then for no apparent reason it will cease accepting new messages from our producers. The consumers will drain the queue empty, but then nothing new gets accepted into Active MQ. Restarting Active MQ and then restarting all the producers and consumers allows things to proceed as normal.
> Below is our activemq.xml.
> Thanks for any help.
> -M@
>
> <!--
> Licensed to the Apache Software Foundation (ASF) under one or more
> contributor license agreements. See the NOTICE file distributed with
> this work for additional information regarding copyright ownership.
> The ASF licenses this file to You under the Apache License, Version 2.0
> (the "License"); you may not use this file except in compliance with
> the License. You may obtain a copy of the License at
>
> http://www.apache.org/licenses/LICENSE-2.0
>
> Unless required by applicable law or agreed to in writing, software
> distributed under the License is distributed on an "AS IS" BASIS,
> WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
> See the License for the specific language governing permissions and
> limitations under the License.
> -->
> <beans
> xmlns="http://www.springframework.org/schema/beans"
> xmlns:amq="http://activemq.apache.org/schema/core"
> xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
> xsi:schemaLocation="http://www.springframework.org/schema/beans http://www.springframework.org/schema/beans/spring-beans-2.0.xsd
> http://activemq.apache.org/schema/core http://activemq.apache.org/schema/core/activemq-core.xsd">
>
> <!-- Allows us to use system properties as variables in this configuration file -->
> <bean class="org.springframework.beans.factory.config.PropertyPlaceholderConfigurer">
> <property name="locations">
> <value>file:${activemq.base}/conf/credentials.properties</value>
> </property>
> </bean>
>
> <!--
> The <broker> element is used to configure the ActiveMQ broker.
> -->
> <broker xmlns="http://activemq.apache.org/schema/core" brokerName="localhost" dataDirectory="${activemq.base}/data" destroyApplicationContextOnStop="true">
>
> <!--
> For better performances use VM cursor and small memory limit.
> For more information, see:
>
> http://activemq.apache.org/message-cursors.html
>
> Also, if your producer is "hanging", it's probably due to producer flow control.
> For more information, see:
> http://activemq.apache.org/producer-flow-control.html
> -->
>
> <destinationPolicy>
> <policyMap>
> <policyEntries>
> <policyEntry topic=">" producerFlowControl="false" memoryLimit="100mb">
> <pendingSubscriberPolicy>
> <vmCursor />
> </pendingSubscriberPolicy>
> </policyEntry>
> <policyEntry queue=">" producerFlowControl="true" memoryLimit="1mb">
> <!-- Use VM cursor for better latency
> For more information, see:
>
> http://activemq.apache.org/message-cursors.html
>
> <pendingQueuePolicy>
> <vmQueueCursor/>
> </pendingQueuePolicy>
> -->
> </policyEntry>
> </policyEntries>
> </policyMap>
> </destinationPolicy>
>
>
> <!--
> The managementContext is used to configure how ActiveMQ is exposed in
> JMX. By default, ActiveMQ uses the MBean server that is started by
> the JVM. For more information, see:
>
> http://activemq.apache.org/jmx.html
> -->
> <managementContext>
> <managementContext createConnector="false"/>
> </managementContext>
>
> <!--
> Configure message persistence for the broker. The default persistence
> mechanism is the KahaDB store (identified by the kahaDB tag).
> For more information, see:
>
> http://activemq.apache.org/persistence.html
> -->
> <persistenceAdapter>
> <kahaDB directory="${activemq.base}/data/kahadb"/>
> </persistenceAdapter>
>
>
> <!--
> The systemUsage controls the maximum amount of space the broker will
> use before slowing down producers. For more information, see:
>
> http://activemq.apache.org/producer-flow-control.html
>
> <systemUsage>
> <systemUsage>
> <memoryUsage>
> <memoryUsage limit="100 mb"/>
> </memoryUsage>
> <storeUsage>
> <storeUsage limit="10 gb"/>
> </storeUsage>
> <tempUsage>
> <tempUsage limit="100 mb"/>
> </tempUsage>
> </systemUsage>
> </systemUsage>
> -->
>
> <!--
> The transport connectors expose ActiveMQ over a given protocol to
> clients and other brokers. For more information, see:
>
> http://activemq.apache.org/configuring-transports.html
> -->
> <transportConnectors>
> <transportConnector name="openwire" uri="tcp://0.0.0.0:61616"/>
> </transportConnectors>
>
> </broker>
>
> <!--
> Uncomment to enable Camel
> Take a look at activemq-camel.xml for more details
>
> <import resource="camel.xml"/>
> -->
>
> <!--
> Enable web consoles, REST and Ajax APIs and demos
> Take a look at activemq-jetty.xml for more details
> -->
> <import resource="jetty.xml"/>
>
> </beans>