You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@activemq.apache.org by Tomáš Martinec <to...@d3s.mff.cuni.cz> on 2012/12/04 22:01:29 UTC
Activemq 5.4.2 hangs when the temp disk usage is used
Hello,
I am trying to setup activemq 5.4.2 and I experience hangs when the
broker fills its heap and starts using storage. In my scenario I have
one message producer and one message consumer connected. The producer is
blocked and the consumer does not receive any messages. I have been able
to run the broker under debug mode and identify what the relevant
threads are doing:
The task for iteration of the queue waits for the messagesLock on the
line 1404 in
http://svn.apache.org/viewvc/activemq/tags/activemq-5.4.2/activemq-core/src/main/java/org/apache/activemq/broker/region/Queue.java?view=markup.
The thread that stores messages into the queue waits on the line 219 of
http://svn.apache.org/viewvc/activemq/tags/activemq-5.4.2/activemq-core/src/main/java/org/apache/activemq/broker/region/cursors/FilePendingMessageCursor.java?view=markup
for free space. Note that the thread has the messagesLock locked and the
temporaly usage is always 103%, and therefore, it never returns.
Is it possible that I encountered something like a deadlock? Perhaps, I
messed up something in the configuration.
I will be thankful for advice.
Tomas
Here is the configuration of the broker:
<beans
xmlns="http://www.springframework.org/schema/beans"
xmlns:amq="http://activemq.apache.org/schema/core"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://www.springframework.org/schema/beans
http://www.springframework.org/schema/beans/spring-beans-2.0.xsd
http://activemq.apache.org/schema/core
http://activemq.apache.org/schema/core/activemq-core.xsd">
<!-- Allows us to use system properties as variables in this
configuration file -->
<bean
class="org.springframework.beans.factory.config.PropertyPlaceholderConfigurer"
/>
<!--
The <broker> element is used to configure the ActiveMQ broker.
advisories incurr a hit with every add
connection|destination|producer|consumer
- client side: providerUrl =
tcp://localhost:61616?jms.watchTopicAdvisories=false
statistics have a small impact on concurrency so disable
schedualer has a small impact as it checks for the presence
three properties on each message
jmx adds a bunch of wrapper classes
ACTIVEMQ_OPTS_MEMORY="-Xmx11g -Xms6g
-Dorg.apache.activemq.UseDedicatedTaskRunner=false
-Djava.util.logging.config.file=logging.properties -XX:+UseLargePages"
-->
<broker xmlns="http://activemq.apache.org/schema/core"
brokerName="localhost"
dataDirectory="/dasd/martinec/kahadb-storage/activemq-data"
destroyApplicationContextOnStop="true"
deleteAllMessagesOnStartup="true"
advisorySupport="false"
enableStatistics="false"
schedulerSupport="false"
useJmx="true"
persistent="true">
<destinationPolicy>
<policyMap>
<policyEntries>
<!--
producerFlowControll thread unnecessary with no
memory limits
with no failover, no need to suppress duplicates so
audit can be disabled
message expiry thread unnecessary as normal
dispatch will handle expiry
-->
<policyEntry topic=">" producerFlowControl="false"
enableAudit="false" expireMessagesPeriod="0"></policyEntry>
<policyEntry queue=">" producerFlowControl="true"
memoryLimit="50mb" enableAudit="false" expireMessagesPeriod="0">
</policyEntry>
</policyEntries>
</policyMap>
</destinationPolicy>
<!--
The managementContext is used to configure how ActiveMQ is
exposed in
JMX. By default, ActiveMQ uses the MBean server that is
started by
the JVM. For more information, see:
http://activemq.apache.org/jmx.html
-->
<managementContext>
<managementContext createConnector="true"
connectorPort="1399" rmiServerPort="1399"/>
</managementContext>
<persistenceAdapter>
<kahaPersistenceAdapter directory="/dasd/martinec/kahadb-storage"/>
</persistenceAdapter>
<systemUsage>
<systemUsage>
<memoryUsage>
<!-- the only queue has 50 mb, so this should be enough -->
<memoryUsage limit="200 mb"/>
</memoryUsage>
<storeUsage>
<!-- no special reason for this -->
<storeUsage limit="1 gb"/>
</storeUsage>
<tempUsage>
<!-- this MUST be above the journal file size; see
http://tmielke.blogspot.cz/2011/02/observations-on-activemqs-temp-storage.html
-->
<tempUsage limit="100 mb"/>
</tempUsage>
</systemUsage>
</systemUsage>
<!--
The transport connectors expose ActiveMQ over a given
protocol to
clients and other brokers. For more information, see:
http://activemq.apache.org/configuring-transports.html
-->
<transportConnectors>
<transportConnector name="openwire" uri="tcp://0.0.0.0:61616"/>
</transportConnectors>
</broker>
<import resource="jetty.xml"/>
</beans>
Re: Activemq 5.4.2 hangs when the temp disk usage is used
Posted by Christian Posta <ch...@gmail.com>.
Actually, your config *could* cause a problem.
If you have>100MB of messages in memory and you try to write them to temp
storage (let's say on a Topic) then you might have problems since temp
store is only 100MB (1/2 of what memory is).
On Wed, Dec 5, 2012 at 6:35 AM, Christian Posta
<ch...@gmail.com>wrote:
> Interesting. Your scenario sounds like PFC is blocking the connection. Not
> sure yet why that would happen. Your config looks okay. Do you still have
> the thread dumps?
>
> Are the producer/consumer on the same connection? If you have a test case
> you can post, please do. Would make it easy to figure what the problem is.
>
>
> On Tue, Dec 4, 2012 at 2:01 PM, Tomáš Martinec <
> tomas.martinec@d3s.mff.cuni.cz> wrote:
>
>> Hello,
>>
>> I am trying to setup activemq 5.4.2 and I experience hangs when the
>> broker fills its heap and starts using storage. In my scenario I have one
>> message producer and one message consumer connected. The producer is
>> blocked and the consumer does not receive any messages. I have been able to
>> run the broker under debug mode and identify what the relevant threads are
>> doing:
>>
>> The task for iteration of the queue waits for the messagesLock on the
>> line 1404 in http://svn.apache.org/viewvc/**activemq/tags/activemq-5.4.2/
>> **activemq-core/src/main/java/**org/apache/activemq/broker/**
>> region/Queue.java?view=markup<http://svn.apache.org/viewvc/activemq/tags/activemq-5.4.2/activemq-core/src/main/java/org/apache/activemq/broker/region/Queue.java?view=markup>.
>>
>>
>> The thread that stores messages into the queue waits on the line 219 of
>> http://svn.apache.org/viewvc/**activemq/tags/activemq-5.4.2/**
>> activemq-core/src/main/java/**org/apache/activemq/broker/**
>> region/cursors/**FilePendingMessageCursor.java?**view=markup<http://svn.apache.org/viewvc/activemq/tags/activemq-5.4.2/activemq-core/src/main/java/org/apache/activemq/broker/region/cursors/FilePendingMessageCursor.java?view=markup>for free space. Note that the thread has the messagesLock locked and the
>> temporaly usage is always 103%, and therefore, it never returns.
>>
>> Is it possible that I encountered something like a deadlock? Perhaps, I
>> messed up something in the configuration.
>>
>> I will be thankful for advice.
>> Tomas
>>
>> Here is the configuration of the broker:
>>
>> <beans
>> xmlns="http://www.**springframework.org/schema/**beans<http://www.springframework.org/schema/beans>
>> "
>> xmlns:amq="http://activemq.**apache.org/schema/core<http://activemq.apache.org/schema/core>
>> "
>> xmlns:xsi="http://www.w3.org/**2001/XMLSchema-instance<http://www.w3.org/2001/XMLSchema-instance>
>> "
>> xsi:schemaLocation="http://**www.springframework.org/**schema/beans<http://www.springframework.org/schema/beans>
>> http://www.springframework.**org/schema/beans/spring-beans-**2.0.xsd<http://www.springframework.org/schema/beans/spring-beans-2.0.xsd>
>> http://activemq.apache.org/**schema/core<http://activemq.apache.org/schema/core>
>> http://activemq.apache.org/**schema/core/activemq-core.xsd<http://activemq.apache.org/schema/core/activemq-core.xsd>
>> "**>
>>
>> <!-- Allows us to use system properties as variables in this
>> configuration file -->
>> <bean class="org.springframework.**beans.factory.config.**PropertyPlaceholderConfigurer"
>> />
>>
>> <!--
>> The <broker> element is used to configure the ActiveMQ broker.
>>
>> advisories incurr a hit with every add connection|destination|**
>> producer|consumer
>> - client side: providerUrl = tcp://localhost:61616?jms.**
>> watchTopicAdvisories=false
>> statistics have a small impact on concurrency so disable
>> schedualer has a small impact as it checks for the presence three
>> properties on each message
>> jmx adds a bunch of wrapper classes
>>
>> ACTIVEMQ_OPTS_MEMORY="-Xmx11g -Xms6g -Dorg.apache.activemq.**UseDedicatedTaskRunner=false
>> -Djava.util.logging.config.**file=logging.properties -XX:+UseLargePages"
>> -->
>> <broker xmlns="http://activemq.apache.**org/schema/core<http://activemq.apache.org/schema/core>"
>> brokerName="localhost"
>> dataDirectory="/dasd/martinec/**kahadb-storage/activemq-data"
>> destroyApplicationContextOnSto**p="true"
>> deleteAllMessagesOnStartup="**true"
>> advisorySupport="false"
>> enableStatistics="false"
>> schedulerSupport="false"
>> useJmx="true"
>> persistent="true">
>>
>> <destinationPolicy>
>> <policyMap>
>> <policyEntries>
>> <!--
>> producerFlowControll thread unnecessary with no
>> memory limits
>> with no failover, no need to suppress duplicates so
>> audit can be disabled
>> message expiry thread unnecessary as normal dispatch
>> will handle expiry
>> -->
>> <policyEntry topic=">" producerFlowControl="false"
>> enableAudit="false" expireMessagesPeriod="0"></**policyEntry>
>> <policyEntry queue=">" producerFlowControl="true"
>> memoryLimit="50mb" enableAudit="false" expireMessagesPeriod="0">
>>
>> </policyEntry>
>> </policyEntries>
>> </policyMap>
>> </destinationPolicy>
>>
>> <!--
>> The managementContext is used to configure how ActiveMQ is
>> exposed in
>> JMX. By default, ActiveMQ uses the MBean server that is
>> started by
>> the JVM. For more information, see:
>>
>> http://activemq.apache.org/**jmx.html<http://activemq.apache.org/jmx.html>
>> -->
>> <managementContext>
>> <managementContext createConnector="true"
>> connectorPort="1399" rmiServerPort="1399"/>
>> </managementContext>
>>
>> <persistenceAdapter>
>> <kahaPersistenceAdapter directory="/dasd/martinec/**
>> kahadb-storage"/>
>> </persistenceAdapter>
>>
>> <systemUsage>
>> <systemUsage>
>> <memoryUsage>
>> <!-- the only queue has 50 mb, so this should be
>> enough -->
>> <memoryUsage limit="200 mb"/>
>> </memoryUsage>
>> <storeUsage>
>> <!-- no special reason for this -->
>> <storeUsage limit="1 gb"/>
>> </storeUsage>
>> <tempUsage>
>> <!-- this MUST be above the journal file size; see
>> http://tmielke.blogspot.cz/**2011/02/observations-on-**
>> activemqs-temp-storage.html<http://tmielke.blogspot.cz/2011/02/observations-on-activemqs-temp-storage.html>
>> -->
>> <tempUsage limit="100 mb"/>
>> </tempUsage>
>> </systemUsage>
>> </systemUsage>
>>
>> <!--
>> The transport connectors expose ActiveMQ over a given
>> protocol to
>> clients and other brokers. For more information, see:
>>
>> http://activemq.apache.org/**configuring-transports.html<http://activemq.apache.org/configuring-transports.html>
>> -->
>> <transportConnectors>
>> <transportConnector name="openwire" uri="tcp://0.0.0.0:61616
>> "/>
>> </transportConnectors>
>> </broker>
>>
>> <import resource="jetty.xml"/>
>>
>> </beans>
>>
>
>
>
> --
> *Christian Posta*
> http://www.christianposta.com/blog
> twitter: @christianposta
>
>
--
*Christian Posta*
http://www.christianposta.com/blog
twitter: @christianposta
Re: Activemq 5.4.2 hangs when the temp disk usage is used
Posted by Christian Posta <ch...@gmail.com>.
Interesting. Your scenario sounds like PFC is blocking the connection. Not
sure yet why that would happen. Your config looks okay. Do you still have
the thread dumps?
Are the producer/consumer on the same connection? If you have a test case
you can post, please do. Would make it easy to figure what the problem is.
On Tue, Dec 4, 2012 at 2:01 PM, Tomáš Martinec <
tomas.martinec@d3s.mff.cuni.cz> wrote:
> Hello,
>
> I am trying to setup activemq 5.4.2 and I experience hangs when the broker
> fills its heap and starts using storage. In my scenario I have one message
> producer and one message consumer connected. The producer is blocked and
> the consumer does not receive any messages. I have been able to run the
> broker under debug mode and identify what the relevant threads are doing:
>
> The task for iteration of the queue waits for the messagesLock on the line
> 1404 in http://svn.apache.org/viewvc/**activemq/tags/activemq-5.4.2/**
> activemq-core/src/main/java/**org/apache/activemq/broker/**
> region/Queue.java?view=markup<http://svn.apache.org/viewvc/activemq/tags/activemq-5.4.2/activemq-core/src/main/java/org/apache/activemq/broker/region/Queue.java?view=markup>.
>
>
> The thread that stores messages into the queue waits on the line 219 of
> http://svn.apache.org/viewvc/**activemq/tags/activemq-5.4.2/**
> activemq-core/src/main/java/**org/apache/activemq/broker/**region/cursors/
> **FilePendingMessageCursor.java?**view=markup<http://svn.apache.org/viewvc/activemq/tags/activemq-5.4.2/activemq-core/src/main/java/org/apache/activemq/broker/region/cursors/FilePendingMessageCursor.java?view=markup>for free space. Note that the thread has the messagesLock locked and the
> temporaly usage is always 103%, and therefore, it never returns.
>
> Is it possible that I encountered something like a deadlock? Perhaps, I
> messed up something in the configuration.
>
> I will be thankful for advice.
> Tomas
>
> Here is the configuration of the broker:
>
> <beans
> xmlns="http://www.**springframework.org/schema/**beans<http://www.springframework.org/schema/beans>
> "
> xmlns:amq="http://activemq.**apache.org/schema/core<http://activemq.apache.org/schema/core>
> "
> xmlns:xsi="http://www.w3.org/**2001/XMLSchema-instance<http://www.w3.org/2001/XMLSchema-instance>
> "
> xsi:schemaLocation="http://**www.springframework.org/**schema/beans<http://www.springframework.org/schema/beans>
> http://www.springframework.**org/schema/beans/spring-beans-**2.0.xsd<http://www.springframework.org/schema/beans/spring-beans-2.0.xsd>
> http://activemq.apache.org/**schema/core<http://activemq.apache.org/schema/core>
> http://activemq.apache.org/**schema/core/activemq-core.xsd<http://activemq.apache.org/schema/core/activemq-core.xsd>
> "**>
>
> <!-- Allows us to use system properties as variables in this
> configuration file -->
> <bean class="org.springframework.**beans.factory.config.**PropertyPlaceholderConfigurer"
> />
>
> <!--
> The <broker> element is used to configure the ActiveMQ broker.
>
> advisories incurr a hit with every add connection|destination|**
> producer|consumer
> - client side: providerUrl = tcp://localhost:61616?jms.**
> watchTopicAdvisories=false
> statistics have a small impact on concurrency so disable
> schedualer has a small impact as it checks for the presence three
> properties on each message
> jmx adds a bunch of wrapper classes
>
> ACTIVEMQ_OPTS_MEMORY="-Xmx11g -Xms6g -Dorg.apache.activemq.**UseDedicatedTaskRunner=false
> -Djava.util.logging.config.**file=logging.properties -XX:+UseLargePages"
> -->
> <broker xmlns="http://activemq.apache.**org/schema/core<http://activemq.apache.org/schema/core>"
> brokerName="localhost"
> dataDirectory="/dasd/martinec/**kahadb-storage/activemq-data"
> destroyApplicationContextOnSto**p="true"
> deleteAllMessagesOnStartup="**true"
> advisorySupport="false"
> enableStatistics="false"
> schedulerSupport="false"
> useJmx="true"
> persistent="true">
>
> <destinationPolicy>
> <policyMap>
> <policyEntries>
> <!--
> producerFlowControll thread unnecessary with no memory
> limits
> with no failover, no need to suppress duplicates so
> audit can be disabled
> message expiry thread unnecessary as normal dispatch
> will handle expiry
> -->
> <policyEntry topic=">" producerFlowControl="false"
> enableAudit="false" expireMessagesPeriod="0"></**policyEntry>
> <policyEntry queue=">" producerFlowControl="true"
> memoryLimit="50mb" enableAudit="false" expireMessagesPeriod="0">
>
> </policyEntry>
> </policyEntries>
> </policyMap>
> </destinationPolicy>
>
> <!--
> The managementContext is used to configure how ActiveMQ is
> exposed in
> JMX. By default, ActiveMQ uses the MBean server that is
> started by
> the JVM. For more information, see:
>
> http://activemq.apache.org/**jmx.html<http://activemq.apache.org/jmx.html>
> -->
> <managementContext>
> <managementContext createConnector="true" connectorPort="1399"
> rmiServerPort="1399"/>
> </managementContext>
>
> <persistenceAdapter>
> <kahaPersistenceAdapter directory="/dasd/martinec/**
> kahadb-storage"/>
> </persistenceAdapter>
>
> <systemUsage>
> <systemUsage>
> <memoryUsage>
> <!-- the only queue has 50 mb, so this should be
> enough -->
> <memoryUsage limit="200 mb"/>
> </memoryUsage>
> <storeUsage>
> <!-- no special reason for this -->
> <storeUsage limit="1 gb"/>
> </storeUsage>
> <tempUsage>
> <!-- this MUST be above the journal file size; see
> http://tmielke.blogspot.cz/**2011/02/observations-on-**
> activemqs-temp-storage.html<http://tmielke.blogspot.cz/2011/02/observations-on-activemqs-temp-storage.html>
> -->
> <tempUsage limit="100 mb"/>
> </tempUsage>
> </systemUsage>
> </systemUsage>
>
> <!--
> The transport connectors expose ActiveMQ over a given protocol
> to
> clients and other brokers. For more information, see:
>
> http://activemq.apache.org/**configuring-transports.html<http://activemq.apache.org/configuring-transports.html>
> -->
> <transportConnectors>
> <transportConnector name="openwire" uri="tcp://0.0.0.0:61616
> "/>
> </transportConnectors>
> </broker>
>
> <import resource="jetty.xml"/>
>
> </beans>
>
--
*Christian Posta*
http://www.christianposta.com/blog
twitter: @christianposta