You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@activemq.apache.org by Denary <de...@gmail.com> on 2015/05/15 10:17:54 UTC
Memory leak in JDBC Message Store
Hello. Recently I've faced with weird problem, actually I'm not sure if it's
a bug or misconfiguration or something else. But since my broker has
"almost" default configuration I think this is bug.
Brief configuration description(full configuration xml file at the end of
issue):
- Persistent storage: MySQL Db
- Message type: only persistent Text Messages
- Subscribers: only durable topic subscribers
So, when I started broker I monitored its cpu/memory usage and here is what
I noticed, memory chart below:(for servers monitoring I'm using New Relic):
<http://activemq.2283324.n4.nabble.com/file/n4696558/amq_memory_leak_chart.png>
As you can see usage of memory slowly but constantly growing.
I decided to find out what exactly stored in activemq's heap, so I dumped
heap and analyzed it with Eclipse MAT. Screenshots below:
http://s4.postimg.org/nqaw7ml31/amq_leak_suspect.png
<http://s4.postimg.org/nqaw7ml31/amq_leak_suspect.png>
As we can see almost all of the memory occupied in JournalTopicMessageStore.
Going deeper:
http://s4.postimg.org/p6r03iddp/amq_problem_suspect_00_00.png
<http://s4.postimg.org/p6r03iddp/amq_problem_suspect_00_00.png>
We can see *completions* list in TransactionContext, which has size of 38590
objects. This heap dump I made at 00:00.
Below is a heap dump that I made 4 hours later:
http://s4.postimg.org/e8fqlbosd/amq_problem_suspect_04_00.png
<http://s4.postimg.org/e8fqlbosd/amq_problem_suspect_04_00.png>
The picture is the same, but now this list contains 63770 objecs.
Analyzed code a little bit, I noticed that this list is never cleared and
hence its size always growth. Runnables that are stored in this list have
link to Message object and hence this message is never will be GC'ed, even
after ActiveMQ puts it to db. Of course I don't know probably this is a
regular behavior that objects from this list are never removed, but it's
looks really strange that object that contains this list is never recreated
for 10 hours.
Below I paste configuration of my broker:
{code:title="broker configuration"|collapse}
<beans
xmlns="http://www.springframework.org/schema/beans"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://www.springframework.org/schema/beans
http://www.springframework.org/schema/beans/spring-beans.xsd
http://activemq.apache.org/schema/core
http://activemq.apache.org/schema/core/activemq-core.xsd">
<bean
class="org.springframework.beans.factory.config.PropertyPlaceholderConfigurer">
<property name="locations">
<value>file:${activemq.conf}/credentials.properties</value>
</property>
</bean>
<bean id="logQuery" class="io.fabric8.insight.log.log4j.Log4jLogQuery"
lazy-init="false" scope="singleton"
init-method="start" destroy-method="stop">
</bean>
<broker xmlns="http://activemq.apache.org/schema/core"
brokerName="broker"
persistent="true"
schedulerSupport="true"
advisorySupport="false"
dataDirectory="${activemq.data}">
<destinationPolicy>
<policyMap>
<policyEntries>
<policyEntry topic=">" producerFlowControl="false"
expireMessagesPeriod="0">
<deadLetterStrategy>
<sharedDeadLetterStrategy processExpired="false"
processNonPersistent="false"/>
</deadLetterStrategy>
</policyEntry>
<policyEntry queue=">" optimizedDispatch="true"
producerFlowControl="false">
<deadLetterStrategy>
<sharedDeadLetterStrategy processExpired="false"
processNonPersistent="false"/>
</deadLetterStrategy>
</policyEntry>
</policyEntries>
</policyMap>
</destinationPolicy>
<managementContext>
<managementContext createConnector="false"/>
</managementContext>
<persistenceFactory>
<journalPersistenceAdapterFactory
journalLogFiles="8"
journalLogFileSize="268435456"
useJournal="true"
useQuickJournal="false"
dataSource="#mysql-ds"
dataDirectory="${activemq.data}"/>
</persistenceFactory>
<systemUsage>
<systemUsage>
<memoryUsage>
<memoryUsage limit="64 mb"/>
</memoryUsage>
<storeUsage>
<storeUsage limit="100 gb"/>
</storeUsage>
<tempUsage>
<tempUsage limit="50 gb"/>
</tempUsage>
</systemUsage>
</systemUsage>
<transportConnectors>
<transportConnector name="default"
uri="nio://activemq.lc:61616?maximumConnections=1000&connectionTimeout=600000&wireFormat.maxFrameSize=104857600"/>
</transportConnectors>
<plugins>
<loggingBrokerPlugin logAll="true"/>
<redeliveryPlugin sendToDlqIfMaxRetriesExceeded="true">
<redeliveryPolicyMap>
<redeliveryPolicyMap>
<defaultEntry>
<redeliveryPolicy maximumRedeliveries="4"
initialRedeliveryDelay="5000" redeliveryDelay="10000"
useExponentialBackOff="true" />
</defaultEntry>
</redeliveryPolicyMap>
</redeliveryPolicyMap>
</redeliveryPlugin>
</plugins>
<shutdownHooks>
<bean xmlns="http://www.springframework.org/schema/beans"
class="org.apache.activemq.hooks.SpringContextHook"/>
</shutdownHooks>
</broker>
<bean id="mysql-ds"
class="org.apache.commons.dbcp.BasicDataSource"
destroy-method="close">
<property name="driverClassName" value="com.mysql.jdbc.Driver"/>
<property name="url"
value="jdbc:mysql://localhost:3306/activemq?relaxAutoCommit=true&autoReconnect=true"/>
<property name="username" value="root"/>
<property name="password" value=""/>
<property name="poolPreparedStatements" value="true"/>
<property name="maxActive" value="40"/>
</bean>
<import resource="jetty.xml"/>
</beans>
{code}
Please help me fiund out is it configuration problem or it is really a bug,
because I suffer with this problem for about 2 weeks.
Thanks!
--
View this message in context: http://activemq.2283324.n4.nabble.com/Memory-leak-in-JDBC-Message-Store-tp4696558.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.
Re: Memory leak in JDBC Message Store
Posted by Gary Tully <ga...@gmail.com>.
did you try running without the journal? ie: replace
<persistenceFactory>
<journalPersistenceAdapterFactory> ...
with:
<persistenceAdapter>
<jdbcPersistenceAdapter> ....
On 15 May 2015 at 09:17, Denary <de...@gmail.com> wrote:
> Hello. Recently I've faced with weird problem, actually I'm not sure if it's
> a bug or misconfiguration or something else. But since my broker has
> "almost" default configuration I think this is bug.
> Brief configuration description(full configuration xml file at the end of
> issue):
> - Persistent storage: MySQL Db
> - Message type: only persistent Text Messages
> - Subscribers: only durable topic subscribers
>
> So, when I started broker I monitored its cpu/memory usage and here is what
> I noticed, memory chart below:(for servers monitoring I'm using New Relic):
> <http://activemq.2283324.n4.nabble.com/file/n4696558/amq_memory_leak_chart.png>
> As you can see usage of memory slowly but constantly growing.
> I decided to find out what exactly stored in activemq's heap, so I dumped
> heap and analyzed it with Eclipse MAT. Screenshots below:
> http://s4.postimg.org/nqaw7ml31/amq_leak_suspect.png
> <http://s4.postimg.org/nqaw7ml31/amq_leak_suspect.png>
> As we can see almost all of the memory occupied in JournalTopicMessageStore.
> Going deeper:
> http://s4.postimg.org/p6r03iddp/amq_problem_suspect_00_00.png
> <http://s4.postimg.org/p6r03iddp/amq_problem_suspect_00_00.png>
> We can see *completions* list in TransactionContext, which has size of 38590
> objects. This heap dump I made at 00:00.
> Below is a heap dump that I made 4 hours later:
> http://s4.postimg.org/e8fqlbosd/amq_problem_suspect_04_00.png
> <http://s4.postimg.org/e8fqlbosd/amq_problem_suspect_04_00.png>
>
> The picture is the same, but now this list contains 63770 objecs.
> Analyzed code a little bit, I noticed that this list is never cleared and
> hence its size always growth. Runnables that are stored in this list have
> link to Message object and hence this message is never will be GC'ed, even
> after ActiveMQ puts it to db. Of course I don't know probably this is a
> regular behavior that objects from this list are never removed, but it's
> looks really strange that object that contains this list is never recreated
> for 10 hours.
>
> Below I paste configuration of my broker:
> {code:title="broker configuration"|collapse}
> <beans
> xmlns="http://www.springframework.org/schema/beans"
> xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
> xsi:schemaLocation="http://www.springframework.org/schema/beans
> http://www.springframework.org/schema/beans/spring-beans.xsd
> http://activemq.apache.org/schema/core
> http://activemq.apache.org/schema/core/activemq-core.xsd">
>
>
> <bean
> class="org.springframework.beans.factory.config.PropertyPlaceholderConfigurer">
> <property name="locations">
> <value>file:${activemq.conf}/credentials.properties</value>
> </property>
> </bean>
>
>
> <bean id="logQuery" class="io.fabric8.insight.log.log4j.Log4jLogQuery"
> lazy-init="false" scope="singleton"
> init-method="start" destroy-method="stop">
> </bean>
>
>
> <broker xmlns="http://activemq.apache.org/schema/core"
> brokerName="broker"
> persistent="true"
> schedulerSupport="true"
> advisorySupport="false"
> dataDirectory="${activemq.data}">
>
> <destinationPolicy>
> <policyMap>
> <policyEntries>
> <policyEntry topic=">" producerFlowControl="false"
> expireMessagesPeriod="0">
> <deadLetterStrategy>
> <sharedDeadLetterStrategy processExpired="false"
> processNonPersistent="false"/>
> </deadLetterStrategy>
> </policyEntry>
> <policyEntry queue=">" optimizedDispatch="true"
> producerFlowControl="false">
> <deadLetterStrategy>
> <sharedDeadLetterStrategy processExpired="false"
> processNonPersistent="false"/>
> </deadLetterStrategy>
> </policyEntry>
> </policyEntries>
> </policyMap>
> </destinationPolicy>
>
>
>
> <managementContext>
> <managementContext createConnector="false"/>
> </managementContext>
>
>
> <persistenceFactory>
> <journalPersistenceAdapterFactory
> journalLogFiles="8"
> journalLogFileSize="268435456"
> useJournal="true"
> useQuickJournal="false"
> dataSource="#mysql-ds"
> dataDirectory="${activemq.data}"/>
> </persistenceFactory>
>
>
> <systemUsage>
> <systemUsage>
> <memoryUsage>
> <memoryUsage limit="64 mb"/>
> </memoryUsage>
> <storeUsage>
> <storeUsage limit="100 gb"/>
> </storeUsage>
> <tempUsage>
> <tempUsage limit="50 gb"/>
> </tempUsage>
> </systemUsage>
> </systemUsage>
>
>
> <transportConnectors>
>
> <transportConnector name="default"
> uri="nio://activemq.lc:61616?maximumConnections=1000&connectionTimeout=600000&wireFormat.maxFrameSize=104857600"/>
> </transportConnectors>
>
>
> <plugins>
> <loggingBrokerPlugin logAll="true"/>
> <redeliveryPlugin sendToDlqIfMaxRetriesExceeded="true">
> <redeliveryPolicyMap>
> <redeliveryPolicyMap>
> <defaultEntry>
> <redeliveryPolicy maximumRedeliveries="4"
> initialRedeliveryDelay="5000" redeliveryDelay="10000"
> useExponentialBackOff="true" />
> </defaultEntry>
> </redeliveryPolicyMap>
> </redeliveryPolicyMap>
> </redeliveryPlugin>
> </plugins>
>
>
> <shutdownHooks>
> <bean xmlns="http://www.springframework.org/schema/beans"
> class="org.apache.activemq.hooks.SpringContextHook"/>
> </shutdownHooks>
>
> </broker>
>
> <bean id="mysql-ds"
> class="org.apache.commons.dbcp.BasicDataSource"
> destroy-method="close">
> <property name="driverClassName" value="com.mysql.jdbc.Driver"/>
> <property name="url"
> value="jdbc:mysql://localhost:3306/activemq?relaxAutoCommit=true&autoReconnect=true"/>
> <property name="username" value="root"/>
> <property name="password" value=""/>
> <property name="poolPreparedStatements" value="true"/>
> <property name="maxActive" value="40"/>
> </bean>
>
>
> <import resource="jetty.xml"/>
>
> </beans>
> {code}
>
> Please help me fiund out is it configuration problem or it is really a bug,
> because I suffer with this problem for about 2 weeks.
> Thanks!
>
>
>
> --
> View this message in context: http://activemq.2283324.n4.nabble.com/Memory-leak-in-JDBC-Message-Store-tp4696558.html
> Sent from the ActiveMQ - User mailing list archive at Nabble.com.