You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@activemq.apache.org by "Gary Tully (JIRA)" <ji...@apache.org> on 2016/03/02 12:23:18 UTC
[jira] [Commented] (AMQ-5773) Memory leak in JDBC Message Store
[ https://issues.apache.org/jira/browse/AMQ-5773?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15175454#comment-15175454 ]
Gary Tully commented on AMQ-5773:
---------------------------------
I think this is resolved by https://git-wip-us.apache.org/repos/asf?p=activemq.git;h=7c11663
it was introduced via - https://issues.apache.org/jira/browse/AMQ-5266
> Memory leak in JDBC Message Store
> ---------------------------------
>
> Key: AMQ-5773
> URL: https://issues.apache.org/jira/browse/AMQ-5773
> Project: ActiveMQ
> Issue Type: Bug
> Components: JDBC
> Affects Versions: 5.11.1
> Environment: $ java -version
> java version "1.8.0_20"
> Java(TM) SE Runtime Environment (build 1.8.0_40-b26)
> Java HotSpot(TM) 64-Bit Server VM (build 25.20-b23, mixed mode)
> $ uname -a
> Linux mint 3.13.0-24-generic #46-Ubuntu SMP Thu Apr 10 19:11:08 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
> Reporter: Dmitry
>
> Hello. Recently I've faced with weird problem, actually I'm not sure if it's a bug or misconfiguration or something else. But since my broker has "almost" default configuration I think this is bug.
> Brief configuration description(full configuration xml file at the end of issue):
> - Persistent storage: MySQL Db
> - Message type: only persistent Text Messages
> - Subscribers: only durable topic subscribers
> So, when I started broker I monitored its cpu/memory usage and here is what I noticed, memory chart below:(for servers monitoring I'm using New Relic):
> [Heap Memory Chart | http://s4.postimg.org/fjiwg1v0d/amq_memory_leak_chart.png]
> As you can see usage of memory slowly but constantly growing.
> I decided to find out what exactly stored in activemq's heap, so I dumped heap and analyzed it with Eclipse MAT. Screenshots below:
> [Leak Analysis | http://s4.postimg.org/nqaw7ml31/amq_leak_suspect.png]
> As we can see almost all of the memory occupied in JournalTopicMessageStore. Going deeper:
> [First heapdump | http://s4.postimg.org/p6r03iddp/amq_problem_suspect_00_00.png]
> We can see *completions* list in TransactionContext, which has size of 38590 objects. This heap dump I made at 00:00.
> Below is a heap dump that I made 4 hours later:
> [Second heapdump | http://s4.postimg.org/e8fqlbosd/amq_problem_suspect_04_00.png]
> The picture is the same, but now this list contains 63770 objecs.
> Analyzed code a little bit, I noticed that this list is never cleared and hence its size always growth. Runnables that are stored in this list have link to Message object and hence this message is never will be GC'ed, even after ActiveMQ puts it to db. Of course I don't know probably this is a regular behavior that objects from this list are never removed, but it's looks really strange that object that contains this list is never recreated for 10 hours.
> Below I paste configuration of my broker:
> {code:title="broker configuration"|collapse}
> <beans
> xmlns="http://www.springframework.org/schema/beans"
> xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
> xsi:schemaLocation="http://www.springframework.org/schema/beans http://www.springframework.org/schema/beans/spring-beans.xsd
> http://activemq.apache.org/schema/core http://activemq.apache.org/schema/core/activemq-core.xsd">
> <!-- Allows us to use system properties as variables in this configuration file -->
> <bean class="org.springframework.beans.factory.config.PropertyPlaceholderConfigurer">
> <property name="locations">
> <value>file:${activemq.conf}/credentials.properties</value>
> </property>
> </bean>
> <!-- Allows accessing the server log -->
> <bean id="logQuery" class="io.fabric8.insight.log.log4j.Log4jLogQuery"
> lazy-init="false" scope="singleton"
> init-method="start" destroy-method="stop">
> </bean>
> <!--
> The <broker> element is used to configure the ActiveMQ broker.
> -->
> <broker xmlns="http://activemq.apache.org/schema/core"
> brokerName="broker"
> persistent="true"
> schedulerSupport="true"
> advisorySupport="false"
> dataDirectory="${activemq.data}">
> <destinationPolicy>
> <policyMap>
> <policyEntries>
> <policyEntry topic=">" producerFlowControl="false" expireMessagesPeriod="0">
> <deadLetterStrategy>
> <sharedDeadLetterStrategy processExpired="false" processNonPersistent="false"/>
> </deadLetterStrategy>
> </policyEntry>
> <policyEntry queue=">" optimizedDispatch="true" producerFlowControl="false">
> <deadLetterStrategy>
> <sharedDeadLetterStrategy processExpired="false" processNonPersistent="false"/>
> </deadLetterStrategy>
> </policyEntry>
> </policyEntries>
> </policyMap>
> </destinationPolicy>
> <!--
> The managementContext is used to configure how ActiveMQ is exposed in
> JMX. By default, ActiveMQ uses the MBean server that is started by
> the JVM. For more information, see:
> http://activemq.apache.org/jmx.html
> -->
> <managementContext>
> <managementContext createConnector="false"/>
> </managementContext>
> <!--
> Configure message persistence for the broker. The default persistence
> mechanism is MySQL.
> For more information, see:
> http://activemq.apache.org/persistence.html
> -->
> <persistenceFactory>
> <journalPersistenceAdapterFactory
> journalLogFiles="8"
> journalLogFileSize="268435456"
> useJournal="true"
> useQuickJournal="false"
> dataSource="#mysql-ds"
> dataDirectory="${activemq.data}"/>
> </persistenceFactory>
> <!--
> The systemUsage controls the maximum amount of space the broker will
> use before disabling caching and/or slowing down producers. For more information, see:
> http://activemq.apache.org/producer-flow-control.html
> -->
> <systemUsage>
> <systemUsage>
> <memoryUsage>
> <memoryUsage limit="64 mb"/>
> </memoryUsage>
> <storeUsage>
> <storeUsage limit="100 gb"/>
> </storeUsage>
> <tempUsage>
> <tempUsage limit="50 gb"/>
> </tempUsage>
> </systemUsage>
> </systemUsage>
> <!--
> The transport connectors expose ActiveMQ over a given protocol to
> clients and other brokers. For more information, see:
> http://activemq.apache.org/configuring-transports.html
> -->
> <transportConnectors>
> <!-- DOS protection, limit concurrent connections to 1000 and frame size to 100MB -->
> <transportConnector name="default" uri="nio://activemq.lc:61616?maximumConnections=1000&connectionTimeout=600000&wireFormat.maxFrameSize=104857600"/>
> </transportConnectors>
> <!-- redelivery configuration-->
> <plugins>
> <loggingBrokerPlugin logAll="true"/>
> <redeliveryPlugin sendToDlqIfMaxRetriesExceeded="true">
> <redeliveryPolicyMap>
> <redeliveryPolicyMap>
> <defaultEntry>
> <redeliveryPolicy maximumRedeliveries="4" initialRedeliveryDelay="5000" redeliveryDelay="10000" useExponentialBackOff="true" />
> </defaultEntry>
> </redeliveryPolicyMap>
> </redeliveryPolicyMap>
> </redeliveryPlugin>
> </plugins>
> <!-- destroy the spring context on shutdown to stop jetty -->
> <shutdownHooks>
> <bean xmlns="http://www.springframework.org/schema/beans" class="org.apache.activemq.hooks.SpringContextHook"/>
> </shutdownHooks>
> </broker>
> <bean id="mysql-ds"
> class="org.apache.commons.dbcp.BasicDataSource"
> destroy-method="close">
> <property name="driverClassName" value="com.mysql.jdbc.Driver"/>
> <property name="url" value="jdbc:mysql://localhost:3306/activemq?relaxAutoCommit=true&autoReconnect=true"/>
> <property name="username" value="root"/>
> <property name="password" value=""/>
> <property name="poolPreparedStatements" value="true"/>
> <property name="maxActive" value="40"/>
> </bean>
> <!--
> Enable web consoles, REST and Ajax APIs and demos
> The web consoles requires by default login, you can disable this in the jetty.xml file
> Take a look at ${ACTIVEMQ_HOME}/conf/jetty.xml for more details
> -->
> <import resource="jetty.xml"/>
> </beans>
> {code}
> Please help me fiund out is it configuration problem or it is really a bug, because I suffer with this problem for about 2 weeks.
> Thanks!
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)