You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@activemq.apache.org by "art.licis" <ar...@gmail.com> on 2018/07/19 14:13:21 UTC
ActiveMQ 5.11.0: Out Of Memory, caused by PageFile object tree
filling up heap
Dear community,
We've been evaluation AMQ which is based on version 5.11.0. Several tests in
staging environments which would have a big number of producers and
consumers ended up with OutOfMemory: GC Overhead Limit exceeded exception.
However, it looks like it's just one slow consumer that is causing this
memory problem (details below).
Initially, we tried running AMQ with 4Gb of max heap, later with 8Gb but it
had the same outcome.
Configuration: we have intentionally producer flow control turned off, and
expecting temp store to fill up before the producer is blocked
(non-persistent topic messages). I can simulate this with a simple and
controlled tests involving very fast producer, and several fast and some
slow consumers. With manual testing we achieve the desired result: when temp
store reaches the limit, the producer is blocked.
We analyzed GC outputs and heap dump which we got upon the crash. Time spent
in GC keeps significantly increasing before the crash, and heap dump shows
the majority of memory occupied by PageFile object tree (lots of
PageFile$PageWrite objects, which indicate that too much data remains in
memory uncommitted to temp store; also related DataFileAppender$WriteBatch).
PageFile is the correct one, the one used in PListStoreImpl (i.e. temp store
for non-persistent messages).
During my analysis I also discovered some bug that was fixed recently -
*AMQ-6815*. It looks like it affects the exact class family that could have
caused the memory leak.
I'd be grateful for any help or suggestions. I'm currently preparing the
bigger test environment with many producers/consumers (and I'm also gonna
run Activemq in docker to limit I/O speed while I'm sure it wasn't the
issue), but the fact is that out of potentially thousands of
producers/consumers there was only one slow consumer. Eventually, we will
try running whole system against latest ActiveMQ, however, it will take lots
of arrangements and I'd like to see if I can make reproducable and
controlled test environment, and if there're some pitfalls in current setup.
activemq.xml is below (the slow consumer was not the one listening to
virtual topic). And I will post more info if I'm able to solve the issue or
at least get more hints.
Thank you.
-Art
<beans
xmlns="http://www.springframework.org/schema/beans"
xmlns:amq="http://activemq.apache.org/schema/core"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://www.springframework.org/schema/beans
http://www.springframework.org/schema/beans/spring-beans.xsd
http://activemq.apache.org/schema/core
http://activemq.apache.org/schema/core/activemq-core.xsd">
<bean
class="org.springframework.beans.factory.config.PropertyPlaceholderConfigurer">
<property name="properties">
<bean class="io.fabric8.mq.fabric.ConfigurationProperties"/>
</property>
</bean>
<broker xmlns="http://activemq.apache.org/schema/core"
brokerName="${broker-name}" dataDirectory="${data}" start="false"
restartAllowed="false">
<destinationPolicy>
<policyMap>
<policyEntries>
<policyEntry topic=">" producerFlowControl="false">
</policyEntry>
<policyEntry queue=">" producerFlowControl="false"
optimizedDispatch="true" >
</policyEntry>
</policyEntries>
</policyMap>
</destinationPolicy>
<managementContext>
<managementContext createConnector="true" connectorPort="9001"/>
</managementContext>
<ioExceptionHandler>
<defaultIOExceptionHandler ignoreNoSpaceErrors="false"/>
</ioExceptionHandler>
<persistenceAdapter>
<kahaDB directory="${data}/kahadb"/>
</persistenceAdapter>
<plugins>
</plugins>
<systemUsage>
<systemUsage>
<memoryUsage>
<memoryUsage limit="300 mb"/>
</memoryUsage>
<storeUsage>
<storeUsage limit="100 gb"/>
</storeUsage>
<tempUsage>
<tempUsage limit="10 gb"/>
</tempUsage>
</systemUsage>
</systemUsage>
<transportConnectors>
<transportConnector name="openwire"
uri="tcp://0.0.0.0:0?maximumConnections=1000&wireFormat.maxFrameSize=104857600"/>
</transportConnectors>
<destinationInterceptors>
<virtualDestinationInterceptor>
<virtualDestinations>
<compositeQueue name="rpc.request.>"
forwardOnly="false">
<forwardTo>
<topic
physicalName="rpc.request.virtual.notifications" />
</forwardTo>
</compositeQueue>
<compositeQueue name="rpc.response.>"
forwardOnly="false">
<forwardTo>
<topic
physicalName="rpc.response.virtual.notifications" />
</forwardTo>
</compositeQueue>
</virtualDestinations>
</virtualDestinationInterceptor>
</destinationInterceptors>
</broker>
</beans>
--
Sent from: http://activemq.2283324.n4.nabble.com/ActiveMQ-User-f2341805.html
Re: ActiveMQ 5.11.0: Out Of Memory, caused by PageFile object tree
filling up heap
Posted by Tom Hall <th...@concentricsky.com>.
Hi Art,
What per destination limit are you looking for? The per destination limits will deal with memory used per queue.
http://activemq.apache.org/per-destination-policies.html <http://activemq.apache.org/per-destination-policies.html>
<policyEntry queue=“queue" producerFlowControl="true" memoryLimit="100 kb">
-Tom
> On Jul 20, 2018, at 3:50 AM, art.licis <ar...@gmail.com> wrote:
>
> 300mb memory limit is on a VM having -XmX 4Gb for heap; on 8Gb (most likely)
> it was more. While I totally agree 300 Mb memory limit is far from the
> optimal setting, and even better per-destination limits are needed.
>
> However, I have stress tested ActiveMQ with much less memory, and I've seen
> the following in logs:
>
> topic://topic-3, Usage Manager memory
> limit reached 5242880. Producers will be throttled to the rate at which
> messages are removed from this destination to prevent flooding it.
>
> << just 5Mb of memory limit on this test!
>
> .. and eventually this:
>
> TopicSubscription: consumer=ID:a-thinkpad-44715-1532031318503-1:1:1:1,
> destinations=1, dispatched=32766, delivered=15963293, matched=1460,
> discarded=0, prefetchExtension=0: Pending message cursor
> [org.apache.activemq.broker.region.cursors.FilePendingMessageCursor@6bba423c]
> is full, temp usag (100%) or memory usage (0%) limit reached, blocking
> message add() pending the release of resources.
>
> << all 10Gb of temp store filled!
>
>
> Which means filling up temp store and blocking the producer, but NOT
> crashing with out of memory, having all heap filled with PageFile-related
> data.
>
> - Art
>
>
>
> --
> Sent from: http://activemq.2283324.n4.nabble.com/ActiveMQ-User-f2341805.html
Re: ActiveMQ 5.11.0: Out Of Memory, caused by PageFile object tree
filling up heap
Posted by "art.licis" <ar...@gmail.com>.
300mb memory limit is on a VM having -XmX 4Gb for heap; on 8Gb (most likely)
it was more. While I totally agree 300 Mb memory limit is far from the
optimal setting, and even better per-destination limits are needed.
However, I have stress tested ActiveMQ with much less memory, and I've seen
the following in logs:
topic://topic-3, Usage Manager memory
limit reached 5242880. Producers will be throttled to the rate at which
messages are removed from this destination to prevent flooding it.
<< just 5Mb of memory limit on this test!
.. and eventually this:
TopicSubscription: consumer=ID:a-thinkpad-44715-1532031318503-1:1:1:1,
destinations=1, dispatched=32766, delivered=15963293, matched=1460,
discarded=0, prefetchExtension=0: Pending message cursor
[org.apache.activemq.broker.region.cursors.FilePendingMessageCursor@6bba423c]
is full, temp usag (100%) or memory usage (0%) limit reached, blocking
message add() pending the release of resources.
<< all 10Gb of temp store filled!
Which means filling up temp store and blocking the producer, but NOT
crashing with out of memory, having all heap filled with PageFile-related
data.
- Art
--
Sent from: http://activemq.2283324.n4.nabble.com/ActiveMQ-User-f2341805.html
Re: ActiveMQ 5.11.0: Out Of Memory, caused by PageFile object tree
filling up heap
Posted by Tom Hall <th...@concentricsky.com>.
Are you really limiting ram to 300 mb? Why so low if you are on a 8G box?
Thanks,
Tom
> On Jul 19, 2018, at 7:13 AM, art.licis <ar...@gmail.com> wrote:
>
> Dear community,
>
> We've been evaluation AMQ which is based on version 5.11.0. Several tests in
> staging environments which would have a big number of producers and
> consumers ended up with OutOfMemory: GC Overhead Limit exceeded exception.
> However, it looks like it's just one slow consumer that is causing this
> memory problem (details below).
>
> Initially, we tried running AMQ with 4Gb of max heap, later with 8Gb but it
> had the same outcome.
>
> Configuration: we have intentionally producer flow control turned off, and
> expecting temp store to fill up before the producer is blocked
> (non-persistent topic messages). I can simulate this with a simple and
> controlled tests involving very fast producer, and several fast and some
> slow consumers. With manual testing we achieve the desired result: when temp
> store reaches the limit, the producer is blocked.
>
> We analyzed GC outputs and heap dump which we got upon the crash. Time spent
> in GC keeps significantly increasing before the crash, and heap dump shows
> the majority of memory occupied by PageFile object tree (lots of
> PageFile$PageWrite objects, which indicate that too much data remains in
> memory uncommitted to temp store; also related DataFileAppender$WriteBatch).
> PageFile is the correct one, the one used in PListStoreImpl (i.e. temp store
> for non-persistent messages).
>
> During my analysis I also discovered some bug that was fixed recently -
> *AMQ-6815*. It looks like it affects the exact class family that could have
> caused the memory leak.
>
> I'd be grateful for any help or suggestions. I'm currently preparing the
> bigger test environment with many producers/consumers (and I'm also gonna
> run Activemq in docker to limit I/O speed while I'm sure it wasn't the
> issue), but the fact is that out of potentially thousands of
> producers/consumers there was only one slow consumer. Eventually, we will
> try running whole system against latest ActiveMQ, however, it will take lots
> of arrangements and I'd like to see if I can make reproducable and
> controlled test environment, and if there're some pitfalls in current setup.
>
> activemq.xml is below (the slow consumer was not the one listening to
> virtual topic). And I will post more info if I'm able to solve the issue or
> at least get more hints.
>
> Thank you.
> -Art
>
> <beans
> xmlns="http://www.springframework.org/schema/beans"
> xmlns:amq="http://activemq.apache.org/schema/core"
> xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
> xsi:schemaLocation="http://www.springframework.org/schema/beans
> http://www.springframework.org/schema/beans/spring-beans.xsd
> http://activemq.apache.org/schema/core
> http://activemq.apache.org/schema/core/activemq-core.xsd">
>
>
> <bean
> class="org.springframework.beans.factory.config.PropertyPlaceholderConfigurer">
> <property name="properties">
> <bean class="io.fabric8.mq.fabric.ConfigurationProperties"/>
> </property>
> </bean>
>
> <broker xmlns="http://activemq.apache.org/schema/core"
> brokerName="${broker-name}" dataDirectory="${data}" start="false"
> restartAllowed="false">
>
> <destinationPolicy>
> <policyMap>
> <policyEntries>
> <policyEntry topic=">" producerFlowControl="false">
> </policyEntry>
> <policyEntry queue=">" producerFlowControl="false"
> optimizedDispatch="true" >
> </policyEntry>
> </policyEntries>
> </policyMap>
> </destinationPolicy>
>
> <managementContext>
> <managementContext createConnector="true" connectorPort="9001"/>
> </managementContext>
>
> <ioExceptionHandler>
> <defaultIOExceptionHandler ignoreNoSpaceErrors="false"/>
> </ioExceptionHandler>
>
> <persistenceAdapter>
> <kahaDB directory="${data}/kahadb"/>
> </persistenceAdapter>
>
> <plugins>
> </plugins>
>
> <systemUsage>
> <systemUsage>
> <memoryUsage>
> <memoryUsage limit="300 mb"/>
> </memoryUsage>
> <storeUsage>
> <storeUsage limit="100 gb"/>
> </storeUsage>
> <tempUsage>
> <tempUsage limit="10 gb"/>
> </tempUsage>
> </systemUsage>
> </systemUsage>
>
> <transportConnectors>
> <transportConnector name="openwire"
> uri="tcp://0.0.0.0:0?maximumConnections=1000&wireFormat.maxFrameSize=104857600"/>
> </transportConnectors>
>
> <destinationInterceptors>
> <virtualDestinationInterceptor>
> <virtualDestinations>
> <compositeQueue name="rpc.request.>"
> forwardOnly="false">
> <forwardTo>
> <topic
> physicalName="rpc.request.virtual.notifications" />
> </forwardTo>
> </compositeQueue>
> <compositeQueue name="rpc.response.>"
> forwardOnly="false">
> <forwardTo>
> <topic
> physicalName="rpc.response.virtual.notifications" />
> </forwardTo>
> </compositeQueue>
> </virtualDestinations>
> </virtualDestinationInterceptor>
> </destinationInterceptors>
>
> </broker>
>
> </beans>
>
>
>
> --
> Sent from: http://activemq.2283324.n4.nabble.com/ActiveMQ-User-f2341805.html