You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@activemq.apache.org by Ishitori <Is...@bk.ru> on 2013/02/25 14:45:19 UTC
OutOfMemoryError due to TopicSubscription
Hi,
Recently our live AMQ 5.7.0 server on Windows experienced OutOfMemory
exception. I've read the article
http://blog.garytully.com/2009/07/apache-activemq-out-of-memory.html that
states that it is usually a problem of configuration. I also read FAQ page
http://activemq.apache.org/javalangoutofmemory.html to understand what kind
of configuration parameters should I tune.
But what I can't understand is why TopicSubscription is mentioned in the
log? I don't use topics at all, I use queues only. But the log message
states that TopicSubscription was the reason I got the exception:
2013-02-22 00:27:09,330 | INFO | TopicSubscription:
consumer=ID:AMQ1-49531-1360656683385-98:2:1:1, destinations=7, dispatched=1,
de
livered=16458, matched=51393, discarded=0: Pending message cursor
[org.apache.activemq.broker.region.cursors.VMPendingMessageCursor@13e97dd]
is full, temp usage (59%) or memory usage (101%) limit reached, blocking
message add() pending the release of resources. | org.apache.activ
emq.broker.region.TopicSubscription | ActiveMQ NIO Worker 168983
So, I can guess that cursor got to the limit of memory and stopped
processing. But why is it TopicSubscription? If it is a queue then what is
the name of that queue? And am I right that I should increase <memoryUsage
limit="64 mb"/> to solve the problem in future?
--
View this message in context: http://activemq.2283324.n4.nabble.com/OutOfMemoryError-due-to-TopicSubscription-tp4664025.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.
Re: OutOfMemoryError due to TopicSubscription
Posted by Ishitori <Is...@bk.ru>.
Thanks for the help! I would re-read these articles. But could you draft the
worthwhile configuration for me? Is it enough to just increase <memoryUsage
limit="> up to 128 mb? Or the whole configuration is wrong?
--
View this message in context: http://activemq.2283324.n4.nabble.com/OutOfMemoryError-due-to-TopicSubscription-tp4664025p4664128.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.
Re: OutOfMemoryError due to TopicSubscription
Posted by Christian Posta <ch...@gmail.com>.
bridge per destination is unnecessary. especially if you're using the
default mem options which are much to low. also, why are you altering the
prefetch settings on the bridge and setting to 1? looks like someone wasn't
very sure about the config options that were chosen... you're using default
memory configs, unnecessarily limiting the bridge prefetch, producer flow
control is false, etc, etc, etc. this config is ripe for OOM.
It would be very worthwhile to understand all of the config settings from
the following wiki docs and make very calculated decisions about which ones
to change:
http://activemq.apache.org/message-cursors.html
http://activemq.apache.org/javalangoutofmemory.html
http://activemq.apache.org/producer-flow-control.html
http://activemq.apache.org/networks-of-brokers.html
hope that helps..
On Mon, Feb 25, 2013 at 1:34 PM, Ishitori <Is...@bk.ru> wrote:
> I am not sure why it uses a bridge to destination feature. Is there another
> way to configure it? (sysadmin did it that way, not me)
>
> But I wonder is 6 bridges is really a high number? How many resources does
> it take to support a separate advisory topic subs? Here is the full config:
>
> <beans
> xmlns="http://www.springframework.org/schema/beans"
> xmlns:amq="http://activemq.apache.org/schema/core"
> xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
> xsi:schemaLocation="http://www.springframework.org/schema/beans
> http://www.springframework.org/schema/beans/spring-beans-2.0.xsd
> http://activemq.apache.org/schema/core
> http://activemq.apache.org/schema/core/activemq-core.xsd">
>
> <bean
>
> class="org.springframework.beans.factory.config.PropertyPlaceholderConfigurer">
> <property name="locations">
> <value>credentials_path</value>
> </property>
> </bean>
>
> <broker xmlns="http://activemq.apache.org/schema/core"
> brokerName="AMQ_Broker_1" useJmx="true"
> persistent="true" dataDirectory="<path>"
> useShutdownHook="true" schedulerSupport="true"
> schedulerDirectory="<scheduler_path>" >
>
> <destinationPolicy>
> <policyMap>
> <policyEntries>
> <policyEntry topic=">" producerFlowControl="false"
> memoryLimit="1mb">
> <pendingSubscriberPolicy>
> <vmCursor />
> </pendingSubscriberPolicy>
> </policyEntry>
> <policyEntry queue=">" producerFlowControl="false"
> memoryLimit="10mb" enableAudit="false">
> <networkBridgeFilterFactory>
> <conditionalNetworkBridgeFilterFactory
> replayWhenNoConsumers="true" />
> </networkBridgeFilterFactory>
> <deadLetterStrategy>
> <individualDeadLetterStrategy queuePrefix="DLQ."
> processExpired="false"/>
> </deadLetterStrategy>
> <pendingQueuePolicy>
> <fileQueueCursor/>
> </pendingQueuePolicy>
> </policyEntry>
> </policyEntries>
> </policyMap>
> </destinationPolicy>
>
> <managementContext>
> <managementContext createConnector="true" connectorHost="host"
> connectorPort="port" rmiServerPort="rmiPort"/>
> </managementContext>
>
> <networkConnectors>
> <networkConnector
> name="TopicsOnly-duplex"
> uri="static:(nio://ip)"
> networkTTL="2"
> prefetchSize="1"
> duplex="true">
> <dynamicallyIncludedDestinations>
> <topic physicalName=">"/>
> </dynamicallyIncludedDestinations>
> </networkConnector>
>
> <networkConnector
> name="queue1-duplex"
> uri="static:(nio://ip)"
> networkTTL="2"
> prefetchSize="1"
> duplex="true"
> decreaseNetworkConsumerPriority = "true"
> suppressDuplicateQueueSubscriptions = "true"
> conduitSubscriptions="false">
> <dynamicallyIncludedDestinations>
> <queue physicalName="queue1"/>
> </dynamicallyIncludedDestinations>
> </networkConnector>
>
> <networkConnector
> name="queue2-duplex"
> uri="static:(nio://ip)"
> networkTTL="2"
> prefetchSize="1"
> duplex="true"
> decreaseNetworkConsumerPriority = "true"
> suppressDuplicateQueueSubscriptions = "true"
> conduitSubscriptions="false">
> <dynamicallyIncludedDestinations>
> <queue physicalName="queue2"/>
> </dynamicallyIncludedDestinations>
> </networkConnector>
>
> <networkConnector
> name="queue3-duplex"
> uri="static:(nio://ip)"
> networkTTL="2"
> prefetchSize="1"
> duplex="true"
> decreaseNetworkConsumerPriority = "true"
> suppressDuplicateQueueSubscriptions = "true"
> conduitSubscriptions="false">
> <dynamicallyIncludedDestinations>
> <queue physicalName="queue3"/>
> </dynamicallyIncludedDestinations>
> </networkConnector>
>
> <networkConnector
> name="queue4-duplex"
> uri="static:(nio://ip)"
> networkTTL="2"
> prefetchSize="1"
> duplex="true"
> decreaseNetworkConsumerPriority = "true"
> suppressDuplicateQueueSubscriptions = "true"
> conduitSubscriptions="false">
> <dynamicallyIncludedDestinations>
> <queue physicalName="queue4"/>
> </dynamicallyIncludedDestinations>
> </networkConnector>
>
> <networkConnector
> name="queue5-duplex"
> uri="static:(nio://ip)"
> networkTTL="2"
> prefetchSize="1"
> duplex="true"
> decreaseNetworkConsumerPriority = "true"
> suppressDuplicateQueueSubscriptions = "true"
> conduitSubscriptions="false">
> <dynamicallyIncludedDestinations>
> <queue physicalName="queue5"/>
> </dynamicallyIncludedDestinations>
> </networkConnector>
> </networkConnectors>
>
> <persistenceAdapter>
> <kahaDB directory="path"/>
> </persistenceAdapter>
>
> <systemUsage>
> <systemUsage>
> <memoryUsage>
> <memoryUsage limit="64 mb"/>
> </memoryUsage>
> <storeUsage>
> <storeUsage limit="45 gb"/>
> </storeUsage>
> <tempUsage>
> <tempUsage limit="2 gb"/>
> </tempUsage>
> </systemUsage>
> </systemUsage>
>
> <transportConnectors>
> <transportConnector name="nio" uri="nio://ip"
> auditNetworkProducers="true"/>
> </transportConnectors>
>
> </broker>
> <import resource="jetty.xml"/>
>
> </beans>
>
>
>
> --
> View this message in context:
> http://activemq.2283324.n4.nabble.com/OutOfMemoryError-due-to-TopicSubscription-tp4664025p4664054.html
> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>
--
*Christian Posta*
http://www.christianposta.com/blog
twitter: @christianposta
Re: OutOfMemoryError due to TopicSubscription
Posted by Ishitori <Is...@bk.ru>.
I am not sure why it uses a bridge to destination feature. Is there another
way to configure it? (sysadmin did it that way, not me)
But I wonder is 6 bridges is really a high number? How many resources does
it take to support a separate advisory topic subs? Here is the full config:
<beans
xmlns="http://www.springframework.org/schema/beans"
xmlns:amq="http://activemq.apache.org/schema/core"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://www.springframework.org/schema/beans
http://www.springframework.org/schema/beans/spring-beans-2.0.xsd
http://activemq.apache.org/schema/core
http://activemq.apache.org/schema/core/activemq-core.xsd">
<bean
class="org.springframework.beans.factory.config.PropertyPlaceholderConfigurer">
<property name="locations">
<value>credentials_path</value>
</property>
</bean>
<broker xmlns="http://activemq.apache.org/schema/core"
brokerName="AMQ_Broker_1" useJmx="true"
persistent="true" dataDirectory="<path>"
useShutdownHook="true" schedulerSupport="true"
schedulerDirectory="<scheduler_path>" >
<destinationPolicy>
<policyMap>
<policyEntries>
<policyEntry topic=">" producerFlowControl="false"
memoryLimit="1mb">
<pendingSubscriberPolicy>
<vmCursor />
</pendingSubscriberPolicy>
</policyEntry>
<policyEntry queue=">" producerFlowControl="false"
memoryLimit="10mb" enableAudit="false">
<networkBridgeFilterFactory>
<conditionalNetworkBridgeFilterFactory
replayWhenNoConsumers="true" />
</networkBridgeFilterFactory>
<deadLetterStrategy>
<individualDeadLetterStrategy queuePrefix="DLQ."
processExpired="false"/>
</deadLetterStrategy>
<pendingQueuePolicy>
<fileQueueCursor/>
</pendingQueuePolicy>
</policyEntry>
</policyEntries>
</policyMap>
</destinationPolicy>
<managementContext>
<managementContext createConnector="true" connectorHost="host"
connectorPort="port" rmiServerPort="rmiPort"/>
</managementContext>
<networkConnectors>
<networkConnector
name="TopicsOnly-duplex"
uri="static:(nio://ip)"
networkTTL="2"
prefetchSize="1"
duplex="true">
<dynamicallyIncludedDestinations>
<topic physicalName=">"/>
</dynamicallyIncludedDestinations>
</networkConnector>
<networkConnector
name="queue1-duplex"
uri="static:(nio://ip)"
networkTTL="2"
prefetchSize="1"
duplex="true"
decreaseNetworkConsumerPriority = "true"
suppressDuplicateQueueSubscriptions = "true"
conduitSubscriptions="false">
<dynamicallyIncludedDestinations>
<queue physicalName="queue1"/>
</dynamicallyIncludedDestinations>
</networkConnector>
<networkConnector
name="queue2-duplex"
uri="static:(nio://ip)"
networkTTL="2"
prefetchSize="1"
duplex="true"
decreaseNetworkConsumerPriority = "true"
suppressDuplicateQueueSubscriptions = "true"
conduitSubscriptions="false">
<dynamicallyIncludedDestinations>
<queue physicalName="queue2"/>
</dynamicallyIncludedDestinations>
</networkConnector>
<networkConnector
name="queue3-duplex"
uri="static:(nio://ip)"
networkTTL="2"
prefetchSize="1"
duplex="true"
decreaseNetworkConsumerPriority = "true"
suppressDuplicateQueueSubscriptions = "true"
conduitSubscriptions="false">
<dynamicallyIncludedDestinations>
<queue physicalName="queue3"/>
</dynamicallyIncludedDestinations>
</networkConnector>
<networkConnector
name="queue4-duplex"
uri="static:(nio://ip)"
networkTTL="2"
prefetchSize="1"
duplex="true"
decreaseNetworkConsumerPriority = "true"
suppressDuplicateQueueSubscriptions = "true"
conduitSubscriptions="false">
<dynamicallyIncludedDestinations>
<queue physicalName="queue4"/>
</dynamicallyIncludedDestinations>
</networkConnector>
<networkConnector
name="queue5-duplex"
uri="static:(nio://ip)"
networkTTL="2"
prefetchSize="1"
duplex="true"
decreaseNetworkConsumerPriority = "true"
suppressDuplicateQueueSubscriptions = "true"
conduitSubscriptions="false">
<dynamicallyIncludedDestinations>
<queue physicalName="queue5"/>
</dynamicallyIncludedDestinations>
</networkConnector>
</networkConnectors>
<persistenceAdapter>
<kahaDB directory="path"/>
</persistenceAdapter>
<systemUsage>
<systemUsage>
<memoryUsage>
<memoryUsage limit="64 mb"/>
</memoryUsage>
<storeUsage>
<storeUsage limit="45 gb"/>
</storeUsage>
<tempUsage>
<tempUsage limit="2 gb"/>
</tempUsage>
</systemUsage>
</systemUsage>
<transportConnectors>
<transportConnector name="nio" uri="nio://ip"
auditNetworkProducers="true"/>
</transportConnectors>
</broker>
<import resource="jetty.xml"/>
</beans>
--
View this message in context: http://activemq.2283324.n4.nabble.com/OutOfMemoryError-due-to-TopicSubscription-tp4664025p4664054.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.
Re: OutOfMemoryError due to TopicSubscription
Posted by Christian Posta <ch...@gmail.com>.
The network of brokers relies on advisory topics. Any good reason why you
have a separate bridge per destination?
My *guess* is you're using default config with default memory limits, and
since you have a lot of network bridges, thus separate advisory topic subs,
you're running out of memory.
If you post your complete config, we can see.
On Mon, Feb 25, 2013 at 9:45 AM, Ishitori <Is...@bk.ru> wrote:
> I don't use advisory topics, but I do use network of brokers. There are 2
> brokers and they are connected by network connector per queue basis. Here
> is
> the example:
>
> <networkConnector
> name="TopicsOnly-duplex"
> uri="static:(nio://<ip>:62626)"
> networkTTL="2"
> prefetchSize="1"
> duplex="true">
> <dynamicallyIncludedDestinations>
> <topic physicalName=">"/>
> </dynamicallyIncludedDestinations>
> </networkConnector>
>
> <networkConnector
> name="SpecificQueue-duplex"
> uri="static:(nio://<ip>:62626)"
> networkTTL="2"
> prefetchSize="1"
> duplex="true"
> decreaseNetworkConsumerPriority = "true"
> suppressDuplicateQueueSubscriptions = "true"
> conduitSubscriptions="false">
> <dynamicallyIncludedDestinations>
> <queue physicalName="<queue_name_here>"/>
> </dynamicallyIncludedDestinations>
> </networkConnector>
>
> And there are more networkConnector for each queue I have with same
> parameters. About 5 of them.
>
>
>
> --
> View this message in context:
> http://activemq.2283324.n4.nabble.com/OutOfMemoryError-due-to-TopicSubscription-tp4664025p4664043.html
> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>
--
*Christian Posta*
http://www.christianposta.com/blog
twitter: @christianposta
Re: OutOfMemoryError due to TopicSubscription
Posted by Ishitori <Is...@bk.ru>.
I don't use advisory topics, but I do use network of brokers. There are 2
brokers and they are connected by network connector per queue basis. Here is
the example:
<networkConnector
name="TopicsOnly-duplex"
uri="static:(nio://<ip>:62626)"
networkTTL="2"
prefetchSize="1"
duplex="true">
<dynamicallyIncludedDestinations>
<topic physicalName=">"/>
</dynamicallyIncludedDestinations>
</networkConnector>
<networkConnector
name="SpecificQueue-duplex"
uri="static:(nio://<ip>:62626)"
networkTTL="2"
prefetchSize="1"
duplex="true"
decreaseNetworkConsumerPriority = "true"
suppressDuplicateQueueSubscriptions = "true"
conduitSubscriptions="false">
<dynamicallyIncludedDestinations>
<queue physicalName="<queue_name_here>"/>
</dynamicallyIncludedDestinations>
</networkConnector>
And there are more networkConnector for each queue I have with same
parameters. About 5 of them.
--
View this message in context: http://activemq.2283324.n4.nabble.com/OutOfMemoryError-due-to-TopicSubscription-tp4664025p4664043.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.
Re: OutOfMemoryError due to TopicSubscription
Posted by Christian Posta <ch...@gmail.com>.
Maybe you can post your complete config file? Are you using network of
brokers? Or using advisory topics?
On Mon, Feb 25, 2013 at 6:45 AM, Ishitori <Is...@bk.ru> wrote:
> Hi,
>
> Recently our live AMQ 5.7.0 server on Windows experienced OutOfMemory
> exception. I've read the article
> http://blog.garytully.com/2009/07/apache-activemq-out-of-memory.html that
> states that it is usually a problem of configuration. I also read FAQ page
> http://activemq.apache.org/javalangoutofmemory.html to understand what
> kind
> of configuration parameters should I tune.
>
> But what I can't understand is why TopicSubscription is mentioned in the
> log? I don't use topics at all, I use queues only. But the log message
> states that TopicSubscription was the reason I got the exception:
>
> 2013-02-22 00:27:09,330 | INFO | TopicSubscription:
> consumer=ID:AMQ1-49531-1360656683385-98:2:1:1, destinations=7,
> dispatched=1,
> de
> livered=16458, matched=51393, discarded=0: Pending message cursor
> [org.apache.activemq.broker.region.cursors.VMPendingMessageCursor@13e97dd]
> is full, temp usage (59%) or memory usage (101%) limit reached, blocking
> message add() pending the release of resources. | org.apache.activ
> emq.broker.region.TopicSubscription | ActiveMQ NIO Worker 168983
>
> So, I can guess that cursor got to the limit of memory and stopped
> processing. But why is it TopicSubscription? If it is a queue then what is
> the name of that queue? And am I right that I should increase <memoryUsage
> limit="64 mb"/> to solve the problem in future?
>
>
>
>
>
> --
> View this message in context:
> http://activemq.2283324.n4.nabble.com/OutOfMemoryError-due-to-TopicSubscription-tp4664025.html
> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>
--
*Christian Posta*
http://www.christianposta.com/blog
twitter: @christianposta