You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@activemq.apache.org by kureckam <mk...@fractech.net> on 2012/10/29 18:04:12 UTC
Stopped waiting for space in pendingMessage cursor messages
For about 26 minutes I received approximately 500,000 of the following
messages:
2012-10-28 11:19:00,877 | WARN | TopicSubscription:
consumer=ID:CR-xxxx-49190-1351302775232-0:1:4:1, destinations=1,
dispatched=32766, delivered=2728701, matched=29318, discarded=0: stopped
waiting for space in pendingMessage cursor for:
ID:CR-xxxx-49177-1351302708839-0:1:10:1:2647613 |
org.apache.activemq.broker.region.TopicSubscription | ActiveMQ NIO Worker
What does this message mean and what caused it?
--
View this message in context: http://activemq.2283324.n4.nabble.com/Stopped-waiting-for-space-in-pendingMessage-cursor-messages-tp4658396.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.
Re: Stopped waiting for space in pendingMessage cursor messages
Posted by Torsten Mielke <to...@fusesource.com>.
On Oct 30, 2012, at 1:18 PM, kureckam wrote:
> Below is the config file. I don't have persistence set to true and the
> ActiveMQ heap size is 1024mb.
Sorry what exactly do you refer to when saying "don't have persistence set to true"? Do you refer to the persistence attribute on the broker xml element? If sets not set it defaults to true.
Your memoryUsage configuration is well beyond the JVMs max heap size. So you should not run out of memory because of keeping too many messages in the brokers memory. The rest of the broker config looks ok as well.
Do you by any chance have a large number of client connections to the broker? Each client connection results in two threads being created which obviously also requires a some amount of Java heap space.
I suggest you monitor your broker usage closely using jconsole and check for any unusual number (i.e. jvm heap size increasing over time, the Broker MBean statistics, destination statistic, number of subscriptions). One of these should probably indicate if and where there is a problem.
Regards,
Torsten Mielke
torsten@fusesource.com
tmielke.blogspot.com
Re: Stopped waiting for space in pendingMessage cursor messages
Posted by kureckam <mk...@fractech.net>.
Below is the config file. I don't have persistence set to true and the
ActiveMQ heap size is 1024mb.
<beans xmlns="http://www.springframework.org/schema/beans"
xmlns:amq="http://activemq.apache.org/schema/core"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://www.springframework.org/schema/beans
http://www.springframework.org/schema/beans/spring-beans-2.0.xsd
http://activemq.apache.org/schema/core
http://activemq.apache.org/schema/core/activemq-core.xsd">
<bean
class="org.springframework.beans.factory.config.PropertyPlaceholderConfigurer">
<property name="locations">
<value>file:${activemq.base}/conf/credentials.properties</value>
</property>
</bean>
<broker xmlns="http://activemq.apache.org/schema/core"
brokerName="pBroker"
dataDirectory="${activemq.base}/data">
<destinationPolicy>
<policyMap>
<policyEntries>
<policyEntry queue=">"
producerFlowControl="true"
optimizedDispatch="true" />
<policyEntry topic=">"
topicPrefetch="1000"
producerFlowControl="true"
advisoryForSlowConsumers="true"
advisoryForDiscardingMessages="true"
advisoryForConsumed="true"
advisoryWhenFull="true" />
</policyEntries>
</policyMap>
</destinationPolicy>
<managementContext>
<managementContext connectorPort="1099"
jmxDomainName="pDomain"/>
</managementContext>
<persistenceAdapter>
<kahaDB directory="${activemq.base}/data/pData"
journalMaxFileLength="3 mb" />
</persistenceAdapter>
<systemUsage>
<systemUsage sendFailIfNoSpace="true">
<memoryUsage>
<memoryUsage limit="600 mb"/>
</memoryUsage>
<storeUsage>
<storeUsage limit="500 mb" />
</storeUsage>
<tempUsage>
<tempUsage limit="500 mb"/>
</tempUsage>
</systemUsage>
</systemUsage>
<transportConnectors>
<transportConnector name="openwire_p"
uri="nio://0.0.0.0:61616" />
</transportConnectors>
</broker>
<broker xmlns="http://activemq.apache.org/schema/core"
brokerName="fBroker" dataDirectory="${activemq.base}/data">
<destinationPolicy>
<policyMap>
<policyEntries>
<policyEntry queue=">"
producerFlowControl="true"
optimizedDispatch="true" />
<policyEntry topic=">"
producerFlowControl="true" />
</policyEntries>
</policyMap>
</destinationPolicy>
<managementContext>
<managementContext connectorPort="1099"
jmxDomainName="fDomain"/>
</managementContext>
<networkConnectors>
<networkConnector
uri="static:(tcp://xxx.xxx.xxx.xxx:61616?wireFormat.maxInactivityDuration=3600000)"
name="fBridge"
conduitSubscriptions="false"
decreaseNetworkConsumerPriority="false"
duplex="true" />
</networkConnectors>
<persistenceAdapter>
<kahaDB directory="${activemq.base}/data/fData"
journalMaxFileLength="10 mb" />
</persistenceAdapter>
<systemUsage>
<systemUsage sendFailIfNoSpace="true">
<memoryUsage>
<memoryUsage limit="168 mb"/>
</memoryUsage>
<storeUsage>
<storeUsage limit="2 gb" />
</storeUsage>
<tempUsage>
<tempUsage limit="50 mb"/>
</tempUsage>
</systemUsage>
</systemUsage>
<transportConnectors>
<transportConnector name="openwire_f"
uri="nio://0.0.0.0:61617" />
</transportConnectors>
</broker>
<import resource="jetty.xml" />
</beans>
--
View this message in context: http://activemq.2283324.n4.nabble.com/Stopped-waiting-for-space-in-pendingMessage-cursor-messages-tp4658396p4658445.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.
Re: Stopped waiting for space in pendingMessage cursor messages
Posted by Torsten Mielke <to...@fusesource.com>.
If your broker JVM is running out of memory, then your broker configuration is most likely not appropriate for your JVM settings (JVM max heap size).
What JVM max heap size do you use and what is the <systemUsage> section configured for?
Your memoryUsage setting must be lower than the JVMs max heap size.
Regards,
Torsten Mielke
torsten@fusesource.com
tmielke@blogspot.com
On Oct 29, 2012, at 8:18 PM, kureckam wrote:
> I had this message 35 minutes prior to the flood of other messages. But why
> would it take 35 minutes for the other messages to occur after the OOM? We
> use non-durable topics so I don't believe temp memory would be used, would
> it? ActiveMQ has since been restarted so I can't check the JMX stats.
>
> java.lang.OutOfMemoryError: unable to create new native thread
> at java.lang.Thread.start0(Native Method)
> at java.lang.Thread.start(Unknown Source)
> at
> java.util.concurrent.ThreadPoolExecutor.addIfUnderMaximumPoolSize(Unknown
> Source)
> at java.util.concurrent.ThreadPoolExecutor.execute(Unknown Source)
> at
> org.apache.activemq.thread.TaskRunnerFactory.execute(TaskRunnerFactory.java:92)
> at
> org.apache.activemq.broker.TransportConnection.stopAsync(TransportConnection.java:947)
> at
> org.apache.activemq.broker.TransportConnection.serviceTransportException(TransportConnection.java:234)
> at
> org.apache.activemq.broker.TransportConnection$1.onException(TransportConnection.java:192)
> at
> org.apache.activemq.transport.TransportFilter.onException(TransportFilter.java:101)
> at
> org.apache.activemq.transport.TransportFilter.onException(TransportFilter.java:101)
> at
> org.apache.activemq.transport.WireFormatNegotiator.onException(WireFormatNegotiator.java:160)
> at
> org.apache.activemq.transport.InactivityMonitor.onException(InactivityMonitor.java:265)
> at
> org.apache.activemq.transport.TransportSupport.onException(TransportSupport.java:96)
> at
> org.apache.activemq.transport.nio.NIOTransport$1.onError(NIOTransport.java:75)
> at
> org.apache.activemq.transport.nio.SelectorSelection.onError(SelectorSelection.java:98)
> at
> org.apache.activemq.transport.nio.SelectorWorker.run(SelectorWorker.java:128)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown Source)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
> at java.lang.Thread.run(Unknown Source)
>
>
>
>
> --
> View this message in context: http://activemq.2283324.n4.nabble.com/Stopped-waiting-for-space-in-pendingMessage-cursor-messages-tp4658396p4658403.html
> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
Re: Stopped waiting for space in pendingMessage cursor messages
Posted by Christian Posta <ch...@gmail.com>.
Å FilePendingMessageCursor would be used by default if <brokerService ...
persistent="true" /> in your config.
This cursor would use temp space. What are the settings for the temp space?
I had this message 35 minutes prior to the flood of other messages. But why
> would it take 35 minutes for the other messages to occur after the OOM?
Can you explain what you mean here?
What version are you using?
On Mon, Oct 29, 2012 at 12:18 PM, kureckam <mk...@fractech.net> wrote:
> I had this message 35 minutes prior to the flood of other messages. But
> why
> would it take 35 minutes for the other messages to occur after the OOM? We
> use non-durable topics so I don't believe temp memory would be used, would
> it? ActiveMQ has since been restarted so I can't check the JMX stats.
>
> java.lang.OutOfMemoryError: unable to create new native thread
> at java.lang.Thread.start0(Native Method)
> at java.lang.Thread.start(Unknown Source)
> at
> java.util.concurrent.ThreadPoolExecutor.addIfUnderMaximumPoolSize(Unknown
> Source)
> at java.util.concurrent.ThreadPoolExecutor.execute(Unknown Source)
> at
>
> org.apache.activemq.thread.TaskRunnerFactory.execute(TaskRunnerFactory.java:92)
> at
>
> org.apache.activemq.broker.TransportConnection.stopAsync(TransportConnection.java:947)
> at
>
> org.apache.activemq.broker.TransportConnection.serviceTransportException(TransportConnection.java:234)
> at
>
> org.apache.activemq.broker.TransportConnection$1.onException(TransportConnection.java:192)
> at
>
> org.apache.activemq.transport.TransportFilter.onException(TransportFilter.java:101)
> at
>
> org.apache.activemq.transport.TransportFilter.onException(TransportFilter.java:101)
> at
>
> org.apache.activemq.transport.WireFormatNegotiator.onException(WireFormatNegotiator.java:160)
> at
>
> org.apache.activemq.transport.InactivityMonitor.onException(InactivityMonitor.java:265)
> at
>
> org.apache.activemq.transport.TransportSupport.onException(TransportSupport.java:96)
> at
>
> org.apache.activemq.transport.nio.NIOTransport$1.onError(NIOTransport.java:75)
> at
>
> org.apache.activemq.transport.nio.SelectorSelection.onError(SelectorSelection.java:98)
> at
>
> org.apache.activemq.transport.nio.SelectorWorker.run(SelectorWorker.java:128)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown
> Source)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown
> Source)
> at java.lang.Thread.run(Unknown Source)
>
>
>
>
> --
> View this message in context:
> http://activemq.2283324.n4.nabble.com/Stopped-waiting-for-space-in-pendingMessage-cursor-messages-tp4658396p4658403.html
> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>
--
*Christian Posta*
http://www.christianposta.com/blog
twitter: @christianposta
Re: Stopped waiting for space in pendingMessage cursor messages
Posted by kureckam <mk...@fractech.net>.
I had this message 35 minutes prior to the flood of other messages. But why
would it take 35 minutes for the other messages to occur after the OOM? We
use non-durable topics so I don't believe temp memory would be used, would
it? ActiveMQ has since been restarted so I can't check the JMX stats.
java.lang.OutOfMemoryError: unable to create new native thread
at java.lang.Thread.start0(Native Method)
at java.lang.Thread.start(Unknown Source)
at
java.util.concurrent.ThreadPoolExecutor.addIfUnderMaximumPoolSize(Unknown
Source)
at java.util.concurrent.ThreadPoolExecutor.execute(Unknown Source)
at
org.apache.activemq.thread.TaskRunnerFactory.execute(TaskRunnerFactory.java:92)
at
org.apache.activemq.broker.TransportConnection.stopAsync(TransportConnection.java:947)
at
org.apache.activemq.broker.TransportConnection.serviceTransportException(TransportConnection.java:234)
at
org.apache.activemq.broker.TransportConnection$1.onException(TransportConnection.java:192)
at
org.apache.activemq.transport.TransportFilter.onException(TransportFilter.java:101)
at
org.apache.activemq.transport.TransportFilter.onException(TransportFilter.java:101)
at
org.apache.activemq.transport.WireFormatNegotiator.onException(WireFormatNegotiator.java:160)
at
org.apache.activemq.transport.InactivityMonitor.onException(InactivityMonitor.java:265)
at
org.apache.activemq.transport.TransportSupport.onException(TransportSupport.java:96)
at
org.apache.activemq.transport.nio.NIOTransport$1.onError(NIOTransport.java:75)
at
org.apache.activemq.transport.nio.SelectorSelection.onError(SelectorSelection.java:98)
at
org.apache.activemq.transport.nio.SelectorWorker.run(SelectorWorker.java:128)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at java.lang.Thread.run(Unknown Source)
--
View this message in context: http://activemq.2283324.n4.nabble.com/Stopped-waiting-for-space-in-pendingMessage-cursor-messages-tp4658396p4658403.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.
Re: Stopped waiting for space in pendingMessage cursor messages
Posted by Torsten Mielke <to...@fusesource.com>.
Not 100%ly sure but did the cursor run out of memory and perhaps even temp space because of too many messages and slow subscribers? You may want to check JMX statistics of the broker and the topic in question if it hits any limits.
Regards,
Torsten Mielke
torsten@fusesource.com
tmielke@blogspot.com
On Oct 29, 2012, at 6:04 PM, kureckam wrote:
> For about 26 minutes I received approximately 500,000 of the following
> messages:
>
> 2012-10-28 11:19:00,877 | WARN | TopicSubscription:
> consumer=ID:CR-xxxx-49190-1351302775232-0:1:4:1, destinations=1,
> dispatched=32766, delivered=2728701, matched=29318, discarded=0: stopped
> waiting for space in pendingMessage cursor for:
> ID:CR-xxxx-49177-1351302708839-0:1:10:1:2647613 |
> org.apache.activemq.broker.region.TopicSubscription | ActiveMQ NIO Worker
>
> What does this message mean and what caused it?
>
>
>
> --
> View this message in context: http://activemq.2283324.n4.nabble.com/Stopped-waiting-for-space-in-pendingMessage-cursor-messages-tp4658396.html
> Sent from the ActiveMQ - User mailing list archive at Nabble.com.