You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@activemq.apache.org by Jim Alateras <ji...@comware.com.au> on 2009/03/17 14:21:29 UTC
Large number of messages in the queue
I have around 60K messages in one of my queues that I want to process.
The problem is that after around 10 minutes I get an
OutOfMemoryError. I get a number of exceptions such as
2009-03-17 23:37:29,354 [VMTransport] ERROR - Failed to page in more
queue messages
java.lang.OutOfMemoryError: Java heap space
or
java.lang.OutOfMemoryError: Java heap space
at
java
.lang.AbstractStringBuilder.expandCapacity(AbstractStringBuilder.java:
99)
at java.lang.AbstractStringBuilder.append(AbstractStringBuilder.java:
393)
at java.lang.StringBuffer.append(StringBuffer.java:225)
at java.io.StringWriter.write(StringWriter.java:90)
at java.io.PrintWriter.write(PrintWriter.java:384)
at java.io.PrintWriter.write(PrintWriter.java:401)
at java.io.PrintWriter.print(PrintWriter.java:532)
at java.io.PrintWriter.println(PrintWriter.java:669)
at java.lang.Throwable.printStackTrace(Throwable.java:513)
at
org
.apache
.derby.iapi.services.context.ErrorStringBuilder.stackTrace(Unknown
Source)
at
org
.apache
.derby.iapi.services.context.ContextManager.cleanupOnError(Unknown
Source)
at
org
.apache
.derby.impl.jdbc.TransactionResourceImpl.handleException(Unknown Source)
at org.apache.derby.impl.jdbc.EmbedConnection.handleException(Unknown
Source)
at org.apache.derby.impl.jdbc.ConnectionChild.handleException(Unknown
Source)
at org.apache.derby.impl.jdbc.EmbedStatement.executeStatement(Unknown
Source)
at
org
.apache
.derby.impl.jdbc.EmbedPreparedStatement.executeStatement(Unknown Source)
at
org.apache.derby.impl.jdbc.EmbedPreparedStatement.executeQuery(Unknown
Source)
at
org
.apache
.commons
.dbcp
.DelegatingPreparedStatement
.executeQuery(DelegatingPreparedStatement.java:92)
at
org
.apache
.activemq
.store
.jdbc
.adapter
.DefaultJDBCAdapter.doRecoverNextMessages(DefaultJDBCAdapter.java:724)
at
org
.apache
.activemq
.store.jdbc.JDBCMessageStore.recoverNextMessages(JDBCMessageStore.java:
214)
at
org
.apache
.activemq
.store.ProxyMessageStore.recoverNextMessages(ProxyMessageStore.java:87)
at
org
.apache
.activemq
.broker
.region.cursors.QueueStorePrefetch.doFillBatch(QueueStorePrefetch.java:
84)
at
org
.apache
.activemq
.broker
.region.cursors.AbstractStoreCursor.fillBatch(AbstractStoreCursor.java:
236)
at
org
.apache
.activemq
.broker
.region.cursors.AbstractStoreCursor.hasNext(AbstractStoreCursor.java:
136)
at
org
.apache
.activemq
.broker.region.cursors.StoreQueueCursor.hasNext(StoreQueueCursor.java:
131)
at org.apache.activemq.broker.region.Queue.doPageIn(Queue.java:1252)
at org.apache.activemq.broker.region.Queue.pageInMessages(Queue.java:
1391)
at org.apache.activemq.broker.region.Queue.iterate(Queue.java:1075)
at
org
.apache
.activemq
.thread.DeterministicTaskRunner.runTask(DeterministicTaskRunner.java:84)
at org.apache.activemq.thread.DeterministicTaskRunner
$1.run(DeterministicTaskRunner.java:41)
at java.util.concurrent.ThreadPoolExecutor
$Worker.runTask(ThreadPoolExecutor.java:650)
at java.util.concurrent.ThreadPoolExecutor
$Worker.run(ThreadPoolExecutor.java:675)
depending on my configuration. What is see is that the inflight count
tends to always increase, which is consuming memory and was wondering
under what circumstances it gets capped.
Here is my broker configuration. Any advice would be much appreciated.
<bean id="broker"
class="org.apache.activemq.xbean.XBeanBrokerService" lazy-init="true">
<property name="brokerName" value="localhost" />
<property name="advisorySupport" value="false" />
<property name="systemUsage">
<bean id="systemUsage"
class="org.apache.activemq.usage.SystemUsage">
<property name="sendFailIfNoSpace" value="true" />
<property name="memoryUsage">
<bean id="memoryUsage"
class="org.apache.activemq.usage.MemoryUsage">
<property name="limit" value="10000000" />
</bean>
</property>
<property name="tempUsage">
<bean id="tempUsage"
class="org.apache.activemq.usage.TempUsage">
<property name="limit" value="50000000" />
</bean>
</property>
</bean>
</property>
<property name="destinationPolicy">
<bean class="org.apache.activemq.broker.region.policy.PolicyMap">
<property name="defaultEntry">
<bean
class="org.apache.activemq.broker.region.policy.PolicyEntry">
<property name="destination">
<bean class="org.apache.activemq.command.ActiveMQQueue">
<constructor-arg index="0" value="*" />
</bean>
</property>
<property name="deadLetterStrategy">
<bean
class
=
"au
.com
.observant.ringocore.dispatcher.activemq.DefaultDeadLetterStrategy" />
</property>
<property name="producerFlowControl" value="false" />
<property name="enableAudit" value="false" />
</bean>
</property>
</bean>
</property>
<property name="persistenceAdapter" ref="persistenceAdaptor" />
<!-- we don't want to start it yet, we may have to change
persistenceAdaptor -->
<property name="start" value="false" />
</bean>
cheers
</jima>
Re: Large number of messages in the queue
Posted by Andreas Gies <ag...@progress.com>.
Hi there,
are there any consumers consuming messages from that queue? - Can they
keep up with the producer speed?
If there are 60k msg in the queue you seen to encounter at least a
peak in messaging traffic.
It would be worth to check the memory settings of your JVM.
If the no of inflight messages is rising - what is the acknowledgment
strategy of your clients? Inflight messages have been delivered to
the client side, but have not been acknowledged yet.
It would also be helpful if you could let us know which version
version you are using.
Best regards
Andreas
On Mar 17, 2009, at 2:21 PM, Jim Alateras wrote:
> I have around 60K messages in one of my queues that I want to
> process. The problem is that after around 10 minutes I get an
> OutOfMemoryError. I get a number of exceptions such as
>
> 2009-03-17 23:37:29,354 [VMTransport] ERROR - Failed to page in more
> queue messages
> java.lang.OutOfMemoryError: Java heap space
>
>
> or
>
> java.lang.OutOfMemoryError: Java heap space
> at
> java
> .lang
> .AbstractStringBuilder.expandCapacity(AbstractStringBuilder.java:99)
> at
> java.lang.AbstractStringBuilder.append(AbstractStringBuilder.java:393)
> at java.lang.StringBuffer.append(StringBuffer.java:225)
> at java.io.StringWriter.write(StringWriter.java:90)
> at java.io.PrintWriter.write(PrintWriter.java:384)
> at java.io.PrintWriter.write(PrintWriter.java:401)
> at java.io.PrintWriter.print(PrintWriter.java:532)
> at java.io.PrintWriter.println(PrintWriter.java:669)
> at java.lang.Throwable.printStackTrace(Throwable.java:513)
> at
> org
> .apache
> .derby.iapi.services.context.ErrorStringBuilder.stackTrace(Unknown
> Source)
> at
> org
> .apache
> .derby.iapi.services.context.ContextManager.cleanupOnError(Unknown
> Source)
> at
> org
> .apache
> .derby.impl.jdbc.TransactionResourceImpl.handleException(Unknown
> Source)
> at
> org.apache.derby.impl.jdbc.EmbedConnection.handleException(Unknown
> Source)
> at
> org.apache.derby.impl.jdbc.ConnectionChild.handleException(Unknown
> Source)
> at
> org.apache.derby.impl.jdbc.EmbedStatement.executeStatement(Unknown
> Source)
> at
> org
> .apache
> .derby.impl.jdbc.EmbedPreparedStatement.executeStatement(Unknown
> Source)
> at
> org
> .apache.derby.impl.jdbc.EmbedPreparedStatement.executeQuery(Unknown
> Source)
> at
> org
> .apache
> .commons
> .dbcp
> .DelegatingPreparedStatement
> .executeQuery(DelegatingPreparedStatement.java:92)
> at
> org
> .apache
> .activemq
> .store
> .jdbc
> .adapter
> .DefaultJDBCAdapter.doRecoverNextMessages(DefaultJDBCAdapter.java:724)
> at
> org
> .apache
> .activemq
> .store
> .jdbc.JDBCMessageStore.recoverNextMessages(JDBCMessageStore.java:214)
> at
> org
> .apache
> .activemq
> .store.ProxyMessageStore.recoverNextMessages(ProxyMessageStore.java:
> 87)
> at
> org
> .apache
> .activemq
> .broker
> .region
> .cursors.QueueStorePrefetch.doFillBatch(QueueStorePrefetch.java:84)
> at
> org
> .apache
> .activemq
> .broker
> .region
> .cursors.AbstractStoreCursor.fillBatch(AbstractStoreCursor.java:236)
> at
> org
> .apache
> .activemq
> .broker
> .region.cursors.AbstractStoreCursor.hasNext(AbstractStoreCursor.java:
> 136)
> at
> org
> .apache
> .activemq
> .broker
> .region.cursors.StoreQueueCursor.hasNext(StoreQueueCursor.java:131)
> at org.apache.activemq.broker.region.Queue.doPageIn(Queue.java:1252)
> at
> org.apache.activemq.broker.region.Queue.pageInMessages(Queue.java:
> 1391)
> at org.apache.activemq.broker.region.Queue.iterate(Queue.java:1075)
> at
> org
> .apache
> .activemq
> .thread.DeterministicTaskRunner.runTask(DeterministicTaskRunner.java:
> 84)
> at org.apache.activemq.thread.DeterministicTaskRunner
> $1.run(DeterministicTaskRunner.java:41)
> at java.util.concurrent.ThreadPoolExecutor
> $Worker.runTask(ThreadPoolExecutor.java:650)
> at java.util.concurrent.ThreadPoolExecutor
> $Worker.run(ThreadPoolExecutor.java:675)
>
>
> depending on my configuration. What is see is that the inflight
> count tends to always increase, which is consuming memory and was
> wondering under what circumstances it gets capped.
>
> Here is my broker configuration. Any advice would be much appreciated.
>
> <bean id="broker"
> class="org.apache.activemq.xbean.XBeanBrokerService" lazy-init="true">
> <property name="brokerName" value="localhost" />
> <property name="advisorySupport" value="false" />
> <property name="systemUsage">
> <bean id="systemUsage"
> class="org.apache.activemq.usage.SystemUsage">
> <property name="sendFailIfNoSpace" value="true" />
> <property name="memoryUsage">
> <bean id="memoryUsage"
> class="org.apache.activemq.usage.MemoryUsage">
> <property name="limit" value="10000000" />
> </bean>
> </property>
> <property name="tempUsage">
> <bean id="tempUsage"
> class="org.apache.activemq.usage.TempUsage">
> <property name="limit" value="50000000" />
> </bean>
> </property>
> </bean>
> </property>
> <property name="destinationPolicy">
> <bean class="org.apache.activemq.broker.region.policy.PolicyMap">
> <property name="defaultEntry">
> <bean
> class="org.apache.activemq.broker.region.policy.PolicyEntry">
> <property name="destination">
> <bean class="org.apache.activemq.command.ActiveMQQueue">
> <constructor-arg index="0" value="*" />
> </bean>
> </property>
> <property name="deadLetterStrategy">
> <bean
> class
> =
> "au
> .com
> .observant.ringocore.dispatcher.activemq.DefaultDeadLetterStrategy" />
> </property>
> <property name="producerFlowControl" value="false" />
> <property name="enableAudit" value="false" />
> </bean>
> </property>
> </bean>
> </property>
> <property name="persistenceAdapter" ref="persistenceAdaptor" />
> <!-- we don't want to start it yet, we may have to change
> persistenceAdaptor -->
> <property name="start" value="false" />
> </bean>
>
> cheers
> </jima>
>
>
>
---
Mit freundlichen Grüssen - Kind Regards
Andreas Gies
Principal Consultant
Open Source Center of Competence
Progress Software GmbH
Agrippinawerft 26
50678 Köln
E-Mail agies@progress.com
Direct Line +49 (0)9953 980349
Mobile +49 (0)170 5759611
Skype +44 (0)20 3239 2922
Skype +353 (0)1 443 4971
Skype +1 (0)781 262 0168
http://www.progress.com
http://fusesource.com
http://open-source-adventures.blogspot.com
-------------------------------------------------------
Progress Software GmbH
Sitz der Gesellschaft: Agrippinawerft 26, 50678 Koeln;
Niederlassung: Fuerstenrieder Str. 279, 81377 Muenchen
Amtsgericht Koeln, HRB 15620;
Geschaeftsfuehrung: David Ireland
-------------------------------------------------------