You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@activemq.apache.org by victorbp <di...@hotmail.com> on 2008/10/06 18:33:17 UTC
ActiveMQ Blocked when memory is full
Hello!
I'm sending non persistent messages to queue and topics (non durable)
through ActiveMQ. I'm trying to use the spooling to disk when memory is full
but every time that I reach the memory limit, seems like the broker gets
blocked and I'm not able to send or receive more messages.
I can see how the broker works fine with a large number of messages without
reach the limit of memory (I use JConsole to see the memory percentage
usage), but if I increase the number of messages and the memory reach 100%
usage, I just see that the memory doesn't decrease, remains at 100% even if
I wait long time, seems like the consumers get blocked and they dont read
the messages.
I get this behaviour using VMCursors and FileCursors, the only difference is
that when I use FileCursors I can see some temp files created in the
tempDataStore directory.
What I understand about spooling to disk is that when the memory limit is
reached, the next messages will be saved in disk until the messages in
memory are read, but instead I'm getting the system blocked.
Here is my config. Does anyone have an idea about what is happening or if I
am missing some extra config?
Thanks a lot!
<bean id="brokerService"
class="org.apache.activemq.broker.BrokerService" init-method="start"
destroy-method="stop">
<property name="brokerName" value="localhost" />
<property name="persistent" value="false" />
<property name="useJmx" value="true" />
<property name="managementContext">
<bean
class="org.apache.activemq.broker.jmx.ManagementContext">
<property name="createMBeanServer" value="false" />
<property name="createConnector" value="false" />
<property name="findTigerMbeanServer" value="true" />
</bean>
</property>
<property name="advisorySupport" value="false" />
<property name="systemUsage">
<bean class="org.apache.activemq.usage.SystemUsage">
<property name="memoryUsage">
<bean
class="org.apache.activemq.usage.MemoryUsage">
<property name="limit"
value="200000" />
</bean>
</property>
<property name="storeUsage">
<bean class="org.apache.activemq.usage.StoreUsage">
<property name="limit"
value="100000000" />
</bean>
</property>
<property name="tempUsage">
<bean class="org.apache.activemq.usage.TempUsage">
<property name="limit"
value="100000000" />
</bean>
</property>
</bean>
</property>
<property name="transportConnectorURIs">
<list>
<value>${broker.url}</value>
</list>
</property>
<property name="tempDataStore">
<bean class="org.apache.activemq.kaha.impl.KahaStore">
<constructor-arg>
<bean class="java.io.File">
<constructor-arg
value="temp/" />
</bean>
</constructor-arg>
<constructor-arg value="rw" />
</bean>
</property>
<property name="destinationPolicy">
<bean
class="org.apache.activemq.broker.region.policy.PolicyMap">
<property name="defaultEntry">
<bean
class="org.apache.activemq.broker.region.policy.PolicyEntry">
<property name="subscriptionRecoveryPolicy">
<bean
class="org.apache.activemq.broker.region.policy.NoSubscriptionRecoveryPolicy">
</bean>
</property>
<property name="messageEvictionStrategy">
<bean
class="org.apache.activemq.broker.region.policy.OldestMessageEvictionStrategy">
</bean>
</property>
<property name="pendingMessageLimitStrategy">
<bean
class="org.apache.activemq.broker.region.policy.ConstantPendingMessageLimitStrategy">
</bean>
</property>
<property name="destination">
<bean
class="org.apache.activemq.command.ActiveMQTopic">
<property name="physicalName"
value="topic" />
</bean>
</property>
<property name="pendingSubscriberPolicy">
<bean
class="org.apache.activemq.broker.region.policy.FilePendingSubscriberMessageStoragePolicy">
</bean>
</property>
<property
name="pendingDurableSubscriberPolicy">
<bean
class="org.apache.activemq.broker.region.policy.FilePendingDurableSubscriberMessageStoragePolicy">
</bean>
</property>
<property name="pendingQueuePolicy">
<bean
class="org.apache.activemq.broker.region.policy.FilePendingQueueMessageStoragePolicy">
</bean>
</property>
</bean>
</property>
</bean>
--
View this message in context: http://www.nabble.com/ActiveMQ-Blocked-when-memory-is-full-tp19841470p19841470.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.
Re: ActiveMQ Blocked when memory is full
Posted by Jim Gomes <e....@gmail.com>.
This looks suspiciously related to a problem that we have been encountering,
but have been unable to narrow down. I don't know if there is an existing
JIRA for this, but if there is, I'd vote for it. Top priority.
Thanks for reporting your insights on this issue.
On Thu, Feb 19, 2009 at 2:57 AM, fehm <Fe...@cern.ch> wrote:
>
> Hi,
>
> seems like its still a problem in 5.2.0 (fuse 5.2.0.2).
> The problem is quite easy to reproduce:
> 1.Limit the memoryUsage in activemq.xml to -say- 50MB.
> 2.set producer flow control=false and disable saving messages to disk.
> 2.Start one producer publishing non-persistent messages with TTL=5sec in a
> loop.
> 3.One/Several consumers receiving one message using consumer.receive() and
> then sleep for a second (also in a loop).
>
> You can sit back and watch the memory percent usage in JMX increasing. When
> 100% is reached the consumer stops receiving messages, but the producer
> keeps running and therefore increasing the memory.
> Starting a new consumer will result in this exception :
>
>
> javax.jms.JMSException: Unexpected error occured
> at
>
> org.apache.activemq.util.JMSExceptionSupport.create(JMSExceptionSupport.java:49)
> at
>
> org.apache.activemq.ActiveMQConnection.onAsyncException(ActiveMQConnection.java:1773)
> at
>
> org.apache.activemq.ActiveMQConnection.onException(ActiveMQConnection.java:1790)
> at
>
> org.apache.activemq.transport.TransportFilter.onException(TransportFilter.java:99)
> at
>
> org.apache.activemq.transport.ResponseCorrelator.onException(ResponseCorrelator.java:114)
> at
>
> org.apache.activemq.transport.TransportFilter.onException(TransportFilter.java:99)
> at
>
> org.apache.activemq.transport.TransportFilter.onException(TransportFilter.java:99)
> at
>
> org.apache.activemq.transport.WireFormatNegotiator.onException(WireFormatNegotiator.java:152)
> at
>
> org.apache.activemq.transport.InactivityMonitor.onException(InactivityMonitor.java:244)
> at
>
> org.apache.activemq.transport.TransportSupport.onException(TransportSupport.java:96)
> at
> org.apache.activemq.transport.tcp.TcpTransport.run(TcpTransport.java:194)
> at java.lang.Thread.run(Thread.java:619)
> Caused by: java.io.IOException: Unexpected error occured
> at
> org.apache.activemq.transport.tcp.TcpTransport.run(TcpTransport.java:192)
> ... 1 more
> Caused by: java.lang.ClassCastException:
> org.apache.activemq.command.BrokerId cannot be cast to
> org.apache.activemq.command.ConsumerId
> at
>
> org.apache.activemq.openwire.v4.MessageMarshaller.tightUnmarshal(MessageMarshaller.java:75)
> at
>
> org.apache.activemq.openwire.v4.ActiveMQMessageMarshaller.tightUnmarshal(ActiveMQMessageMarshaller.java:66)
> at
>
> org.apache.activemq.openwire.v4.ActiveMQTextMessageMarshaller.tightUnmarshal(ActiveMQTextMessageMarshaller.java:66)
> at
>
> org.apache.activemq.openwire.OpenWireFormat.tightUnmarshalNestedObject(OpenWireFormat.java:453)
> at
>
> org.apache.activemq.openwire.v4.BaseDataStreamMarshaller.tightUnmarsalNestedObject(BaseDataStreamMarshaller.java:126)
> at
>
> org.apache.activemq.openwire.v4.MessageDispatchMarshaller.tightUnmarshal(MessageDispatchMarshaller.java:71)
> at
>
> org.apache.activemq.openwire.OpenWireFormat.doUnmarshal(OpenWireFormat.java:362)
> at
>
> org.apache.activemq.openwire.OpenWireFormat.unmarshal(OpenWireFormat.java:276)
> at
>
> org.apache.activemq.transport.tcp.TcpTransport.readCommand(TcpTransport.java:210)
> at
> org.apache.activemq.transport.tcp.TcpTransport.doRun(TcpTransport.java:202)
> at
> org.apache.activemq.transport.tcp.TcpTransport.run(TcpTransport.java:185)
> ... 1 more
>
>
> Sadly, only a broker restart solves the problem.
>
> Cheers, Felix
>
>
>
>
> rajdavies wrote:
> >
> > To be honest this looks like a bug in the Memory Usage stats :( -
> > though I believe its fixed in the 5.2 release
> >
> > thanks,
> >
> > Rob
> >
> > On 7 Oct 2008, at 22:38, victorbp wrote:
> >
> >>
> >> Thanks Rob!
> >>
> >> Now I'm able to receive all the messages using disk spooling, but I
> >> see an
> >> odd behaviour: when the memory is full and the broker starts to save
> >> the
> >> messages in disk, I can see in JConsole that the MemoryPercentUsage
> >> property
> >> increases significantly, much higher than 100% (In some test the
> >> value is
> >> close to 3000%).
> >>
> >> I'm not sure about this value, I guess that ActiveMQ takes all the
> >> space
> >> (memory and disk) like the same and thats why I see this increase,
> >> but what
> >> worries me is that once that the broker start to save in disk and
> >> the memory
> >> is increased, it doesn't decrease, I mean, all the time is like the
> >> memory
> >> was full, even if I don't send messages and all the previous
> >> messages was
> >> consumed. Do you know if this is a normal behaviour or how can I
> >> decrease
> >> this memory usage?
> >>
> >> Thanks a lot!
> >> --
> >> View this message in context:
> >>
> http://www.nabble.com/ActiveMQ-Blocked-when-memory-is-full-tp19841470p19867887.html
> >> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
> >>
> >
> >
> >
>
> --
> View this message in context:
> http://www.nabble.com/ActiveMQ-Blocked-when-memory-is-full-tp19841470p22098033.html
> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>
>
Re: ActiveMQ Blocked when memory is full
Posted by fehm <Fe...@cern.ch>.
Hi,
seems like its still a problem in 5.2.0 (fuse 5.2.0.2).
The problem is quite easy to reproduce:
1.Limit the memoryUsage in activemq.xml to -say- 50MB.
2.set producer flow control=false and disable saving messages to disk.
2.Start one producer publishing non-persistent messages with TTL=5sec in a
loop.
3.One/Several consumers receiving one message using consumer.receive() and
then sleep for a second (also in a loop).
You can sit back and watch the memory percent usage in JMX increasing. When
100% is reached the consumer stops receiving messages, but the producer
keeps running and therefore increasing the memory.
Starting a new consumer will result in this exception :
javax.jms.JMSException: Unexpected error occured
at
org.apache.activemq.util.JMSExceptionSupport.create(JMSExceptionSupport.java:49)
at
org.apache.activemq.ActiveMQConnection.onAsyncException(ActiveMQConnection.java:1773)
at
org.apache.activemq.ActiveMQConnection.onException(ActiveMQConnection.java:1790)
at
org.apache.activemq.transport.TransportFilter.onException(TransportFilter.java:99)
at
org.apache.activemq.transport.ResponseCorrelator.onException(ResponseCorrelator.java:114)
at
org.apache.activemq.transport.TransportFilter.onException(TransportFilter.java:99)
at
org.apache.activemq.transport.TransportFilter.onException(TransportFilter.java:99)
at
org.apache.activemq.transport.WireFormatNegotiator.onException(WireFormatNegotiator.java:152)
at
org.apache.activemq.transport.InactivityMonitor.onException(InactivityMonitor.java:244)
at
org.apache.activemq.transport.TransportSupport.onException(TransportSupport.java:96)
at
org.apache.activemq.transport.tcp.TcpTransport.run(TcpTransport.java:194)
at java.lang.Thread.run(Thread.java:619)
Caused by: java.io.IOException: Unexpected error occured
at
org.apache.activemq.transport.tcp.TcpTransport.run(TcpTransport.java:192)
... 1 more
Caused by: java.lang.ClassCastException:
org.apache.activemq.command.BrokerId cannot be cast to
org.apache.activemq.command.ConsumerId
at
org.apache.activemq.openwire.v4.MessageMarshaller.tightUnmarshal(MessageMarshaller.java:75)
at
org.apache.activemq.openwire.v4.ActiveMQMessageMarshaller.tightUnmarshal(ActiveMQMessageMarshaller.java:66)
at
org.apache.activemq.openwire.v4.ActiveMQTextMessageMarshaller.tightUnmarshal(ActiveMQTextMessageMarshaller.java:66)
at
org.apache.activemq.openwire.OpenWireFormat.tightUnmarshalNestedObject(OpenWireFormat.java:453)
at
org.apache.activemq.openwire.v4.BaseDataStreamMarshaller.tightUnmarsalNestedObject(BaseDataStreamMarshaller.java:126)
at
org.apache.activemq.openwire.v4.MessageDispatchMarshaller.tightUnmarshal(MessageDispatchMarshaller.java:71)
at
org.apache.activemq.openwire.OpenWireFormat.doUnmarshal(OpenWireFormat.java:362)
at
org.apache.activemq.openwire.OpenWireFormat.unmarshal(OpenWireFormat.java:276)
at
org.apache.activemq.transport.tcp.TcpTransport.readCommand(TcpTransport.java:210)
at
org.apache.activemq.transport.tcp.TcpTransport.doRun(TcpTransport.java:202)
at
org.apache.activemq.transport.tcp.TcpTransport.run(TcpTransport.java:185)
... 1 more
Sadly, only a broker restart solves the problem.
Cheers, Felix
rajdavies wrote:
>
> To be honest this looks like a bug in the Memory Usage stats :( -
> though I believe its fixed in the 5.2 release
>
> thanks,
>
> Rob
>
> On 7 Oct 2008, at 22:38, victorbp wrote:
>
>>
>> Thanks Rob!
>>
>> Now I'm able to receive all the messages using disk spooling, but I
>> see an
>> odd behaviour: when the memory is full and the broker starts to save
>> the
>> messages in disk, I can see in JConsole that the MemoryPercentUsage
>> property
>> increases significantly, much higher than 100% (In some test the
>> value is
>> close to 3000%).
>>
>> I'm not sure about this value, I guess that ActiveMQ takes all the
>> space
>> (memory and disk) like the same and thats why I see this increase,
>> but what
>> worries me is that once that the broker start to save in disk and
>> the memory
>> is increased, it doesn't decrease, I mean, all the time is like the
>> memory
>> was full, even if I don't send messages and all the previous
>> messages was
>> consumed. Do you know if this is a normal behaviour or how can I
>> decrease
>> this memory usage?
>>
>> Thanks a lot!
>> --
>> View this message in context:
>> http://www.nabble.com/ActiveMQ-Blocked-when-memory-is-full-tp19841470p19867887.html
>> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>>
>
>
>
--
View this message in context: http://www.nabble.com/ActiveMQ-Blocked-when-memory-is-full-tp19841470p22098033.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.
Re: ActiveMQ Blocked when memory is full
Posted by Rob Davies <ra...@gmail.com>.
To be honest this looks like a bug in the Memory Usage stats :( -
though I believe its fixed in the 5.2 release
thanks,
Rob
On 7 Oct 2008, at 22:38, victorbp wrote:
>
> Thanks Rob!
>
> Now I'm able to receive all the messages using disk spooling, but I
> see an
> odd behaviour: when the memory is full and the broker starts to save
> the
> messages in disk, I can see in JConsole that the MemoryPercentUsage
> property
> increases significantly, much higher than 100% (In some test the
> value is
> close to 3000%).
>
> I'm not sure about this value, I guess that ActiveMQ takes all the
> space
> (memory and disk) like the same and thats why I see this increase,
> but what
> worries me is that once that the broker start to save in disk and
> the memory
> is increased, it doesn't decrease, I mean, all the time is like the
> memory
> was full, even if I don't send messages and all the previous
> messages was
> consumed. Do you know if this is a normal behaviour or how can I
> decrease
> this memory usage?
>
> Thanks a lot!
> --
> View this message in context: http://www.nabble.com/ActiveMQ-Blocked-when-memory-is-full-tp19841470p19867887.html
> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>
Re: ActiveMQ Blocked when memory is full
Posted by victorbp <di...@hotmail.com>.
Thanks Rob!
Now I'm able to receive all the messages using disk spooling, but I see an
odd behaviour: when the memory is full and the broker starts to save the
messages in disk, I can see in JConsole that the MemoryPercentUsage property
increases significantly, much higher than 100% (In some test the value is
close to 3000%).
I'm not sure about this value, I guess that ActiveMQ takes all the space
(memory and disk) like the same and thats why I see this increase, but what
worries me is that once that the broker start to save in disk and the memory
is increased, it doesn't decrease, I mean, all the time is like the memory
was full, even if I don't send messages and all the previous messages was
consumed. Do you know if this is a normal behaviour or how can I decrease
this memory usage?
Thanks a lot!
--
View this message in context: http://www.nabble.com/ActiveMQ-Blocked-when-memory-is-full-tp19841470p19867887.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.
Re: ActiveMQ Blocked when memory is full
Posted by Rob Davies <ra...@gmail.com>.
turn off producer flow control - see http://activemq.apache.org/producer-flow-control.html
cheers,
Rob
On 6 Oct 2008, at 17:33, victorbp wrote:
>
> Hello!
>
> I'm sending non persistent messages to queue and topics (non durable)
> through ActiveMQ. I'm trying to use the spooling to disk when memory
> is full
> but every time that I reach the memory limit, seems like the broker
> gets
> blocked and I'm not able to send or receive more messages.
>
> I can see how the broker works fine with a large number of messages
> without
> reach the limit of memory (I use JConsole to see the memory percentage
> usage), but if I increase the number of messages and the memory
> reach 100%
> usage, I just see that the memory doesn't decrease, remains at 100%
> even if
> I wait long time, seems like the consumers get blocked and they dont
> read
> the messages.
>
> I get this behaviour using VMCursors and FileCursors, the only
> difference is
> that when I use FileCursors I can see some temp files created in the
> tempDataStore directory.
>
> What I understand about spooling to disk is that when the memory
> limit is
> reached, the next messages will be saved in disk until the messages in
> memory are read, but instead I'm getting the system blocked.
>
> Here is my config. Does anyone have an idea about what is happening
> or if I
> am missing some extra config?
>
> Thanks a lot!
>
> <bean id="brokerService"
> class="org.apache.activemq.broker.BrokerService" init-method="start"
> destroy-method="stop">
> <property name="brokerName" value="localhost" />
> <property name="persistent" value="false" />
> <property name="useJmx" value="true" />
> <property name="managementContext">
> <bean
> class="org.apache.activemq.broker.jmx.ManagementContext">
> <property name="createMBeanServer" value="false" />
> <property name="createConnector" value="false" />
> <property name="findTigerMbeanServer" value="true" />
> </bean>
> </property>
> <property name="advisorySupport" value="false" />
> <property name="systemUsage">
> <bean class="org.apache.activemq.usage.SystemUsage">
> <property name="memoryUsage">
> <bean
> class="org.apache.activemq.usage.MemoryUsage">
> <property name="limit"
> value="200000" />
> </bean>
> </property>
> <property name="storeUsage">
> <bean class="org.apache.activemq.usage.StoreUsage">
> <property name="limit"
> value="100000000" />
> </bean>
> </property>
> <property name="tempUsage">
> <bean class="org.apache.activemq.usage.TempUsage">
> <property name="limit"
> value="100000000" />
> </bean>
> </property>
> </bean>
> </property>
> <property name="transportConnectorURIs">
> <list>
> <value>${broker.url}</value>
> </list>
> </property>
>
> <property name="tempDataStore">
> <bean class="org.apache.activemq.kaha.impl.KahaStore">
> <constructor-arg>
> <bean class="java.io.File">
> <constructor-arg
> value="temp/" />
> </bean>
> </constructor-arg>
> <constructor-arg value="rw" />
> </bean>
> </property>
> <property name="destinationPolicy">
> <bean
> class="org.apache.activemq.broker.region.policy.PolicyMap">
> <property name="defaultEntry">
> <bean
> class="org.apache.activemq.broker.region.policy.PolicyEntry">
> <property name="subscriptionRecoveryPolicy">
> <bean
>
> class
> =
> "org
> .apache.activemq.broker.region.policy.NoSubscriptionRecoveryPolicy">
> </bean>
> </property>
> <property name="messageEvictionStrategy">
> <bean
>
> class
> =
> "org
> .apache.activemq.broker.region.policy.OldestMessageEvictionStrategy">
> </bean>
> </property>
> <property name="pendingMessageLimitStrategy">
> <bean
>
> class
> =
> "org
> .apache
> .activemq.broker.region.policy.ConstantPendingMessageLimitStrategy">
> </bean>
> </property>
> <property name="destination">
> <bean
> class="org.apache.activemq.command.ActiveMQTopic">
> <property name="physicalName"
> value="topic" />
> </bean>
> </property>
> <property name="pendingSubscriberPolicy">
> <bean
>
> class
> =
> "org
> .apache
> .activemq
> .broker.region.policy.FilePendingSubscriberMessageStoragePolicy">
> </bean>
> </property>
> <property
> name="pendingDurableSubscriberPolicy">
> <bean
>
> class
> =
> "org
> .apache
> .activemq
> .broker
> .region.policy.FilePendingDurableSubscriberMessageStoragePolicy">
> </bean>
> </property>
> <property name="pendingQueuePolicy">
> <bean
>
> class
> =
> "org
> .apache
> .activemq.broker.region.policy.FilePendingQueueMessageStoragePolicy">
> </bean>
> </property>
> </bean>
> </property>
> </bean>
>
> --
> View this message in context: http://www.nabble.com/ActiveMQ-Blocked-when-memory-is-full-tp19841470p19841470.html
> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>