You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@activemq.apache.org by skarthik <ka...@necam.com> on 2006/05/31 00:42:31 UTC
4.0 Consumer OutOfMemoryError bug?
Hi,
I am using the latest activemq-4.0 (downloaded today 5/30/2006) The
following problem exists in activemq-4.0 downloaded on 5/8/2006 as well, but
NOT in RC2.
The setup includes one activemq server, one simple producer process and one
simple consumer process. After the consumer receives around 64MB of messages
(6200 10KB messages or 998 64KB messages or 634 100KB messages), it throws
the following error on its console
Exception in thread "ActiveMQ Transport: tcp:///143.102.32.173:8000"
java.lang.OutOfMemoryError: Java heap space
The producer publishes the messages at a rate of 10 messages per second on a
Topic using DeliveryMode.PERSISTENT. The test programs are very simple and
only just log the messages sent and recieved. So they should not have any
memory leaks. The JMS server uses default configuration (persistent broker)
Is there any configuration that needs to be specified to eliminate this
problem? Like I mentioned at the start, the exact same test code works fine
with RC2 version (which was verified/re-tested today).
thanks
karthik
--
View this message in context: http://www.nabble.com/4.0+Consumer+OutOfMemoryError+bug--t1707655.html#a4636181
Sent from the ActiveMQ - User forum at Nabble.com.
Re: 4.0 Consumer OutOfMemoryError bug?
Posted by James Strachan <ja...@gmail.com>.
On 6/13/06, jevans12 <je...@harris.com> wrote:
> Are prefetch policy and optimized ack's Autonomous?
Yes, they are independent features. Prefetch policy is the maximum
number of unacknowledged messages delivered to a client by the broker.
optimized ack is an option to delay sending individual acks but send batches.
> As I see a consumer will
> ask for 65% of the prefetch policy messages.
The broker sends the entire prefetch size of messages to a client as
fast as possible; the client doesn't ask for messages per se - other
than subscribing with a prefetch value.
> Does the batching of ack's
> follow this? as I receive the 65th (of 100) message a batch ack will be sent
> for those 65?
I'd have to noodle around the code but yes if enabled then optimize
acks sends in batches.
Though folks have experienced some problems with optimize acks...
http://issues.apache.org/activemq/browse/AMQ-716
so it might be worth disabling them (I've added this to the 4.0 release notes).
--
James
-------
http://radio.weblogs.com/0112098/
Re: 4.0 Consumer OutOfMemoryError bug?
Posted by jevans12 <je...@harris.com>.
Are prefetch policy and optimized ack's Autonomous? As I see a consumer will
ask for 65% of the prefetch policy messages. Does the batching of ack's
follow this? as I receive the 65th (of 100) message a batch ack will be sent
for those 65?
--
View this message in context: http://www.nabble.com/4.0-Consumer-OutOfMemoryError-bug--t1707655.html#a4847821
Sent from the ActiveMQ - User forum at Nabble.com.
Re: 4.0 Consumer OutOfMemoryError bug?
Posted by jsalvo <je...@communicator.com.au>.
The JMS client also logged the following exception:
javax.jms.JMSException: Java heap space
at
org.apache.activemq.util.JMSExceptionSupport.create(JMSExceptionSupport.java:45)
at
org.apache.activemq.ActiveMQConnection.syncSendPacket(ActiveMQConnection.java:1137)
at
org.apache.activemq.ActiveMQSession.syncSendPacket(ActiveMQSession.java:1667)
at
org.apache.activemq.ActiveMQMessageConsumer.<init>(ActiveMQMessageConsumer.java:196)
at
org.apache.activemq.ActiveMQSession.createConsumer(ActiveMQSession.java:840)
at
org.apache.activemq.ActiveMQSession.createConsumer(ActiveMQSession.java:800)
at
ActiveMQConsumerMultipleConnections.run(ActiveMQConsumerMultipleConnections.java:42)
at java.lang.Thread.run(Thread.java:595)
Caused by: java.lang.OutOfMemoryError: Java heap space
--
View this message in context: http://www.nabble.com/4.0-Consumer-OutOfMemoryError-bug--t1707655.html#a4894261
Sent from the ActiveMQ - User forum at Nabble.com.
Re: 4.0 Consumer OutOfMemoryError bug?
Posted by "Christopher G. Stach II" <cg...@ldsys.net>.
jsalvo wrote:
> I am seeing this problem as well ... and I have optimizedAcknowledge disabled
> in my jndi.properties:
>
> java.naming.provider.url =
> tcp://localhost:61616?jms.optimizeAcknowledge=false
>
> I have this happened while testing ActiveMQ 4.0.
>
> If I have one Connection, then multiple Sessions from the same Connection,
> and each Session creating its own Consumer, then the problem DOES NOT
> happen.
>
> If I have multiple Connections, then one Session from each Connection, and
> each Session creating its own Consumer, then the problem DOES happen.
>
> The first sign of problem from the logs was being unable to create the
> dynamic destination:
>
> 2006-06-16 12:48:52,778 [127.0.0.1:59267] ERROR ManagedRegionBroker
> - Failed to register destination queue://SampleDynamicQueue
> javax.management.InstanceAlreadyExistsException:
> org.apache.activemq:BrokerName=localhost,Type=Queue,Destination=SampleDynamicQueue
> at
> com.sun.jmx.mbeanserver.RepositorySupport.addMBean(RepositorySupport.java:452)
> at
> com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.internal_addObject(DefaultMBeanServerInterceptor.java:1410)
> at
> com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerObject(DefaultMBeanServerInterceptor.java:936)
> at
> com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerMBean(DefaultMBeanServerInterceptor.java:337)
> at
> com.sun.jmx.mbeanserver.JmxMBeanServer.registerMBean(JmxMBeanServer.java:497)
> at
> org.apache.activemq.broker.jmx.ManagedRegionBroker.registerDestination(ManagedRegionBroker.java:215)
> at
> org.apache.activemq.broker.jmx.ManagedRegionBroker.register(ManagedRegionBroker.java:143)
> at
> org.apache.activemq.broker.jmx.ManagedQueueRegion.createDestination(ManagedQueueRegion.java:57)
> at
> org.apache.activemq.broker.region.AbstractRegion.addDestination(AbstractRegion.java:77)
> at
> org.apache.activemq.broker.region.RegionBroker.addDestination(RegionBroker.java:217)
> at
> org.apache.activemq.broker.BrokerFilter.addDestination(BrokerFilter.java:129)
> at
> org.apache.activemq.advisory.AdvisoryBroker.addDestination(AdvisoryBroker.java:141)
> at
> org.apache.activemq.broker.BrokerFilter.addDestination(BrokerFilter.java:129)
> at
> org.apache.activemq.broker.MutableBrokerFilter.addDestination(MutableBrokerFilter.java:142)
> at
> org.apache.activemq.broker.region.AbstractRegion.addConsumer(AbstractRegion.java:161)
> at
> org.apache.activemq.broker.region.RegionBroker.addConsumer(RegionBroker.java:293)
> at
> org.apache.activemq.broker.BrokerFilter.addConsumer(BrokerFilter.java:73)
> at
> org.apache.activemq.advisory.AdvisoryBroker.addConsumer(AdvisoryBroker.java:77)
> at
> org.apache.activemq.broker.BrokerFilter.addConsumer(BrokerFilter.java:73)
> at
> org.apache.activemq.broker.MutableBrokerFilter.addConsumer(MutableBrokerFilter.java:86)
> at
> org.apache.activemq.broker.AbstractConnection.processAddConsumer(AbstractConnection.java:427)
> at
> org.apache.activemq.command.ConsumerInfo.visit(ConsumerInfo.java:295)
> at
> org.apache.activemq.broker.AbstractConnection.service(AbstractConnection.java:201)
> at
> org.apache.activemq.broker.TransportConnection$1.onCommand(TransportConnection.java:62)
> at
> org.apache.activemq.transport.ResponseCorrelator.onCommand(ResponseCorrelator.java:97)
> at
> org.apache.activemq.transport.TransportFilter.onCommand(TransportFilter.java:63)
> at
> org.apache.activemq.transport.WireFormatNegotiator.onCommand(WireFormatNegotiator.java:114)
> at
> org.apache.activemq.transport.InactivityMonitor.onCommand(InactivityMonitor.java:122)
> at
> org.apache.activemq.transport.TransportSupport.doConsume(TransportSupport.java:87)
> at
> org.apache.activemq.transport.tcp.TcpTransport.run(TcpTransport.java:143)
> at java.lang.Thread.run(Thread.java:595)
>
> This exception was logged 10 times ( 10 times because there were 10
> connections ?? )
>
> Then after that, I have:
> 2006-06-16 12:52:28,656 [127.0.0.1:59272] INFO Service
> - Sync error occurred: java.lang.OutOfMemoryError: Java heap space
> java.lang.OutOfMemoryError: Java heap space
>
>
> The following was logged on the console but not on the log file:
> Exception in thread "ActiveMQ Transport: tcp:///127.0.0.1:59273" Exception
> in thread "MulticastDiscovery: tcp://jsalvo-desktop:61616"
> java.lang.OutOfMemoryError: Java heap space
> Exception in thread "MulticastDiscovery: null" java.lang.OutOfMemoryError:
> Java heap space
>
>
>
> --
> View this message in context: http://www.nabble.com/4.0-Consumer-OutOfMemoryError-bug--t1707655.html#a4894234
> Sent from the ActiveMQ - User forum at Nabble.com.
>
We're getting very similar memory consumption until permgen space fills
up and MBean registration exceptions, but only in the broker which runs
in its own JVM. We don't use discovery, though. We also have
optimizeAcknowledge turned off. Somehow, we run out of 512MB with fewer
than ten thousand very small messages (just a couple of Integers.)
--
Christopher G. Stach II
Re: 4.0 Consumer OutOfMemoryError bug?
Posted by jsalvo <je...@communicator.com.au>.
> With regards memory consumption; you are saying RAM usage grows as you
> produce messages. This makes sense since we tend to cache messages
> around so that they are in RAM when consumers start consuming them.
> For non-pesistent messaging all messages stay in RAM. So firstly are
> you using persistent messaging? (i.e. what is the delivery mode on the
> producer).
I was using PERSISTENT mode:
Connection conn = this.factory.createConnection();
Session session = conn.createSession( false, Session.AUTO_ACKNOWLEDGE
);
MessageProducer qProducer = session.createProducer( this.myQueue);
qProducer.setDeliveryMode(DeliveryMode.PERSISTENT);
> Secondly the journal caches messages and acks around in RAM until a
> checkpoint to JDBC occurs (for persistent messaging) - so you should
> see RAM use go down after you've sent a bunch of persistent messags,
> then some time later the checkpoint happens.
This does not seem to be the case for me. Having said that, when I was
running all of these test, I was actually using kaha persistence:
<persistenceAdapter>
<kahaPersistentAdaptor dir = "activemq-data"/>
</persistenceAdapter>
I'll try to see if it happens as well with the original config:
<persistenceAdapter>
<journaledJDBC journalLogFiles="5" dataDirectory="../activemq-data"/>
</persistenceAdapter>
--
View this message in context: http://www.nabble.com/4.0-Consumer-OutOfMemoryError-bug--t1707655.html#a4896797
Sent from the ActiveMQ - User forum at Nabble.com.
Re: 4.0 Consumer OutOfMemoryError bug?
Posted by James Strachan <ja...@gmail.com>.
If you can figure out how to reproduce the JMX exception in a JUnit
test case we can fix it.
With regards memory consumption; you are saying RAM usage grows as you
produce messages. This makes sense since we tend to cache messages
around so that they are in RAM when consumers start consuming them.
For non-pesistent messaging all messages stay in RAM. So firstly are
you using persistent messaging? (i.e. what is the delivery mode on the
producer).
Secondly the journal caches messages and acks around in RAM until a
checkpoint to JDBC occurs (for persistent messaging) - so you should
see RAM use go down after you've sent a bunch of persistent messags,
then some time later the checkpoint happens.
On 6/16/06, jsalvo <je...@communicator.com.au> wrote:
>
> After restarting the broker and starting the test client, I was not able to
> reproduce the problem until after some significant amount of time.
>
> I believe that the reason is not because of the number of connections ...
> The problem is also not related to consumers ( even with consumers not
> cleanup cleanly ).
>
> Instead, the problem seems to be with producers.
>
> After running a JMS client producer with produced lots of messages (
> non-transacted, auto-acknowledged ), JConsole shows the memory heap usage
> DOES NOT go down, even after telling JConsole to do a garbage collection.
>
> However, after running a JMS client consumer, JConsole shows the memory heap
> usage DOES go down after telling JConsole to do a garbage collection ...
> even if the JMS client producer did not clean up properly (e.g. testing how
> resilient the JMS provider is when a JMS client JVM is killed, etc.. )
>
> I ran this test a couple of times, and the evidence so far shows that the
> broker is somehow not releasing some resource that the JMS client producer
> had even after the JMS client has exited / disconnected.
>
> Apart from that, even when there are no JMS clients, the memory usage goes
> up continuously ... but on slightly. Then again, even after I tell JConsole
> to do a GC, the heap memory usage graph does not go down as well ... So that
> maybe another case separate from the producer case.
>
>
>
>
> --
> View this message in context: http://www.nabble.com/4.0-Consumer-OutOfMemoryError-bug--t1707655.html#a4895366
> Sent from the ActiveMQ - User forum at Nabble.com.
>
>
--
James
-------
http://radio.weblogs.com/0112098/
Re: 4.0 Consumer OutOfMemoryError bug?
Posted by jsalvo <je...@communicator.com.au>.
After restarting the broker and starting the test client, I was not able to
reproduce the problem until after some significant amount of time.
I believe that the reason is not because of the number of connections ...
The problem is also not related to consumers ( even with consumers not
cleanup cleanly ).
Instead, the problem seems to be with producers.
After running a JMS client producer with produced lots of messages (
non-transacted, auto-acknowledged ), JConsole shows the memory heap usage
DOES NOT go down, even after telling JConsole to do a garbage collection.
However, after running a JMS client consumer, JConsole shows the memory heap
usage DOES go down after telling JConsole to do a garbage collection ...
even if the JMS client producer did not clean up properly (e.g. testing how
resilient the JMS provider is when a JMS client JVM is killed, etc.. )
I ran this test a couple of times, and the evidence so far shows that the
broker is somehow not releasing some resource that the JMS client producer
had even after the JMS client has exited / disconnected.
Apart from that, even when there are no JMS clients, the memory usage goes
up continuously ... but on slightly. Then again, even after I tell JConsole
to do a GC, the heap memory usage graph does not go down as well ... So that
maybe another case separate from the producer case.
--
View this message in context: http://www.nabble.com/4.0-Consumer-OutOfMemoryError-bug--t1707655.html#a4895366
Sent from the ActiveMQ - User forum at Nabble.com.
Re: 4.0 Consumer OutOfMemoryError bug?
Posted by jsalvo <je...@communicator.com.au>.
> The first sign of problem from the logs was being unable to create the
dynamic destination:
>
> 2006-06-16 12:48:52,778 [127.0.0.1:59267] ERROR ManagedRegionBroker
> - Failed to register destination queue://SampleDynamicQueue
> javax.management.InstanceAlreadyExistsException:
Well I guess this exception is really expected, since there is more than 1
connection after all ... and 1 of the 10 connections has already created the
destination.
As an aside ... what -Xms and -Xmx settings do you guys have in your
production environment ? I am currently just using the default, which is
-Xmx512M
--
View this message in context: http://www.nabble.com/4.0-Consumer-OutOfMemoryError-bug--t1707655.html#a4894475
Sent from the ActiveMQ - User forum at Nabble.com.
Re: 4.0 Consumer OutOfMemoryError bug?
Posted by jsalvo <je...@communicator.com.au>.
I am seeing this problem as well ... and I have optimizedAcknowledge disabled
in my jndi.properties:
java.naming.provider.url =
tcp://localhost:61616?jms.optimizeAcknowledge=false
I have this happened while testing ActiveMQ 4.0.
If I have one Connection, then multiple Sessions from the same Connection,
and each Session creating its own Consumer, then the problem DOES NOT
happen.
If I have multiple Connections, then one Session from each Connection, and
each Session creating its own Consumer, then the problem DOES happen.
The first sign of problem from the logs was being unable to create the
dynamic destination:
2006-06-16 12:48:52,778 [127.0.0.1:59267] ERROR ManagedRegionBroker
- Failed to register destination queue://SampleDynamicQueue
javax.management.InstanceAlreadyExistsException:
org.apache.activemq:BrokerName=localhost,Type=Queue,Destination=SampleDynamicQueue
at
com.sun.jmx.mbeanserver.RepositorySupport.addMBean(RepositorySupport.java:452)
at
com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.internal_addObject(DefaultMBeanServerInterceptor.java:1410)
at
com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerObject(DefaultMBeanServerInterceptor.java:936)
at
com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerMBean(DefaultMBeanServerInterceptor.java:337)
at
com.sun.jmx.mbeanserver.JmxMBeanServer.registerMBean(JmxMBeanServer.java:497)
at
org.apache.activemq.broker.jmx.ManagedRegionBroker.registerDestination(ManagedRegionBroker.java:215)
at
org.apache.activemq.broker.jmx.ManagedRegionBroker.register(ManagedRegionBroker.java:143)
at
org.apache.activemq.broker.jmx.ManagedQueueRegion.createDestination(ManagedQueueRegion.java:57)
at
org.apache.activemq.broker.region.AbstractRegion.addDestination(AbstractRegion.java:77)
at
org.apache.activemq.broker.region.RegionBroker.addDestination(RegionBroker.java:217)
at
org.apache.activemq.broker.BrokerFilter.addDestination(BrokerFilter.java:129)
at
org.apache.activemq.advisory.AdvisoryBroker.addDestination(AdvisoryBroker.java:141)
at
org.apache.activemq.broker.BrokerFilter.addDestination(BrokerFilter.java:129)
at
org.apache.activemq.broker.MutableBrokerFilter.addDestination(MutableBrokerFilter.java:142)
at
org.apache.activemq.broker.region.AbstractRegion.addConsumer(AbstractRegion.java:161)
at
org.apache.activemq.broker.region.RegionBroker.addConsumer(RegionBroker.java:293)
at
org.apache.activemq.broker.BrokerFilter.addConsumer(BrokerFilter.java:73)
at
org.apache.activemq.advisory.AdvisoryBroker.addConsumer(AdvisoryBroker.java:77)
at
org.apache.activemq.broker.BrokerFilter.addConsumer(BrokerFilter.java:73)
at
org.apache.activemq.broker.MutableBrokerFilter.addConsumer(MutableBrokerFilter.java:86)
at
org.apache.activemq.broker.AbstractConnection.processAddConsumer(AbstractConnection.java:427)
at
org.apache.activemq.command.ConsumerInfo.visit(ConsumerInfo.java:295)
at
org.apache.activemq.broker.AbstractConnection.service(AbstractConnection.java:201)
at
org.apache.activemq.broker.TransportConnection$1.onCommand(TransportConnection.java:62)
at
org.apache.activemq.transport.ResponseCorrelator.onCommand(ResponseCorrelator.java:97)
at
org.apache.activemq.transport.TransportFilter.onCommand(TransportFilter.java:63)
at
org.apache.activemq.transport.WireFormatNegotiator.onCommand(WireFormatNegotiator.java:114)
at
org.apache.activemq.transport.InactivityMonitor.onCommand(InactivityMonitor.java:122)
at
org.apache.activemq.transport.TransportSupport.doConsume(TransportSupport.java:87)
at
org.apache.activemq.transport.tcp.TcpTransport.run(TcpTransport.java:143)
at java.lang.Thread.run(Thread.java:595)
This exception was logged 10 times ( 10 times because there were 10
connections ?? )
Then after that, I have:
2006-06-16 12:52:28,656 [127.0.0.1:59272] INFO Service
- Sync error occurred: java.lang.OutOfMemoryError: Java heap space
java.lang.OutOfMemoryError: Java heap space
The following was logged on the console but not on the log file:
Exception in thread "ActiveMQ Transport: tcp:///127.0.0.1:59273" Exception
in thread "MulticastDiscovery: tcp://jsalvo-desktop:61616"
java.lang.OutOfMemoryError: Java heap space
Exception in thread "MulticastDiscovery: null" java.lang.OutOfMemoryError:
Java heap space
--
View this message in context: http://www.nabble.com/4.0-Consumer-OutOfMemoryError-bug--t1707655.html#a4894234
Sent from the ActiveMQ - User forum at Nabble.com.
Re: 4.0 Consumer OutOfMemoryError bug?
Posted by skarthik <ka...@necam.com>.
Thanks James,
It looks like batch acknowledgement was the problem. Once
omtimizeAcknowledgeis turned OFF, the consumer memory never grows beyond 1-2
MB which is what were expecting. We did NOT have to tune the prefetch size
(which BTW we found out is 32000 by default).
I don't know what is the algorithm used for determining when the grouped
messages need to be acknowledged, but if in case you guys are going to
revisit it, maybe it should take available heap size into consideration
(just a suggestion).
Thanks again for your time.
karthik
--
View this message in context: http://www.nabble.com/4.0-Consumer-OutOfMemoryError-bug--t1707655.html#a4831142
Sent from the ActiveMQ - User forum at Nabble.com.
Re: 4.0 Consumer OutOfMemoryError bug?
Posted by James Strachan <ja...@gmail.com>.
For high performance we have an optimised acknowledgement mode which
is an attempt at acknowledging in batches rather than on every single
message. This is on by default in 4.0 but we've since disabled it as
some folks have experienced problems in some scenarios. Batching
acknowledgements causes the client to keep consumed messages around
until they are acknowledged; which further increases the RAM usage.
Given you are consumed over heap size you might want to disable this
feature...
http://incubator.apache.org/activemq/connection-configuration-uri.html
so connect via http://localhost:61616?jms.optimizeAcknowledge=false
(in 4.0.1 this will be the default value).
Other than that as I said, if you don't want a heap of such a large
size then just reduce down the prefetech; things won't run quite as
fast but if you've such little RAM on the boxes you are running
ActiveMQ then you have to trade off RAM with performance.
On 6/12/06, skarthik <ka...@necam.com> wrote:
>
> James,
>
> Thanks for your valuable time on following up on this issue.
>
> Well, I tried out some more test scenarios over the weekend and now DON'T
> think it is a memory leak problem, but definitely some kind of cleanup
> problem. Also I would like to reiterate that these issuses did not exist in
> RC2 build.
>
> I ran the test programs submitted in JIRA, with following parameters.
>
> Producer publishes messages of 10K size.
>
> The consumer heap was increased to 220 MB, to allow the consumer to store
> more than 20,000 messages (if required). Used the default topic prefetch
> values. (i think you mentioned it was 4000).
> With this heap size, the consumer runs out of memory after consuming around
> 19,000 messages.
>
> Now increased the consumer heap size to 225MB. Then a peculiar thing
> happens. After using up almost all of the heap memory (225 MB, and consuming
> about 21500-odd messages), the whole heap (about 210MB) is freed up in one
> shot. And this cycle (of using upto 225 MB and then freeing up all of it)
> continues after consuming every 21500-odd messages and the consumer never
> runs out of memory.
>
> So the question is why does it wait for upto 225 MB of memory to be used up
> before some kind of cleanup kicks in.
>
> BTW, since the memory profile test case reported by Feng (my co-worker) in
> the JIRA issue-tracker, involved messages of 64K and a heap size of only
> 64MB, I can see why it might be difficult to convince that a problem exists.
> But I don't think the prefetch size is an issue, since the consumer consumes
> the messages at the same rate as the producer, unless the messages are kept
> around even after the consumer has processed it (my understanding of
> activemq internals/behavior is not deep). Also for our application (not the
> test code), we would like to keep the default prefetch size (around
> 4000-5000).
>
> Anyways, hope you guys get a chance to look at this problem as time is
> running out for us (we have a deadline to deilver by this month-end). Also
> would like to mention that the reason we are stressing on this issue is,
> that using activemq 4.0, our actual application (not the test code
> submitted) does not pass through our QA department. RC2 build does, but it
> has some known issues.
>
> Thanks,
> Karthik
>
> --
> View this message in context: http://www.nabble.com/4.0-Consumer-OutOfMemoryError-bug--t1707655.html#a4822824
> Sent from the ActiveMQ - User forum at Nabble.com.
>
>
--
James
-------
http://radio.weblogs.com/0112098/
Re: 4.0 Consumer OutOfMemoryError bug?
Posted by skarthik <ka...@necam.com>.
James,
Thanks for your valuable time on following up on this issue.
Well, I tried out some more test scenarios over the weekend and now DON'T
think it is a memory leak problem, but definitely some kind of cleanup
problem. Also I would like to reiterate that these issuses did not exist in
RC2 build.
I ran the test programs submitted in JIRA, with following parameters.
Producer publishes messages of 10K size.
The consumer heap was increased to 220 MB, to allow the consumer to store
more than 20,000 messages (if required). Used the default topic prefetch
values. (i think you mentioned it was 4000).
With this heap size, the consumer runs out of memory after consuming around
19,000 messages.
Now increased the consumer heap size to 225MB. Then a peculiar thing
happens. After using up almost all of the heap memory (225 MB, and consuming
about 21500-odd messages), the whole heap (about 210MB) is freed up in one
shot. And this cycle (of using upto 225 MB and then freeing up all of it)
continues after consuming every 21500-odd messages and the consumer never
runs out of memory.
So the question is why does it wait for upto 225 MB of memory to be used up
before some kind of cleanup kicks in.
BTW, since the memory profile test case reported by Feng (my co-worker) in
the JIRA issue-tracker, involved messages of 64K and a heap size of only
64MB, I can see why it might be difficult to convince that a problem exists.
But I don't think the prefetch size is an issue, since the consumer consumes
the messages at the same rate as the producer, unless the messages are kept
around even after the consumer has processed it (my understanding of
activemq internals/behavior is not deep). Also for our application (not the
test code), we would like to keep the default prefetch size (around
4000-5000).
Anyways, hope you guys get a chance to look at this problem as time is
running out for us (we have a deadline to deilver by this month-end). Also
would like to mention that the reason we are stressing on this issue is,
that using activemq 4.0, our actual application (not the test code
submitted) does not pass through our QA department. RC2 build does, but it
has some known issues.
Thanks,
Karthik
--
View this message in context: http://www.nabble.com/4.0-Consumer-OutOfMemoryError-bug--t1707655.html#a4822824
Sent from the ActiveMQ - User forum at Nabble.com.
Re: 4.0 Consumer OutOfMemoryError bug?
Posted by James Strachan <ja...@gmail.com>.
On 6/9/06, FWang <fe...@necam.com> wrote:
>
>
> I have post more info on http://issues.apache.org/activemq/browse/AMQ-735.
> Has anyone investigate the issue?
I added a comment to the issue. I'm not yet convinced you are
experiencing any memory leak; its just the effect of the pre-fetch. If
you want to use a small heap size with messages of around 10K then
you'll probably need to reduce the pre-fetch values for topics to
something like 10
http://incubator.apache.org/activemq/what-is-the-prefetch-limit-for.html
--
James
-------
http://radio.weblogs.com/0112098/
Re: 4.0 Consumer OutOfMemoryError bug?
Posted by FWang <fe...@necam.com>.
I have post more info on http://issues.apache.org/activemq/browse/AMQ-735.
Has anyone investigate the issue?
thanks,
--
View this message in context: http://www.nabble.com/4.0-Consumer-OutOfMemoryError-bug--t1707655.html#a4798340
Sent from the ActiveMQ - User forum at Nabble.com.
Re: 4.0 Consumer OutOfMemoryError bug?
Posted by James Strachan <ja...@gmail.com>.
On 6/7/06, FWang <fe...@necam.com> wrote:
>
>
> I ran JProbe on karthik's code and I suspect that MessageDispath has the
> memory leak. Please see the attached screen shot from "Memory Leak Detector"
> in JProbe.
Thanks for doing that!
Though I don't really understand the pictures; if my reading of them
is correct the test program has created 124596 JMS Session objects;
I'm not surprised that ran out of RAM - thats a massive number of
sessions; you generally only need at most one per thread.
JMS Sessions are created by JMS client code - they are not controlled
by the JMS provider - so if too many of those are created I suspect
its your application code thats at fault.
Also imagine when you start consuming topics on that many sessions;
each consumer on the session will receive its own copy of the message
- then we prefech a few thousand messages by default per consumer - so
running out of RAM with that configuration doesnt' seem like a memory
leak - it seems like normal behaviour.
http://incubator.apache.org/activemq/what-is-the-prefetch-limit-for.html
(Unless the number next to the objects in the pictures is not the
number of instances created in which case please disregard this entire
email :)
--
James
-------
http://radio.weblogs.com/0112098/
Re: 4.0 Consumer OutOfMemoryError bug?
Posted by FWang <fe...@necam.com>.
I ran JProbe on karthik's code and I suspect that MessageDispath has the
memory leak. Please see the attached screen shot from "Memory Leak Detector"
in JProbe.
I have put a copy of zip file (activemq_735.zip) on Jira as well.
(http://issues.apache.org/activemq/browse/AMQ-735)
thanks,
--
View this message in context: http://www.nabble.com/4.0-Consumer-OutOfMemoryError-bug--t1707655.html#a4761710
Sent from the ActiveMQ - User forum at Nabble.com.
Re: 4.0 Consumer OutOfMemoryError bug?
Posted by James Strachan <ja...@gmail.com>.
BTW in any kind of memory leak type situation, it can help us to help
you quicker if you run a profiler on the test case which will then
report the types of objects which are leaking. e.g. if you run
JProfiler or YourKit on the test case and attach the memory output
file to the JIRA (probably zipped up if its huge) then we can
typically diagnose the issue quicker.
On 6/6/06, Hiram Chirino <hi...@hiramchirino.com> wrote:
> Sorry.. not yet. I've been swamped.. I'll try hooking up a memory
> profiler to your test case perhaps early next week.
>
> On 6/6/06, skarthik <ka...@necam.com> wrote:
> >
> > Hi,
> >
> > Just following up. Has anybody from the development team had a chance to
> > look at this problem and the test code I posted? Was wondering if this is a
> > geniune problem or is it something I am doing wrong?
> >
> > thanks,
> > karthik
> > --
> > View this message in context: http://www.nabble.com/4.0-Consumer-OutOfMemoryError-bug--t1707655.html#a4733986
> > Sent from the ActiveMQ - User forum at Nabble.com.
> >
> >
>
>
> --
> Regards,
> Hiram
>
--
James
-------
http://radio.weblogs.com/0112098/
Re: 4.0 Consumer OutOfMemoryError bug?
Posted by Hiram Chirino <hi...@hiramchirino.com>.
Sorry.. not yet. I've been swamped.. I'll try hooking up a memory
profiler to your test case perhaps early next week.
On 6/6/06, skarthik <ka...@necam.com> wrote:
>
> Hi,
>
> Just following up. Has anybody from the development team had a chance to
> look at this problem and the test code I posted? Was wondering if this is a
> geniune problem or is it something I am doing wrong?
>
> thanks,
> karthik
> --
> View this message in context: http://www.nabble.com/4.0-Consumer-OutOfMemoryError-bug--t1707655.html#a4733986
> Sent from the ActiveMQ - User forum at Nabble.com.
>
>
--
Regards,
Hiram
Re: 4.0 Consumer OutOfMemoryError bug?
Posted by skarthik <ka...@necam.com>.
Hi,
Just following up. Has anybody from the development team had a chance to
look at this problem and the test code I posted? Was wondering if this is a
geniune problem or is it something I am doing wrong?
thanks,
karthik
--
View this message in context: http://www.nabble.com/4.0-Consumer-OutOfMemoryError-bug--t1707655.html#a4733986
Sent from the ActiveMQ - User forum at Nabble.com.
Re: 4.0 Consumer OutOfMemoryError bug?
Posted by skarthik <ka...@necam.com>.
I have raised the follwing issue in JIRA, which has the test code attached.
http://issues.apache.org/activemq/browse/AMQ-735
karthik
--
View this message in context: http://www.nabble.com/4.0+Consumer+OutOfMemoryError+bug--t1707655.html#a4669553
Sent from the ActiveMQ - User forum at Nabble.com.
Re: 4.0 Consumer OutOfMemoryError bug?
Posted by James Strachan <ja...@gmail.com>.
On 5/31/06, skarthik <ka...@necam.com> wrote:
>
> HI,
>
> I would like send in the test code files as zip file (along with some script
> files to compile and run it). Is there anyway I can attach it or any email
> address to send it to?
The easiest way is to raise a JIRA issue in our issue tracker then
attach whatever source files & config files to that then we can try
replicate your issue
http://incubator.apache.org/activemq/support.html
--
James
-------
http://radio.weblogs.com/0112098/
Re: 4.0 Consumer OutOfMemoryError bug?
Posted by skarthik <ka...@necam.com>.
HI,
I would like send in the test code files as zip file (along with some script
files to compile and run it). Is there anyway I can attach it or any email
address to send it to?
Thanks
karthik
--
View this message in context: http://www.nabble.com/4.0+Consumer+OutOfMemoryError+bug--t1707655.html#a4652810
Sent from the ActiveMQ - User forum at Nabble.com.
Re: 4.0 Consumer OutOfMemoryError bug?
Posted by amerigo5 <am...@hotmail.com>.
Can you post your code and activemq.conf?
Also, make sure that your client code is using the correct version of jar
files.
Thanks.
--
View this message in context: http://www.nabble.com/4.0+Consumer+OutOfMemoryError+bug--t1707655.html#a4650934
Sent from the ActiveMQ - User forum at Nabble.com.
Re: 4.0 Consumer OutOfMemoryError bug?
Posted by skarthik <ka...@necam.com>.
Hi James,
Well I read the information on prefetch limit (the link you specified) and
iIMHO I don't believe that is the issue.
First, I lowered the rate of messages published to 1 per second and I still
see the heap size growing at the same rate (per KB of messages received) and
it eventually hits the OutOfMemoryError. There is only one publisher and one
consumer connected the JMS server.
Second my consumer is a test program and is very fast. All it does is
display the messages received. Also the log time of the published messages
(logged by producer before publish) and received messages (logged by
consumer after received) are the same (down to the last millisecond).
I do feel there maybe a critical memory leak somewhere. Also as I mentioned
in my previous post, the RC2 build does not have this problem.
thanks
karthik
--
View this message in context: http://www.nabble.com/4.0+Consumer+OutOfMemoryError+bug--t1707655.html#a4647038
Sent from the ActiveMQ - User forum at Nabble.com.
Re: 4.0 Consumer OutOfMemoryError bug?
Posted by James Strachan <ja...@gmail.com>.
You could try lowering the prefetch size to reduce the amount of
Message objects which are loaded into the consumer....
http://incubator.apache.org/activemq/what-is-the-prefetch-limit-for.html
Try set it to a low value like 100 and see if that fixes it.
On 5/31/06, skarthik <ka...@necam.com> wrote:
>
> Only the consumer JVM (JDK 1.5) runs out of memory. I am using the default
> JVM heap sizes which I think is 64MB.
>
> But I put some logging statements in the consumer code to display memory
> statistics after receiving every message (using Runtime.maxMemory(),
> Runtime.totalMemory() and Runtime.freeMemory()). I see that max memory is
> 64MB and the total memory used slowly creeps up to that max mark and then
> encounters the OutOfMemoryError.
>
> Also I am using a non-durable subscriber, and am using subscriber.receive
> () with timeout to pull the messages from the Topic (NOT using
> MessageListener interface). Also the TopicSessions used are non-transacted
> with Session.AUTO_ACKNOWLEDGE mode.
>
> Hope this helps.
> karthik
> --
> View this message in context: http://www.nabble.com/4.0+Consumer+OutOfMemoryError+bug--t1707655.html#a4645070
> Sent from the ActiveMQ - User forum at Nabble.com.
>
>
--
James
-------
http://radio.weblogs.com/0112098/
Re: 4.0 Consumer OutOfMemoryError bug?
Posted by skarthik <ka...@necam.com>.
Only the consumer JVM (JDK 1.5) runs out of memory. I am using the default
JVM heap sizes which I think is 64MB.
But I put some logging statements in the consumer code to display memory
statistics after receiving every message (using Runtime.maxMemory(),
Runtime.totalMemory() and Runtime.freeMemory()). I see that max memory is
64MB and the total memory used slowly creeps up to that max mark and then
encounters the OutOfMemoryError.
Also I am using a non-durable subscriber, and am using subscriber.receive
() with timeout to pull the messages from the Topic (NOT using
MessageListener interface). Also the TopicSessions used are non-transacted
with Session.AUTO_ACKNOWLEDGE mode.
Hope this helps.
karthik
--
View this message in context: http://www.nabble.com/4.0+Consumer+OutOfMemoryError+bug--t1707655.html#a4645070
Sent from the ActiveMQ - User forum at Nabble.com.
Re: 4.0 Consumer OutOfMemoryError bug?
Posted by James Strachan <ja...@gmail.com>.
On 5/30/06, skarthik <ka...@necam.com> wrote:
>
> Hi,
>
> I am using the latest activemq-4.0 (downloaded today 5/30/2006) The
> following problem exists in activemq-4.0 downloaded on 5/8/2006 as well, but
> NOT in RC2.
>
> The setup includes one activemq server, one simple producer process and one
> simple consumer process. After the consumer receives around 64MB of messages
> (6200 10KB messages or 998 64KB messages or 634 100KB messages), it throws
> the following error on its console
>
> Exception in thread "ActiveMQ Transport: tcp:///143.102.32.173:8000"
> java.lang.OutOfMemoryError: Java heap space
>
> The producer publishes the messages at a rate of 10 messages per second on a
> Topic using DeliveryMode.PERSISTENT. The test programs are very simple and
> only just log the messages sent and recieved. So they should not have any
> memory leaks. The JMS server uses default configuration (persistent broker)
>
> Is there any configuration that needs to be specified to eliminate this
> problem? Like I mentioned at the start, the exact same test code works fine
> with RC2 version (which was verified/re-tested today).
So the JVM that runs out of memory - is that just the consumer, or is
it the consumer, the broker and producer? Whats the heap size of that
JVM? What acknowldgement mode are you using?
--
James
-------
http://radio.weblogs.com/0112098/