You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@activemq.apache.org by WHIRLYCOTT <ph...@whirlycott.com> on 2007/10/01 00:51:08 UTC
Crashing an AMQ producer in ~12 seconds
I'm attaching a pretty straightforward example of some code that
exhibits a problem that I'm having trouble fixing. I'm hoping
someone can either point out my error or at least help me diagnose
the problem. Basically, the code simply sends ~5000 messages to a
broker. Yes, it's not using a connection pool. That doesn't seem to
make any difference and I'm looking for the simplest possible
example. After about ~1300 messages, the client runs out of memory
and dies (jvm default is 64Mb, iirc).
I've tested this with recent AMQ nightly builds or with 5.0.0.2-fuse
and I get the same result in both cases. In running this code, I'm
also seeing several exceptions like this (maybe ~10 by the time I get
the OOME):
[ActiveMQ Transport: tcp://localhost/127.0.0.1:61616] WARN
org.apache.activemq.ActiveMQConnection - Async exception with no
exception listener: java.net.SocketException: Socket closed
Here's the offending code. Any help would be super.
phil.
import javax.jms.Connection;
import javax.jms.DeliveryMode;
import javax.jms.JMSException;
import javax.jms.MapMessage;
import javax.jms.MessageProducer;
import javax.jms.Session;
import javax.jms.Topic;
import org.apache.activemq.ActiveMQConnectionFactory;
import org.apache.log4j.Logger;
public class ActiveMQClientDyingExample {
private static final Logger log = Logger.getLogger
(ActiveMQClientDyingExample.class);
public static void main(final String args[]) {
final String url = "tcp://localhost:61616";
final String topicName = "test/foo";
log.debug("Initializing pooled connection factory for JMS to URL: "
+ url);
final ActiveMQConnectionFactory normalFactory = new
ActiveMQConnectionFactory();
normalFactory.setBrokerURL(url);
for (int i = 0; i < 5000; i++) {
if (i % 100 ==0)
log.debug(i);
Connection conn = null;
try {
conn = normalFactory.createConnection();
final Session session = conn.createSession(false,
Session.AUTO_ACKNOWLEDGE);
final Topic topic = session.createTopic(topicName);
final MessageProducer producer = session.createProducer(topic);
producer.setDeliveryMode(DeliveryMode.PERSISTENT);
final MapMessage m = session.createMapMessage();
m.setInt("hey", i);
producer.send(m);
} catch (JMSException e) {
log.warn(e.getMessage(), e);
} finally {
if (conn != null)
try {
conn.close();
} catch (JMSException e) {
log.warn(e.getMessage(), e);
}
}
}
}
}
--
Whirlycott
Philip Jacob
phil@whirlycott.com
http://www.whirlycott.com/phil/
Re: Crashing an AMQ producer in ~12 seconds
Posted by Rob Davies <ra...@gmail.com>.
k - this is fixed in main trunk now
On Oct 2, 2007, at 1:23 PM, WHIRLYCOTT wrote:
> On Oct 1, 2007, at 8:27 AM, Rob Davies wrote:
>
>> I'm going to see if I can reproduce on OS/X
>
> Any luck?
>
> phil.
>
> --
> Whirlycott
> Philip Jacob
> phil@whirlycott.com
> http://www.whirlycott.com/phil/
>
>
>
Re: Crashing an AMQ producer in ~12 seconds
Posted by Rob Davies <ra...@gmail.com>.
reproduced it - am shaking out the problem at the mo
On Oct 2, 2007, at 1:23 PM, WHIRLYCOTT wrote:
> On Oct 1, 2007, at 8:27 AM, Rob Davies wrote:
>
>> I'm going to see if I can reproduce on OS/X
>
> Any luck?
>
> phil.
>
> --
> Whirlycott
> Philip Jacob
> phil@whirlycott.com
> http://www.whirlycott.com/phil/
>
>
>
Re: Crashing an AMQ producer in ~12 seconds
Posted by Rob Davies <ra...@gmail.com>.
This is now fixed in the lasted svn - there was a few things needed
tidying up on the broker side too
thanks,
Rob
On Oct 4, 2007, at 3:26 AM, WHIRLYCOTT wrote:
> Rob, thanks for looking into this. I'll provide a summary of what
> I found here for anybody out there doing searches who might run
> across this issue:
>
> Basically, if you create several (or many) connection objects, you
> end up with these threads piling up in the background. I've also
> experienced this same problem with the connection pool supplied
> with ActiveMQ.
>
> What I've done for my producers is to only have one connection
> object per application lifecycle. This already seems to be working
> fine for me. It's not intuitive and, as Rob has confirmed, there's
> definitely a problem with AMQ that he is still fixing.
>
> Thanks again.
>
> phil.
>
> On Oct 3, 2007, at 3:25 PM, Rob Davies wrote:
>
>> er - kinda fixed - there's another leak just need to plug ...
>> On Oct 2, 2007, at 1:23 PM, WHIRLYCOTT wrote:
>>
>>> On Oct 1, 2007, at 8:27 AM, Rob Davies wrote:
>>>
>>>> I'm going to see if I can reproduce on OS/X
>>>
>>> Any luck?
>>>
>>> phil.
>>>
>>> --
>>> Whirlycott
>>> Philip Jacob
>>> phil@whirlycott.com
>>> http://www.whirlycott.com/phil/
>>>
>>>
>>>
>>
>
>
>
> --
> Whirlycott
> Philip Jacob
> phil@whirlycott.com
> http://www.whirlycott.com/phil/
>
>
>
Re: Crashing an AMQ producer in ~12 seconds
Posted by WHIRLYCOTT <ph...@whirlycott.com>.
Rob, thanks for looking into this. I'll provide a summary of what I
found here for anybody out there doing searches who might run across
this issue:
Basically, if you create several (or many) connection objects, you
end up with these threads piling up in the background. I've also
experienced this same problem with the connection pool supplied with
ActiveMQ.
What I've done for my producers is to only have one connection object
per application lifecycle. This already seems to be working fine for
me. It's not intuitive and, as Rob has confirmed, there's definitely
a problem with AMQ that he is still fixing.
Thanks again.
phil.
On Oct 3, 2007, at 3:25 PM, Rob Davies wrote:
> er - kinda fixed - there's another leak just need to plug ...
> On Oct 2, 2007, at 1:23 PM, WHIRLYCOTT wrote:
>
>> On Oct 1, 2007, at 8:27 AM, Rob Davies wrote:
>>
>>> I'm going to see if I can reproduce on OS/X
>>
>> Any luck?
>>
>> phil.
>>
>> --
>> Whirlycott
>> Philip Jacob
>> phil@whirlycott.com
>> http://www.whirlycott.com/phil/
>>
>>
>>
>
--
Whirlycott
Philip Jacob
phil@whirlycott.com
http://www.whirlycott.com/phil/
Re: Crashing an AMQ producer in ~12 seconds
Posted by Rob Davies <ra...@gmail.com>.
er - kinda fixed - there's another leak just need to plug ...
On Oct 2, 2007, at 1:23 PM, WHIRLYCOTT wrote:
> On Oct 1, 2007, at 8:27 AM, Rob Davies wrote:
>
>> I'm going to see if I can reproduce on OS/X
>
> Any luck?
>
> phil.
>
> --
> Whirlycott
> Philip Jacob
> phil@whirlycott.com
> http://www.whirlycott.com/phil/
>
>
>
Re: Crashing an AMQ producer in ~12 seconds
Posted by WHIRLYCOTT <ph...@whirlycott.com>.
On Oct 1, 2007, at 8:27 AM, Rob Davies wrote:
> I'm going to see if I can reproduce on OS/X
Any luck?
phil.
--
Whirlycott
Philip Jacob
phil@whirlycott.com
http://www.whirlycott.com/phil/
Re: Crashing an AMQ producer in ~12 seconds
Posted by Rob Davies <ra...@gmail.com>.
I'm going to see if I can reproduce on OS/X
cheers,
Rob
http://rajdavies.blogspot.com/
On Oct 1, 2007, at 1:06 PM, WHIRLYCOTT wrote:
> Thanks for your response, Saqib,
>
> On Oct 1, 2007, at 6:24 AM, srasul wrote:
>
>> from what i can see, you are sending persistant messages to a
>> topic, this
>> means that if no one is listening on that topic, your messages
>> will he
>> stored (aka held in memory) until someone listens on that topic.
>> this would
>> explain why you run out of memory. Essentially you experiencing
>> the "slow
>> consumer" issue: http://activemq.apache.org/slow-consumers.html
>
> My reading of that document is that non-durable (i.e. non-
> persistent) topics can present an issue, but that's not my case:
>
> "Non-durable topics are the scenario which is most affected by slow
> consumers since the messages are not persistent and messages
> generally go to all consumers."
>
> Also:
>
> http://activemq.apache.org/slow-consumer-handling.html
>
> "Slow Consumers can cause problems on non-durable topics since they
> can force the broker to keep old messages in RAM which once it
> fills up, forces the broker to slow down producers, causing the
> fast consumers to be slowed down."
>
> Please note that the problem I'm experiencing is in the *client*,
> not in the broker.
>
>> another issue might be that you are creating connections to
>> activeMQ for
>> each message. this may be an issue on windows machines (atleast i
>> know this
>> issue on WinXP) where it runs out of ports to assign for your
>> connection. so
>> your might be better off using one connection to send 5000
>> messages atleast
>> on Windows machines.
>
> I'm running on OS/X and Linux.
>
> phil.
>
> --
> Whirlycott
> Philip Jacob
> phil@whirlycott.com
> http://www.whirlycott.com/phil/
>
>
Re: Crashing an AMQ producer in ~12 seconds
Posted by WHIRLYCOTT <ph...@whirlycott.com>.
Thanks for your response, Saqib,
On Oct 1, 2007, at 6:24 AM, srasul wrote:
> from what i can see, you are sending persistant messages to a
> topic, this
> means that if no one is listening on that topic, your messages will he
> stored (aka held in memory) until someone listens on that topic.
> this would
> explain why you run out of memory. Essentially you experiencing the
> "slow
> consumer" issue: http://activemq.apache.org/slow-consumers.html
My reading of that document is that non-durable (i.e. non-persistent)
topics can present an issue, but that's not my case:
"Non-durable topics are the scenario which is most affected by slow
consumers since the messages are not persistent and messages
generally go to all consumers."
Also:
http://activemq.apache.org/slow-consumer-handling.html
"Slow Consumers can cause problems on non-durable topics since they
can force the broker to keep old messages in RAM which once it fills
up, forces the broker to slow down producers, causing the fast
consumers to be slowed down."
Please note that the problem I'm experiencing is in the *client*, not
in the broker.
> another issue might be that you are creating connections to
> activeMQ for
> each message. this may be an issue on windows machines (atleast i
> know this
> issue on WinXP) where it runs out of ports to assign for your
> connection. so
> your might be better off using one connection to send 5000 messages
> atleast
> on Windows machines.
I'm running on OS/X and Linux.
phil.
--
Whirlycott
Philip Jacob
phil@whirlycott.com
http://www.whirlycott.com/phil/
Re: Crashing an AMQ producer in ~12 seconds
Posted by WHIRLYCOTT <ph...@whirlycott.com>.
On Oct 1, 2007, at 8:21 AM, ttmdev wrote:
> The messages are persistent, so shouldn't this give the broker the
> option of
> sending them to the store, thus freeing up memory? Only if the
> messages are
> non-persistent does the broker have no choice but to keep them in-
> memory.
That is my understanding.
> Saqib is right, you should only have to create one connection.
> Connections
> are heavy-weight objects, opening and closing one for each send
> puts a lot
> of strain on the system. Because you're sending to the same topic,
> you can
> also get away with having to create just one session and publisher.
Yep, that's right. In production, I use a PooledConnectionFactory.
The reason that I'm not using a pool in this example is precisely
because this reproduces exactly the problem that I see in production,
where ~1500 threads pile up and the jvm runs out of memory (max heap
is ~2Gb in prod).
In the example that I sent to the list earlier, if you kill -3 the
jvm pid after about ~1300 messages have been sent, you'll see a ton
of threads that are actually the source of the problem.
"AcitveMQ Connection Worker: tcp://localhost/127.0.0.1:61616" daemon
prio=5 tid=0x00511d40 nid=0x1845c00 waiting on condition
[0xb1112000..0xb1112d10]
While I do accept that creating a regular connection with each
request is not a good practice and is not the type of thing that one
would want to do in production, it seems to me that the behavior
ought to be correct and that this should still function, albeit
slowly. Currently, it does not appear to be so.
phil.
--
Whirlycott
Philip Jacob
phil@whirlycott.com
http://www.whirlycott.com/phil/
Re: Crashing an AMQ producer in ~12 seconds
Posted by ttmdev <jo...@ttmsolutions.com>.
The messages are persistent, so shouldn't this give the broker the option of
sending them to the store, thus freeing up memory? Only if the messages are
non-persistent does the broker have no choice but to keep them in-memory.
Saqib is right, you should only have to create one connection. Connections
are heavy-weight objects, opening and closing one for each send puts a lot
of strain on the system. Because you're sending to the same topic, you can
also get away with having to create just one session and publisher.
Regards,
Joe
srasul wrote:
>
> Hi Phill,
>
> from what i can see, you are sending persistant messages to a topic, this
> means that if no one is listening on that topic, your messages will he
> stored (aka held in memory) until someone listens on that topic. this
> would explain why you run out of memory. Essentially you experiencing the
> "slow consumer" issue: http://activemq.apache.org/slow-consumers.html
>
> another issue might be that you are creating connections to activeMQ for
> each message. this may be an issue on windows machines (atleast i know
> this issue on WinXP) where it runs out of ports to assign for your
> connection. so your might be better off using one connection to send 5000
> messages atleast on Windows machines.
>
> hope this helps,
>
> Regards,
>
> Saqib
>
>
> WHIRLYCOTT wrote:
>>
>> I'm attaching a pretty straightforward example of some code that
>> exhibits a problem that I'm having trouble fixing. I'm hoping
>> someone can either point out my error or at least help me diagnose
>> the problem. Basically, the code simply sends ~5000 messages to a
>> broker. Yes, it's not using a connection pool. That doesn't seem to
>> make any difference and I'm looking for the simplest possible
>> example. After about ~1300 messages, the client runs out of memory
>> and dies (jvm default is 64Mb, iirc).
>>
>> I've tested this with recent AMQ nightly builds or with 5.0.0.2-fuse
>> and I get the same result in both cases. In running this code, I'm
>> also seeing several exceptions like this (maybe ~10 by the time I get
>> the OOME):
>>
>> [ActiveMQ Transport: tcp://localhost/127.0.0.1:61616] WARN
>> org.apache.activemq.ActiveMQConnection - Async exception with no
>> exception listener: java.net.SocketException: Socket closed
>>
>> Here's the offending code. Any help would be super.
>>
>> phil.
>>
>> import javax.jms.Connection;
>> import javax.jms.DeliveryMode;
>> import javax.jms.JMSException;
>> import javax.jms.MapMessage;
>> import javax.jms.MessageProducer;
>> import javax.jms.Session;
>> import javax.jms.Topic;
>>
>> import org.apache.activemq.ActiveMQConnectionFactory;
>> import org.apache.log4j.Logger;
>>
>> public class ActiveMQClientDyingExample {
>>
>> private static final Logger log = Logger.getLogger
>> (ActiveMQClientDyingExample.class);
>>
>> public static void main(final String args[]) {
>>
>> final String url = "tcp://localhost:61616";
>> final String topicName = "test/foo";
>>
>> log.debug("Initializing pooled connection factory for JMS to URL: "
>> + url);
>> final ActiveMQConnectionFactory normalFactory = new
>> ActiveMQConnectionFactory();
>> normalFactory.setBrokerURL(url);
>>
>> for (int i = 0; i < 5000; i++) {
>>
>> if (i % 100 ==0)
>> log.debug(i);
>>
>> Connection conn = null;
>> try {
>>
>> conn = normalFactory.createConnection();
>> final Session session = conn.createSession(false,
>> Session.AUTO_ACKNOWLEDGE);
>> final Topic topic = session.createTopic(topicName);
>> final MessageProducer producer = session.createProducer(topic);
>> producer.setDeliveryMode(DeliveryMode.PERSISTENT);
>>
>> final MapMessage m = session.createMapMessage();
>> m.setInt("hey", i);
>>
>> producer.send(m);
>>
>> } catch (JMSException e) {
>> log.warn(e.getMessage(), e);
>> } finally {
>> if (conn != null)
>> try {
>> conn.close();
>> } catch (JMSException e) {
>> log.warn(e.getMessage(), e);
>> }
>> }
>>
>> }
>> }
>>
>> }
>>
>>
>>
>> --
>> Whirlycott
>> Philip Jacob
>> phil@whirlycott.com
>> http://www.whirlycott.com/phil/
>>
>>
>>
>>
>
>
--
View this message in context: http://www.nabble.com/Crashing-an-AMQ-producer-in-%7E12-seconds-tf4545474s2354.html#a12978205
Sent from the ActiveMQ - User mailing list archive at Nabble.com.
Re: Crashing an AMQ producer in ~12 seconds
Posted by srasul <sa...@gmail.com>.
Hi Phill,
from what i can see, you are sending persistant messages to a topic, this
means that if no one is listening on that topic, your messages will he
stored (aka held in memory) until someone listens on that topic. this would
explain why you run out of memory. Essentially you experiencing the "slow
consumer" issue: http://activemq.apache.org/slow-consumers.html
another issue might be that you are creating connections to activeMQ for
each message. this may be an issue on windows machines (atleast i know this
issue on WinXP) where it runs out of ports to assign for your connection. so
your might be better off using one connection to send 5000 messages atleast
on Windows machines.
hope this helps,
Regards,
Saqib
WHIRLYCOTT wrote:
>
> I'm attaching a pretty straightforward example of some code that
> exhibits a problem that I'm having trouble fixing. I'm hoping
> someone can either point out my error or at least help me diagnose
> the problem. Basically, the code simply sends ~5000 messages to a
> broker. Yes, it's not using a connection pool. That doesn't seem to
> make any difference and I'm looking for the simplest possible
> example. After about ~1300 messages, the client runs out of memory
> and dies (jvm default is 64Mb, iirc).
>
> I've tested this with recent AMQ nightly builds or with 5.0.0.2-fuse
> and I get the same result in both cases. In running this code, I'm
> also seeing several exceptions like this (maybe ~10 by the time I get
> the OOME):
>
> [ActiveMQ Transport: tcp://localhost/127.0.0.1:61616] WARN
> org.apache.activemq.ActiveMQConnection - Async exception with no
> exception listener: java.net.SocketException: Socket closed
>
> Here's the offending code. Any help would be super.
>
> phil.
>
> import javax.jms.Connection;
> import javax.jms.DeliveryMode;
> import javax.jms.JMSException;
> import javax.jms.MapMessage;
> import javax.jms.MessageProducer;
> import javax.jms.Session;
> import javax.jms.Topic;
>
> import org.apache.activemq.ActiveMQConnectionFactory;
> import org.apache.log4j.Logger;
>
> public class ActiveMQClientDyingExample {
>
> private static final Logger log = Logger.getLogger
> (ActiveMQClientDyingExample.class);
>
> public static void main(final String args[]) {
>
> final String url = "tcp://localhost:61616";
> final String topicName = "test/foo";
>
> log.debug("Initializing pooled connection factory for JMS to URL: "
> + url);
> final ActiveMQConnectionFactory normalFactory = new
> ActiveMQConnectionFactory();
> normalFactory.setBrokerURL(url);
>
> for (int i = 0; i < 5000; i++) {
>
> if (i % 100 ==0)
> log.debug(i);
>
> Connection conn = null;
> try {
>
> conn = normalFactory.createConnection();
> final Session session = conn.createSession(false,
> Session.AUTO_ACKNOWLEDGE);
> final Topic topic = session.createTopic(topicName);
> final MessageProducer producer = session.createProducer(topic);
> producer.setDeliveryMode(DeliveryMode.PERSISTENT);
>
> final MapMessage m = session.createMapMessage();
> m.setInt("hey", i);
>
> producer.send(m);
>
> } catch (JMSException e) {
> log.warn(e.getMessage(), e);
> } finally {
> if (conn != null)
> try {
> conn.close();
> } catch (JMSException e) {
> log.warn(e.getMessage(), e);
> }
> }
>
> }
> }
>
> }
>
>
>
> --
> Whirlycott
> Philip Jacob
> phil@whirlycott.com
> http://www.whirlycott.com/phil/
>
>
>
>
--
View this message in context: http://www.nabble.com/Crashing-an-AMQ-producer-in-%7E12-seconds-tf4545474s2354.html#a12976645
Sent from the ActiveMQ - User mailing list archive at Nabble.com.