You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@servicemix.apache.org by Bruce Snyder <br...@gmail.com> on 2008/08/22 06:11:20 UTC
Re: ActiveMQ causes producer to hang when no consumer on topic after a few minutes.
On Thu, Aug 21, 2008 at 9:43 PM, Ryan Moquin <fr...@gmail.com> wrote:
> Hi Bruce,
>
> I tried my service with the two settings:
>
> deliveryMode="1"
> timeToLive="1000"
>
> The scenario is an external JMS client sends a message to a JMS consumer SU
> in servicemix. It forwards that to my service which does some processing
> with the incoming message, and then creates a new message that it sends to a
> JMS provider SU. I ran this using Servicemix 3.3.
>
> I loaded up jconsole and watched my provider topic. As soon as my Topic
> size hit 1321, my service all of a sudden stopped receiving messages from my
> external sender and went completely silent. If I look at the topic that my
> external sender, is sending to, it is still filling up with messages that
> are no longer being received by my hanging service. When my external sender
> finished, I see that 1500 messages were enqueued and dequeued on my consumer
> JMS SU, but only 1340 were enqueued on my provider JMS SU.
>
> So, I did a stack dump on the JVM and noticed there are 35 hanging threads
> that are all hanging trying to send a message to the JMS provider. So
> apparently, the:
>
> <property name="useAsyncSend" value="true" />
>
> is working and I just didn't notice that there were 35 hanging threads. The
> problem is, I guess once enough threads are tied up because the messages
> can't be sent, then everything starts hanging. I have the message time to
> live sent to a 1000. You said that it worked in 5.1, but it doesn't seem to
> be working for me. Is that value really in milliseconds? Is there
> something else I need? The non persistent delivery mode obviously isn't
> working either. Do I have any options for preventing this situation from
> happening? I don't see any other options in the config I can try.
>
> Here is the stack dump that is the same for all 35 threads:
>
> "pool-component.servicemix-jms-thread-3" prio=6 tid=0x39a4b400 nid=0x2724
> waiting on condition [0x3f58e000..0x3f58fb14]
> java.lang.Thread.State: WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for <0x1774ee48> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:19
I'm thinking that the issues you're experiencing are ServiceMix
related, not ActiveMQ related. It sounds like messages may not be
consumed fast enough which could cause the slowdown you describe, but
I'm not sure yet. I'd like to see how you have the thread pooling
configured in ServiceMix, so please post your servicemix.properties
file. Also, please provide two full thread dumps (kill -3 <pid>) of
ServiceMix that are, maybe say, 20-30 seconds apart. These three items
should help us to troubleshoot the situation a bit further.
Bruce
--
perl -e 'print unpack("u30","D0G)U8V4\@4VYY9&5R\"F)R=6-E+G-N>61E<D\!G;6%I;\"YC;VT*"
);'
Apache ActiveMQ - http://activemq.org/
Apache Camel - http://activemq.org/camel/
Apache ServiceMix - http://servicemix.org/
Blog: http://bruceblog.org/
Re: ActiveMQ causes producer to hang when no consumer on topic after a few minutes.
Posted by Bruce Snyder <br...@gmail.com>.
On Fri, Aug 22, 2008 at 11:33 AM, Ryan Moquin <fr...@gmail.com> wrote:
> Ok, I am using the default servicemix settings without any tweaks. I try to
> avoid whenever possible making any changes to the default config since it's
> just one more thing for me to have to maintain and remember it's there in
> case I hit other problems.
>
> Sure, I'll make several threads dumps.. do you want me to do it after the
> threads starting hanging? or during the time that they are starting to hang?
Yes, definitely grab the thread dumps while the server is hanging,
maybe even just before it starts to hang as well. This will give us a
much better picture of what's happening. Also, if you can, try to map
each one to a time if possible so we know in what order they were
taken.
Bruce
--
perl -e 'print unpack("u30","D0G)U8V4\@4VYY9&5R\"F)R=6-E+G-N>61E<D\!G;6%I;\"YC;VT*"
);'
Apache ActiveMQ - http://activemq.org/
Apache Camel - http://activemq.org/camel/
Apache ServiceMix - http://servicemix.org/
Blog: http://bruceblog.org/
Re: ActiveMQ causes producer to hang when no consumer on topic after a few minutes.
Posted by Ryan Moquin <fr...@gmail.com>.
Ok, I am using the default servicemix settings without any tweaks. I try to
avoid whenever possible making any changes to the default config since it's
just one more thing for me to have to maintain and remember it's there in
case I hit other problems.
Sure, I'll make several threads dumps.. do you want me to do it after the
threads starting hanging? or during the time that they are starting to hang?
On Fri, Aug 22, 2008 at 12:11 AM, Bruce Snyder <br...@gmail.com>wrote:
> On Thu, Aug 21, 2008 at 9:43 PM, Ryan Moquin <fr...@gmail.com>
> wrote:
> > Hi Bruce,
> >
> > I tried my service with the two settings:
> >
> > deliveryMode="1"
> > timeToLive="1000"
> >
> > The scenario is an external JMS client sends a message to a JMS consumer
> SU
> > in servicemix. It forwards that to my service which does some processing
> > with the incoming message, and then creates a new message that it sends
> to a
> > JMS provider SU. I ran this using Servicemix 3.3.
> >
> > I loaded up jconsole and watched my provider topic. As soon as my Topic
> > size hit 1321, my service all of a sudden stopped receiving messages from
> my
> > external sender and went completely silent. If I look at the topic that
> my
> > external sender, is sending to, it is still filling up with messages that
> > are no longer being received by my hanging service. When my external
> sender
> > finished, I see that 1500 messages were enqueued and dequeued on my
> consumer
> > JMS SU, but only 1340 were enqueued on my provider JMS SU.
> >
> > So, I did a stack dump on the JVM and noticed there are 35 hanging
> threads
> > that are all hanging trying to send a message to the JMS provider. So
> > apparently, the:
> >
> > <property name="useAsyncSend" value="true" />
> >
> > is working and I just didn't notice that there were 35 hanging threads.
> The
> > problem is, I guess once enough threads are tied up because the messages
> > can't be sent, then everything starts hanging. I have the message time
> to
> > live sent to a 1000. You said that it worked in 5.1, but it doesn't seem
> to
> > be working for me. Is that value really in milliseconds? Is there
> > something else I need? The non persistent delivery mode obviously isn't
> > working either. Do I have any options for preventing this situation from
> > happening? I don't see any other options in the config I can try.
> >
> > Here is the stack dump that is the same for all 35 threads:
> >
> > "pool-component.servicemix-jms-thread-3" prio=6 tid=0x39a4b400 nid=0x2724
> > waiting on condition [0x3f58e000..0x3f58fb14]
> > java.lang.Thread.State: WAITING (parking)
> > at sun.misc.Unsafe.park(Native Method)
> > - parking to wait for <0x1774ee48> (a
> > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> > at
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> > at
> >
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:19
>
> I'm thinking that the issues you're experiencing are ServiceMix
> related, not ActiveMQ related. It sounds like messages may not be
> consumed fast enough which could cause the slowdown you describe, but
> I'm not sure yet. I'd like to see how you have the thread pooling
> configured in ServiceMix, so please post your servicemix.properties
> file. Also, please provide two full thread dumps (kill -3 <pid>) of
> ServiceMix that are, maybe say, 20-30 seconds apart. These three items
> should help us to troubleshoot the situation a bit further.
>
> Bruce
> --
> perl -e 'print
> unpack("u30","D0G)U8V4\@4VYY9&5R\"F)R=6-E+G-N>61E<D\!G;6%I;\"YC;VT*"
> );'
>
> Apache ActiveMQ - http://activemq.org/
> Apache Camel - http://activemq.org/camel/
> Apache ServiceMix - http://servicemix.org/
>
> Blog: http://bruceblog.org/
>