You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@activemq.apache.org by Ryan Moquin <fr...@gmail.com> on 2008/08/20 02:54:20 UTC
ActiveMQ causes producer to hang when no consumer on topic after a few minutes.
Hello,
I'm trying to understand how to configure ActiveMQ (4.1.1 is what Servicemix
is using) properly so that my Servicemix services will stop hanging after a
few minutes of sending messages to a provider endpoint. The ActiveMQ
provider endpoint is consistently deadlocking after a short period of time
(about 10 minutes, 1500 messages). It appears as though activemq is
configured out of the box (or maybe Servicemix configures it this way) to
persistent every single message in memory whether or not it's been delivered
(or atleast maybe the ones that aren't delivered, not completely sure).
Basically, the messages I'm producing should be thrown away if there is no
consumer. But currently the messages just pile up on my destination topic
in ActiveMQ until it starts refusing to accept anymore messages. This
happens extremely quickly, as I mentioned above, when there are no consumers
subscribed to the topic.
I've tried setting the TTL for the messages to 1000, I've tried setting the
deliverymode to non-persistent, but no matter what, my service will
deadlock. When I look at the topic in the JMX manager, I will see a very
high enqueue count, which I'm assuming means my messages are just sitting
there, waiting for a consumer. Is there any way to make activemq not keep
these messages around?
Thanks!
Re: ActiveMQ causes producer to hang when no consumer on topic after a few minutes.
Posted by Ryan Moquin <fr...@gmail.com>.
Are there any memory or other considerations to worry about under a high
volume of messages related to the DLQ? Once I figure out my current issue,
I want to make sure I know any other options I might want to consider that
are within my control.
On Fri, Aug 22, 2008 at 9:49 PM, Joe Fernandez <
joe.fernandez@ttmsolutions.com> wrote:
>
> Try setting the processExpired attribute to 'false' for the
> sharedDeadLetterStrategy . For example,
>
> <destinationPolicy>
> <policyMap>
> <policyEntries>
> <policyEntry queue=">" >
> <deadLetterStrategy>
> <sharedDeadLetterStrategy processExpired="false"/>
> </deadLetterStrategy>
> </policyEntry>
> </policyEntries>
> </policyMap>
> </destinationPolicy>
>
>
> Joe
> Free AMQ reference guide @ http://www.ttmsolutions.com
>
>
> David Crisp wrote:
> >
> >
> > Hi Bruce,
> > How can I configure the broker to simply drop messages when they
> > expire rather than sending them to the DLQ? Is there a
> > "BitBucketDeadLetterStrategy" or something?
> >
> > -D
> >
> >
> > bsnyder wrote:
> >>
> >> On Wed, Aug 20, 2008 at 2:49 PM, Ryan Moquin <fr...@gmail.com>
> >> wrote:
> >>> I don't think I saw that option? I assumed the timetolive would have
> >>> that
> >>> effect? I'll take another look at the settings, I may have overlooked
> >>> that
> >>> one. I hope so because I'm getting pretty desperate and I'm positive
> >>> that
> >>> I've correctly tried all the other options correctly, which is why I'm
> >>> so
> >>> puzzled.
> >>
> >> OK, my mistake in suggesting the use of the expiration. I just tested
> >> the expiration and the timeToLive on the JMS message in ActiveMQ 4.1.1
> >> and 5.1. The expiration doesn't seem to have any affect on either
> >> broker, but the timeToLive does. In ActiveMQ 4.1.1, the timeToLive
> >> doesn't affect message expiration but in ActiveMQ 5.1, setting a
> >> timeToLive will cause the message to be delivered to the DLQ if the
> >> timeToLive surpasses and the message is still pending.
> >>
> >> Bruce
> >> --
> >> perl -e 'print
> >> unpack("u30","D0G)U8V4\@4VYY9&5R\"F)R=6-E+G-N>61E<D\!G;6%I;\"YC;VT*"
> >> );'
> >>
> >> Apache ActiveMQ - http://activemq.org/
> >> Apache Camel - http://activemq.org/camel/
> >> Apache ServiceMix - http://servicemix.org/
> >>
> >> Blog: http://bruceblog.org/
> >>
> >>
> >
> >
>
> --
> View this message in context:
> http://www.nabble.com/ActiveMQ-causes-producer-to-hang-when-no-consumer-on-topic-after-a-few-minutes.-tp19061289p19117910.html
> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>
>
Re: ActiveMQ causes producer to hang when no consumer on topic after a few minutes.
Posted by Rob Davies <ra...@gmail.com>.
That's the default behaviour in 5.2 (or trunk) btw
cheers,
Rob
On 23 Aug 2008, at 02:49, Joe Fernandez wrote:
>
> Try setting the processExpired attribute to 'false' for the
> sharedDeadLetterStrategy . For example,
>
> <destinationPolicy>
> <policyMap>
> <policyEntries>
> <policyEntry queue=">" >
> <deadLetterStrategy>
> <sharedDeadLetterStrategy processExpired="false"/>
> </deadLetterStrategy>
> </policyEntry>
> </policyEntries>
> </policyMap>
> </destinationPolicy>
>
>
> Joe
> Free AMQ reference guide @ http://www.ttmsolutions.com
>
>
> David Crisp wrote:
>>
>>
>> Hi Bruce,
>> How can I configure the broker to simply drop messages when they
>> expire rather than sending them to the DLQ? Is there a
>> "BitBucketDeadLetterStrategy" or something?
>>
>> -D
>>
>>
>> bsnyder wrote:
>>>
>>> On Wed, Aug 20, 2008 at 2:49 PM, Ryan Moquin
>>> <fr...@gmail.com>
>>> wrote:
>>>> I don't think I saw that option? I assumed the timetolive would
>>>> have
>>>> that
>>>> effect? I'll take another look at the settings, I may have
>>>> overlooked
>>>> that
>>>> one. I hope so because I'm getting pretty desperate and I'm
>>>> positive
>>>> that
>>>> I've correctly tried all the other options correctly, which is
>>>> why I'm
>>>> so
>>>> puzzled.
>>>
>>> OK, my mistake in suggesting the use of the expiration. I just
>>> tested
>>> the expiration and the timeToLive on the JMS message in ActiveMQ
>>> 4.1.1
>>> and 5.1. The expiration doesn't seem to have any affect on either
>>> broker, but the timeToLive does. In ActiveMQ 4.1.1, the timeToLive
>>> doesn't affect message expiration but in ActiveMQ 5.1, setting a
>>> timeToLive will cause the message to be delivered to the DLQ if the
>>> timeToLive surpasses and the message is still pending.
>>>
>>> Bruce
>>> --
>>> perl -e 'print
>>> unpack("u30","D0G)U8V4\@4VYY9&5R\"F)R=6-E+G-N>61E<D\!G;6%I;\"YC;VT*"
>>> );'
>>>
>>> Apache ActiveMQ - http://activemq.org/
>>> Apache Camel - http://activemq.org/camel/
>>> Apache ServiceMix - http://servicemix.org/
>>>
>>> Blog: http://bruceblog.org/
>>>
>>>
>>
>>
>
> --
> View this message in context: http://www.nabble.com/ActiveMQ-causes-producer-to-hang-when-no-consumer-on-topic-after-a-few-minutes.-tp19061289p19117910.html
> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>
Re: ActiveMQ causes producer to hang when no consumer on topic
after a few minutes.
Posted by Joe Fernandez <jo...@ttmsolutions.com>.
Try setting the processExpired attribute to 'false' for the
sharedDeadLetterStrategy . For example,
<destinationPolicy>
<policyMap>
<policyEntries>
<policyEntry queue=">" >
<deadLetterStrategy>
<sharedDeadLetterStrategy processExpired="false"/>
</deadLetterStrategy>
</policyEntry>
</policyEntries>
</policyMap>
</destinationPolicy>
Joe
Free AMQ reference guide @ http://www.ttmsolutions.com
David Crisp wrote:
>
>
> Hi Bruce,
> How can I configure the broker to simply drop messages when they
> expire rather than sending them to the DLQ? Is there a
> "BitBucketDeadLetterStrategy" or something?
>
> -D
>
>
> bsnyder wrote:
>>
>> On Wed, Aug 20, 2008 at 2:49 PM, Ryan Moquin <fr...@gmail.com>
>> wrote:
>>> I don't think I saw that option? I assumed the timetolive would have
>>> that
>>> effect? I'll take another look at the settings, I may have overlooked
>>> that
>>> one. I hope so because I'm getting pretty desperate and I'm positive
>>> that
>>> I've correctly tried all the other options correctly, which is why I'm
>>> so
>>> puzzled.
>>
>> OK, my mistake in suggesting the use of the expiration. I just tested
>> the expiration and the timeToLive on the JMS message in ActiveMQ 4.1.1
>> and 5.1. The expiration doesn't seem to have any affect on either
>> broker, but the timeToLive does. In ActiveMQ 4.1.1, the timeToLive
>> doesn't affect message expiration but in ActiveMQ 5.1, setting a
>> timeToLive will cause the message to be delivered to the DLQ if the
>> timeToLive surpasses and the message is still pending.
>>
>> Bruce
>> --
>> perl -e 'print
>> unpack("u30","D0G)U8V4\@4VYY9&5R\"F)R=6-E+G-N>61E<D\!G;6%I;\"YC;VT*"
>> );'
>>
>> Apache ActiveMQ - http://activemq.org/
>> Apache Camel - http://activemq.org/camel/
>> Apache ServiceMix - http://servicemix.org/
>>
>> Blog: http://bruceblog.org/
>>
>>
>
>
--
View this message in context: http://www.nabble.com/ActiveMQ-causes-producer-to-hang-when-no-consumer-on-topic-after-a-few-minutes.-tp19061289p19117910.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.
Re: ActiveMQ causes producer to hang when no consumer on topic
after a few minutes.
Posted by David Crisp <da...@yahoo.com>.
Hi Bruce,
How can I configure the broker to simply drop messages when they expire
rather than sending them to the DLQ? Is there a
"BitBucketDeadLetterStrategy" or something?
-D
bsnyder wrote:
>
> On Wed, Aug 20, 2008 at 2:49 PM, Ryan Moquin <fr...@gmail.com>
> wrote:
>> I don't think I saw that option? I assumed the timetolive would have
>> that
>> effect? I'll take another look at the settings, I may have overlooked
>> that
>> one. I hope so because I'm getting pretty desperate and I'm positive
>> that
>> I've correctly tried all the other options correctly, which is why I'm so
>> puzzled.
>
> OK, my mistake in suggesting the use of the expiration. I just tested
> the expiration and the timeToLive on the JMS message in ActiveMQ 4.1.1
> and 5.1. The expiration doesn't seem to have any affect on either
> broker, but the timeToLive does. In ActiveMQ 4.1.1, the timeToLive
> doesn't affect message expiration but in ActiveMQ 5.1, setting a
> timeToLive will cause the message to be delivered to the DLQ if the
> timeToLive surpasses and the message is still pending.
>
> Bruce
> --
> perl -e 'print
> unpack("u30","D0G)U8V4\@4VYY9&5R\"F)R=6-E+G-N>61E<D\!G;6%I;\"YC;VT*"
> );'
>
> Apache ActiveMQ - http://activemq.org/
> Apache Camel - http://activemq.org/camel/
> Apache ServiceMix - http://servicemix.org/
>
> Blog: http://bruceblog.org/
>
>
--
View this message in context: http://www.nabble.com/ActiveMQ-causes-producer-to-hang-when-no-consumer-on-topic-after-a-few-minutes.-tp19061289p19116956.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.
Re: ActiveMQ causes producer to hang when no consumer on topic after a few minutes.
Posted by Bruce Snyder <br...@gmail.com>.
On Fri, Aug 22, 2008 at 11:33 AM, Ryan Moquin <fr...@gmail.com> wrote:
> Ok, I am using the default servicemix settings without any tweaks. I try to
> avoid whenever possible making any changes to the default config since it's
> just one more thing for me to have to maintain and remember it's there in
> case I hit other problems.
>
> Sure, I'll make several threads dumps.. do you want me to do it after the
> threads starting hanging? or during the time that they are starting to hang?
Yes, definitely grab the thread dumps while the server is hanging,
maybe even just before it starts to hang as well. This will give us a
much better picture of what's happening. Also, if you can, try to map
each one to a time if possible so we know in what order they were
taken.
Bruce
--
perl -e 'print unpack("u30","D0G)U8V4\@4VYY9&5R\"F)R=6-E+G-N>61E<D\!G;6%I;\"YC;VT*"
);'
Apache ActiveMQ - http://activemq.org/
Apache Camel - http://activemq.org/camel/
Apache ServiceMix - http://servicemix.org/
Blog: http://bruceblog.org/
Re: ActiveMQ causes producer to hang when no consumer on topic after a few minutes.
Posted by Ryan Moquin <fr...@gmail.com>.
Ok, I am using the default servicemix settings without any tweaks. I try to
avoid whenever possible making any changes to the default config since it's
just one more thing for me to have to maintain and remember it's there in
case I hit other problems.
Sure, I'll make several threads dumps.. do you want me to do it after the
threads starting hanging? or during the time that they are starting to hang?
On Fri, Aug 22, 2008 at 12:11 AM, Bruce Snyder <br...@gmail.com>wrote:
> On Thu, Aug 21, 2008 at 9:43 PM, Ryan Moquin <fr...@gmail.com>
> wrote:
> > Hi Bruce,
> >
> > I tried my service with the two settings:
> >
> > deliveryMode="1"
> > timeToLive="1000"
> >
> > The scenario is an external JMS client sends a message to a JMS consumer
> SU
> > in servicemix. It forwards that to my service which does some processing
> > with the incoming message, and then creates a new message that it sends
> to a
> > JMS provider SU. I ran this using Servicemix 3.3.
> >
> > I loaded up jconsole and watched my provider topic. As soon as my Topic
> > size hit 1321, my service all of a sudden stopped receiving messages from
> my
> > external sender and went completely silent. If I look at the topic that
> my
> > external sender, is sending to, it is still filling up with messages that
> > are no longer being received by my hanging service. When my external
> sender
> > finished, I see that 1500 messages were enqueued and dequeued on my
> consumer
> > JMS SU, but only 1340 were enqueued on my provider JMS SU.
> >
> > So, I did a stack dump on the JVM and noticed there are 35 hanging
> threads
> > that are all hanging trying to send a message to the JMS provider. So
> > apparently, the:
> >
> > <property name="useAsyncSend" value="true" />
> >
> > is working and I just didn't notice that there were 35 hanging threads.
> The
> > problem is, I guess once enough threads are tied up because the messages
> > can't be sent, then everything starts hanging. I have the message time
> to
> > live sent to a 1000. You said that it worked in 5.1, but it doesn't seem
> to
> > be working for me. Is that value really in milliseconds? Is there
> > something else I need? The non persistent delivery mode obviously isn't
> > working either. Do I have any options for preventing this situation from
> > happening? I don't see any other options in the config I can try.
> >
> > Here is the stack dump that is the same for all 35 threads:
> >
> > "pool-component.servicemix-jms-thread-3" prio=6 tid=0x39a4b400 nid=0x2724
> > waiting on condition [0x3f58e000..0x3f58fb14]
> > java.lang.Thread.State: WAITING (parking)
> > at sun.misc.Unsafe.park(Native Method)
> > - parking to wait for <0x1774ee48> (a
> > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> > at
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> > at
> >
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:19
>
> I'm thinking that the issues you're experiencing are ServiceMix
> related, not ActiveMQ related. It sounds like messages may not be
> consumed fast enough which could cause the slowdown you describe, but
> I'm not sure yet. I'd like to see how you have the thread pooling
> configured in ServiceMix, so please post your servicemix.properties
> file. Also, please provide two full thread dumps (kill -3 <pid>) of
> ServiceMix that are, maybe say, 20-30 seconds apart. These three items
> should help us to troubleshoot the situation a bit further.
>
> Bruce
> --
> perl -e 'print
> unpack("u30","D0G)U8V4\@4VYY9&5R\"F)R=6-E+G-N>61E<D\!G;6%I;\"YC;VT*"
> );'
>
> Apache ActiveMQ - http://activemq.org/
> Apache Camel - http://activemq.org/camel/
> Apache ServiceMix - http://servicemix.org/
>
> Blog: http://bruceblog.org/
>
Re: ActiveMQ causes producer to hang when no consumer on topic after a few minutes.
Posted by Bruce Snyder <br...@gmail.com>.
On Thu, Aug 21, 2008 at 9:43 PM, Ryan Moquin <fr...@gmail.com> wrote:
> Hi Bruce,
>
> I tried my service with the two settings:
>
> deliveryMode="1"
> timeToLive="1000"
>
> The scenario is an external JMS client sends a message to a JMS consumer SU
> in servicemix. It forwards that to my service which does some processing
> with the incoming message, and then creates a new message that it sends to a
> JMS provider SU. I ran this using Servicemix 3.3.
>
> I loaded up jconsole and watched my provider topic. As soon as my Topic
> size hit 1321, my service all of a sudden stopped receiving messages from my
> external sender and went completely silent. If I look at the topic that my
> external sender, is sending to, it is still filling up with messages that
> are no longer being received by my hanging service. When my external sender
> finished, I see that 1500 messages were enqueued and dequeued on my consumer
> JMS SU, but only 1340 were enqueued on my provider JMS SU.
>
> So, I did a stack dump on the JVM and noticed there are 35 hanging threads
> that are all hanging trying to send a message to the JMS provider. So
> apparently, the:
>
> <property name="useAsyncSend" value="true" />
>
> is working and I just didn't notice that there were 35 hanging threads. The
> problem is, I guess once enough threads are tied up because the messages
> can't be sent, then everything starts hanging. I have the message time to
> live sent to a 1000. You said that it worked in 5.1, but it doesn't seem to
> be working for me. Is that value really in milliseconds? Is there
> something else I need? The non persistent delivery mode obviously isn't
> working either. Do I have any options for preventing this situation from
> happening? I don't see any other options in the config I can try.
>
> Here is the stack dump that is the same for all 35 threads:
>
> "pool-component.servicemix-jms-thread-3" prio=6 tid=0x39a4b400 nid=0x2724
> waiting on condition [0x3f58e000..0x3f58fb14]
> java.lang.Thread.State: WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for <0x1774ee48> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:19
I'm thinking that the issues you're experiencing are ServiceMix
related, not ActiveMQ related. It sounds like messages may not be
consumed fast enough which could cause the slowdown you describe, but
I'm not sure yet. I'd like to see how you have the thread pooling
configured in ServiceMix, so please post your servicemix.properties
file. Also, please provide two full thread dumps (kill -3 <pid>) of
ServiceMix that are, maybe say, 20-30 seconds apart. These three items
should help us to troubleshoot the situation a bit further.
Bruce
--
perl -e 'print unpack("u30","D0G)U8V4\@4VYY9&5R\"F)R=6-E+G-N>61E<D\!G;6%I;\"YC;VT*"
);'
Apache ActiveMQ - http://activemq.org/
Apache Camel - http://activemq.org/camel/
Apache ServiceMix - http://servicemix.org/
Blog: http://bruceblog.org/
Re: ActiveMQ causes producer to hang when no consumer on topic after a few minutes.
Posted by Bruce Snyder <br...@gmail.com>.
On Thu, Aug 21, 2008 at 9:43 PM, Ryan Moquin <fr...@gmail.com> wrote:
> Hi Bruce,
>
> I tried my service with the two settings:
>
> deliveryMode="1"
> timeToLive="1000"
>
> The scenario is an external JMS client sends a message to a JMS consumer SU
> in servicemix. It forwards that to my service which does some processing
> with the incoming message, and then creates a new message that it sends to a
> JMS provider SU. I ran this using Servicemix 3.3.
>
> I loaded up jconsole and watched my provider topic. As soon as my Topic
> size hit 1321, my service all of a sudden stopped receiving messages from my
> external sender and went completely silent. If I look at the topic that my
> external sender, is sending to, it is still filling up with messages that
> are no longer being received by my hanging service. When my external sender
> finished, I see that 1500 messages were enqueued and dequeued on my consumer
> JMS SU, but only 1340 were enqueued on my provider JMS SU.
>
> So, I did a stack dump on the JVM and noticed there are 35 hanging threads
> that are all hanging trying to send a message to the JMS provider. So
> apparently, the:
>
> <property name="useAsyncSend" value="true" />
>
> is working and I just didn't notice that there were 35 hanging threads. The
> problem is, I guess once enough threads are tied up because the messages
> can't be sent, then everything starts hanging. I have the message time to
> live sent to a 1000. You said that it worked in 5.1, but it doesn't seem to
> be working for me. Is that value really in milliseconds? Is there
> something else I need? The non persistent delivery mode obviously isn't
> working either. Do I have any options for preventing this situation from
> happening? I don't see any other options in the config I can try.
>
> Here is the stack dump that is the same for all 35 threads:
>
> "pool-component.servicemix-jms-thread-3" prio=6 tid=0x39a4b400 nid=0x2724
> waiting on condition [0x3f58e000..0x3f58fb14]
> java.lang.Thread.State: WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for <0x1774ee48> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:19
I'm thinking that the issues you're experiencing are ServiceMix
related, not ActiveMQ related. It sounds like messages may not be
consumed fast enough which could cause the slowdown you describe, but
I'm not sure yet. I'd like to see how you have the thread pooling
configured in ServiceMix, so please post your servicemix.properties
file. Also, please provide two full thread dumps (kill -3 <pid>) of
ServiceMix that are, maybe say, 20-30 seconds apart. These three items
should help us to troubleshoot the situation a bit further.
Bruce
--
perl -e 'print unpack("u30","D0G)U8V4\@4VYY9&5R\"F)R=6-E+G-N>61E<D\!G;6%I;\"YC;VT*"
);'
Apache ActiveMQ - http://activemq.org/
Apache Camel - http://activemq.org/camel/
Apache ServiceMix - http://servicemix.org/
Blog: http://bruceblog.org/
Re: ActiveMQ causes producer to hang when no consumer on topic after a few minutes.
Posted by Ryan Moquin <fr...@gmail.com>.
Hi Bruce,
I tried my service with the two settings:
deliveryMode="1"
timeToLive="1000"
The scenario is an external JMS client sends a message to a JMS consumer SU
in servicemix. It forwards that to my service which does some processing
with the incoming message, and then creates a new message that it sends to a
JMS provider SU. I ran this using Servicemix 3.3.
I loaded up jconsole and watched my provider topic. As soon as my Topic
size hit 1321, my service all of a sudden stopped receiving messages from my
external sender and went completely silent. If I look at the topic that my
external sender, is sending to, it is still filling up with messages that
are no longer being received by my hanging service. When my external sender
finished, I see that 1500 messages were enqueued and dequeued on my consumer
JMS SU, but only 1340 were enqueued on my provider JMS SU.
So, I did a stack dump on the JVM and noticed there are 35 hanging threads
that are all hanging trying to send a message to the JMS provider. So
apparently, the:
<property name="useAsyncSend" value="true" />
is working and I just didn't notice that there were 35 hanging threads. The
problem is, I guess once enough threads are tied up because the messages
can't be sent, then everything starts hanging. I have the message time to
live sent to a 1000. You said that it worked in 5.1, but it doesn't seem to
be working for me. Is that value really in milliseconds? Is there
something else I need? The non persistent delivery mode obviously isn't
working either. Do I have any options for preventing this situation from
happening? I don't see any other options in the config I can try.
Here is the stack dump that is the same for all 35 threads:
"pool-component.servicemix-jms-thread-3" prio=6 tid=0x39a4b400 nid=0x2724
waiting on condition [0x3f58e000..0x3f58fb14]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x1774ee48> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:19
25)
at
java.util.concurrent.ArrayBlockingQueue.put(ArrayBlockingQueue.java:252)
at
org.apache.servicemix.jbi.messaging.DeliveryChannelImpl.processInBound(DeliveryChannelImpl.java:663)
at
org.apache.servicemix.jbi.nmr.flow.AbstractFlow.doRouting(AbstractFlow.java:172)
at
org.apache.servicemix.jbi.nmr.flow.seda.SedaFlow.doRouting(SedaFlow.java:168)
at
org.apache.servicemix.jbi.nmr.flow.seda.SedaQueue$1.run(SedaQueue.java:134)
at
java.util.concurrent.ThreadPoolExecutor$CallerRunsPolicy.rejectedExecution(ThreadPoolExecutor.java:1737)
at
java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:767)
at
java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:658)
at
org.apache.servicemix.executors.impl.ExecutorImpl.execute(ExecutorImpl.java:43)
at
org.apache.servicemix.jbi.nmr.flow.seda.SedaQueue.enqueue(SedaQueue.java:128)
at
org.apache.servicemix.jbi.nmr.flow.seda.SedaFlow.enqueuePacket(SedaFlow.java:183)
at
org.apache.servicemix.jbi.nmr.flow.seda.SedaFlow.doSend(SedaFlow.java:163)
at
org.apache.servicemix.jbi.nmr.flow.AbstractFlow.send(AbstractFlow.java:123)
at
org.apache.servicemix.jbi.nmr.DefaultBroker.sendExchangePacket(DefaultBroker.java:283)
at
org.apache.servicemix.jbi.security.SecuredBroker.sendExchangePacket(SecuredBroker.java:88)
at
org.apache.servicemix.jbi.container.JBIContainer.sendExchange(JBIContainer.java:843)
at
org.apache.servicemix.jbi.messaging.DeliveryChannelImpl.doSend(DeliveryChannelImpl.java:395)
at
org.apache.servicemix.jbi.messaging.DeliveryChannelImpl.send(DeliveryChannelImpl.java:431)
at
org.apache.servicemix.common.EndpointDeliveryChannel.send(EndpointDeliveryChannel.java:88)
at
org.apache.servicemix.common.endpoints.SimpleEndpoint.send(SimpleEndpoint.java:67)
at
org.apache.servicemix.common.endpoints.SimpleEndpoint.done(SimpleEndpoint.java:78)
at
org.apache.servicemix.common.endpoints.ProviderEndpoint.process(ProviderEndpoint.java:101)
at
org.apache.servicemix.common.AsyncBaseLifeCycle.doProcess(AsyncBaseLifeCycle.java:597)
at
org.apache.servicemix.common.AsyncBaseLifeCycle.processExchange(AsyncBaseLifeCycle.java:548)
at
org.apache.servicemix.common.AsyncBaseLifeCycle.onMessageExchange(AsyncBaseLifeCycle.java:502)
at
org.apache.servicemix.common.SyncLifeCycleWrapper.onMessageExchange(SyncLifeCycleWrapper.java:60)
at
org.apache.servicemix.jbi.messaging.DeliveryChannelImpl.processInBound(DeliveryChannelImpl.java:610)
at
org.apache.servicemix.jbi.nmr.flow.AbstractFlow.doRouting(AbstractFlow.java:172)
at
org.apache.servicemix.jbi.nmr.flow.seda.SedaFlow.doRouting(SedaFlow.java:168)
at
org.apache.servicemix.jbi.nmr.flow.seda.SedaQueue$1.run(SedaQueue.java:134)
at
java.util.concurrent.ThreadPoolExecutor$CallerRunsPolicy.rejectedExecution(ThreadPoolExecutor.java:1737)
at
java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:767)
at
java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:658)
at
org.apache.servicemix.executors.impl.ExecutorImpl.execute(ExecutorImpl.java:43)
at
org.apache.servicemix.jbi.nmr.flow.seda.SedaQueue.enqueue(SedaQueue.java:128)
at
org.apache.servicemix.jbi.nmr.flow.seda.SedaFlow.enqueuePacket(SedaFlow.java:183)
at
org.apache.servicemix.jbi.nmr.flow.seda.SedaFlow.doSend(SedaFlow.java:163)
at
org.apache.servicemix.jbi.nmr.flow.AbstractFlow.send(AbstractFlow.java:123)
at
org.apache.servicemix.jbi.nmr.DefaultBroker.sendExchangePacket(DefaultBroker.java:283)
at
org.apache.servicemix.jbi.security.SecuredBroker.sendExchangePacket(SecuredBroker.java:88)
at
org.apache.servicemix.jbi.container.JBIContainer.sendExchange(JBIContainer.java:843)
at
org.apache.servicemix.jbi.messaging.DeliveryChannelImpl.doSend(DeliveryChannelImpl.java:395)
at
org.apache.servicemix.jbi.messaging.DeliveryChannelImpl.send(DeliveryChannelImpl.java:431)
at
org.apache.servicemix.client.DefaultServiceMixClient.send(DefaultServiceMixClient.java:147)
at
org.apache.servicemix.client.DefaultServiceMixClient.send(DefaultServiceMixClient.java:211)
On Wed, Aug 20, 2008 at 11:39 PM, Ryan Moquin <fr...@gmail.com>wrote:
> I should mention though that I did see settings for the new servicemix-jms
> endpoints actually that should solve the problems I'm having, but I don't
> think any of them work. I'll try them again and let you know which ones and
> then you can tell me if I'm screwing up and using them wrong, though they
> aren't complicated so I can't imagine I'm using them wrong.
>
>
> On Wed, Aug 20, 2008 at 5:05 PM, Bruce Snyder <br...@gmail.com>wrote:
>
>> On Wed, Aug 20, 2008 at 2:49 PM, Ryan Moquin <fr...@gmail.com>
>> wrote:
>> > I don't think I saw that option? I assumed the timetolive would have
>> that
>> > effect? I'll take another look at the settings, I may have overlooked
>> that
>> > one. I hope so because I'm getting pretty desperate and I'm positive
>> that
>> > I've correctly tried all the other options correctly, which is why I'm
>> so
>> > puzzled.
>>
>> OK, my mistake in suggesting the use of the expiration. I just tested
>> the expiration and the timeToLive on the JMS message in ActiveMQ 4.1.1
>> and 5.1. The expiration doesn't seem to have any affect on either
>> broker, but the timeToLive does. In ActiveMQ 4.1.1, the timeToLive
>> doesn't affect message expiration but in ActiveMQ 5.1, setting a
>> timeToLive will cause the message to be delivered to the DLQ if the
>> timeToLive surpasses and the message is still pending.
>>
>> Bruce
>> --
>> perl -e 'print
>> unpack("u30","D0G)U8V4\@4VYY9&5R\"F)R=6-E+G-N>61E<D\!G;6%I;\"YC;VT*"
>> );'
>>
>> Apache ActiveMQ - http://activemq.org/
>> Apache Camel - http://activemq.org/camel/
>> Apache ServiceMix - http://servicemix.org/
>>
>> Blog: http://bruceblog.org/
>>
>
>
Re: ActiveMQ causes producer to hang when no consumer on topic after a few minutes.
Posted by Ryan Moquin <fr...@gmail.com>.
I should mention though that I did see settings for the new servicemix-jms
endpoints actually that should solve the problems I'm having, but I don't
think any of them work. I'll try them again and let you know which ones and
then you can tell me if I'm screwing up and using them wrong, though they
aren't complicated so I can't imagine I'm using them wrong.
On Wed, Aug 20, 2008 at 5:05 PM, Bruce Snyder <br...@gmail.com>wrote:
> On Wed, Aug 20, 2008 at 2:49 PM, Ryan Moquin <fr...@gmail.com>
> wrote:
> > I don't think I saw that option? I assumed the timetolive would have
> that
> > effect? I'll take another look at the settings, I may have overlooked
> that
> > one. I hope so because I'm getting pretty desperate and I'm positive
> that
> > I've correctly tried all the other options correctly, which is why I'm so
> > puzzled.
>
> OK, my mistake in suggesting the use of the expiration. I just tested
> the expiration and the timeToLive on the JMS message in ActiveMQ 4.1.1
> and 5.1. The expiration doesn't seem to have any affect on either
> broker, but the timeToLive does. In ActiveMQ 4.1.1, the timeToLive
> doesn't affect message expiration but in ActiveMQ 5.1, setting a
> timeToLive will cause the message to be delivered to the DLQ if the
> timeToLive surpasses and the message is still pending.
>
> Bruce
> --
> perl -e 'print
> unpack("u30","D0G)U8V4\@4VYY9&5R\"F)R=6-E+G-N>61E<D\!G;6%I;\"YC;VT*"
> );'
>
> Apache ActiveMQ - http://activemq.org/
> Apache Camel - http://activemq.org/camel/
> Apache ServiceMix - http://servicemix.org/
>
> Blog: http://bruceblog.org/
>
Re: ActiveMQ causes producer to hang when no consumer on topic after a few minutes.
Posted by Ryan Moquin <fr...@gmail.com>.
Interesting. Ok, I have that on, but I don't think I tried without a
consumer in 3.3. The only problem in that case then, is that if my producer
sends 1300 messages and a consumer receives those 1300 messages, my producer
starts to block in that case as well. It seems like no matter what, my
producer blocks at either 1100 or 1300 whether or not there is a consumer.
I'm not sure why ActiveMQ never seems to throw a message away. Since
ActiveMQ seems to hold onto every single message, it seems like the
timetolive should also cause those messages to be killed instead of them
persisting forever. This behavior explains why if somehow I don't hit the
producer hanging first, under a high load Servicemix will always eventually
run out of memory because ActiveMQ never discards any of the messages on any
of the destinations... I'm not sure what to do about that either. People
obviously use ActiveMQ in production, so there has to be some way to rectify
the situation.
On Wed, Aug 20, 2008 at 5:05 PM, Bruce Snyder <br...@gmail.com>wrote:
> On Wed, Aug 20, 2008 at 2:49 PM, Ryan Moquin <fr...@gmail.com>
> wrote:
> > I don't think I saw that option? I assumed the timetolive would have
> that
> > effect? I'll take another look at the settings, I may have overlooked
> that
> > one. I hope so because I'm getting pretty desperate and I'm positive
> that
> > I've correctly tried all the other options correctly, which is why I'm so
> > puzzled.
>
> OK, my mistake in suggesting the use of the expiration. I just tested
> the expiration and the timeToLive on the JMS message in ActiveMQ 4.1.1
> and 5.1. The expiration doesn't seem to have any affect on either
> broker, but the timeToLive does. In ActiveMQ 4.1.1, the timeToLive
> doesn't affect message expiration but in ActiveMQ 5.1, setting a
> timeToLive will cause the message to be delivered to the DLQ if the
> timeToLive surpasses and the message is still pending.
>
> Bruce
> --
> perl -e 'print
> unpack("u30","D0G)U8V4\@4VYY9&5R\"F)R=6-E+G-N>61E<D\!G;6%I;\"YC;VT*"
> );'
>
> Apache ActiveMQ - http://activemq.org/
> Apache Camel - http://activemq.org/camel/
> Apache ServiceMix - http://servicemix.org/
>
> Blog: http://bruceblog.org/
>
Re: ActiveMQ causes producer to hang when no consumer on topic after a few minutes.
Posted by Bruce Snyder <br...@gmail.com>.
On Wed, Aug 20, 2008 at 2:49 PM, Ryan Moquin <fr...@gmail.com> wrote:
> I don't think I saw that option? I assumed the timetolive would have that
> effect? I'll take another look at the settings, I may have overlooked that
> one. I hope so because I'm getting pretty desperate and I'm positive that
> I've correctly tried all the other options correctly, which is why I'm so
> puzzled.
OK, my mistake in suggesting the use of the expiration. I just tested
the expiration and the timeToLive on the JMS message in ActiveMQ 4.1.1
and 5.1. The expiration doesn't seem to have any affect on either
broker, but the timeToLive does. In ActiveMQ 4.1.1, the timeToLive
doesn't affect message expiration but in ActiveMQ 5.1, setting a
timeToLive will cause the message to be delivered to the DLQ if the
timeToLive surpasses and the message is still pending.
Bruce
--
perl -e 'print unpack("u30","D0G)U8V4\@4VYY9&5R\"F)R=6-E+G-N>61E<D\!G;6%I;\"YC;VT*"
);'
Apache ActiveMQ - http://activemq.org/
Apache Camel - http://activemq.org/camel/
Apache ServiceMix - http://servicemix.org/
Blog: http://bruceblog.org/
Re: ActiveMQ causes producer to hang when no consumer on topic after a few minutes.
Posted by Ryan Moquin <fr...@gmail.com>.
I don't think I saw that option? I assumed the timetolive would have that
effect? I'll take another look at the settings, I may have overlooked that
one. I hope so because I'm getting pretty desperate and I'm positive that
I've correctly tried all the other options correctly, which is why I'm so
puzzled.
On Wed, Aug 20, 2008 at 4:03 PM, Bruce Snyder <br...@gmail.com>wrote:
> On Tue, Aug 19, 2008 at 8:25 PM, Ryan Moquin <fr...@gmail.com>
> wrote:
> > So I guess the issue is that this is a behavior of ActiveMQ by default.
> > Does the non-persistent deliveryMode work in activemq 4.1.1? I'm stuck
> on
> > that version since that is what the latest Servicemix uses, but when I
> > attempt to set the jms:provider to use a deliverymode of "1", my producer
> > will still hang when the topic message enqueue count goes above 1100.
> Are
> > there any other options for having these messages not hang around on the
> > topic? I tried the TTL, but that setting does absolutely nothing as
> well.
> >
> > Any suggestions?
>
> Yes, ActiveMQ holds pending messages (i.e., messages not yet delivered
> to a consumer) in memory by default until they're consumed. ActiveMQ
> 5.1 has various strategies for spooling them to disk, but these aren't
> available in 4.x. How about setting the expiration (JMSExpiration) on
> the messages so that the broker won't deliver them if they're expired?
>
> Bruce
> --
> perl -e 'print
> unpack("u30","D0G)U8V4\@4VYY9&5R\"F)R=6-E+G-N>61E<D\!G;6%I;\"YC;VT*"
> );'
>
> Apache ActiveMQ - http://activemq.org/
> Apache Camel - http://activemq.org/camel/
> Apache ServiceMix - http://servicemix.org/
>
> Blog: http://bruceblog.org/
>
Re: ActiveMQ causes producer to hang when no consumer on topic after a few minutes.
Posted by Bruce Snyder <br...@gmail.com>.
On Tue, Aug 19, 2008 at 8:25 PM, Ryan Moquin <fr...@gmail.com> wrote:
> So I guess the issue is that this is a behavior of ActiveMQ by default.
> Does the non-persistent deliveryMode work in activemq 4.1.1? I'm stuck on
> that version since that is what the latest Servicemix uses, but when I
> attempt to set the jms:provider to use a deliverymode of "1", my producer
> will still hang when the topic message enqueue count goes above 1100. Are
> there any other options for having these messages not hang around on the
> topic? I tried the TTL, but that setting does absolutely nothing as well.
>
> Any suggestions?
Yes, ActiveMQ holds pending messages (i.e., messages not yet delivered
to a consumer) in memory by default until they're consumed. ActiveMQ
5.1 has various strategies for spooling them to disk, but these aren't
available in 4.x. How about setting the expiration (JMSExpiration) on
the messages so that the broker won't deliver them if they're expired?
Bruce
--
perl -e 'print unpack("u30","D0G)U8V4\@4VYY9&5R\"F)R=6-E+G-N>61E<D\!G;6%I;\"YC;VT*"
);'
Apache ActiveMQ - http://activemq.org/
Apache Camel - http://activemq.org/camel/
Apache ServiceMix - http://servicemix.org/
Blog: http://bruceblog.org/
Re: ActiveMQ causes producer to hang when no consumer on topic after a few minutes.
Posted by Ryan Moquin <fr...@gmail.com>.
So I guess the issue is that this is a behavior of ActiveMQ by default.
Does the non-persistent deliveryMode work in activemq 4.1.1? I'm stuck on
that version since that is what the latest Servicemix uses, but when I
attempt to set the jms:provider to use a deliverymode of "1", my producer
will still hang when the topic message enqueue count goes above 1100. Are
there any other options for having these messages not hang around on the
topic? I tried the TTL, but that setting does absolutely nothing as well.
Any suggestions?
On Tue, Aug 19, 2008 at 8:54 PM, Ryan Moquin <fr...@gmail.com> wrote:
> Hello,
>
> I'm trying to understand how to configure ActiveMQ (4.1.1 is what
> Servicemix is using) properly so that my Servicemix services will stop
> hanging after a few minutes of sending messages to a provider endpoint. The
> ActiveMQ provider endpoint is consistently deadlocking after a short period
> of time (about 10 minutes, 1500 messages). It appears as though activemq is
> configured out of the box (or maybe Servicemix configures it this way) to
> persistent every single message in memory whether or not it's been delivered
> (or atleast maybe the ones that aren't delivered, not completely sure).
> Basically, the messages I'm producing should be thrown away if there is no
> consumer. But currently the messages just pile up on my destination topic
> in ActiveMQ until it starts refusing to accept anymore messages. This
> happens extremely quickly, as I mentioned above, when there are no consumers
> subscribed to the topic.
>
> I've tried setting the TTL for the messages to 1000, I've tried setting the
> deliverymode to non-persistent, but no matter what, my service will
> deadlock. When I look at the topic in the JMX manager, I will see a very
> high enqueue count, which I'm assuming means my messages are just sitting
> there, waiting for a consumer. Is there any way to make activemq not keep
> these messages around?
>
> Thanks!
>