You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@activemq.apache.org by nbreau <nb...@rogers.com> on 2006/10/05 15:00:27 UTC
Hang on publish of non-persistent message
Hi All,
I've noticed that ActiveMQ has been occasionaly hanging causing deadlock in
our application when trying to publish a message. While debugging i noticed
that when the publish becomes hung if i kill activemq and restart it the
message completes its publishing and the code continues to execute.
The deliver mode is non persistent, the producer connection is set to
connectionFactory.setUseAsyncSend(true) and the consumer connection is
connectionFactory.setAsyncDispatch(true).
At this point I'm not sure where to look to solve this issue.. since there
is no actuall exception being thrown or stack trace/message to look out.
thanks,
Nick.
--
View this message in context: http://www.nabble.com/Hang-on-publish-of-non-persistent-message-tf2388290.html#a6658067
Sent from the ActiveMQ - User mailing list archive at Nabble.com.
Re: Hang on publish of non-persistent message
Posted by James Strachan <ja...@gmail.com>.
On 10/6/06, Hiram Chirino <hi...@hiramchirino.com> wrote:
> On 10/5/06, John Heitmann <jh...@gmail.com> wrote:
> >
> > On Oct 5, 2006, at 6:40 AM, nbreau wrote:
> >
> > > I'm not sure if this issue is related.... my issue occurs randomly,
> > > within
> > > the first 50-100 messages that get sent, and it's only a single
> > > producer
> > > that hangs, but over time all my producer threads would become hung.
> > >
> > > Is there some kind of timeout i can apply to a publish call ?
> >
> > Unfortunately the useAsyncSend setting doesn't have the entire
> > semantic you might expect. At the lowest level it will still perform
> > synchronous network operations in the same thread as your publish.
> > This means that if there's a TCP backlog (because of a slow network,
> > down network, throttling broker, buggy broker, etc. etc.) then even
> > the async send can block.
> >
> > One thing to check would be to connect to your broker with JMX and
> > look at the thread handling the hung connection and see if and where
> > it's blocked. If it's blocked on the UsageManager then you need to
> > either raise your UsageManager memory limit or turn on the new non-
> > blocking UsageManager flag.
> >
> > If it's not blocked on the UsageManager then you're probably hitting
> > an unknown problem. If you could find the blockage and let us know
> > where it is that would rock (assuming it's not just a problem on your
> > network).
> >
> > I've got a transport filter I've been meaning to submit that sits in
> > front of almost all other transport filters and provides true
> > asynchronous sending by adding a tiny client-side queue. This helps
> > avoid bugs in transport code, hitches in the network, and hitches in
> > the broker that could otherwise lead to publish blocking. I'll get
> > that dusted off and published. The downside to it is that it means
> > publishes are 100% unthrottled, which can easily lead to dropped
> > messages if you publish too fast (it throws and exception so you know
> > what's happening at least).
>
> Awesome Idea. Please send us the patch!
Agreed - and that Jedi patch too :)
>
> Hopefully when you submit the patch it will add a sendBufferSize
> property. Where it's value helps us configure if a send is 'truely
> async' or not.
> The sendBufferSize would be the the maximum number of messages that
> will buffer for your 'true' async sends. This setting should default
> to 0 and in essence disable 'true' async sends. It should be a
> property on both the connection factory and the connection.
>
> And hopefully it also has a sendTimeout property on the connection
> factory, connection, and producer destination option. It would only
> be used if 'true' async sends are being used and it values should work
> something like:
> if sendTimeout=0 then sends block until more space is available (like
> we do today).
> if sendTimeout<0 then if the client side send buffer is full, we throw
> and exception right away
> if sendTimeout>0 then the send waits at least sendTimeout ms
> attempting to get space on that buffer before throwing a timeout.
Yeah. I guess using a BlockingQueue and offer() in the calling thread
then a separate thread doing the actual writing to the socket. Then
the caller thread could use a timeout based offer
http://java.sun.com/j2se/1.5.0/docs/api/java/util/concurrent/BlockingQueue.html#offer(E,%20long,%20java.util.concurrent.TimeUnit)
to avoid blocking the calling thread too long then failing if the
timeout expires.
--
James
-------
http://radio.weblogs.com/0112098/
Re: Hang on publish of non-persistent message
Posted by Hiram Chirino <hi...@hiramchirino.com>.
On 10/5/06, John Heitmann <jh...@gmail.com> wrote:
>
> On Oct 5, 2006, at 6:40 AM, nbreau wrote:
>
> > I'm not sure if this issue is related.... my issue occurs randomly,
> > within
> > the first 50-100 messages that get sent, and it's only a single
> > producer
> > that hangs, but over time all my producer threads would become hung.
> >
> > Is there some kind of timeout i can apply to a publish call ?
>
> Unfortunately the useAsyncSend setting doesn't have the entire
> semantic you might expect. At the lowest level it will still perform
> synchronous network operations in the same thread as your publish.
> This means that if there's a TCP backlog (because of a slow network,
> down network, throttling broker, buggy broker, etc. etc.) then even
> the async send can block.
>
> One thing to check would be to connect to your broker with JMX and
> look at the thread handling the hung connection and see if and where
> it's blocked. If it's blocked on the UsageManager then you need to
> either raise your UsageManager memory limit or turn on the new non-
> blocking UsageManager flag.
>
> If it's not blocked on the UsageManager then you're probably hitting
> an unknown problem. If you could find the blockage and let us know
> where it is that would rock (assuming it's not just a problem on your
> network).
>
> I've got a transport filter I've been meaning to submit that sits in
> front of almost all other transport filters and provides true
> asynchronous sending by adding a tiny client-side queue. This helps
> avoid bugs in transport code, hitches in the network, and hitches in
> the broker that could otherwise lead to publish blocking. I'll get
> that dusted off and published. The downside to it is that it means
> publishes are 100% unthrottled, which can easily lead to dropped
> messages if you publish too fast (it throws and exception so you know
> what's happening at least).
Awesome Idea. Please send us the patch!
Hopefully when you submit the patch it will add a sendBufferSize
property. Where it's value helps us configure if a send is 'truely
async' or not.
The sendBufferSize would be the the maximum number of messages that
will buffer for your 'true' async sends. This setting should default
to 0 and in essence disable 'true' async sends. It should be a
property on both the connection factory and the connection.
And hopefully it also has a sendTimeout property on the connection
factory, connection, and producer destination option. It would only
be used if 'true' async sends are being used and it values should work
something like:
if sendTimeout=0 then sends block until more space is available (like
we do today).
if sendTimeout<0 then if the client side send buffer is full, we throw
and exception right away
if sendTimeout>0 then the send waits at least sendTimeout ms
attempting to get space on that buffer before throwing a timeout.
>
> John
>
>
>
>
--
Regards,
Hiram
Blog: http://hiramchirino.com
Re: Hang on publish of non-persistent message
Posted by John Heitmann <jh...@gmail.com>.
On Oct 5, 2006, at 6:40 AM, nbreau wrote:
> I'm not sure if this issue is related.... my issue occurs randomly,
> within
> the first 50-100 messages that get sent, and it's only a single
> producer
> that hangs, but over time all my producer threads would become hung.
>
> Is there some kind of timeout i can apply to a publish call ?
Unfortunately the useAsyncSend setting doesn't have the entire
semantic you might expect. At the lowest level it will still perform
synchronous network operations in the same thread as your publish.
This means that if there's a TCP backlog (because of a slow network,
down network, throttling broker, buggy broker, etc. etc.) then even
the async send can block.
One thing to check would be to connect to your broker with JMX and
look at the thread handling the hung connection and see if and where
it's blocked. If it's blocked on the UsageManager then you need to
either raise your UsageManager memory limit or turn on the new non-
blocking UsageManager flag.
If it's not blocked on the UsageManager then you're probably hitting
an unknown problem. If you could find the blockage and let us know
where it is that would rock (assuming it's not just a problem on your
network).
I've got a transport filter I've been meaning to submit that sits in
front of almost all other transport filters and provides true
asynchronous sending by adding a tiny client-side queue. This helps
avoid bugs in transport code, hitches in the network, and hitches in
the broker that could otherwise lead to publish blocking. I'll get
that dusted off and published. The downside to it is that it means
publishes are 100% unthrottled, which can easily lead to dropped
messages if you publish too fast (it throws and exception so you know
what's happening at least).
John
Re: Hang on publish of non-persistent message
Posted by nbreau <nb...@rogers.com>.
I'm not sure if this issue is related.... my issue occurs randomly, within
the first 50-100 messages that get sent, and it's only a single producer
that hangs, but over time all my producer threads would become hung.
Is there some kind of timeout i can apply to a publish call ?
C. Benson Manica-2 wrote:
>
> I'm not sure if the issue you're reporting is related, but as it
> happens I have been experiencing some hanging issues myself which led
> me to
>
> http://www.nabble.com/activemq-clients-all-hang-t1950608.html
>
> and
>
> https://issues.apache.org/activemq/browse/AMQ-845
>
> , which unfortunately do not make for comforting reading. HTH...
>
> On Thu, Oct 05, 2006 at 06:00:27AM -0700, nbreau wrote:
>> Hi All,
>>
>> I've noticed that ActiveMQ has been occasionaly hanging causing deadlock
>> in
>> our application when trying to publish a message. While debugging i
>> noticed
>> that when the publish becomes hung if i kill activemq and restart it the
>> message completes its publishing and the code continues to execute.
>>
>> The deliver mode is non persistent, the producer connection is set to
>> connectionFactory.setUseAsyncSend(true) and the consumer connection is
>> connectionFactory.setAsyncDispatch(true).
>>
>> At this point I'm not sure where to look to solve this issue.. since
>> there
>> is no actuall exception being thrown or stack trace/message to look out.
>>
>> thanks,
>> Nick.
>>
>> --
>> View this message in context:
>> http://www.nabble.com/Hang-on-publish-of-non-persistent-message-tf2388290.html#a6658067
>> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>
> --
> C. Benson Manica
> ataru(at)sdf.lonestar.org
>
>
--
View this message in context: http://www.nabble.com/Hang-on-publish-of-non-persistent-message-tf2388290.html#a6658737
Sent from the ActiveMQ - User mailing list archive at Nabble.com.
Re: Hang on publish of non-persistent message
Posted by "C. Benson Manica" <at...@sdf.lonestar.org>.
I'm not sure if the issue you're reporting is related, but as it
happens I have been experiencing some hanging issues myself which led
me to
http://www.nabble.com/activemq-clients-all-hang-t1950608.html
and
https://issues.apache.org/activemq/browse/AMQ-845
, which unfortunately do not make for comforting reading. HTH...
On Thu, Oct 05, 2006 at 06:00:27AM -0700, nbreau wrote:
> Hi All,
>
> I've noticed that ActiveMQ has been occasionaly hanging causing deadlock in
> our application when trying to publish a message. While debugging i noticed
> that when the publish becomes hung if i kill activemq and restart it the
> message completes its publishing and the code continues to execute.
>
> The deliver mode is non persistent, the producer connection is set to
> connectionFactory.setUseAsyncSend(true) and the consumer connection is
> connectionFactory.setAsyncDispatch(true).
>
> At this point I'm not sure where to look to solve this issue.. since there
> is no actuall exception being thrown or stack trace/message to look out.
>
> thanks,
> Nick.
>
> --
> View this message in context: http://www.nabble.com/Hang-on-publish-of-non-persistent-message-tf2388290.html#a6658067
> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
--
C. Benson Manica
ataru(at)sdf.lonestar.org