You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@activemq.apache.org by Jason Rosenberg <jr...@shopzilla.com> on 2007/10/17 22:02:43 UTC
Redelivery sticks to and blocks a consumer thread
All,
I have been experimenting with using exponentialBackOff as a redelivery
policy. I have multiple parallel consumers. I'm using AMQ 4.1.1, using
Spring's DefaultMessageListener, within Tomcat 6.
I've noticed that when a message consumer has a failure, and throws an
exception, and the message gets scheduled for retry, the particular consumer
thread gets tied to the redelivery, and it doesn't process any more messages
until it's gone through the full redelivery schedule for the error prone
message. So, with the exponentialBackOff mode, it's easy to set it to wait
10 minutes before redelivering a failed message, and in that time that
consumer will be blocked.
If the consumer has a pre-fetch greater than 1, this means other messages
which might be perfectly valid for processing are blocked until the troubled
message goes through it's full redelivery cycle (which could be
infinite!)....
I'm wondering if there's an alternate way to configure things, so that a
message marked for redelivery goes back on the queue to be scheduled by the
next available consumer (and not until it's next scheduled redelivery time
has arrived!)....
If I have a queue with multiple consumers, and perhaps even a small
percentage of the messages are corrupt and force multiple retries, it could
eventually happen that all consumers are blocked waiting idly to retry the
message at the top of it's list, even though other messages are readily
available. This would seem to be the case even when the prefetch limit is
1.
Thoughts?
Jason
--
View this message in context: http://www.nabble.com/Redelivery-sticks-to-and-blocks-a-consumer-thread-tf4642986s2354.html#a13261850
Sent from the ActiveMQ - User mailing list archive at Nabble.com.
Re: Redelivery sticks to and blocks a consumer thread
Posted by Demian Mrakovich <ne...@hotmail.com>.
OK, I found some time to test this on 5.1 myself. Unfortunately the issue is
NOT resolved as far as I can tell.
I don't see how anyone can live with this...
Consider a system which processes 100 messages/sec, with MaximumRedeliveries
to 8 for a total of let's say an hour (in case of network failure against
remote services etc.) before going to DLQ. Then we get about 20 corrupt
messages in a row going into redelivery for a full 8 retries. The remaining
incoming messages are fine, but what is the condition of our system now?
With thousands of messages on queue, which could have been processed in the
mean time, the system is effectively down for a full hour!
I understand this blocking mechanism is by design to ensure correct order of
messages, but I think this should be optional, because having the system
down for a full hour is much worse than a few messages out of order (at
least for some systems) . So, is there a configuration option I have missed
somewhere or is this still a firm design decision?
Hoping for a reply,
Demian
rajdavies wrote:
>
> I'd certainly try 5.1
>
> cheers,
>
> Rob
> On 6 Jun 2008, at 14:05, Demian Mrakovich wrote:
>
>>
>> Hi, bumping this one... Has this (I'd have to say major) issue been
>> addressed
>> in newer versions of ActiveMQ?
>>
>>
>> Jason Rosenberg wrote:
>>>
>>> All,
>>>
>>> I have been experimenting with using exponentialBackOff as a
>>> redelivery
>>> policy. I have multiple parallel consumers. I'm using AMQ 4.1.1,
>>> using
>>> Spring's DefaultMessageListener, within Tomcat 6.
>>>
>>> I've noticed that when a message consumer has a failure, and throws
>>> an
>>> exception, and the message gets scheduled for retry, the particular
>>> consumer thread gets tied to the redelivery, and it doesn't process
>>> any
>>> more messages until it's gone through the full redelivery schedule
>>> for the
>>> error prone message. So, with the exponentialBackOff mode, it's
>>> easy to
>>> set it to wait 10 minutes before redelivering a failed message, and
>>> in
>>> that time that consumer will be blocked.
>>>
>>> If the consumer has a pre-fetch greater than 1, this means other
>>> messages
>>> which might be perfectly valid for processing are blocked until the
>>> troubled message goes through it's full redelivery cycle (which
>>> could be
>>> infinite!)....
>>>
>>> I'm wondering if there's an alternate way to configure things, so
>>> that a
>>> message marked for redelivery goes back on the queue to be
>>> scheduled by
>>> the next available consumer (and not until it's next scheduled
>>> redelivery
>>> time has arrived!)....
>>>
>>> If I have a queue with multiple consumers, and perhaps even a small
>>> percentage of the messages are corrupt and force multiple retries, it
>>> could eventually happen that all consumers are blocked waiting idly
>>> to
>>> retry the message at the top of it's list, even though other
>>> messages are
>>> readily available. This would seem to be the case even when the
>>> prefetch
>>> limit is 1.
>>>
>>> Thoughts?
>>>
>>> Jason
>>>
>>
>> --
>> View this message in context:
>> http://www.nabble.com/Redelivery-sticks-to-and-blocks-a-consumer-thread-tp13261850s2354p17691674.html
>> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>>
>
>
>
--
View this message in context: http://www.nabble.com/Redelivery-sticks-to-and-blocks-a-consumer-thread-tp13261850p18029513.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.
Re: Redelivery sticks to and blocks a consumer thread
Posted by Demian Mrakovich <ne...@hotmail.com>.
Thanks,
So you can confirm that threads are no longer blocked during retries in 5.1?
Can I find any docs/info on the retry mechanism in 5.1 anywhere?
rajdavies wrote:
>
> I'd certainly try 5.1
>
> cheers,
>
> Rob
> On 6 Jun 2008, at 14:05, Demian Mrakovich wrote:
>
>>
>> Hi, bumping this one... Has this (I'd have to say major) issue been
>> addressed
>> in newer versions of ActiveMQ?
>>
>>
>> Jason Rosenberg wrote:
>>>
>>> All,
>>>
>>> I have been experimenting with using exponentialBackOff as a
>>> redelivery
>>> policy. I have multiple parallel consumers. I'm using AMQ 4.1.1,
>>> using
>>> Spring's DefaultMessageListener, within Tomcat 6.
>>>
>>> I've noticed that when a message consumer has a failure, and throws
>>> an
>>> exception, and the message gets scheduled for retry, the particular
>>> consumer thread gets tied to the redelivery, and it doesn't process
>>> any
>>> more messages until it's gone through the full redelivery schedule
>>> for the
>>> error prone message. So, with the exponentialBackOff mode, it's
>>> easy to
>>> set it to wait 10 minutes before redelivering a failed message, and
>>> in
>>> that time that consumer will be blocked.
>>>
>>> If the consumer has a pre-fetch greater than 1, this means other
>>> messages
>>> which might be perfectly valid for processing are blocked until the
>>> troubled message goes through it's full redelivery cycle (which
>>> could be
>>> infinite!)....
>>>
>>> I'm wondering if there's an alternate way to configure things, so
>>> that a
>>> message marked for redelivery goes back on the queue to be
>>> scheduled by
>>> the next available consumer (and not until it's next scheduled
>>> redelivery
>>> time has arrived!)....
>>>
>>> If I have a queue with multiple consumers, and perhaps even a small
>>> percentage of the messages are corrupt and force multiple retries, it
>>> could eventually happen that all consumers are blocked waiting idly
>>> to
>>> retry the message at the top of it's list, even though other
>>> messages are
>>> readily available. This would seem to be the case even when the
>>> prefetch
>>> limit is 1.
>>>
>>> Thoughts?
>>>
>>> Jason
>>>
>>
>> --
>> View this message in context:
>> http://www.nabble.com/Redelivery-sticks-to-and-blocks-a-consumer-thread-tp13261850s2354p17691674.html
>> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>>
>
>
>
--
View this message in context: http://www.nabble.com/Redelivery-sticks-to-and-blocks-a-consumer-thread-tp13261850s2354p17693688.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.
Re: Redelivery sticks to and blocks a consumer thread
Posted by Rob Davies <ra...@gmail.com>.
I'd certainly try 5.1
cheers,
Rob
On 6 Jun 2008, at 14:05, Demian Mrakovich wrote:
>
> Hi, bumping this one... Has this (I'd have to say major) issue been
> addressed
> in newer versions of ActiveMQ?
>
>
> Jason Rosenberg wrote:
>>
>> All,
>>
>> I have been experimenting with using exponentialBackOff as a
>> redelivery
>> policy. I have multiple parallel consumers. I'm using AMQ 4.1.1,
>> using
>> Spring's DefaultMessageListener, within Tomcat 6.
>>
>> I've noticed that when a message consumer has a failure, and throws
>> an
>> exception, and the message gets scheduled for retry, the particular
>> consumer thread gets tied to the redelivery, and it doesn't process
>> any
>> more messages until it's gone through the full redelivery schedule
>> for the
>> error prone message. So, with the exponentialBackOff mode, it's
>> easy to
>> set it to wait 10 minutes before redelivering a failed message, and
>> in
>> that time that consumer will be blocked.
>>
>> If the consumer has a pre-fetch greater than 1, this means other
>> messages
>> which might be perfectly valid for processing are blocked until the
>> troubled message goes through it's full redelivery cycle (which
>> could be
>> infinite!)....
>>
>> I'm wondering if there's an alternate way to configure things, so
>> that a
>> message marked for redelivery goes back on the queue to be
>> scheduled by
>> the next available consumer (and not until it's next scheduled
>> redelivery
>> time has arrived!)....
>>
>> If I have a queue with multiple consumers, and perhaps even a small
>> percentage of the messages are corrupt and force multiple retries, it
>> could eventually happen that all consumers are blocked waiting idly
>> to
>> retry the message at the top of it's list, even though other
>> messages are
>> readily available. This would seem to be the case even when the
>> prefetch
>> limit is 1.
>>
>> Thoughts?
>>
>> Jason
>>
>
> --
> View this message in context: http://www.nabble.com/Redelivery-sticks-to-and-blocks-a-consumer-thread-tp13261850s2354p17691674.html
> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>
Re: Redelivery sticks to and blocks a consumer thread
Posted by Demian Mrakovich <ne...@hotmail.com>.
Hi, bumping this one... Has this (I'd have to say major) issue been addressed
in newer versions of ActiveMQ?
Jason Rosenberg wrote:
>
> All,
>
> I have been experimenting with using exponentialBackOff as a redelivery
> policy. I have multiple parallel consumers. I'm using AMQ 4.1.1, using
> Spring's DefaultMessageListener, within Tomcat 6.
>
> I've noticed that when a message consumer has a failure, and throws an
> exception, and the message gets scheduled for retry, the particular
> consumer thread gets tied to the redelivery, and it doesn't process any
> more messages until it's gone through the full redelivery schedule for the
> error prone message. So, with the exponentialBackOff mode, it's easy to
> set it to wait 10 minutes before redelivering a failed message, and in
> that time that consumer will be blocked.
>
> If the consumer has a pre-fetch greater than 1, this means other messages
> which might be perfectly valid for processing are blocked until the
> troubled message goes through it's full redelivery cycle (which could be
> infinite!)....
>
> I'm wondering if there's an alternate way to configure things, so that a
> message marked for redelivery goes back on the queue to be scheduled by
> the next available consumer (and not until it's next scheduled redelivery
> time has arrived!)....
>
> If I have a queue with multiple consumers, and perhaps even a small
> percentage of the messages are corrupt and force multiple retries, it
> could eventually happen that all consumers are blocked waiting idly to
> retry the message at the top of it's list, even though other messages are
> readily available. This would seem to be the case even when the prefetch
> limit is 1.
>
> Thoughts?
>
> Jason
>
--
View this message in context: http://www.nabble.com/Redelivery-sticks-to-and-blocks-a-consumer-thread-tp13261850s2354p17691674.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.