You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@camel.apache.org by Lorrin Nelson <lh...@nerdylorrin.net> on 2010/11/23 02:42:34 UTC
DefaultShutdownStrategy can't shutdown DefaultErrorHandler's retries?
The interaction between DefaultErrorHandler retries and DefaultShutdownStrategy seems broken. What I want:
* long running infrequent retries. This seems like what DefaultErrorHandler is built for, thanks to it's exponential back-off feature
* shutdown as soon as no retry is in-flight
Instead, I get:
* DefaultShutdownStrategy wants to wait until all future retries have been attempted! I've configured this to be days! But at least there's a timeout on the DefaultShutdownStrategy, so after a while of waiting around (while no retries are actually occurring), it proceeds.
* DefaultShutdownStrategry logs "Timeout occurred. Now forcing the routes to be shutdown now.", but actually does nothing. The route keeps retrying and Tomcat still can't shutdown.
Is this broken or have I misconfigured somehow?
-Lorrin
Re: DefaultShutdownStrategy can't shutdown DefaultErrorHandler's retries?
Posted by Claus Ibsen <cl...@gmail.com>.
Hi
Yeah topics has its issues in the messaging world. There is a little
wiki page from Apache ActiveMQ for a concept they have that's improves
topics:
http://activemq.apache.org/virtual-destinations.html
The idea of having a persistent redelivery error handler in Camel is
interesting.
It bring in its number of complexities which you would have to
consider in the design.
For example for a transacted JMS session, you can't really have that
running for 24 hours, and it cant "survive" if the server is restarted
or crashed and recovered. So you may have to regard the transaction as
complete, when the persistent error handler have stored the exchange
in its repository.
This also seems what you want, as the consumers on the topic then
would be able to read the next message, while the previous is
scheduled for redelivery and handled by the error handler.
The error handler should also be able to "recover" on startup and
bring back the "state" of each exchange as they were. For example it
should remember the number of redeliveries it have tried before. Then
it will be able to pickup where it left.
And you may want to let the persistent error handler act transactional
as well, for example similar to what the HawtDB offers for the
aggregator EIP pattern. In fact it could be a good idea to check some
of its logic and see what can be reused for your idea.
Are there other who think that a persistent error handler is useable
in Camel? Then we could create a JIRA ticket. And with your work we
may be able to implement it into the camel-core. However error
handling is hard, so it's not that easy.
On Sat, Feb 19, 2011 at 12:22 AM, Rosmon Sidhik <ro...@gmail.com> wrote:
> Hi Claus,
> Thanks for the quick response.
> I'm using OpenMq and I have noticed that when use a queue with
> errorhandler and redeliveryPolicy configured,
> the message seems to stay on the queue till one of the retries passes
> or all of them fail and then gets put on the deadletteruri.
>
> I'm facing this issue when i'm using topics and have durableconsumers
> subscribing to the topic that I have the problem.
> I'm using onException block in the route where the durableconsumer is
> defined and it is transacted. I also have a redeliverPolicy defined
> here (same as for queue).
> Once the exception happens the message doesn't go back on the topic
> (i'm guessing that's the logical way it's supposed to work),
> but now if for some reason the camel context goes down then we loose
> the message that's being redelivered/retried.
>
> This is the reason I was looking for a persistent errorhandler.
> Regards,
> Rosmon Sidhik
>
>
>
> On Thu, Feb 17, 2011 at 10:41 PM, Claus Ibsen <cl...@gmail.com> wrote:
>> On Fri, Feb 18, 2011 at 1:14 AM, rosmons <ro...@gmail.com> wrote:
>>>
>>> Hi Claus,
>>> I've been trying to figure out if the TransactionErrorHandler actually
>>> persists the message in between retries.
>>> I can see from your post here that it's probably not implemented yet. I have
>>> used the hawtdb as repo for Aggregation, but was wondering if you could
>>> point me in the right way to do the same for the TransactionErrorHandler.
>>> I wouldn't mind writing the additional code and could probably upload a
>>> patch that can be used for future release if you find it useful.
>>> The reason we need this is for the following scenario:
>>> We have a topic which has durable consumers.
>>> - The durable consumers receive a message
>>> - Encounters an error while transmitting to a route
>>> - in the OnException definition we have a RedeliveryPolicy which could run
>>> upto 24hrs
>>> - Let's say the camel context shutsdown due to some reason we loose the
>>> message that is being redelivered
>>>
>>
>> Hi
>>
>> Usually the message broker have options for redelivery. By leveraging
>> those instead, you ensure the message is kept persisted in the broker.
>> I suggest you take a look at what the broker offers, instead of using
>> Camel's redelivery.
>>
>>
>>> if i can add persistence to the errorhandler it would solve this issue for
>>> me.
>>>
>>> thanks
>>> Rosmon
>>>
>>> --
>>> View this message in context: http://camel.465427.n5.nabble.com/DefaultShutdownStrategy-can-t-shutdown-DefaultErrorHandler-s-retries-tp3276249p3390369.html
>>> Sent from the Camel - Users mailing list archive at Nabble.com.
>>>
>>
>>
>>
>> --
>> Claus Ibsen
>> -----------------
>> FuseSource
>> Email: cibsen@fusesource.com
>> Web: http://fusesource.com
>> Twitter: davsclaus
>> Blog: http://davsclaus.blogspot.com/
>> Author of Camel in Action: http://www.manning.com/ibsen/
>>
>
--
Claus Ibsen
-----------------
FuseSource
Email: cibsen@fusesource.com
Web: http://fusesource.com
Twitter: davsclaus
Blog: http://davsclaus.blogspot.com/
Author of Camel in Action: http://www.manning.com/ibsen/
Re: DefaultShutdownStrategy can't shutdown DefaultErrorHandler's retries?
Posted by Rosmon Sidhik <ro...@gmail.com>.
Hi Claus,
Thanks for the quick response.
I'm using OpenMq and I have noticed that when use a queue with
errorhandler and redeliveryPolicy configured,
the message seems to stay on the queue till one of the retries passes
or all of them fail and then gets put on the deadletteruri.
I'm facing this issue when i'm using topics and have durableconsumers
subscribing to the topic that I have the problem.
I'm using onException block in the route where the durableconsumer is
defined and it is transacted. I also have a redeliverPolicy defined
here (same as for queue).
Once the exception happens the message doesn't go back on the topic
(i'm guessing that's the logical way it's supposed to work),
but now if for some reason the camel context goes down then we loose
the message that's being redelivered/retried.
This is the reason I was looking for a persistent errorhandler.
Regards,
Rosmon Sidhik
On Thu, Feb 17, 2011 at 10:41 PM, Claus Ibsen <cl...@gmail.com> wrote:
> On Fri, Feb 18, 2011 at 1:14 AM, rosmons <ro...@gmail.com> wrote:
>>
>> Hi Claus,
>> I've been trying to figure out if the TransactionErrorHandler actually
>> persists the message in between retries.
>> I can see from your post here that it's probably not implemented yet. I have
>> used the hawtdb as repo for Aggregation, but was wondering if you could
>> point me in the right way to do the same for the TransactionErrorHandler.
>> I wouldn't mind writing the additional code and could probably upload a
>> patch that can be used for future release if you find it useful.
>> The reason we need this is for the following scenario:
>> We have a topic which has durable consumers.
>> - The durable consumers receive a message
>> - Encounters an error while transmitting to a route
>> - in the OnException definition we have a RedeliveryPolicy which could run
>> upto 24hrs
>> - Let's say the camel context shutsdown due to some reason we loose the
>> message that is being redelivered
>>
>
> Hi
>
> Usually the message broker have options for redelivery. By leveraging
> those instead, you ensure the message is kept persisted in the broker.
> I suggest you take a look at what the broker offers, instead of using
> Camel's redelivery.
>
>
>> if i can add persistence to the errorhandler it would solve this issue for
>> me.
>>
>> thanks
>> Rosmon
>>
>> --
>> View this message in context: http://camel.465427.n5.nabble.com/DefaultShutdownStrategy-can-t-shutdown-DefaultErrorHandler-s-retries-tp3276249p3390369.html
>> Sent from the Camel - Users mailing list archive at Nabble.com.
>>
>
>
>
> --
> Claus Ibsen
> -----------------
> FuseSource
> Email: cibsen@fusesource.com
> Web: http://fusesource.com
> Twitter: davsclaus
> Blog: http://davsclaus.blogspot.com/
> Author of Camel in Action: http://www.manning.com/ibsen/
>
Re: DefaultShutdownStrategy can't shutdown DefaultErrorHandler's retries?
Posted by Claus Ibsen <cl...@gmail.com>.
On Fri, Feb 18, 2011 at 1:14 AM, rosmons <ro...@gmail.com> wrote:
>
> Hi Claus,
> I've been trying to figure out if the TransactionErrorHandler actually
> persists the message in between retries.
> I can see from your post here that it's probably not implemented yet. I have
> used the hawtdb as repo for Aggregation, but was wondering if you could
> point me in the right way to do the same for the TransactionErrorHandler.
> I wouldn't mind writing the additional code and could probably upload a
> patch that can be used for future release if you find it useful.
> The reason we need this is for the following scenario:
> We have a topic which has durable consumers.
> - The durable consumers receive a message
> - Encounters an error while transmitting to a route
> - in the OnException definition we have a RedeliveryPolicy which could run
> upto 24hrs
> - Let's say the camel context shutsdown due to some reason we loose the
> message that is being redelivered
>
Hi
Usually the message broker have options for redelivery. By leveraging
those instead, you ensure the message is kept persisted in the broker.
I suggest you take a look at what the broker offers, instead of using
Camel's redelivery.
> if i can add persistence to the errorhandler it would solve this issue for
> me.
>
> thanks
> Rosmon
>
> --
> View this message in context: http://camel.465427.n5.nabble.com/DefaultShutdownStrategy-can-t-shutdown-DefaultErrorHandler-s-retries-tp3276249p3390369.html
> Sent from the Camel - Users mailing list archive at Nabble.com.
>
--
Claus Ibsen
-----------------
FuseSource
Email: cibsen@fusesource.com
Web: http://fusesource.com
Twitter: davsclaus
Blog: http://davsclaus.blogspot.com/
Author of Camel in Action: http://www.manning.com/ibsen/
Re: DefaultShutdownStrategy can't shutdown DefaultErrorHandler's
retries?
Posted by rosmons <ro...@gmail.com>.
Hi Claus,
I've been trying to figure out if the TransactionErrorHandler actually
persists the message in between retries.
I can see from your post here that it's probably not implemented yet. I have
used the hawtdb as repo for Aggregation, but was wondering if you could
point me in the right way to do the same for the TransactionErrorHandler.
I wouldn't mind writing the additional code and could probably upload a
patch that can be used for future release if you find it useful.
The reason we need this is for the following scenario:
We have a topic which has durable consumers.
- The durable consumers receive a message
- Encounters an error while transmitting to a route
- in the OnException definition we have a RedeliveryPolicy which could run
upto 24hrs
- Let's say the camel context shutsdown due to some reason we loose the
message that is being redelivered
if i can add persistence to the errorhandler it would solve this issue for
me.
thanks
Rosmon
--
View this message in context: http://camel.465427.n5.nabble.com/DefaultShutdownStrategy-can-t-shutdown-DefaultErrorHandler-s-retries-tp3276249p3390369.html
Sent from the Camel - Users mailing list archive at Nabble.com.
Re: DefaultShutdownStrategy can't shutdown DefaultErrorHandler's retries?
Posted by Claus Ibsen <cl...@gmail.com>.
Hi
It really depends on your business requirements. If you use JMS then
the messages on the queue is ordered.
And if you use transaction then the consumer will rollback the message
in case of an error, and redelivery the same message again.
The message broker have option to control this, for example to move
the failed message to a dead letter queue after X failed attempts.
But there is nothing, AFAIR, on the JMS broker to say that on rollback
the message should be moved to the tail of the queue. The message
keeps its position, and thus the same message will be redelivered.
However you can use concurrent consumers or group ids, to have other
consumers process messages, while a given message on one consumer is
kept being redelivered.
Is ordering of the messages important to your business? If it doesn't
matter then you got more options, as opposed to that the message must
be processed in a given order.
We have thought of adding persistent store to the Camel error handler,
so you could have it instructed to use long lasting redelivery
strategies.
However that persistent storage may require your operations team to
run and maintain it as well, if they want something like JDBC or JMS.
We got a light weight alternative based on file which is called HawtDB
which would be usable. Also there are a number of other NoSQL out
there
which could be interested to integrate and offer for persistent
storage of Camel messages.
On Sat, Nov 27, 2010 at 12:47 AM, Lorrin Nelson
<lh...@nerdylorrin.net> wrote:
> Great, thanks Claus!
>
> I've been thinking about your response. Although our volume is low and we can probably get away with it, I appreciate your point that long retries create the risk of big memory usage spikes as things pile up. Also I imagine it might make it a little difficult to ascertain the current state of the system.
>
> Our use case is that we've got some messages going to systems with unpredictable uptime. The long retries seemed like a simple way to ensure that messages would be delivered eventually, even if it took a while for the recipient to come online.
>
> What would be a good alternative? Sounds like there's nothing built into Camel, but maybe I could create something easily enough:
>
> route#1: from(<some external producer> to(<local JMS queue>)
>
> route#2: transacted receipt from local JMS queue with simple delivery attempt. If failed, either:
> * update exchange to indicate failure and place back on JMS queue
> * or decide message has been in system for too many days, move to dead letter queue
>
> Is it safe to have route #2 sometimes place exchanges back on the same JMS queue it's reading from? Would something else be better suited than JMS?
>
> Cheers
> -Lorrin
>
> On Nov 24, 2010, at 7:37 AM, Claus Ibsen wrote:
>
>> Hi
>>
>> I created a ticket
>> https://issues.apache.org/activemq/browse/CAMEL-3364
>>
>> On Tue, Nov 23, 2010 at 11:55 AM, Claus Ibsen <cl...@gmail.com> wrote:
>>> On Tue, Nov 23, 2010 at 2:42 AM, Lorrin Nelson
>>> <lh...@nerdylorrin.net> wrote:
>>>> The interaction between DefaultErrorHandler retries and DefaultShutdownStrategy seems broken. What I want:
>>>> * long running infrequent retries. This seems like what DefaultErrorHandler is built for, thanks to it's exponential back-off feature
>>>> * shutdown as soon as no retry is in-flight
>>>>
>>>> Instead, I get:
>>>> * DefaultShutdownStrategy wants to wait until all future retries have been attempted! I've configured this to be days! But at least there's a timeout on the DefaultShutdownStrategy, so after a while of waiting around (while no retries are actually occurring), it proceeds.
>>>> * DefaultShutdownStrategry logs "Timeout occurred. Now forcing the routes to be shutdown now.", but actually does nothing. The route keeps retrying and Tomcat still can't shutdown.
>>>>
>>>> Is this broken or have I misconfigured somehow?
>>>>
>>>
>>> This is the intended behavior. Its generally not a good idea to have
>>> redelivery lasting for days. The exchange then has to be stored in
>>> memory until it has to do redelivery the next day. Its generally
>>> better to try for a limited period, and then fail if still a problem.
>>>
>>> We could introduce some option to instruct the error handler to stop
>>> attempt redeliveries if a shutdown has been commenced and then
>>> indicate those exchanges failed by setting an exception on that.
>>>
>>> Fell free to create a JIRA for such an enhancement.
>>>
>>>
>>>> -Lorrin
>>>
>>>
>>>
>>> --
>>> Claus Ibsen
>>> -----------------
>>> FuseSource
>>> Email: cibsen@fusesource.com
>>> Web: http://fusesource.com
>>> Twitter: davsclaus
>>> Blog: http://davsclaus.blogspot.com/
>>> Author of Camel in Action: http://www.manning.com/ibsen/
>>>
>>
>>
>>
>> --
>> Claus Ibsen
>> -----------------
>> FuseSource
>> Email: cibsen@fusesource.com
>> Web: http://fusesource.com
>> Twitter: davsclaus
>> Blog: http://davsclaus.blogspot.com/
>> Author of Camel in Action: http://www.manning.com/ibsen/
>>
>
>
--
Claus Ibsen
-----------------
FuseSource
Email: cibsen@fusesource.com
Web: http://fusesource.com
Twitter: davsclaus
Blog: http://davsclaus.blogspot.com/
Author of Camel in Action: http://www.manning.com/ibsen/
Re: DefaultShutdownStrategy can't shutdown DefaultErrorHandler's retries?
Posted by Lorrin Nelson <lh...@nerdylorrin.net>.
Great, thanks Claus!
I've been thinking about your response. Although our volume is low and we can probably get away with it, I appreciate your point that long retries create the risk of big memory usage spikes as things pile up. Also I imagine it might make it a little difficult to ascertain the current state of the system.
Our use case is that we've got some messages going to systems with unpredictable uptime. The long retries seemed like a simple way to ensure that messages would be delivered eventually, even if it took a while for the recipient to come online.
What would be a good alternative? Sounds like there's nothing built into Camel, but maybe I could create something easily enough:
route#1: from(<some external producer> to(<local JMS queue>)
route#2: transacted receipt from local JMS queue with simple delivery attempt. If failed, either:
* update exchange to indicate failure and place back on JMS queue
* or decide message has been in system for too many days, move to dead letter queue
Is it safe to have route #2 sometimes place exchanges back on the same JMS queue it's reading from? Would something else be better suited than JMS?
Cheers
-Lorrin
On Nov 24, 2010, at 7:37 AM, Claus Ibsen wrote:
> Hi
>
> I created a ticket
> https://issues.apache.org/activemq/browse/CAMEL-3364
>
> On Tue, Nov 23, 2010 at 11:55 AM, Claus Ibsen <cl...@gmail.com> wrote:
>> On Tue, Nov 23, 2010 at 2:42 AM, Lorrin Nelson
>> <lh...@nerdylorrin.net> wrote:
>>> The interaction between DefaultErrorHandler retries and DefaultShutdownStrategy seems broken. What I want:
>>> * long running infrequent retries. This seems like what DefaultErrorHandler is built for, thanks to it's exponential back-off feature
>>> * shutdown as soon as no retry is in-flight
>>>
>>> Instead, I get:
>>> * DefaultShutdownStrategy wants to wait until all future retries have been attempted! I've configured this to be days! But at least there's a timeout on the DefaultShutdownStrategy, so after a while of waiting around (while no retries are actually occurring), it proceeds.
>>> * DefaultShutdownStrategry logs "Timeout occurred. Now forcing the routes to be shutdown now.", but actually does nothing. The route keeps retrying and Tomcat still can't shutdown.
>>>
>>> Is this broken or have I misconfigured somehow?
>>>
>>
>> This is the intended behavior. Its generally not a good idea to have
>> redelivery lasting for days. The exchange then has to be stored in
>> memory until it has to do redelivery the next day. Its generally
>> better to try for a limited period, and then fail if still a problem.
>>
>> We could introduce some option to instruct the error handler to stop
>> attempt redeliveries if a shutdown has been commenced and then
>> indicate those exchanges failed by setting an exception on that.
>>
>> Fell free to create a JIRA for such an enhancement.
>>
>>
>>> -Lorrin
>>
>>
>>
>> --
>> Claus Ibsen
>> -----------------
>> FuseSource
>> Email: cibsen@fusesource.com
>> Web: http://fusesource.com
>> Twitter: davsclaus
>> Blog: http://davsclaus.blogspot.com/
>> Author of Camel in Action: http://www.manning.com/ibsen/
>>
>
>
>
> --
> Claus Ibsen
> -----------------
> FuseSource
> Email: cibsen@fusesource.com
> Web: http://fusesource.com
> Twitter: davsclaus
> Blog: http://davsclaus.blogspot.com/
> Author of Camel in Action: http://www.manning.com/ibsen/
>
Re: DefaultShutdownStrategy can't shutdown DefaultErrorHandler's retries?
Posted by Claus Ibsen <cl...@gmail.com>.
Hi
I created a ticket
https://issues.apache.org/activemq/browse/CAMEL-3364
On Tue, Nov 23, 2010 at 11:55 AM, Claus Ibsen <cl...@gmail.com> wrote:
> On Tue, Nov 23, 2010 at 2:42 AM, Lorrin Nelson
> <lh...@nerdylorrin.net> wrote:
>> The interaction between DefaultErrorHandler retries and DefaultShutdownStrategy seems broken. What I want:
>> * long running infrequent retries. This seems like what DefaultErrorHandler is built for, thanks to it's exponential back-off feature
>> * shutdown as soon as no retry is in-flight
>>
>> Instead, I get:
>> * DefaultShutdownStrategy wants to wait until all future retries have been attempted! I've configured this to be days! But at least there's a timeout on the DefaultShutdownStrategy, so after a while of waiting around (while no retries are actually occurring), it proceeds.
>> * DefaultShutdownStrategry logs "Timeout occurred. Now forcing the routes to be shutdown now.", but actually does nothing. The route keeps retrying and Tomcat still can't shutdown.
>>
>> Is this broken or have I misconfigured somehow?
>>
>
> This is the intended behavior. Its generally not a good idea to have
> redelivery lasting for days. The exchange then has to be stored in
> memory until it has to do redelivery the next day. Its generally
> better to try for a limited period, and then fail if still a problem.
>
> We could introduce some option to instruct the error handler to stop
> attempt redeliveries if a shutdown has been commenced and then
> indicate those exchanges failed by setting an exception on that.
>
> Fell free to create a JIRA for such an enhancement.
>
>
>> -Lorrin
>
>
>
> --
> Claus Ibsen
> -----------------
> FuseSource
> Email: cibsen@fusesource.com
> Web: http://fusesource.com
> Twitter: davsclaus
> Blog: http://davsclaus.blogspot.com/
> Author of Camel in Action: http://www.manning.com/ibsen/
>
--
Claus Ibsen
-----------------
FuseSource
Email: cibsen@fusesource.com
Web: http://fusesource.com
Twitter: davsclaus
Blog: http://davsclaus.blogspot.com/
Author of Camel in Action: http://www.manning.com/ibsen/
Re: DefaultShutdownStrategy can't shutdown DefaultErrorHandler's retries?
Posted by Claus Ibsen <cl...@gmail.com>.
On Tue, Nov 23, 2010 at 2:42 AM, Lorrin Nelson
<lh...@nerdylorrin.net> wrote:
> The interaction between DefaultErrorHandler retries and DefaultShutdownStrategy seems broken. What I want:
> * long running infrequent retries. This seems like what DefaultErrorHandler is built for, thanks to it's exponential back-off feature
> * shutdown as soon as no retry is in-flight
>
> Instead, I get:
> * DefaultShutdownStrategy wants to wait until all future retries have been attempted! I've configured this to be days! But at least there's a timeout on the DefaultShutdownStrategy, so after a while of waiting around (while no retries are actually occurring), it proceeds.
> * DefaultShutdownStrategry logs "Timeout occurred. Now forcing the routes to be shutdown now.", but actually does nothing. The route keeps retrying and Tomcat still can't shutdown.
>
> Is this broken or have I misconfigured somehow?
>
This is the intended behavior. Its generally not a good idea to have
redelivery lasting for days. The exchange then has to be stored in
memory until it has to do redelivery the next day. Its generally
better to try for a limited period, and then fail if still a problem.
We could introduce some option to instruct the error handler to stop
attempt redeliveries if a shutdown has been commenced and then
indicate those exchanges failed by setting an exception on that.
Fell free to create a JIRA for such an enhancement.
> -Lorrin
--
Claus Ibsen
-----------------
FuseSource
Email: cibsen@fusesource.com
Web: http://fusesource.com
Twitter: davsclaus
Blog: http://davsclaus.blogspot.com/
Author of Camel in Action: http://www.manning.com/ibsen/