[DISCUSS] - Error handler and redelivery with a Processor - Time to take action

[Claus Ibsen <cl...@gmail.com>]

Hi

I am working on
https://issues.apache.org/jira/browse/CAMEL-4117

This ticket is about an issue in Camel error handler and doing
redelivery when using a Processor in the route (eg
.process(myProcessor)).
When Camel performs a redelivery it will redeliver the same Exchange
again, with whatever state was done in the pervious invocation.

So if the end user changes some state on the Exchange such as setting
an OUT message, a FAULT, or change the message body etc.
Those changes are kept and used for redeliveries.

This is not how that works, when you use eg a Bean, To etc in the
route as Camel will only update the Exchange if the operation was a
success.

We have in the past looked into this when we were busy doing Camel 2.0
but hadn't the luxury and time to get this sorted back then.
We captured this in ticket: https://issues.apache.org/jira/browse/CAMEL-1817
And wrote down some ideas for improving this:
http://camel.apache.org/camel-2x-speed-optimizations.html
And as a bonus the changes should improve the memory footprint and
performance during routing.

Those ideas have been postponed to Camel 3.0, but since Camel 3.0 is
not around the corner. I suggest to move forward some of these ideas,
to correct CAMEL-4117 but also look into reducing the need for a
defensive copy of the Exchange in the Pipeline, as that logic is not
only needed
by the RedeliveryErrorHandler (CAMEL-1817).

Let me try to summarize
=================
- CAMEL-4117: Shows a bug/issue with redelivery when using .processor,
the Exchange will carry over dirty content between redeliveries (this
is not expected by end users)
- CAMEL-1817: Moves the defensive copy from Pipeline to
RedeliveryErrorHandler (well it frankly belong in redelivery error
handler, as this is where we need it).
  And this also improves memory footprint and performance. We can
further optimize RedeliveryErrorHandler to only do defensive copy if
needed (eg redelivery is enabled)

So why now?
==========
Well in Camel 2.8 we have fixed/improved some other issues with error
handling (eg what to do when a 2nd exception occurs while handling an
exception, such as when doing onException).
And there has been other slight fixes/improvements as well. So we may
as well further improve error handling in this release.

Camel 3.0 has not been started yet and even if we start today, its
most likely it would take 6+ months to get that work done. So I start
to fell its becoming a bit too later to wait with correcting this.
Especially if we can also in general reduce memory footprint and
improve performance a bit (notice the gain is due reduced memory usage
during routing).


Okay my ramblings is over. Let me dig into the code.

Any thoughts?



-- 
Claus Ibsen
-----------------
FuseSource
Email: cibsen@fusesource.com
Web: http://fusesource.com
Twitter: davsclaus, fusenews
Blog: http://davsclaus.blogspot.com/
Author of Camel in Action: http://www.manning.com/ibsen/

Re: [DISCUSS] - Error handler and redelivery with a Processor - Time to take action

[Claus Ibsen <cl...@gmail.com>]

Hi

For the past two days I have been working on this and running many
tests to ensure my changes are working as expected.
I actually spotted a few unit tests that was wrong and could correct
those as well.

I have just committed the changes as I feel it fixes issues with
redelivery and also improves the the routing engine, to reduce the
defensive copy of exchange in the pipeline, which we had on our
roadmap for a long time (CAMEL-1817).

In fact this was just the first step, and it opens a door for a few
further optimizations we can do. I will take a look at one of those,
which is to only do a defensive copy in the RedeliveryErrorHandler if
redelivery is enabled at all. No need to do that if no redelivery is
every going to be executed. In fact thats the default setting (no
redelivery) so there is a good potential benefit here.





On Sun, Jun 19, 2011 at 11:11 AM, Claus Ibsen <cl...@gmail.com> wrote:
> Hi
>
> I am working on
> https://issues.apache.org/jira/browse/CAMEL-4117
>
> This ticket is about an issue in Camel error handler and doing
> redelivery when using a Processor in the route (eg
> .process(myProcessor)).
> When Camel performs a redelivery it will redeliver the same Exchange
> again, with whatever state was done in the pervious invocation.
>
> So if the end user changes some state on the Exchange such as setting
> an OUT message, a FAULT, or change the message body etc.
> Those changes are kept and used for redeliveries.
>
> This is not how that works, when you use eg a Bean, To etc in the
> route as Camel will only update the Exchange if the operation was a
> success.
>
> We have in the past looked into this when we were busy doing Camel 2.0
> but hadn't the luxury and time to get this sorted back then.
> We captured this in ticket: https://issues.apache.org/jira/browse/CAMEL-1817
> And wrote down some ideas for improving this:
> http://camel.apache.org/camel-2x-speed-optimizations.html
> And as a bonus the changes should improve the memory footprint and
> performance during routing.
>
> Those ideas have been postponed to Camel 3.0, but since Camel 3.0 is
> not around the corner. I suggest to move forward some of these ideas,
> to correct CAMEL-4117 but also look into reducing the need for a
> defensive copy of the Exchange in the Pipeline, as that logic is not
> only needed
> by the RedeliveryErrorHandler (CAMEL-1817).
>
> Let me try to summarize
> =================
> - CAMEL-4117: Shows a bug/issue with redelivery when using .processor,
> the Exchange will carry over dirty content between redeliveries (this
> is not expected by end users)
> - CAMEL-1817: Moves the defensive copy from Pipeline to
> RedeliveryErrorHandler (well it frankly belong in redelivery error
> handler, as this is where we need it).
>  And this also improves memory footprint and performance. We can
> further optimize RedeliveryErrorHandler to only do defensive copy if
> needed (eg redelivery is enabled)
>
> So why now?
> ==========
> Well in Camel 2.8 we have fixed/improved some other issues with error
> handling (eg what to do when a 2nd exception occurs while handling an
> exception, such as when doing onException).
> And there has been other slight fixes/improvements as well. So we may
> as well further improve error handling in this release.
>
> Camel 3.0 has not been started yet and even if we start today, its
> most likely it would take 6+ months to get that work done. So I start
> to fell its becoming a bit too later to wait with correcting this.
> Especially if we can also in general reduce memory footprint and
> improve performance a bit (notice the gain is due reduced memory usage
> during routing).
>
>
> Okay my ramblings is over. Let me dig into the code.
>
> Any thoughts?
>
>
>
> --
> Claus Ibsen
> -----------------
> FuseSource
> Email: cibsen@fusesource.com
> Web: http://fusesource.com
> Twitter: davsclaus, fusenews
> Blog: http://davsclaus.blogspot.com/
> Author of Camel in Action: http://www.manning.com/ibsen/
>



-- 
Claus Ibsen
-----------------
FuseSource
Email: cibsen@fusesource.com
Web: http://fusesource.com
Twitter: davsclaus, fusenews
Blog: http://davsclaus.blogspot.com/
Author of Camel in Action: http://www.manning.com/ibsen/