[HEADS UP - About my latest commits]

[Claus Ibsen <cl...@gmail.com>]

Hi

As you might have noticed I have been cut from the internet for the
last 3 days. But now its all running again.

And as such I have worked offline with the camel codebase and as often
I endavour into different paths and thus my list of changed files got
rather big.
So it can be a hassle to split and make sure each tiny changes in
files belong to the correct changelist. So at the end I have commited
3 tickets in one go.

I have created ticket: CAMEL-1638, 1639 and 1640 that covers the major
parts of the commit

So here is a short summary (also with general observations).
=============================================
- Gert V. I managed to create the ExecutorService that is "submit
order" ordered. So the code in MulticastProcessor is a bit easier as
the parallel processing uses the same code, just a different executor
service depending if streaming() is enabled or not. See the
SubmitOrderedCompletionService class.


- Threads created by Camel now uses the ExecutorServiceHelper to
ensure consistent thread names. This was mostly already done. But
please remember to use this helper when creating threads using
ExecutorService. And please do not use TimerTask as its kinda
@deprecated according to Brian Goetz a leading Java Concurrency
specialists. See his book about this. AFAIR we dont have TimerTask
anymore. And in Camel 2.1 we have on the roadmap for much easier
configuration of thread pools in Camel.


- AggragatorStrategy. It took me a bit with surprise that the first
callback was not invoked. So it makes it painful for end users if they
need to do some work on the exchange before its aggregated. Or for
example if they need to sum up an amount based on figures from each
new exchange being aggregated. So I changed it to be invoked also at
first call.
Eg if you aggregate 10 exchanges you get 10 callbacks. Before you only
got 9 callbacks. This is still up for debate, see CAMEL-1638. PS: I am
wondering if we could also add a feature to allow using a POJO as
AggregationStrategy to avoid being dependent on Camel API.


- OnCompletion handover. This is part on the work with the new async
API in Camel. What it means with the handover is that when a route
turns into async using the *async* DSL it hands over the onCompletion
callbacks registered as well. So its the new async thread that
actually invokes the onCompletion callbacks when its complete. This
allows us for instance with the file component to only delete/move the
files when the async thread is actually complete. And in case the
async thread failed the file component will do a rollback.
Without this handover the sync thread will do the on completion and
thus the files is moved/deleted to early.

We should also start migrating existing camel components that uses a
strategy for commit/rollback work. I have also adjusted the idempotent
consumer in Camel for this. But there might be other components that
has this kind of strategy. Anyone remember any component?

As a side note I am in doubt if we should also handover when you use
the SEDA component. So when you route a message to a seda queue should
the original thread or the new thread do the onCompletion work?
Current it does *not* do that. I added a TODO in the SEDA component.
Any thoughts?


I guess that covers my latest commits.


-- 
Claus Ibsen
Apache Camel Committer

Open Source Integration: http://fusesource.com
Blog: http://davsclaus.blogspot.com/
Twitter: http://twitter.com/davsclaus

Re: [HEADS UP - About my latest commits]

[Claus Ibsen <cl...@gmail.com>]

Hi

Me again. I forgot about one ticket

- BatchConsumer
I have introduced a org.apache.camel.BatchConsumer interface for
consumers that can do batching. For instance the file consumer, the
mail consumer etc. What I want is to standardise this in Camel. The
idea is that these consumers gathers the exchanges in the batch before
they are processed. This allows you to know the total number of
exchanges in the batch. Then we can process them one by one in the
batch and have the index and total number in exchange headers.

This allows end user to know how many files was in the batch and can
react to this. Also it allows the Camel aggregator to be able to
aggregate the batch. So I added a *batchConsumer* option on the
Aggregator to turn it into this mode. (Looking for a better name, or
should we auto detect and just do it by itself?). Before hand the
aggregator had either to timeout or reach the batch size threshold
before triggering. With this mode it knows that there are eg 12 files
in the batch so it should trigger when all 12 files have been
aggregated into the aggregator.

By the way the default settings for the aggregator is kinda lame. We
have 1 sec as timeout. Isnt this to low? I would thing that we should
use 20 sec as the JMS component uses for timeout.

And you have to set a timeout, I was wondering what if you want to
disregard the timeout, then you have to set a value like Integer.MAX
etc. Maybe we can support -1 as timeout to indicate its disabled.

The batch consumer is a work in progress as there are other components
that can leverage this. In camel-core its only the file component. But
the mail, rss, atom and some others support batching as well. So there
is still work to migrate these components.




On Sat, May 23, 2009 at 9:32 AM, Claus Ibsen <cl...@gmail.com> wrote:
> Hi
>
> As you might have noticed I have been cut from the internet for the
> last 3 days. But now its all running again.
>
> And as such I have worked offline with the camel codebase and as often
> I endavour into different paths and thus my list of changed files got
> rather big.
> So it can be a hassle to split and make sure each tiny changes in
> files belong to the correct changelist. So at the end I have commited
> 3 tickets in one go.
>
> I have created ticket: CAMEL-1638, 1639 and 1640 that covers the major
> parts of the commit
>
> So here is a short summary (also with general observations).
> =============================================
> - Gert V. I managed to create the ExecutorService that is "submit
> order" ordered. So the code in MulticastProcessor is a bit easier as
> the parallel processing uses the same code, just a different executor
> service depending if streaming() is enabled or not. See the
> SubmitOrderedCompletionService class.
>
>
> - Threads created by Camel now uses the ExecutorServiceHelper to
> ensure consistent thread names. This was mostly already done. But
> please remember to use this helper when creating threads using
> ExecutorService. And please do not use TimerTask as its kinda
> @deprecated according to Brian Goetz a leading Java Concurrency
> specialists. See his book about this. AFAIR we dont have TimerTask
> anymore. And in Camel 2.1 we have on the roadmap for much easier
> configuration of thread pools in Camel.
>
>
> - AggragatorStrategy. It took me a bit with surprise that the first
> callback was not invoked. So it makes it painful for end users if they
> need to do some work on the exchange before its aggregated. Or for
> example if they need to sum up an amount based on figures from each
> new exchange being aggregated. So I changed it to be invoked also at
> first call.
> Eg if you aggregate 10 exchanges you get 10 callbacks. Before you only
> got 9 callbacks. This is still up for debate, see CAMEL-1638. PS: I am
> wondering if we could also add a feature to allow using a POJO as
> AggregationStrategy to avoid being dependent on Camel API.
>
>
> - OnCompletion handover. This is part on the work with the new async
> API in Camel. What it means with the handover is that when a route
> turns into async using the *async* DSL it hands over the onCompletion
> callbacks registered as well. So its the new async thread that
> actually invokes the onCompletion callbacks when its complete. This
> allows us for instance with the file component to only delete/move the
> files when the async thread is actually complete. And in case the
> async thread failed the file component will do a rollback.
> Without this handover the sync thread will do the on completion and
> thus the files is moved/deleted to early.
>
> We should also start migrating existing camel components that uses a
> strategy for commit/rollback work. I have also adjusted the idempotent
> consumer in Camel for this. But there might be other components that
> has this kind of strategy. Anyone remember any component?
>
> As a side note I am in doubt if we should also handover when you use
> the SEDA component. So when you route a message to a seda queue should
> the original thread or the new thread do the onCompletion work?
> Current it does *not* do that. I added a TODO in the SEDA component.
> Any thoughts?
>
>
> I guess that covers my latest commits.
>
>
> --
> Claus Ibsen
> Apache Camel Committer
>
> Open Source Integration: http://fusesource.com
> Blog: http://davsclaus.blogspot.com/
> Twitter: http://twitter.com/davsclaus
>



-- 
Claus Ibsen
Apache Camel Committer

Open Source Integration: http://fusesource.com
Blog: http://davsclaus.blogspot.com/
Twitter: http://twitter.com/davsclaus