batch processing

[Ben van Klinken <bv...@gmail.com>]

Hi,

I'm pretty new to camel so I may be missing something. I have
discovered I can have a batch consumer, which downloads N items from
the source, but I'd like to have a batch processor and batch producer.
It's not very efficient for me to process an message at a time, so I'm
looking to process a batch of items (like instead of enhancing a
single message from a database, it's much more efficient to enhance a
batch of items). And finally I'd like to be able to send the messages
off to a remote destination in a batch.

I tried building a processor based on the BatchingProcessor, but I had
quite a lot of trouble. I then tried to base it on the resequencer,
which is a bit closer. I had problem with the InFlightRepository not
tracking the async messages, so a shutdown was losing messages.

 Anyway, I just wonder if anyone else has solved this problem, or is
there anything in Camel which i don't know about?


Thanks very much!

Ben

Re: batch processing

[Claus Ibsen <cl...@gmail.com>]

Hi

What kinda of consumer are you using? If you build your custom
consumer, you can just let it pickup N messages and store them as a
List in the message body.

You can use the aggregator EIP to aggregate N Camel messages into a
combined single message.
http://camel.apache.org/aggregator2

Take a look at the completionFromBatchConsumer and groupExchanges
options as well.


On Wed, Feb 22, 2012 at 9:54 AM, Ben van Klinken <bv...@gmail.com> wrote:
> Hi,
>
> I'm pretty new to camel so I may be missing something. I have
> discovered I can have a batch consumer, which downloads N items from
> the source, but I'd like to have a batch processor and batch producer.
> It's not very efficient for me to process an message at a time, so I'm
> looking to process a batch of items (like instead of enhancing a
> single message from a database, it's much more efficient to enhance a
> batch of items). And finally I'd like to be able to send the messages
> off to a remote destination in a batch.
>
> I tried building a processor based on the BatchingProcessor, but I had
> quite a lot of trouble. I then tried to base it on the resequencer,
> which is a bit closer. I had problem with the InFlightRepository not
> tracking the async messages, so a shutdown was losing messages.
>
>  Anyway, I just wonder if anyone else has solved this problem, or is
> there anything in Camel which i don't know about?
>
>
> Thanks very much!
>
> Ben



-- 
Claus Ibsen
-----------------
FuseSource
Email: cibsen@fusesource.com
Web: http://fusesource.com
Twitter: davsclaus, fusenews
Blog: http://davsclaus.blogspot.com/
Author of Camel in Action: http://www.manning.com/ibsen/