Buffering messages for batch processing

[Mansour Al Akeel <ma...@gmail.com>]

Hello all,

I got into this similar design few times. I am not sure if I can use
camel to cleanly code it.

I get messages from a source at random times. For example document
changed, record inserted .. etc.

Invoking the processing for a single document is costly. For example
indexing in lucene and committing it. I want to buffer these messages
until a specified number is reached OR a specified time elapsed since
the first document was recieved.

Once we are ready for processing, messages should be read at once (as
a list). Then I can index them, or send them to a WS or DB. For
indexing/searching I know I can use solr commitWithin, and may be
other solutions exsits for other senarios, but this is not what I
want. I can not pull solr.

I can use blocking queues to do this:

* Block until :
* the buffer reached 100 message
* OR
* 300 seconds elapsed.

I am not sure if Camel/activemq can help here to organize things a bit
and separate sources from processing.

Thank you.