Batching events from event driven sources?

[Juhani Connolly <ju...@cyberagent.co.jp>]

Hi,

Raymond's mail to user@flume "performance on RecoverableMemoryChannel vs 
JdbcChannel" got me thinking about how to deal with batching events from 
any type of event driven source. Since we don't have control of when the 
events arrive, they will generally trickle in one at a time.

I have a couple of half-baked ideas to resolve this:

- Logic in the source to keep a transaction open over a few requests... 
Breaks current conventions(since at the end of the request, the event 
hasn't been committed to the channel). Then again in SyslogSource, there 
is no way to communicate to the sender that it wasn't inserted properly, 
so the risk of loss doesn't change much.
- Add some minimum batch size setting to FileChannel, which delays 
flushes until either a) a certain delay since the last one or b) x 
events have been reached.
- Create a BatchingChannel... Basically configure it with a child, and 
it will do receive puts, storing them in memory. After a configured 
number of events are stored, it puts them to the configured child 
channel. This would again be allowing for the loss of up to the 
configured number of events, but no more.

Any other alternatives/ideas? None of the above feel 100% satisfactory 
to me, though I think we will have to make some compromise if we want to 
allow for decent performance between event driven sources and file channel.