event ordering semantics

[Jan Van Besien <ja...@ngdata.com>]

Hi,

I am trying to figure out what are the exact semantics of flume with 
respect to event ordering. I'm interested in both single agent and multi 
agent setups.

The current documentation (http://flume.apache.org/FlumeUserGuide.html) 
doesn't seem to mention anything about this topic.

Older documentation and posts to this mailing list, seem to suggest that 
flume doesn't guarantee in order delivery of events, but also seem to 
suggest that "it depends".

for example:

- https://github.com/cloudera/flume/wiki/FAQ (last topic)
- 
http://mail-archives.apache.org/mod_mbox/flume-user/201210.mbox/%3c507BE336.6000500@cyberagent.co.jp%3e

Can somebody clarify this for me? If flume doesn't generally guarantee 
in order delivery, is it still achievable with a more restricted set of 
features? Or is this simply not a good idea, and should I look elsewhere 
with these kinds of requirements (or relax my requirements)?

Thanks in advance,
Jan

Re: event ordering semantics

[Brock Noland <br...@cloudera.com>]

If everything is working well flume will give you approximately in
order delivery. If transactions are being rolled back that order will
be considerably mixed.

I don't see us supporting delivery order anytime soon. Additionally,
for most flume use cases a global serial number could be assigned at
the source and then a MR job (or hbase) used to reorder the events
inside Hadoop.

Brock

On Tue, Nov 6, 2012 at 6:10 AM, Jan Van Besien <ja...@ngdata.com> wrote:
> Hi,
>
> I am trying to figure out what are the exact semantics of flume with respect
> to event ordering. I'm interested in both single agent and multi agent
> setups.
>
> The current documentation (http://flume.apache.org/FlumeUserGuide.html)
> doesn't seem to mention anything about this topic.
>
> Older documentation and posts to this mailing list, seem to suggest that
> flume doesn't guarantee in order delivery of events, but also seem to
> suggest that "it depends".
>
> for example:
>
> - https://github.com/cloudera/flume/wiki/FAQ (last topic)
> -
> http://mail-archives.apache.org/mod_mbox/flume-user/201210.mbox/%3c507BE336.6000500@cyberagent.co.jp%3e
>
> Can somebody clarify this for me? If flume doesn't generally guarantee in
> order delivery, is it still achievable with a more restricted set of
> features? Or is this simply not a good idea, and should I look elsewhere
> with these kinds of requirements (or relax my requirements)?
>
> Thanks in advance,
> Jan



-- 
Apache MRUnit - Unit testing MapReduce - http://incubator.apache.org/mrunit/