You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@flume.apache.org by "Zamani, Karim" <ka...@viasat.com> on 2012/10/13 01:03:04 UTC
Flume as event/alerts collector
Hi,
Is Flume (or Flume-NG) suitable for collecting events such as alters generated by various nodes in a distributed architecture to a central location?
I read in the wiki FAQ that Flume was not designed to be a message passing or eventing framework due to issues like message duplication, ordering, etc.
Does Flume-NG have any enhancements to support message passing better?
Thanks!
Karim
Re: Flume as event/alerts collector
Posted by Juhani Connolly <ju...@cyberagent.co.jp>.
Hi Karim
On 10/13/2012 08:03 AM, Zamani, Karim wrote:
>
> Hi,
>
> Is Flume (or Flume-NG) suitable for collecting events such as alters
> generated by various nodes in a distributed architecture to a central
> location?
>
> I read in the wiki FAQ that Flume was not designed to be a message
> passing or eventing framework due to issues like message duplication,
> ordering, etc.
>
Flume NG only guarrantees delivery(so long as you use durable channels
on every hop). Avoiding duplication is not a hard objective, and it is
possible. Same applies for message ordering.
That said, not all component will result in message order getting mixed
or duplicates. If you deliver messages in order to flume and sink them
without batching, along a single path, they should be in order and
without duplicates. However this puts some severe restrictions on
multiple delivery paths for redundancy and batching(not batching the
sink at all is going to hurt performance), so I couldn't claim flume is
ideal for a system where ordering/avoiding duplication is necessary.
> Does Flume-NG have any enhancements to support message passing better?
>
> Thanks!
>
> Karim
>
Re: Flume as event/alerts collector
Posted by Brock Noland <br...@cloudera.com>.
Hi,
Which page on the wiki are you referring to? I believe it's there I
just cannot find it. :)
Brock
On Fri, Oct 12, 2012 at 6:03 PM, Zamani, Karim <ka...@viasat.com> wrote:
> Hi,
>
>
>
> Is Flume (or Flume-NG) suitable for collecting events such as alters
> generated by various nodes in a distributed architecture to a central
> location?
>
>
>
> I read in the wiki FAQ that Flume was not designed to be a message passing
> or eventing framework due to issues like message duplication, ordering, etc.
>
>
>
> Does Flume-NG have any enhancements to support message passing better?
>
>
>
> Thanks!
>
> Karim
--
Apache MRUnit - Unit testing MapReduce - http://incubator.apache.org/mrunit/