You are viewing a plain text version of this content. The canonical link for it is here.

Posted to user@flume.apache.org by Pankaj Gupta <pa...@brightroll.com> on 2012/10/16 19:05:34 UTC

Failover in Flume NG

Hi,

We are investigating using Flume for log aggregation in our organization and have this dilemma over going with Flume NG or OG. We really like the centralized configuration management in OG but that's not a must have. What we need most is reliability in face of machines going down (which is quite common on AWS). How is that handled in Flume NG. In OG, I believe, it was possible to define failover semantics so that messages could be routed to another collector if one collector went down. In Flume OG there is no concept of a special collector but collector is just an agent as I understand. I'm wondering how failover semantics are defined for these agents. It would be great if someone could elaborate on that. 

We also want to use a relatively new version of Hadoop e.g. 1.0.3 or higher. Is it possible to use Flume NG (or OG) with the new versions of hadoop. 

Thanks in advance,
Pankaj