You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@flume.apache.org by Pascal Taddei <pa...@amadeus.com> on 2013/10/11 11:21:22 UTC
flume HA
Hello everybody,
I have a small question on best way to implement HA in flume.
I have seen several features in flume to enhance the HA and noloss on
datas:
file or database channels
load balancing and failover capabilities in sinks and flume sdk
transactions which garantees that an event is removed from (n-1) agent
channel only after it is received in channel by agent (n)
I also read a post saying that you could duplicate your flows of data, and
use hadoop to handle the duplicates ....
I would like to know what is the recommended architecture to guarantee
that an event given to flume does arrive to HDFS....even in case of
massive failures, machine crash ... .
Thanks and best regards
Pascal
Re: flume HA
Posted by "Pritchard, Charles X. -ND" <Ch...@disney.com>.
This is not the recommended architecture, just some suggestions.
You could setup a channel which simply sends the avro event to two other machines. This would lead to double-entries in HDFS to be cleaned up later with an MR job. And if those two other machines fail to connect to the HDFS cluster, you're still out of luck. I don't believe slave replication is really part of the Flume scope⦠at that point you may want to point from flume to a Kafka instance.
However, even in that case, as noted in the Jepsen blog-ette, there can be some edge cases:
http://aphyr.com/posts/293-call-me-maybe-kafka
The trade off here with guarantees is speed.. You can go all the way to writing into HDFS before completing the transaction, but you're going to see some slow down in the number of transactions you can handle, and you're going to lose efficiency.
Massive failure and HA are complex topics, there are always trade-offs. This includes HDFS. If you want to stall out your incoming data until it's been successfully written to two separate HDFS regions, you're going to have some latency.
On Oct 11, 2013, at 2:21 AM, Pascal Taddei <pa...@amadeus.com>> wrote:
I would like to know what is the recommended architecture to guarantee that an event given to flume does arrive to HDFS....even in case of massive failures, machine crash ... .