Reliable flows in flume

[Guido Urdaneta <gu...@gmail.com>]

Hi,

I have been trying to configure a reliable flow in Flume NG.
I installed the Flume NG distribution that comes with Cloudera cdh4.
Cloudera's documentation says that JDBC channel is "ideal for flows
where recoverability is important" and that FileChannel "is not yet
ready for use".
However, in some messages in this mailing list  FileChannel has been
recommended as the best option for reliable flows. Is this actually
the case?
Is Cloudera packing an old version of Flume NG with its CDH4 distribution?
Any recommendations?

I have found the performance of JDBC channel to be extremely slow
(about 10,000 times slower than memory channel), with the bottleneck
being CPU usage.

Can anybody give a recommendation?


Thanks in advance,
GU

Re: Reliable flows in flume

[Mike Percy <mp...@apache.org>]

Hi Guido,
There is no release of Flume yet that has a stable FileChannel. If you want
to use FileChannel now, then you will have to build from trunk.

We're planning on doing an Apache Flume 1.2.0 release in a couple of weeks,
which will include a stable FileChannel.

Regards,
Mike

On Thu, Jul 5, 2012 at 5:37 AM, Guido Urdaneta <gu...@gmail.com> wrote:

> Hi,
>
> I have been trying to configure a reliable flow in Flume NG.
> I installed the Flume NG distribution that comes with Cloudera cdh4.
> Cloudera's documentation says that JDBC channel is "ideal for flows
> where recoverability is important" and that FileChannel "is not yet
> ready for use".
> However, in some messages in this mailing list  FileChannel has been
> recommended as the best option for reliable flows. Is this actually
> the case?
> Is Cloudera packing an old version of Flume NG with its CDH4 distribution?
> Any recommendations?
>
> I have found the performance of JDBC channel to be extremely slow
> (about 10,000 times slower than memory channel), with the bottleneck
> being CPU usage.
>
> Can anybody give a recommendation?
>
>
> Thanks in advance,
> GU
>