Binary File Support in Flume

[nikhil goel <ni...@gmail.com>]

Hello,

We have an usecase where we have to transfer binary files remotely to HDFS.
We are considering using Flume for it. We need end to end reliability.
We have not found a source that can do this. We understand Flume was
designed mainly to handle events. Any suggestions? Would appreciate some
pointers if we have to develop our own custom source for this file type.

Thanks

-- 

Nikhil

Re: Binary File Support in Flume

[nikhil goel <ni...@gmail.com>]

Hi Eric,

Thanks for getting back on this. We are dealing with an usecase and trying
to explore flume for the solution. Here is the usecase - We have to
transfer files from remote locations to a Hadoop cluster. These files are
small (in most cases smalled than the default block size). And one of the
file formats is binary. We also want to delete the files from the remote
location once the file has been put into HDFS.

We are exploring the option of creating a custom source on the agent side
that treats these small files or block of certain size as an event. I
understand that we might run into memory issues with queues. To solve that
issue we were thinking if we can use linux file split or read block of
certain size to get the size of file (event) that does not create memory
issues - say 5 MB. Also custom source can delete the file once agent has
made it available in WAL. Are we thinking on the right path here?
Also we do so and are on the right path, do you forsee any possible changes
that we will need to make to current flume architecture, apart from writing
our custom source at agent.

Also wanted to understand that using Flume the way it works today on event
basis on the text file format, is there a reason in E2E reliability mode
that the event will not appear in sequence on HDFS other than
retransmission?

Also wanted to know what kind of throughput should we expect on using
taildir for text files. We are seeing a 2 mb/s throughput. We tried batch
but it did not really help us improving it.

Thanks,




On Wed, Dec 7, 2011 at 9:51 PM, Eric Sammer <es...@cloudera.com> wrote:

> Nikhil:
>
> You'll almost certainly need to create your own custom source. In the
> 0.9.x branch of Flume the notion of input and output formats are inherently
> coupled to the sources and sinks. A source may read from tail -F, but that
> implies text, for instance. This is a hard relationship to decompose. For
> that reason, custom development is usually the only option. There may be
> ways we can improve this going forward, but it will probably happen in
> Flume NG where it's safe to break backward compatibility. When we do this,
> it will probably look something like Hadoop InputFormat and RecordReaders
> (but without the complexity of input split calculation).
>
> The best way to figure out how to write a source or sink is to look at
> some of the simpler ones like the syslog or thrift rpc sources. Do not
> start by looking at the tail source. ;)
>
> On Wed, Dec 7, 2011 at 3:02 PM, nikhil goel <ni...@gmail.com> wrote:
>
>> Hello,
>>
>> We have an usecase where we have to transfer binary files remotely to
>> HDFS. We are considering using Flume for it. We need end to end reliability.
>> We have not found a source that can do this. We understand Flume was
>> designed mainly to handle events. Any suggestions? Would appreciate some
>> pointers if we have to develop our own custom source for this file type.
>>
>> Thanks
>>
>> --
>>
>> Nikhil
>>
>>
>>
>
>
> --
> Eric Sammer
> twitter: esammer
> data: www.cloudera.com
>



-- 
Nikhil

Re: Binary File Support in Flume

[Eric Sammer <es...@cloudera.com>]

Nikhil:

You'll almost certainly need to create your own custom source. In the 0.9.x
branch of Flume the notion of input and output formats are inherently
coupled to the sources and sinks. A source may read from tail -F, but that
implies text, for instance. This is a hard relationship to decompose. For
that reason, custom development is usually the only option. There may be
ways we can improve this going forward, but it will probably happen in
Flume NG where it's safe to break backward compatibility. When we do this,
it will probably look something like Hadoop InputFormat and RecordReaders
(but without the complexity of input split calculation).

The best way to figure out how to write a source or sink is to look at some
of the simpler ones like the syslog or thrift rpc sources. Do not start by
looking at the tail source. ;)

On Wed, Dec 7, 2011 at 3:02 PM, nikhil goel <ni...@gmail.com> wrote:

> Hello,
>
> We have an usecase where we have to transfer binary files remotely to
> HDFS. We are considering using Flume for it. We need end to end reliability.
> We have not found a source that can do this. We understand Flume was
> designed mainly to handle events. Any suggestions? Would appreciate some
> pointers if we have to develop our own custom source for this file type.
>
> Thanks
>
> --
>
> Nikhil
>
>
>


-- 
Eric Sammer
twitter: esammer
data: www.cloudera.com